Smart home lighting control method and system based on user emotion recognition

Abstract

The application provides a smart home lighting control method and system based on user emotion recognition, and relates to the technical field of intelligent control. The method comprises the following steps: acquiring real-time images and historical images containing user expressions, and adding category labels to the historical images; formulating control instructions of smart home lighting modes, and establishing a mapping relationship between the control instructions and the category labels; constructing a CNN model, training the CNN model by using the historical images with added category labels, obtaining the trained CNN model, inputting the real-time images into the CNN model, obtaining the category labels of the real-time images, denoted as real-time classification labels, calling the control instructions corresponding to the real-time classification labels based on the mapping relationship, denoted as target instructions; and controlling the smart home lighting modes by using the target instructions. The application can recognize the emotion categories of users and automatically adjust the lighting modes according to the emotion categories, thereby improving the satisfaction of users.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent control, in particular to a smart home lighting control method and system based on user emotion recognition. BACKGROUND

[0002] Statista data shows that the global smart home market size is expected to exceed 165 billion US dollars in 2024, among which the annual growth rate of emotion recognition and emotional interaction products reaches 18.7%. China, as one of the largest smart home consumer countries in the world, has an increasing demand for intelligent welcome systems. Research shows that in households with emotion perception home entry systems, the average satisfaction of residents is improved by 34%, and the daily emotional stability is improved by 17%. Especially in first-tier cities, the acceptance of emotion interactive smart home among post-90s and post-95s new middle-class families is high, which has become an important driving force for the continuous expansion of the market.

[0003] In today's society, with the acceleration of urbanization and the increasing pace of life, people are facing unprecedented pressure and challenges. Family, as the harbor of everyone's soul and the haven of body and mind, the importance of its comfort and emotional atmosphere is increasingly highlighted. However, the existing intelligent access control system is mostly limited to identity recognition (such as fingerprint, face unlocking) and basic security alarm functions, and has no effect on the emotional state of the user when entering the door. The setting of the home environment, such as light, music, etc., often needs to be manually operated or relies on fixed preset scenes, lacks dynamic adaptability and emotional care, and is difficult to meet the needs of modern families for intelligent and emotional living experience. SUMMARY

[0004] In order to be able to identify the emotional category of the user and actively adjust the lighting mode according to the emotional category, improve the satisfaction of the user, the application provides a smart home lighting control method and system based on user emotion recognition.

[0005] In the first aspect, the application provides a smart home lighting control method based on user emotion recognition, which adopts the following technical scheme:

[0006] The smart home lighting control method based on user emotion recognition comprises the following steps:

[0007] Obtain real-time images and historical images containing user expressions, add category labels to the historical images, formulate control instructions of smart home lighting mode, and establish a mapping relationship between the control instructions and the category labels;

[0008] The CNN model is constructed, the historical image with the added category label is used to train the CNN model, the trained CNN model is obtained, the real-time image is input into the CNN model, the category label of the real-time image is obtained, which is recorded as a real-time classification label, the control instruction corresponding to the real-time classification label is called based on the mapping relationship, which is recorded as a target instruction;

[0009] The target instruction is used to control the intelligent home lighting mode.

[0010] The application identifies the emotion category of the user through the CNN model and automatically adjusts the lighting mode according to the emotion category, realizes the function of providing a lighting environment more suitable for the needs according to the current emotion or state of the user, improves the comfort and experience of the user, and realizes the function of humanized home welcome experience from emotion recognition to active response through the scheme of image acquisition, expression recognition and lighting control.

[0011] Optionally, after adding the category label to the historical image, the method further comprises:

[0012] performing image semantic segmentation processing on the i-th historical image to obtain a semantic label map; merging pixel points with the same semantic label in the semantic label map into one pixel block to obtain m merged image blocks; screening the m merged image blocks based on the semantic label to obtain n target image blocks; extracting target features of each target image block respectively, and performing weighted fusion on all target features using an attention mechanism to obtain fused features;

[0013] constructing a classifier, the classifier comprising a random forest model and a Softmax layer, the random forest model is used to preliminarily classify the fused features to obtain a preliminary probability vector, and the preliminary probability vector is input into the Softmax layer to output an ideal category label of the i-th historical image;

[0014] If the ideal category label is different from the added category label, the ideal category label is used to replace the added category label.

[0015] The present application performs semantic segmentation on historical images to obtain a semantic label map, which can accurately identify different semantic regions in the image and distinguish various parts of a person's face or background. Subsequently, the present application merges pixel points with the same semantic label into a pixel block and filters target image blocks based on the semantic label to focus on areas meaningful for expression classification, such as eyes, mouth, and other parts with obvious expression features, thereby improving the efficiency and accuracy of feature extraction. Subsequently, the present application extracts target features of the target image blocks and uses an attention mechanism to weight and fuse all target features. The attention mechanism can assign different weights to each feature based on its importance to classification, making the fused features highlight key information and enhance their expression capabilities. Subsequently, the present application constructs a classifier containing a random forest model and a Softmax layer. The random forest model can preliminarily classify the fused features to obtain a preliminary probability vector, and the Softmax layer converts the preliminary probability vector into a more accurate class probability distribution to output the ideal class label of the i-th historical image. By comparing and correcting the added class label, the present application improves the accuracy of the historical image class label.

[0016] Through image semantic segmentation and target image block filtering, the present application can focus on the most relevant areas for expression classification, reducing irrelevant information interference, making the extracted features more representative and distinguishable, and the attention mechanism's weighted fusion further highlighting key features, making the fused features better reflect the expression information of the image and helping to improve the accuracy of classification. The classifier structure combining the random forest model and the Softmax layer can more accurately and meticulously classify the fused features. The above scheme can timely discover and correct the initial added incorrect labels, improve the quality of training data, and thus improve the reliability of the CNN model.

[0017] Optionally, the pixel points with the same semantic label in the semantic label map are merged into a pixel block to obtain m merged image blocks, including:

[0018] Setting a vertex: constructing an empty tree structure and taking any pixel point in the historical image as the vertex of the tree structure;

[0019] Calculating similarity: calculating the similarity between the vertex and the j-th adjacent pixel point, and determining whether the similarity is greater than a preset similarity threshold. If yes, the j-th adjacent pixel point is taken as a new node of the tree structure. If no, the j+1-th adjacent pixel point is updated to the j-th adjacent pixel point, and the step is re-executed until all adjacent pixel points are traversed;

[0020] Vertex update: Take any new node in the tree structure as a vertex and perform the similarity calculation step until there are no more pixels in the i-th historical image that meet the merging condition. Then, take the pixels contained in the tree structure as a merged pixel block.

[0021] Iteration: Delete the pixels contained in the tree structure in the i-th historical image, take the remaining pixels as the new i-th historical image, and perform the step of setting vertices.

[0022] This application achieves a more systematic pixel clustering process compared to traditional region growing algorithms by constructing a tree structure to dynamically expand the pixel block merging process. This application combines traditional region growing algorithms with tree topology, replacing static thresholds with vertex-driven dynamic expansion, reducing the blurring of region boundaries caused by random merging, and improving the accuracy of semantic segmentation.

[0023] Optionally, before performing the step of setting vertices, the method further includes:

[0024] Arbitrarily select a pixel from the pixels contained in each semantic label and denote it as the target pixel. Construct a graph structure with the target pixel as the node and the dissimilarity between target pixels as the edge. Based on the graph structure, construct a minimum spanning tree using a greedy algorithm. Integrate the target pixels contained in the minimum spanning tree into a vertex set.

[0025] In the step of setting vertices, any pixel in the vertex set is used as a vertex of the tree structure.

[0026] The iterative steps also include deleting the selected pixels in the vertex set during the vertex setting step.

[0027] Optionally, before extracting the target features for each target image patch separately, the method further includes:

[0028] Repeat check: Check if n is greater than the preset number threshold. If yes, execute the partitioning step; otherwise, do nothing.

[0029] Partitioning: Construct a coordinate system, sequentially obtain the coordinate values ​​of the center pixels of n target image blocks, perform partitioning processing on the n target image blocks based on the coordinate values, obtain the partitioning results, obtain the depth information of each partition, and retain the target image blocks in the partition with the smallest depth information.

[0030] This application, by determining whether semantic labels are identical and performing a partitioning operation, can process target image blocks with repetitive semantics according to regions. In the process of recognizing user expressions, if multiple target image blocks correspond to the mouth region and have the same semantics, this solution allows for subsequent processing based on these regions.

[0031] In the pruning step, the application calculates the center pixel point coordinates by constructing a coordinate system, and determines which region of the target image block to keep according to the depth information, thereby realizing the function of screening based on the spatial position relationship, making the position of the retained target image block in the overall image relatively more reasonable, and better representing the distribution of the semantic region in the image. The application introduces the processing mode of center pixel point coordinates and depth information, fully considers the spatial position information of the target image block in the image, can reduce the situation that the retained target image block is distributed unreasonably in space due to pruning only according to the semantic label, and improves the quality and usability of the data.

[0032] Optionally, before the preliminary probability vector is input into the Softmax layer, the method further includes:

[0033] determining whether the dimension of the preliminary probability vector is the target dimension, if yes, no processing is performed; if no, a fully connected layer is added between the random forest model and the Softmax layer, the preliminary probability vector is input into the fully connected layer, and an intermediate probability vector with a dimension equal to the target dimension is output, and the intermediate probability vector is updated as the preliminary probability vector.

[0034] The application determines whether the dimension of the preliminary probability vector is the target dimension, thereby realizing the automation of dimension adaptation processing, and improving the self-adaptation ability and automation degree of the prediction model.

[0035] Optionally, the method further includes:

[0036] a music library corresponding to the control instruction of each smart home lighting mode and a voice interaction strategy corresponding to each category label are constructed, and the voice interaction strategy corresponding to the real-time category label and the music library corresponding to the target instruction are played.

[0037] Optionally, before playing the voice interaction strategy corresponding to the category label, the method further includes:

[0038] setting a priority of each category label;

[0039] a voice interaction model is constructed by using a deep learning algorithm, a voice signal of the user is acquired, the voice signal is input into the voice interaction model, a category label of the voice signal is obtained, and it is determined whether the category label of the voice signal is same as a category label of a real-time image,

[0040] if yes, no processing is performed;

[0041] if no, a category label with a higher priority between the category label of the voice signal and the category label of the real-time image is retained.

[0042] The speech interaction model is constructed by using a deep learning algorithm, and the feature extraction and classification ability of deep learning is used to more accurately obtain the classification label of the user speech signal. After the speech signal is input into the speech interaction model, the speech interaction model can analyze the acoustic characteristics, semantic information and the like of the speech, and output the corresponding classification label. Subsequently, the present application checks whether the classification label of the speech signal and the classification label of the real-time image are the same, thereby realizing the consistency of the two. If they are the same, no processing is performed, thereby ensuring the simplicity of the process when the two are consistent. If they are different, the classification label with a high priority is retained according to the priority, and the present application can make a reasonable decision in the inconsistent case. The priority setting enables selection according to the importance of the label when the speech and image classification labels are inconsistent, thereby reducing the unreasonable decision that may be caused by random selection of the label.

[0043] Optionally, the method further comprises:

[0044] At least one edge computing node is set, and the CNN model and the mapping relationship between the control instruction and the category label are preloaded to the edge computing node.

[0045] In a second aspect, the present application provides an intelligent home lighting control system based on user emotion recognition, which adopts the following technical solution:

[0046] The intelligent home lighting control system based on user emotion recognition comprises a storage and a processor,

[0047] The storage stores a computer readable storage medium;

[0048] The processor processes the computer program stored on the computer readable storage medium to implement the method of the first aspect.

[0049] In summary, the present application includes at least one of the following beneficial technical effects:

[0050] 1. The present application identifies the emotion category of the user by using the CNN model and automatically adjusts the lighting mode according to the emotion category, thereby realizing the function of providing a lighting environment that is more suitable for the needs of the user according to the current emotion or state of the user, improving the comfort and experience of the user, and realizing the function of a personalized home welcome experience from emotion recognition to active response by using the scheme of image acquisition, expression recognition and lighting control.

[0051] 2. The present application dynamically expands the merging process of the pixel block by constructing a tree structure, which realizes more systematic pixel clustering processing compared with the traditional region growing algorithm. The present application combines the traditional region growing algorithm with the tree topology, replaces the static threshold with the vertex-driven dynamic expansion, reduces the region boundary ambiguity problem caused by random merging, and improves the accuracy of semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a method flowchart of embodiment 1 of the present application;

[0053] Figure 2 is a method flowchart of embodiment 2 of the present application;

[0054] Figure 3 is a method flowchart of S22 merging in embodiment 2 of the present application;

[0055] Figure 4 is a method flowchart of embodiment 3 of the present application. DETAILED DESCRIPTION

[0056] The following is combined Figures 1 to 4 The present application is further described in detail.

[0057] Embodiment 1: The present embodiment discloses a smart home lighting control method based on user emotion recognition, referring to Figure 1 , the method comprises: S11 data acquisition, S12 classification and S13 control, the present embodiment acquires real-time and historical images containing user expressions, gives historical images category labels, formulates smart home lighting control instructions and establishes the mapping relationship thereof with category labels, constructs a CNN model and trains it with labeled historical images, inputs real-time images into the trained model to obtain real-time classification labels, calls corresponding target instructions based on the mapping relationship, to control the smart home lighting mode, the present embodiment comprises the following steps:

[0058] S11 data acquisition, in the smart home environment, the camera deployed at the entrance captures images containing user expressions. In addition to collecting real-time images, historical images within a period of time also need to be collected, historical images can be obtained from local storage devices (such as hard disk, solid state disk) or cloud storage.

[0059] In order to accurately classify the user expressions in the historical images, a set of reasonable expression classification standards need to be formulated. Common expression classification can be based on basic emotion theory, and the expressions can be classified into happy, sad, angry, surprised, frightened and disgusted categories. In other embodiments, more detailed classification can also be made according to the actual application scene, for example, in the smart home environment, calm, focused and other user state related expression categories can also be considered.

[0060] For the collected historical images, class labels can be added in a manner of manual annotation, automatic annotation, or a combination of both. Manual annotation involves professional observation and judgment of user expressions in images and adding corresponding labels according to the established classification standards. Automatic annotation can use existing expression recognition algorithms or models to classify images. The automatic annotation method can use existing expression recognition algorithms or models to preliminarily classify images, and then manually review and correct them, which can improve the annotation efficiency while ensuring the accuracy of the annotation. The present embodiment can formulate a series of control instructions according to different functions and needs of smart home lighting. The control instructions include switch control, brightness adjustment, color change, etc. of the light. For example, define specific control instructions such as "turn on the light", "turn off the light", "adjust the brightness to 50%", "turn on the blue light", etc.

[0061] By analyzing the potential association between different expression categories and user lighting needs, determine the control instructions corresponding to each expression category, and establish the mapping relationship between control instructions and category labels, for example,

[0062] When the user's expression is detected as happy, turn on the yellow light (wavelength of 590-620 nm), and the yellow spectrum can stimulate the secretion of serotonin, enhancing happiness;

[0063] When the user's expression is detected as sad, turn on the green light (wavelength of 500-570 nm), and the green wavelength can regulate the parasympathetic nervous system;

[0064] When the user's expression is detected as angry, turn on the blue light (wavelength of 450-490 nm), reduce heart rate variability, and calm down;

[0065] When the user's expression is detected as fear or surprise, turn on the full-spectrum simulated daylight;

[0066] When the user's expression is detected as focused, turn on the cool white light (6500K color temperature).

[0067] In other embodiments, the control instructions corresponding to each category label can also be set according to the needs.

[0068] S12 classification, when building a convolutional neural network (CNN) model, the corresponding network architecture can be selected according to the needs. Common CNN architectures include LeNet, AlexNet, VGGNet, ResNet, etc. For the user expression classification task, a suitable architecture can be selected according to the size and complexity of the data set.

[0069] For example, if the data set is small and the expression categories are relatively simple, a shallower network architecture such as LeNet can be selected, including 2 convolutional layers, 2 pooling layers, and 2 fully connected layers, with the specific structure as follows:

[0070] First convolutional layer:

[0071] Number of kernels: Set to 6, 6 kernels extract low-level features such as edges, textures, etc. from different directions (e.g. horizontal, vertical, diagonal, etc.);

[0072] Kernel size: Use 5x5 size;

[0073] Stride: Set to 1;

[0074] Padding: Set to 2;

[0075] Bias: Each kernel corresponds to a bias parameter, the initial value can be set to 0, and it is updated during the training process through the backpropagation algorithm.

[0076] Second convolutional layer:

[0077] Number of kernels: Set to 16, 16 kernels can further extract more complex and abstract features such as local shapes, structures, etc.;

[0078] Kernel size: Also use 5x5 size, keep the same kernel size as the first convolutional layer, which helps the model maintain consistency in feature extraction at different levels;

[0079] Stride: Set to 1;

[0080] Padding: Set to 2;

[0081] Bias: Each kernel corresponds to a bias parameter, the initial value is 0, which is optimized during the training process.

[0082] First pooling layer (after the first convolutional layer)

[0083] Pooling type: Max pooling;

[0084] Pooling window size: Set to 2x2, 2x2 pooling window can effectively reduce the size of the feature map while preserving important features;

[0085] Stride: Set to 2.

[0086] Second pooling layer (after the second convolutional layer)

[0087] Pooling type: Max pooling;

[0088] Pooling window size: Set to 2x2;

[0089] Stride: Set to 2.

[0090] First fully connected layer

[0091] Number of neurons: set to 120;

[0092] Weight matrix: if the output feature vector dimension of the previous layer is q, the size of the weight matrix is 120 x q.

[0093] Bias vector: contains 120 bias parameters, initial value is 0, updated during training.

[0094] Second fully connected layer (output layer)

[0095] Number of neurons: equal to the number of expression categories. For example, if there are 6 expression categories (happy, sad, angry, surprised, fearful, disgusted), the number of neurons is 6;

[0096] Weight matrix: 6 x 120;

[0097] Bias vector: contains 6 bias parameters, initial value is 0, optimized during training.

[0098] If the dataset is large and the expression categories are complex, a deeper network architecture such as ResNet can be selected, including 5 convolutional layers, ReLU functions, and Dropout layers, to improve the feature extraction ability and classification accuracy of the model.

[0099] Convolutional layer parameters: the first layer has 64 convolutional kernels, used to extract low-level features of the image such as edges and textures. With the increase of network layers, the number of convolutional kernels of the subsequent four convolutional layers increases to 128, 256, 512, and 1024, respectively;

[0100] The size of the convolutional kernel is 3 x 3;

[0101] The step is 1;

[0102] The padding is 1;

[0103] The dropout probability of the Dropout layer is 0.2-0.5.

[0104] The CNN model is trained using the historical images with added category labels. During the training process, the historical images with added category labels are divided into a training set, a validation set, and a test set. The training set is used to update the parameters of the CNN model, the validation set is used to monitor the training process of the CNN model and adjust the hyperparameters, and the test set is used to evaluate the final performance of the CNN model. The backpropagation algorithm and optimizers (such as stochastic gradient descent, Adam, etc.) are used to update the weights and biases of the CNN model to minimize the loss function of the CNN model. Through multiple iterations of training, until the performance of the CNN model on the validation set no longer improves significantly or reaches the preset number of iterations.

[0105] The collected real-time images are input into the trained CNN model, which classifies the user's expression in the image and outputs the corresponding category label, denoted as the real-time classification label.

[0106] Based on the mapping relationship between control instructions and expression category labels established in the S11 data collection, the corresponding control instruction is called according to the real-time classification label, denoted as the target instruction. For example, if the real-time classification label is "happy", the corresponding control instruction to "turn on yellow light" is called according to the mapping relationship.

[0107] S13 control, after the smart home lighting device receives the target instruction, it parses the target instruction through WiFi / Zigbee and obtains the parsing result. According to the parsing result, the corresponding operation is performed, such as turning on / off the light, adjusting the brightness, setting the color, etc. The control system inside the smart home lighting device parses the target instruction and controls the corresponding hardware modules (such as LED drive chip, dimming circuit, etc.) to realize the adjustment of lighting effect.

[0108] The embodiment can recognize the user's expression and realize intelligent control of lighting mode according to the expression, and provide a more personalized and comfortable lighting environment according to the user's expression state.

[0109] Embodiment 2: Refer to Figure 2 The difference between this embodiment and embodiment 1 is that after adding category labels to historical images in S11 data collection, the method further comprises:

[0110] S21 semantic segmentation, the i-th historical image is subjected to image semantic segmentation processing to obtain a semantic label map.

[0111] Image semantic segmentation is a task of assigning a specific semantic label to each pixel in an image, aiming to let computers understand the meaning of different regions in the image. Image semantic segmentation methods are mainly based on deep learning, especially convolutional neural networks (CNN) and its variants. For example, fully convolutional networks (FCN) replace the fully connected layers at the end of the CNN traditionally used for image classification with convolutional layers, allowing the network to accept input images of arbitrary size and output a semantic label map corresponding to the input image size, with each pixel point corresponding to a class label. The U-Net network has a unique U-shaped structure that combines an encoder-decoder architecture with a skip connection, which can better preserve the spatial information of the image; the Deep Lab series introduces post-processing techniques such as dilated convolution (Dilated Convolution) and conditional random field (CRF), which expand the receptive field and improve the accuracy and boundary clarity of segmentation.

[0112] Select any one of the above models, train the model using the training set, and input the i-th historical image into the trained model. The trained model performs forward propagation calculation on the image, predicts the semantic class to which each pixel belongs, and finally outputs a semantic label map with the same size as the input image. In this semantic label map, the value of each pixel point represents its corresponding semantic class, for example, 0 represents eyes, 1 represents eyebrows, 2 represents mouth, etc.

[0113] S22 merging, merging pixel points with the same semantic label in the semantic label map into a pixel block to obtain m merged image blocks, referring to Figure 2 , this step includes:

[0114] S220 constructing a vertex set, randomly selecting a pixel point as a target pixel point in the pixel point set contained in each semantic label.

[0115] Taking each selected target pixel point as a node in the graph structure, calculating the dissimilarity between any two different target pixel nodes, and taking the dissimilarity as the weight of the edge connecting the two nodes, the dissimilarity is equal to one minus the similarity between the two nodes.

[0116] The similarity calculation method of the two nodes includes Euclidean distance algorithm, cosine similarity algorithm, etc. Taking the Euclidean distance algorithm as an example, the similarity calculation method is as follows:

[0117] ;

[0118] Where A is the similarity; is the feature value of the kth feature of a certain node; is the feature value of the kth feature of another node; K is the total number of features of the node.

[0119] A graph structure is constructed with the target pixel points as nodes and the dissimilarity between the target pixel points as edges. An arbitrary target pixel point is selected and added to a node set S of the minimum spanning tree, and a priority queue is initialized to store edges connected to the nodes in set S, with the weight of the edge as the priority.

[0120] The edge with the minimum weight is taken out of the priority queue, and the edge connects node u (in set S) and node v (not in set S). Node v is added to set S, and the edge is added to the edge set of the minimum spanning tree. For all adjacent edges of node v (i.e. edges between other nodes connected to v), if the other node of the edge is not in set S, the edge is added to the priority queue. When set S contains all nodes in the graph structure, the algorithm terminates, and the node set contained in the minimum spanning tree is denoted as vertex set.

[0121] S221 sets a vertex and constructs an empty tree structure. The tree structure is a nonlinear data structure composed of nodes (including vertices) and edges, and has a hierarchical relationship. An arbitrary pixel point is selected from the vertex set as the vertex of the tree structure, which will serve as the starting point of the merging process.

[0122] S222 calculates the similarity. For the selected vertex, the similarity between the vertex and the jth adjacent pixel point is calculated, and it is determined whether the calculated similarity is greater than a preset similarity threshold. If the similarity is greater than the preset similarity threshold, it indicates that the two pixel points have sufficient similarity, and the jth adjacent pixel point can be connected to the vertex through an edge as a new node of the tree structure. If the similarity is not greater than the preset similarity threshold, the (j+1)th adjacent pixel point is updated to the jth adjacent pixel point, and the step is re-executed until all adjacent pixel points are traversed.

[0123] The similarity calculation method of the vertex and the jth adjacent pixel point is the same as the similarity calculation method of S220 for constructing the vertex set, which is not described again here.

[0124] S223 updates the vertex. Any new node in the tree structure is taken as a new vertex, and S222 is executed again to calculate the similarity between the new vertex and its adjacent pixel points, and determine whether to merge the adjacent pixel points into the tree structure according to the similarity. This process is repeated until there are no more pixel points greater than the preset similarity threshold in the ith historical image. At this time, the pixel points contained in the tree structure are taken as a merged pixel block, and these pixel points have the same semantic label and certain connectivity and similarity in space.

[0125] S224 iteration, delete the pixel points that have been merged in the tree structure in the i-th historical image, and take the remaining pixel points as the new i-th historical image. In the vertex set, delete the selected pixel points in the step of setting the vertex, and then execute S221 again to set the vertex, reselect the target pixel points remaining in the vertex set as the vertices of the new tree structure, and continue the merging process of the pixel blocks. Through continuous iteration, all pixel points in the image are merged into corresponding pixel blocks, and finally m merged image blocks are obtained.

[0126] S23 screening, screening the m merged image blocks based on the semantic labels to obtain n target image blocks, for example, screening out the merged image blocks representing the eyes, eyebrows and mouth.

[0127] S231 repeated judgment, judging whether n is greater than a preset number threshold, if yes, executing S232 partitioning; if no, no processing is performed.

[0128] The preset number threshold is the number of target image blocks that can be contained in one face.

[0129] S232 partitioning, constructing a coordinate system, sequentially obtaining the coordinate values of the center pixel points of the n target image blocks, performing partitioning processing on the n target image blocks based on the coordinate values to obtain a partitioning result, obtaining the depth information of each partition, and retaining the target image blocks in the partition with the smallest depth information.

[0130] In this embodiment, any one of the four corners of the historical image is taken as the coordinate origin, and the length direction and the width direction of the historical image are taken as the directions of the two coordinate axes to construct the coordinate system. In other embodiments, the coordinate system can also be set according to requirements.

[0131] The partitioning processing on the n target image blocks based on the coordinate values to obtain a partitioning result can be to judge according to the Euclidean distance between any two center pixel points, if the Euclidean distance between any two center pixel points is less than a preset Euclidean distance threshold, it is determined to belong to the same partition.

[0132] S24 fusion, respectively extracting the target features of the target image blocks after S231 repeated judgment and S232 partitioning, and using an attention mechanism to weight and fuse all the target features to obtain a fused feature.

[0133] The core of the attention mechanism is to assign a weight to each target feature to represent its importance. In this embodiment, a self-attention mechanism is used to calculate the weight of each target feature.

[0134] For the extracted p target feature vectors F1, F2, …, F p , first calculate the similarity matrix B between them, where Bab The similarity of the target feature vector F a and F b can be calculated using dot product, cosine similarity, etc. Then the similarity matrix B is normalized to obtain the attention weight matrix C, which can be normalized using the Softmax function.

[0135] According to the calculated attention weight matrix C, the target features are weighted and fused. The fused feature F rh is calculated by the following formula:

[0136] ;

[0137] is the sum of the elements of the a-th row of the attention weight matrix C, which represents the contribution weight of the target feature vector F a to the fused feature.

[0138] S25 establishes a classifier, and random forest is an ensemble learning method composed of multiple decision trees. Each decision tree is trained on a randomly selected sample subset and feature subset. In the classification task, random forest obtains the classification result by voting on the classification results of all decision trees.

[0139] The random forest model is used to preliminarily classify the fused feature, and the probability value corresponding to each class is obtained to form a preliminary probability vector.

[0140] The Softmax layer uses the Softmax function to further process the preliminary probability vector output by the random forest model and convert it into a probability distribution. The formula of the Softmax function is as follows:

[0141] ;

[0142] where z is the preliminary probability vector; z f is the f-th element of the preliminary probability vector z; K is the number of classes; is the output probability corresponding to z f .

[0143] Through the Softmax layer, each element in the preliminary probability vector can be mapped to the interval [0, 1], and the sum of all elements is 1, which makes it more consistent with the properties of probability distribution.

[0144] S26 dimension judgment, judges whether the dimension of the preliminary probability vector is consistent with the target dimension. If it is not consistent, it means that the probability vector output by the random forest model cannot directly meet the requirements of subsequent classification, and dimension adjustment is needed.

[0145] The target dimension is determined according to the number of categories in the actual classification task. For example, if the classification task is to classify images into 6 categories, the target dimension is 6.

[0146] In this embodiment, a fully connected layer is added between the random forest model and the Softmax layer, and the output dimension of the fully connected layer is set according to the target dimension. The preliminary probability vector is input into the fully connected layer, and the weights and biases of the fully connected layer are trained to make the dimension of the output intermediate probability vector equal to the target dimension. Then the intermediate probability vector is updated as the preliminary probability vector for subsequent input into the Softmax layer for classification.

[0147] S27 outputs the label, and inputs the preliminary probability vector in the dimension judgment of S26 into the Softmax layer to output the ideal category label of the i-th historical image.

[0148] The preliminary probability vector after dimension judgment and processing is input into the Softmax layer. The Softmax layer calculates the probability of each element value of the input vector belonging to each category, and selects the category with the maximum probability as the final classification result, i.e. the ideal category label of the i-th historical image.

[0149] For example, the probability vector output by the Softmax layer is [0.1, 0.1, 0.5, 0.1, 0.15, 0.05], and the third category is selected as the ideal category label.

[0150] S28 label comparison. The added category label is manually labeled or labeled by other methods in S11 data collection as the historical image label, and there may be some errors. In this embodiment, by comparing with the ideal category label, these errors can be found and corrected in time, improving the accuracy and reliability of image classification.

[0151] If the ideal category label is different from the added category label, it means that the added label may be inaccurate, and the ideal category label is used to replace the added category label.

[0152] This embodiment realizes the verification of the added category label of the historical image, improves the accuracy of the training sample, and improves the reliability of the CNN model.

[0153] Embodiment 3: Refer to Figure 4 The difference between this embodiment and embodiment 1 is that the method further comprises:

[0154] S31 builds a library, and builds a music library corresponding to the control instruction of each smart home lighting mode, and a voice interaction strategy corresponding to each category label.

[0155] The lighting modes in smart home are various, such as yellow light, blue light and green light, etc. Different lighting modes need to create different atmospheres, and music is an effective means to strengthen this atmosphere. According to the above correlation analysis, collect music suitable for different lighting modes from major music platforms and music databases. Organize the collected music, store it according to the lighting mode, and establish a music library corresponding to the control instruction of each smart home lighting mode. At the same time, design a corresponding voice interaction strategy for each category label.

[0156] According to the emotion type represented by the real-time category label, determine the voice interaction strategy that the user may need in this scene, and finally play the voice interaction strategy corresponding to the real-time category label and the music library corresponding to the target instruction.

[0157] S32 sets priority, sets the priority of each category label according to the severity of the emotion represented by each category label, uses RNN or RNN variants to build a voice interaction model, acquires the user's voice signal through the microphone device in the smart home system, inputs the acquired voice signal into the voice interaction model, and the voice interaction model performs feature extraction and classification prediction on the voice signal, and outputs the classification label of the voice signal.

[0158] S33 label judgment, judge whether the classification label of the voice signal is the same as the classification label of the real-time image,

[0159] If the classification label of the voice signal is the same as the classification label of the real-time image, it means that the voice instruction issued by the user matches the current scene, and no additional operation is needed, and no processing is needed.

[0160] If the classification label of the voice signal is different from the classification label of the real-time image, the priority of the two classification labels needs to be compared. According to the priority order set in S32 priority setting, determine which classification label has higher priority. Keep the classification label with higher priority, and play the voice interaction strategy corresponding to the classification label. For example, if the priority of the classification label of the voice signal is higher than that of the real-time image, play the voice interaction strategy according to the classification label corresponding to the voice signal. At the same time, play the music library corresponding to the control instruction corresponding to the classification label with higher priority.

[0161] In other embodiments, the method further comprises: setting at least one edge computing node, and preloading the CNN model and the mapping relationship between the control instruction and the category label to the edge computing node.

[0162] Embodiment 4: The embodiment discloses a smart home lighting control system based on user emotion recognition, the system comprises: a memory and a processor,

[0163] The memory stores a computer readable storage medium;

[0164] The processor processes the computer program stored on the computer readable storage medium to implement the intelligent home lighting control method based on user emotion recognition.

[0165] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, so that: any equivalent changes made in the structure, shape, principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A smart home lighting control method based on user emotion recognition, characterized by, The method comprises the following steps: acquiring real-time images and historical images containing user expressions, and adding category labels to the historical images; formulating control instructions for intelligent home lighting modes, and establishing a mapping relationship between the control instructions and the category labels; constructing a CNN model, training the CNN model using the historical images with added category labels, inputting real-time images into the CNN model, obtaining category labels of the real-time images, denoted as real-time classification labels, and calling control instructions corresponding to the real-time classification labels based on the mapping relationship, denoted as target instructions; controlling the intelligent home lighting modes using the target instructions; after adding the category labels to the historical images, the method further comprises the following steps: performing image semantic segmentation processing on the i-th historical image to obtain a semantic label image; merging pixel points with the same semantic label in the semantic label image into one pixel block to obtain m merged image blocks; screening the m merged image blocks based on the semantic labels to obtain n target image blocks; extracting target features of each target image block respectively, and performing weighted fusion on all the target features using an attention mechanism to obtain fused features; constructing a classifier, wherein the classifier comprises a random forest model and a Softmax layer, the random forest model is used to preliminarily classify the fused features to obtain a preliminary probability vector, and the preliminary probability vector is input into the Softmax layer to output an ideal category label of the i-th historical image; if the ideal category label is different from the added category label, the ideal category label is used to replace the added category label; merging pixel points with the same semantic label in the semantic label image into one pixel block to obtain m merged image blocks, comprising the following steps: setting a vertex: constructing an empty tree structure, and taking any one pixel point in the historical image as a vertex of the tree structure; calculating a similarity: calculating a similarity between the vertex and a j-th adjacent pixel point, determining whether the similarity is greater than a preset similarity threshold, if yes, taking the j-th adjacent pixel point as a new node of the tree structure, if not, updating a j+1-th adjacent pixel point to the j-th adjacent pixel point, and re-executing the step of calculating the similarity until all adjacent pixel points are traversed; vertex updating: taking any one new node in the tree structure as a vertex, and executing the step of calculating the similarity until there is no pixel point in the i-th historical image that meets the merging condition, and taking pixel points contained in the tree structure as one merged pixel block; iteration: deleting the pixel points contained in the tree structure in the i-th historical image, taking the remaining pixel points as a new i-th historical image, and executing the step of setting the vertex. 2.The smart home lighting control method based on user emotion recognition according to claim 1, wherein, Before executing the step of setting the vertex, the method further comprises the following steps: selecting any one pixel point in each semantic label as a target pixel point, constructing a graph structure taking the target pixel point as a node and taking dissimilarity between the target pixel points as edges, and constructing a minimum spanning tree based on the graph structure using a greedy algorithm, and integrating target pixel points contained in the minimum spanning tree into a vertex set; in the step of setting the vertex, taking any one pixel point in the vertex set as a vertex of the tree structure. The step of iteration further comprises deleting the selected pixel points in the step of setting the vertexes from the vertex set. 3.The smart home lighting control method based on user emotion recognition according to claim 1, wherein, Before extracting the target feature of each target image block respectively, the method further comprises: Repeating judgment: judging whether n is greater than a preset number threshold, if yes, executing the step of partitioning; if no, not processing; Partitioning: constructing a coordinate system, sequentially obtaining the coordinate values of the center pixel points of the n target image blocks, performing partitioning processing on the n target image blocks based on the coordinate values, obtaining a partitioning result, obtaining the depth information of each partition, and retaining the target image block in the partition with the smallest depth information. 4.The smart home lighting control method based on user emotion recognition according to claim 1, wherein, Before inputting the preliminary probability vector into the Softmax layer, the method further comprises: judging whether the dimension of the preliminary probability vector is the target dimension, if yes, not processing; if no, adding a fully connected layer between the random forest model and the Softmax layer, inputting the preliminary probability vector into the fully connected layer, outputting an intermediate probability vector with a dimension equal to the target dimension, and updating the intermediate probability vector as the preliminary probability vector. 5.The smart home lighting control method based on user emotion recognition according to any one of claims 1-4, characterized in that, The method further comprises: constructing a music library corresponding to the control instruction of each smart home lighting mode and a voice interaction strategy corresponding to each category label, playing the voice interaction strategy corresponding to the real-time category label, and playing the music library corresponding to the target instruction. 6.The smart home lighting control method based on user emotion recognition according to claim 5, wherein, Before playing the voice interaction strategy corresponding to the category label, the method further comprises: setting the priority of each category label; using a deep learning algorithm to construct a voice interaction model, obtaining a voice signal of a user, inputting the voice signal into the voice interaction model, obtaining a category label of the voice signal, judging whether the category label of the voice signal is the same as the category label of the real-time image, if yes, not processing; if no, retaining the category label with higher priority from the category label of the voice signal and the category label of the real-time image. 7.The smart home lighting control method based on user emotion recognition according to claim 1, wherein, The method further comprises: setting at least one edge computing node, and preloading the CNN model and the mapping relationship between the control instruction and the category label to the edge computing node.

8. A smart home lighting control system based on user emotion recognition, characterized by, comprises: a memory and a processor, the memory stores a computer readable storage medium; the processor processes the computer program stored on the computer readable storage medium to implement the method of any one of claims 1-7.

Images