Smart home lighting control method and system based on user emotion recognition

Abstract

The invention provides an intelligent home lighting control method and system based on user emotion recognition, and relates to the technical field of intelligent control, and the method comprises the steps: obtaining a real-time image containing a user expression and a historical image, and adding a class label for the historical image; formulating a control instruction of a smart home lighting mode, and establishing a mapping relationship between the control instruction and the category label; constructing a CNN model, training the CNN model by adopting the historical image added with the category label to obtain the trained CNN model, inputting the real-time image into the CNN model to obtain the category label of the real-time image, recording the category label as a real-time classification label, and calling a control instruction corresponding to the real-time classification label based on the mapping relationship, and recording the control instruction as a target instruction; and adopting the target instruction to control the intelligent home lighting mode. The emotion category of the user can be recognized, the lighting mode is automatically adjusted according to the emotion category, and the satisfaction degree of the user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent control, and in particular to a method and system for intelligent home lighting control based on user emotion recognition. Background Art

[0002] According to Statista data, the global smart home market is expected to exceed $165 billion in 2024, with emotion recognition and interaction products experiencing an annual growth rate of 18.7%. As one of the world's largest consumers of smart home products, China is experiencing a growing demand for smart home systems. A survey found that households that installed emotion-sensing entry systems experienced an average 34% increase in resident satisfaction and a 17% improvement in daily emotional stability. Particularly in first-tier cities, the post-90s and post-95s middle-class families are highly receptive to emotion-interactive smart homes, making them a key driver of the market's continued expansion.

[0003] In today's society, with accelerating urbanization and an increasingly fast-paced pace of life, people face unprecedented pressures and challenges. As a haven for the soul and a haven for relaxation, the comfort and emotional atmosphere of the home are becoming increasingly important. However, existing smart access control systems are mostly limited to identity recognition (such as fingerprint and facial unlocking) and basic security alarms, but are unable to address the user's emotional state upon entry. Home environment settings, such as lighting and music, often require manual operation or rely on fixed preset scenes. These settings lack dynamic adaptability and emotional considerations, making them difficult to meet the demands of modern families for intelligent and emotional living experiences. Summary of the Invention

[0004] In order to identify the user's emotion category and actively adjust the lighting mode according to the emotion category to improve user satisfaction, this application provides a smart home lighting control method and system based on user emotion recognition.

[0005] In the first aspect, the present application provides a smart home lighting control method based on user emotion recognition, which adopts the following technical solutions: The smart home lighting control method based on user emotion recognition includes the following steps: Acquire real-time images and historical images containing user expressions, and add category labels to the historical images; formulate control instructions for smart home lighting modes, and establish a mapping relationship between control instructions and category labels; Constructing a CNN model, training the CNN model using historical images with added category labels to obtain a trained CNN model, inputting real-time images into the CNN model to obtain category labels for the real-time images, which are recorded as real-time classification labels, and calling control instructions corresponding to the real-time classification labels based on the mapping relationship, which are recorded as target instructions; Use target instructions to control smart home lighting.

[0006] This application uses a CNN model to identify the user's emotion category and automatically adjusts the lighting mode according to the emotion category, thereby realizing the function of providing a lighting environment that is more in line with the user's current mood or state, thereby improving the user's comfort and experience. This application adopts a solution from image acquisition, expression recognition to lighting control to realize the function of a humanized home welcome experience from emotion recognition to active response.

[0007] Optionally, after adding category labels to historical images, also include: Perform image semantic segmentation on the i-th historical image to obtain a semantic label map; merge pixels with the same semantic label in the semantic label map into a pixel block to obtain m merged image blocks; filter the m merged image blocks based on the semantic label to obtain n target image blocks; extract the target features of each target image block respectively, and use the attention mechanism to perform weighted fusion of all target features to obtain fused features; Constructing a classifier, the classifier includes: a random forest model and a Softmax layer, using the random forest model to perform preliminary classification on the fused features to obtain a preliminary probability vector, inputting the preliminary probability vector into the Softmax layer, and outputting an ideal category label for the i-th historical image; If the ideal category label is different from the added category label, the ideal category label is used instead of the added category label.

[0008] The present application performs semantic segmentation on historical images to obtain semantic label maps, which can accurately identify areas with different semantics in the image and distinguish between various parts of the face of a person or the background. Subsequently, the present application merges pixels with the same semantic label into pixel blocks, and filters target image blocks based on semantic labels to focus on areas that are meaningful for expression classification, such as eyes, mouths and other parts with obvious expression features, to improve the efficiency and accuracy of feature extraction. Subsequently, the present application extracts the target features of the target image blocks respectively, and uses the attention mechanism to perform weighted fusion of all target features. The attention mechanism can assign different weights according to the importance of each feature to the classification, so that the fused features can better highlight key information and enhance the expressive power of the features. Subsequently, the present application constructs a classifier including a random forest model and a Softmax layer. The random forest model can perform preliminary classification on the fused features to obtain a preliminary probability vector. The Softmax layer converts the preliminary probability vector into a more accurate category probability distribution and outputs the ideal category label for the i-th historical image. The present application improves the accuracy of the historical image category label by comparing and correcting it with the added category label.

[0009] Through image semantic segmentation and target image block screening, this application can focus on the areas in the image most relevant to expression classification, reducing the interference of irrelevant information, and extracting more representative and discriminative features. The weighted fusion of the attention mechanism further highlights key features, allowing the fused features to better reflect the expression information of the image, helping to improve classification accuracy. The classifier structure combining the random forest model and the softmax layer can more accurately and meticulously classify the fused features. The above solution can promptly detect and correct the initially added erroneous labels, improve the quality of training data, and thus improve the reliability of the CNN model.

[0010] Optionally, pixels with the same semantic label in the semantic label map are merged into a pixel block to obtain m merged image blocks, including: Set vertex: Build an empty tree structure and use any pixel in the historical image as the vertex of the tree structure; Calculate similarity: Calculate the similarity between the vertex and the jth adjacent pixel, and determine whether the similarity is greater than a preset similarity threshold. If so, the jth adjacent pixel is used as a new node in the tree structure; if not, the j+1th adjacent pixel is updated to the jth adjacent pixel, and this step is repeated until all adjacent pixels are traversed. Vertex update: take any new node in the tree structure as a vertex and perform the similarity calculation step until there are no more pixels in the i-th historical image that meet the merging conditions, and take the pixels contained in the tree structure as a merged pixel block; Iteration: Delete the pixels contained in the tree structure in the i-th historical image, use the remaining pixels as the new i-th historical image, and execute the vertex setting step.

[0011] This application achieves a more systematic pixel clustering process by dynamically expanding the merging process of pixel blocks through the construction of a tree structure. Compared with the traditional region growing algorithm, this application combines the traditional region growing algorithm with a tree topology, replacing the static threshold with vertex-driven dynamic expansion, reducing the problem of region boundary blurring caused by random merging and improving the accuracy of semantic segmentation.

[0012] Optionally, before executing the step of setting the vertex, the method further includes: A pixel point is randomly selected from the pixels included in each semantic label and recorded as the target pixel point. A graph structure is constructed with the target pixel point as the node and the dissimilarity between the target pixel points as the edge. A minimum spanning tree is constructed based on the graph structure using a greedy algorithm, and the target pixel points included in the minimum spanning tree are integrated into a vertex set. In the vertex setting step, any pixel point in the vertex set is used as a vertex of the tree structure; The iterative step further includes deleting, from the vertex set, the pixel points selected in the vertex setting step.

[0013] Optionally, before extracting the target features of each target image block respectively, the method further includes: Repeated judgment: judge whether n is greater than the preset number threshold. If so, perform the partitioning step; if not, do nothing; Partitioning: Construct a coordinate system, obtain the coordinate values ​​of the center pixel points of n target image blocks in sequence, partition the n target image blocks based on the coordinate values, obtain the partition results, obtain the depth information of each partition, and retain the target image block in the partition with the minimum depth information.

[0014] This application can process semantically duplicated target image blocks by region by determining whether the semantic labels are identical and performing a partitioning operation. During the process of recognizing user expressions, if multiple target image blocks correspond to the mouth area and have the same semantics, this solution can be used to perform subsequent processing by region.

[0015] In the deletion step, this application calculates the coordinates of the center pixel point by constructing a coordinate system, and determines which area of ​​the target image block to retain based on the depth information, thereby realizing the function of screening based on the spatial position relationship, so that the position of the retained target image block in the overall image is relatively more reasonable, and can better represent the distribution of the semantic area in the image. This application introduces a processing method for the center pixel point coordinates and depth information, fully considering the spatial position information of the target image block in the image, and can reduce the situation where the retained target image block is irrationally distributed in space due to deletion based only on semantic labels, thereby improving the quality and availability of the data.

[0016] Optionally, before the preliminary probability vector is input into the Softmax layer, the following is also included: Determine whether the dimension of the preliminary probability vector is the target dimension. If so, no processing is performed; if not, add a fully connected layer between the random forest model and the Softmax layer, input the preliminary probability vector into the fully connected layer, output an intermediate probability vector with a dimension equal to the target dimension, and update the intermediate probability vector to the preliminary probability vector.

[0017] This application realizes the automation of dimension adaptation processing by judging whether the dimension of the preliminary probability vector is the target dimension, thereby improving the adaptability and automation level of the prediction model.

[0018] Optionally, the method further includes: Build a music library corresponding to the control instructions of each smart home lighting mode and a voice interaction strategy corresponding to each category label, play the voice interaction strategy corresponding to the real-time classification label and the music library corresponding to the target instruction.

[0019] Optionally, before playing the voice interaction strategy corresponding to the classification label, the method further includes: Set the priority of each category label; A deep learning algorithm is used to build a voice interaction model, obtain the user's voice signal, input the voice signal into the voice interaction model, obtain the classification label of the voice signal, and determine whether the classification label of the voice signal is the same as the classification label of the real-time image. If so, no action will be taken; If not, the classification label with a higher priority between the classification label of the speech signal and the classification label of the real-time image is retained.

[0020] This application uses a deep learning algorithm to build a voice interaction model, and uses the feature extraction and classification capabilities of deep learning to more accurately obtain the classification labels of user voice signals. After the voice signal is input into the voice interaction model, the voice interaction model can analyze the acoustic features, semantic information, etc. of the voice, and output the corresponding classification label. Subsequently, this application checks the consistency of the two by judging whether the classification label of the voice signal is the same as the classification label of the real-time image. If they are the same, no processing is performed, ensuring the simplicity of the process when the two are consistent; if they are different, the classification label with a higher priority is retained according to the priority. This application can make reasonable decisions in the case of inconsistency. The priority setting enables selection based on the importance of the label when the voice and image classification labels are inconsistent, reducing the unreasonable decisions that may result from arbitrary selection of labels.

[0021] Optionally, the method further includes: Set up at least one edge computing node and preload the CNN model and the mapping relationship between control instructions and category labels into the edge computing node.

[0022] In a second aspect, this application provides a smart home lighting control system based on user emotion recognition, which adopts the following technical solutions: A smart home lighting control system based on user emotion recognition, comprising: a memory and a processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, the method according to the first aspect is implemented.

[0023] In summary, this application includes at least one of the following beneficial technical effects: 1. This application uses a CNN model to identify the user's emotion category and automatically adjusts the lighting mode according to the emotion category, thereby realizing the function of providing a lighting environment that is more in line with the user's current mood or state, thereby improving the user's comfort and experience. This application adopts a solution from image acquisition, expression recognition to lighting control to realize the function of a humanized home welcome experience from emotion recognition to active response.

[0024] 2. This application achieves a more systematic pixel clustering process compared to traditional region growing algorithms by dynamically expanding the merging process of pixel blocks through the construction of a tree structure. This application combines the traditional region growing algorithm with a tree topology, replacing the static threshold with vertex-driven dynamic expansion, reducing the region boundary blurring caused by random merging and improving the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the method of Example 1 of the present application; Figure 2 is a flow chart of the method of Example 2 of the present application; Figure 3 This is a flow chart of the method for merging S22 in Example 2 of the present application; Figure 4 This is a flow chart of the method of Example 3 of the present application. DETAILED DESCRIPTION

[0026] The following combination Figures 1 to 4 This application is described in further detail.

[0027] Example 1: This example discloses a smart home lighting control method based on user emotion recognition. Figure 1 The method includes: S11 data acquisition, S12 classification, and S13 control. This embodiment obtains real-time and historical images containing user expressions, adds category labels to the historical images, formulates smart home lighting control instructions and establishes a mapping relationship between them and the category labels, builds a CNN model and trains it with the labeled historical images, inputs the real-time images into the trained model to obtain real-time classification labels, and calls corresponding target instructions based on the mapping relationship to control the smart home lighting mode. This embodiment includes the following steps: S11 data collection: In a smart home environment, cameras deployed at the entrance capture images containing user expressions. In addition to real-time images, historical images over a period of time are also collected. These images can be retrieved from local storage devices (such as hard drives or solid-state drives) or cloud storage.

[0028] To accurately classify user expressions in historical images, a reasonable set of expression classification criteria is necessary. Common expression classifications can be based on basic emotion theory, categorizing expressions into categories such as happiness, sadness, anger, surprise, fear, and disgust. In other embodiments, more detailed classifications can be performed based on actual application scenarios. For example, in a smart home environment, expression categories related to user states, such as calmness and concentration, can also be considered.

[0029] For collected historical images, category labels can be added using manual annotation, automatic annotation, or a combination of the two. Manual annotation involves professionals observing and judging user expressions in images and adding corresponding labels based on established classification criteria. Automatic annotation can utilize existing expression recognition algorithms or models to classify images. Automatic annotation can use existing expression recognition algorithms or models to perform preliminary image classification, followed by manual review and correction. This improves annotation efficiency while ensuring annotation accuracy. This embodiment can develop a series of control instructions based on the different functions and needs of smart home lighting. Control instructions include light on / off control, brightness adjustment, color change, etc. For example, specific control instructions such as "turn on lights," "turn off lights," "adjust brightness to 50%," and "turn on blue lights" can be defined.

[0030] By analyzing the potential association between different expression categories and users' lighting needs, the control instructions corresponding to each expression category are determined, and a mapping relationship between control instructions and category labels is established, for example, When the user's expression is detected to be happy, the yellow light (590-620nm wavelength) is turned on. The yellow spectrum can stimulate the secretion of serotonin and enhance the sense of happiness; When the user's expression is detected as sad, the green light (500-570nm wavelength) is turned on. The green wavelength can regulate the parasympathetic nerves; When the user's expression is detected as angry, the blue light (450-490nm wavelength) is turned on to reduce heart rate variability and calm the user's emotions; When the user's expression is detected as fear or surprise, the full spectrum simulated sunlight is turned on; When the user's expression is detected as focused, the cool white light (6500K color temperature) is turned on.

[0031] In other embodiments, the control instructions corresponding to each category label may be set according to requirements.

[0032] For S12 classification, when building a convolutional neural network (CNN) model, you can choose the appropriate network architecture based on your needs. Common CNN architectures include LeNet, AlexNet, VGGNet, and ResNet. For user expression classification tasks, you can choose an appropriate architecture based on the size and complexity of the dataset.

[0033] For example, if the dataset is small and the expression categories are relatively simple, you can choose a shallower network architecture such as LeNet, which includes 2 convolutional layers, 2 pooling layers, and 2 fully connected layers. The specific structure is as follows: First convolutional layer: The number of convolution kernels is set to 6. The 6 convolution kernels initially extract low-level features of the image, such as edges and textures, from different directions (such as horizontal, vertical, and diagonal); Convolution kernel size: 5×5; Stride: set to 1; Padding: set to 2; Bias: Each convolution kernel corresponds to a bias parameter, the initial value can be set to 0, and it is updated through the backpropagation algorithm during training.

[0034] Second convolutional layer: Number of convolution kernels: set to 16. 16 convolution kernels can further extract more complex and abstract features, such as local shape and structure; Convolution kernel size: The size of 5×5 is also used. Keeping the same convolution kernel size as the first convolution layer helps the model maintain consistency in feature extraction at different levels. Stride: set to 1; Padding: set to 2; Bias: Each convolution kernel corresponds to a bias parameter, the initial value is 0, and it is optimized during the training process.

[0035] First pooling layer (after the first convolutional layer) Pooling type: Max Pooling; Pooling window size: set to 2×2. A 2×2 pooling window can effectively reduce the size of the feature map while retaining important features. Step size: set to 2.

[0036] Second pooling layer (after the second convolutional layer) Pooling type: max pooling; Pooling window size: set to 2×2; Step size: set to 2.

[0037] First fully connected layer Number of neurons: set to 120; Weight matrix: If the dimension of the output feature vector of the previous layer is q, the size of the weight matrix is ​​120×q.

[0038] Bias vector: Contains 120 bias parameters, with an initial value of 0, which is updated during training.

[0039] Second fully connected layer (output layer) The number of neurons is equal to the number of expression categories. For example, if there are 6 expression categories (happy, sad, angry, surprised, fear, disgust), the number of neurons is 6; Weight matrix: 6×120; Bias vector: Contains 6 bias parameters, with an initial value of 0, which are optimized during training.

[0040] If the dataset is large and the expression categories are complex, you can choose a deeper network architecture such as ResNet, which includes 5 convolutional layers, ReLU functions, and Dropout layers to improve the model's feature extraction capabilities and classification accuracy.

[0041] Convolutional layer parameters: The first layer has 64 convolution kernels, which are used to initially extract low-level features of the image, such as edges and textures. As the number of network layers increases, the number of convolution kernels in the subsequent four convolutional layers increases to 128, 256, 512, and 1024 respectively; The convolution kernel size is 3×3; The step size is 1; Fill with 1; The dropout probability of the Dropout layer is 0.2-0.5.

[0042] The CNN model is trained using historical images with added category labels. During training, the labeled historical images are divided into a training set, a validation set, and a test set. The training set is used to update the CNN model's parameters, the validation set is used to monitor the CNN model's training progress and adjust hyperparameters, and the test set is used to evaluate the model's final performance. Backpropagation algorithms and optimizers (such as stochastic gradient descent and Adam) are used to update the CNN model's weights and biases to minimize the CNN model's loss function. Training is repeated multiple times until the CNN model's performance on the validation set no longer improves significantly or the preset number of iterations is reached.

[0043] The collected real-time image is input into the trained CNN model. The CNN model will classify the user's expression in the image and output the corresponding category label, which is recorded as the real-time classification label.

[0044] Based on the mapping relationship between control instructions and expression category labels established during S11 data collection, the corresponding control instruction is called according to the real-time classification label, which is recorded as the target instruction. For example, if the real-time classification label is "happy", the corresponding control instruction "turn on the yellow light" is called according to the mapping relationship.

[0045] S13 controls the receiving of target commands. After receiving them, the smart home lighting device analyzes them via WiFi / Zigbee and obtains the analysis results. Based on the analysis results, it performs corresponding actions, such as turning the light on and off, adjusting the brightness, or setting the color. The control system within the smart home lighting device analyzes the target commands and controls the corresponding hardware modules (such as the LED driver chip and dimming circuit) to adjust the lighting effect.

[0046] This embodiment can recognize the user's expression and realize intelligent control of the lighting mode according to the expression, providing a more personalized and comfortable lighting environment according to the user's expression state.

[0047] Example 2: Reference Figure 2 The difference between this embodiment and embodiment 1 is that, after adding category labels to historical images in S11 data collection, the method further includes: S21 semantic segmentation: perform image semantic segmentation on the i-th historical image to obtain a semantic label map.

[0048] Image semantic segmentation is the task of assigning a specific semantic label to each pixel in an image, with the goal of enabling computers to understand the meaning of different regions in the image. Image semantic segmentation methods are primarily based on deep learning, particularly convolutional neural networks (CNNs) and their variants. For example, the fully convolutional network (FCN) replaces the fully connected layer at the end of the traditional CNN used for image classification with a convolutional layer, allowing the network to accept input images of any size and output a semantic label map corresponding to the size of the input image, with each pixel corresponding to a category label. There is also the U-Net network, whose unique U-shaped structure, through an encoder-decoder architecture combined with skip connections, can better preserve the spatial information of the image; the Deep Lab series introduces post-processing techniques such as dilated convolution and conditional random fields (CRFs) to expand the receptive field and improve segmentation accuracy and boundary clarity.

[0049] Select one of the above models, train it using the training set, and input the i-th historical image into the trained model. The trained model performs a forward propagation on the image, predicting its semantic category pixel by pixel, and ultimately outputs a semantic label map of the same size as the input image. In this semantic label map, the value of each pixel represents its corresponding semantic category, for example, 0 represents eyes, 1 represents eyebrows, 2 represents mouth, and so on.

[0050] S22 merges the pixels with the same semantic label in the semantic label image into a pixel block to obtain m merged image blocks. Figure 2 , this step includes: S220 constructs a vertex set and randomly selects a pixel as a target pixel from the set of pixels included in each semantic label.

[0051] Each selected target pixel point is regarded as a node in the graph structure. For any two different target pixel point nodes, the dissimilarity between them is calculated, and the dissimilarity is used as the weight of the edge connecting the two nodes. The dissimilarity is equal to one minus the similarity between the two nodes.

[0052] Methods for calculating the similarity between the two nodes include: Euclidean distance algorithm, cosine similarity algorithm, etc. Taking the Euclidean distance algorithm as an example, the similarity is calculated as follows: ; Among them, A is the similarity; is the eigenvalue of the kth feature of one of the nodes; is the feature value of the kth feature of another node; K is the total number of features of the node.

[0053] The graph structure is constructed using target pixels as nodes and the dissimilarity between target pixels as edges. A target pixel is randomly selected and added to the node set S of the minimum spanning tree. A priority queue is initialized to store edges connected to nodes in set S, with the edge weights serving as priorities.

[0054] Take the edge with the smallest weight from the priority queue. This edge connects node u (in set S) and node v (not in set S). Add node v to set S and add this edge to the edge set of the minimum spanning tree. For all adjacent edges of node v (that is, edges between other nodes connected to v), if the other node of these edges is not in set S, add these edges to the priority queue. When set S contains all the nodes in the graph structure, the algorithm terminates. The set of nodes contained in the minimum spanning tree obtained at this time is recorded as the vertex set.

[0055] S221 sets vertices and constructs an empty tree structure. A tree structure is a nonlinear data structure consisting of nodes (including vertices) and edges with a hierarchical relationship. An arbitrary pixel point is selected from the vertex set as a vertex of the tree structure. This vertex will serve as the starting point of the merging process.

[0056] S222 calculates the similarity. For the selected vertex, calculates the similarity between the vertex and the jth adjacent pixel point, and determines whether the calculated similarity is greater than a preset similarity threshold. If the similarity is greater than the preset similarity threshold, it indicates that the two pixels are sufficiently similar, and the jth adjacent pixel point can be used as a new node in the tree structure and connected to the vertex through an edge. If the similarity is not greater than the preset similarity threshold, the j+1th adjacent pixel point is updated to the jth adjacent pixel point, and this step is executed again until all adjacent pixels are traversed.

[0057] The method for calculating the similarity between the vertex and the j-th adjacent pixel is the same as the method for calculating the similarity in constructing the vertex set in S220, and will not be repeated here.

[0058] S223 vertex update: any new node in the tree structure is used as a new vertex. Then, S222 similarity calculation is performed again. This is to calculate the similarity between the new vertex and its adjacent pixels. Based on the similarity, it is determined whether to merge the adjacent pixels into the tree structure. This process is repeated until no pixels in the i-th historical image have a similarity greater than the preset threshold. At this point, the pixels contained in the tree structure are considered as a merged pixel block. These pixels have the same semantic label and have a certain degree of spatial connectivity and similarity.

[0059] In step S224, the pixels already merged in the tree structure are deleted from the i-th historical image, and the remaining pixels are used as the new i-th historical image. The pixels selected in the vertex setting step are deleted from the vertex set. Then, step S221 is executed again to set the vertices. The remaining target pixels in the vertex set are reselected as the vertices of the new tree structure, and the pixel block merging process continues. This process continues until all pixels in the image are merged into their corresponding pixel blocks, ultimately obtaining m merged image blocks.

[0060] S23 screening, screening the m merged image blocks based on the semantic labels to obtain n target image blocks, for example, screening out the merged image blocks corresponding to the eyes, eyebrows, and mouth.

[0061] S231 repeats the judgment to determine whether n is greater than a preset number threshold. If so, execute S232 partitioning; if not, do not process.

[0062] The preset number threshold is the number of target image blocks that can be contained in a face.

[0063] S232 partitions, constructs a coordinate system, obtains the coordinate values ​​of the central pixel points of n target image blocks in sequence, performs partition processing on the n target image blocks based on the coordinate values, obtains the partition results, obtains the depth information of each partition, and retains the target image blocks in the partition with the minimum depth information.

[0064] In this embodiment, any pixel point at the four corners of the historical image is used as the coordinate origin, and the length and width directions of the historical image are used as the directions of the two coordinate axes to construct a coordinate system. In other embodiments, the coordinate system can also be set according to needs.

[0065] The n target image blocks are partitioned based on the coordinate values ​​to obtain the partition results, which can be judged based on the Euclidean distance between any two center pixel points. If the Euclidean distance between any two center pixel points is less than a preset Euclidean distance threshold, they are determined to belong to the same partition.

[0066] S24 fusion extracts the target features of the target image blocks after repeated judgment in S231 and partitioning in S232, and uses the attention mechanism to perform weighted fusion on all target features to obtain fused features.

[0067] The core of the attention mechanism is to assign a weight to each target feature to indicate its importance. This embodiment uses the self-attention mechanism to calculate the weight of each target feature.

[0068] For the p target feature vectors F1, F2, ..., F extracted p , first calculate the similarity matrix B between them, where B ab Represents the target feature vector F a and F b The similarity can be calculated using methods such as dot product and cosine similarity. Then the similarity matrix B is normalized to obtain the attention weight matrix C, which can be normalized using the Softmax function.

[0069] According to the calculated attention weight matrix C, the target features are weighted fused. Fusion feature F rh Calculated by the following formula: ; is the sum of the elements in row a of the attention weight matrix C, representing the target feature vector F a Contribution weight to the fusion feature.

[0070] S25 builds a classifier. Random forest is an ensemble learning method that consists of multiple decision trees. Each decision tree is trained on a randomly selected subset of samples and features. In a classification task, random forest obtains the classification result by voting on the classification results of all decision trees.

[0071] The random forest model is used to perform preliminary classification of the fusion features, and the probability value corresponding to each category is obtained to form a preliminary probability vector.

[0072] The Softmax layer uses the Softmax function to further process the preliminary probability vector output by the random forest model and convert it into a probability distribution. The formula of the Softmax function is as follows: ; Among them, z is the preliminary probability vector; z f is the fth element of the preliminary probability vector z; K is the number of categories; It is z f The corresponding output probability.

[0073] Through the Softmax layer, each element in the preliminary probability vector can be mapped to the interval [0, 1], and the sum of all elements is 1, making it more consistent with the properties of the probability distribution.

[0074] S26 dimension judgment, judge whether the dimension of the preliminary probability vector is consistent with the target dimension. If not, it means that the probability vector output by the random forest model cannot directly meet the requirements of subsequent classification and dimension adjustment is required.

[0075] The target dimension is determined by the number of categories in the actual classification task. For example, if the classification task is to classify images into 6 categories, then the target dimension is 6.

[0076] This example adds a fully connected layer between the random forest model and the Softmax layer, and sets the output dimension of the fully connected layer according to the target dimension. A preliminary probability vector is input into the fully connected layer, and the weights and biases of the fully connected layer are trained to ensure that the dimension of the output intermediate probability vector equals the target dimension. The intermediate probability vector is then updated to the preliminary probability vector, which is then input into the Softmax layer for classification.

[0077] S27 outputs the label, inputs the preliminary probability vector in the dimension judgment of S26 into the Softmax layer, and outputs the ideal category label of the i-th historical image.

[0078] The preliminary probability vector, after dimension determination and processing, is input into the Softmax layer. The Softmax layer calculates the probability of each element in the input vector belonging to each category and selects the category with the highest probability as the final classification result, which is the ideal category label for the i-th historical image.

[0079] For example, if the probability vector output by the Softmax layer is [0.1, 0.1, 0.5, 0.1, 0.15, 0.05], the third category is selected as the ideal category label.

[0080] S28 label comparison: The added category labels are the labels manually or through other methods annotated for historical images in S11 data collection, which may contain certain errors. This embodiment can detect and correct these errors in a timely manner by comparing them with the ideal category labels, thereby improving the accuracy and reliability of image classification.

[0081] If the ideal category label is different from the added category label, it means that the added label may be inaccurate. In this case, the ideal category label is used instead of the added category label.

[0082] This embodiment verifies the category labels added to historical images and improves the reliability of the CNN model by improving the accuracy of training samples.

[0083] Example 3: Reference Figure 4 The difference between this embodiment and embodiment 1 is that the method further includes: S31 builds a library to construct a music library corresponding to the control instructions of each smart home lighting mode, as well as a voice interaction strategy corresponding to each category label.

[0084] Smart homes feature a wide variety of lighting styles, such as yellow, blue, and green. Different lighting styles create distinct atmospheres, and music is an effective way to enhance these ambiances. Based on the aforementioned correlation analysis, we collected music suitable for different lighting styles from major music platforms and databases. We organized the collected music and categorized it by lighting style, creating a music library corresponding to the control commands for each smart home lighting style. We also designed corresponding voice interaction strategies for each category label.

[0085] Based on the emotion type represented by the real-time classification label, the voice interaction strategy that the user may need in this scenario is determined. Finally, the voice interaction strategy corresponding to the real-time classification label and the music library corresponding to the target instruction are played.

[0086] S32 sets the priority of each category label according to the severity of the emotion represented by each category label, uses RNN or RNN variant to build a voice interaction model, obtains the user's voice signal through the microphone device in the smart home system, and inputs the obtained voice signal into the voice interaction model. The voice interaction model extracts features and classifies and predicts the voice signal, and outputs the classification label of the voice signal.

[0087] S33 label judgment, judge whether the classification label of the voice signal is the same as the classification label of the real-time image, If the classification label of the voice signal is the same as the classification label of the real-time image, it means that the voice command issued by the user matches the current scene and no additional operation is required.

[0088] If the classification label of the voice signal is different from the classification label of the real-time image, the priorities of the two classification labels need to be compared. According to the priority order set in S32, determine which classification label has a higher priority. Keep the classification label with a higher priority and play the voice interaction strategy corresponding to the classification label. For example, if the classification label of the voice signal has a higher priority than the classification label of the real-time image, play the voice interaction strategy according to the classification label corresponding to the voice signal. At the same time, play the music library corresponding to the control instruction corresponding to the classification label with a higher priority.

[0089] In other embodiments, the method further includes: setting at least one edge computing node, and preloading the CNN model and the mapping relationship between the control instructions and the category labels into the edge computing node.

[0090] Example 4: This embodiment discloses a smart home lighting control system based on user emotion recognition, the system comprising: a memory and a processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, it implements a smart home lighting control method based on user emotion recognition.

[0091] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A smart home lighting control method based on user emotion recognition, characterized in that: include: Obtain real-time images and historical images containing user expressions, and add category labels to historical images; Formulate control instructions for smart home lighting modes and establish a mapping relationship between control instructions and category labels; Constructing a CNN model, training the CNN model using historical images with added category labels to obtain a trained CNN model, inputting real-time images into the CNN model to obtain category labels for the real-time images, which are recorded as real-time classification labels, and calling control instructions corresponding to the real-time classification labels based on the mapping relationship, which are recorded as target instructions; Use target instructions to control smart home lighting.

2. The smart home lighting control method based on user emotion recognition according to claim 1, characterized in that: After adding category labels to historical images, also include: Perform image semantic segmentation on the i-th historical image to obtain a semantic label map; merge pixels with the same semantic label in the semantic label map into a pixel block to obtain m merged image blocks; filter the m merged image blocks based on the semantic label to obtain n target image blocks; extract the target features of each target image block respectively, and use the attention mechanism to perform weighted fusion of all target features to obtain fused features; Constructing a classifier, the classifier includes: a random forest model and a Softmax layer, using the random forest model to perform preliminary classification on the fused features to obtain a preliminary probability vector, inputting the preliminary probability vector into the Softmax layer, and outputting an ideal category label for the i-th historical image; If the ideal category label is different from the added category label, the ideal category label is used instead of the added category label.

3. The smart home lighting control method based on user emotion recognition according to claim 2, characterized in that: The pixels with the same semantic label in the semantic label map are merged into a pixel block to obtain m merged image blocks, including: Set vertex: Build an empty tree structure and use any pixel in the historical image as the vertex of the tree structure; Calculate similarity: Calculate the similarity between the vertex and the jth adjacent pixel, and determine whether the similarity is greater than a preset similarity threshold. If so, the jth adjacent pixel is used as a new node in the tree structure; if not, the j+1th adjacent pixel is updated to the jth adjacent pixel, and this step is repeated until all adjacent pixels are traversed. Vertex update: take any new node in the tree structure as a vertex and perform the similarity calculation step until there are no more pixels in the i-th historical image that meet the merging conditions, and take the pixels contained in the tree structure as a merged pixel block; Iteration: Delete the pixels contained in the tree structure in the i-th historical image, use the remaining pixels as the new i-th historical image, and execute the vertex setting step.

4. The smart home lighting control method based on user emotion recognition according to claim 3 is characterized in that: Before executing the step of setting the vertex, the method further includes: A pixel point is randomly selected from the pixels included in each semantic label and recorded as the target pixel point. A graph structure is constructed with the target pixel point as the node and the dissimilarity between the target pixel points as the edge. A minimum spanning tree is constructed based on the graph structure using a greedy algorithm, and the target pixel points included in the minimum spanning tree are integrated into a vertex set. In the vertex setting step, any pixel point in the vertex set is used as a vertex of the tree structure; The iterative step further includes deleting, from the vertex set, the pixel points selected in the vertex setting step.

5. The smart home lighting control method based on user emotion recognition according to claim 2, characterized in that: Before extracting the target features of each target image block respectively, it also includes: Repeated judgment: judge whether n is greater than the preset number threshold. If so, perform the partitioning step; if not, do nothing; Partitioning: Construct a coordinate system, obtain the coordinate values ​​of the center pixel points of n target image blocks in sequence, partition the n target image blocks based on the coordinate values, obtain the partition results, obtain the depth information of each partition, and retain the target image block in the partition with the minimum depth information.

6. The smart home lighting control method based on user emotion recognition according to claim 2, characterized in that: Before the preliminary probability vector is input into the Softmax layer, it also includes: Determine whether the dimension of the preliminary probability vector is the target dimension. If so, no processing is performed; if not, add a fully connected layer between the random forest model and the Softmax layer, input the preliminary probability vector into the fully connected layer, output an intermediate probability vector with a dimension equal to the target dimension, and update the intermediate probability vector to the preliminary probability vector.

7. The smart home lighting control method based on user emotion recognition according to any one of claims 1 to 6, characterized in that: The method further comprises: Build a music library corresponding to the control instructions of each smart home lighting mode and a voice interaction strategy corresponding to each category label, play the voice interaction strategy corresponding to the real-time classification label and the music library corresponding to the target instruction.

8. The smart home lighting control method based on user emotion recognition according to claim 7, characterized in that: Before playing the voice interaction strategy corresponding to the classification label, the method further includes: Set the priority of each category label; A deep learning algorithm is used to build a voice interaction model, obtain the user's voice signal, input the voice signal into the voice interaction model, obtain the classification label of the voice signal, and determine whether the classification label of the voice signal is the same as the classification label of the real-time image. If so, no action will be taken; If not, the classification label with a higher priority between the classification label of the speech signal and the classification label of the real-time image is retained.

9. The smart home lighting control method based on user emotion recognition according to claim 1, characterized in that: The method further comprises: Set up at least one edge computing node and preload the CNN model and the mapping relationship between control instructions and category labels into the edge computing node.

10. A smart home lighting control system based on user emotion recognition, characterized in that: include: memory and processor, The memory stores a computer-readable storage medium; When the processor processes the computer program stored on the computer-readable storage medium, the method according to any one of claims 1 to 9 is implemented.

Images