Related song pushing system and method based on face micro-expression recognition

Abstract

The invention discloses a related song pushing system based on face micro-expression recognition, and relates to the field of song pushing. The problem that limitation exists when the emotional state of the user is deduced through existing voice recognition or text analysis and music is recommended according to the emotional state is solved. The system comprises a data acquisition module, a face detection module, a micro-expression recognition and classification module, an emotional state inference module and a music recommendation module. The data acquisition module acquires a face video stream and transmits the face video stream to the face detection module, the face detection module detects face information in each frame in the face video stream and transmits the face information to the micro-expression recognition and classification module, and the micro-expression recognition and classification module performs micro-expression classification and performs micro-expression classification. The classification result is input to the emotional state inference module; the emotional state deduction module deduces the emotional state of the user and inputs the emotional state of the user to the music recommendation module; and the music recommendation module selects matched music for recommendation according to the emotional state of the user. The method is applied to the music personalized classification field.

Description

Technical Field

[0001] The present invention relates to the field of song push, and in particular to a related song push system based on facial micro-expression recognition. Background Art

[0002] There are currently a variety of music recommendation systems on the market, which usually recommend music based on users' preference history, playback records, or social network data. However, most existing systems do not directly consider the user's current emotional state, which limits their applicability and personalization level in specific contexts.

[0003] Although some more advanced systems have begun to try to use sentiment analysis techniques to improve recommendation results, such as inferring the user's emotional state through speech recognition or text analysis and recommending music accordingly, these technologies still have some limitations.

[0004] First, most speech- or text-based sentiment analysis technologies can only be used in specific interaction modes, such as conversations or users actively providing text input, which limits the applicable scenarios of the system, especially when users are unwilling or inconvenient to interact with voice or text. Secondly, existing sentiment analysis technologies still need to be improved in accurately identifying user emotions. For example, speech and text analysis may be affected by noise interference or differences in language expression, resulting in inaccurate emotion recognition. In addition, many music recommendation systems require users to actively provide emotional state information, making it impossible to respond in real time when users' emotions change. Finally, although existing systems can make recommendations based on historical behavior, they often ignore the immediate emotional changes of users in different situations, resulting in insufficient personalization of recommended content. Summary of the invention

[0005] Aiming at the problem that existing speech recognition or text analysis is used to infer the user's emotional state and recommend music based on it, the present invention proposes a related song push system based on facial micro-expression recognition, the system comprising:

[0006] Data collection module, facial detection module, micro-expression recognition and classification module, emotional state inference module and music recommendation module;

[0007] The data acquisition module is used to collect facial video streams and transmit the facial video streams to the facial detection module, which is used to receive facial video streams and detect facial information in each frame of the facial video stream; the facial information is transmitted to the micro-expression recognition and classification module, which classifies micro-expressions of the detected facial area and inputs the classification results to the emotional state inference module for fusion judgment; the emotional state inference module is used to infer the emotional state of the user and input the emotional state of the user into the music recommendation module; the music recommendation module selects matching music from the music database for recommendation based on the emotional state of the user.

[0008] Furthermore, a preferred embodiment is proposed, in which the frame rate of the facial video stream collected in the data collection module is 30 frames per second and the resolution is 640x480 pixels.

[0009] Furthermore, a preferred method is proposed, wherein the face detection module is formed by a face detection model MTCNN, and the face detection model MTCNN is composed of three sub-networks: P-Net, R-Net and O-Net; the P-Net receives an input image and generates a candidate face region, and the P-Net outputs a confidence of each region. When the output confidence is higher than 0.9, the candidate region is retained and passed to R-Net; the R-Net receives the candidate region passed by P-Net, and performs face detection in the candidate region, and the R-Net outputs a confidence of each region and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face region, and the valid face region is passed to O-Net. The O-Net receives the refined face region output by R-Net, and performs final face key point positioning. When the confidence of the O-Net output is higher than 0.9, the bounding box coordinates of the face are obtained, and the key point positioning results are retained.

[0010] Furthermore, a preferred embodiment is proposed, wherein the micro-expression recognition and classification module is composed of a ResNet50 model and a yolov5-cls model.

[0011] Furthermore, a preferred method is proposed, in which the optimization process of the micro-expression recognition and classification module is as follows:

[0012] We selected 2,000 images from each of the FER2013 dataset, which contains more than 35,000 grayscale images covering seven facial expressions: anger, disgust, fear, happiness, sadness, surprise, and calmness, and the RAF-DB dataset, which contains 30,000 color images with labels including age, gender, head posture, and lighting conditions, to obtain open source datasets.

[0013] Preprocess the open source dataset, trim the images in the dataset into 640x480 pixel labeled images, and scale them into 224x224 pixel square images, obtaining 1,400 expression data;

[0014] The 5400 images were divided into training set, validation set and test set in a ratio of 7:2:1;

[0015] When receiving face information, the ResNet50 model and the YOLOv5-CLS model are called simultaneously for prediction, and the confidence threshold of each model is set to 0.7;

[0016] If the prediction results of the two models are consistent, the results are directly passed to the emotional state inference module;

[0017] If the prediction results of the two models are inconsistent, the weighted average method is used to calculate the final decision, where the ResNet50 weight is 0.9 and the YOLOv5-CLS model weight is 0.85.

[0018] Calculate the weighted average of the two models’ predictions, re-compare the confidence levels, and make a final decision;

[0019] If only one model returns a prediction, the result is passed directly to the emotional state inference module.

[0020] Furthermore, a preferred embodiment is proposed, wherein the music recommendation module comprises:

[0021] According to the user's emotional state, the user's emotional state is converted into a vector representation;

[0022] Label each piece of music in the music database with an emotional tag, and convert the emotional tag of the music into a vector representation;

[0023] The cosine similarity is used to calculate the similarity between the user's emotional state vector and the music emotion label vector;

[0024] Based on the calculated similarity, music with higher similarity is recommended to the user.

[0025] Based on the same inventive concept, the present invention also proposes a method for pushing related songs based on facial micro-expression recognition, which is implemented based on the above-mentioned system and includes:

[0026] Collect the user's facial video stream in real time through a camera or other video acquisition device;

[0027] The face detection module is used to detect the video stream and process it frame by frame. The image is converted into RGB format and passed into the face detection model to obtain the bounding box coordinates and key point information of all faces detected in each frame. The coordinate information is retained and the image in the coordinate box is extracted as a new file. After scaling and adjusting to a square, it is passed into the micro-expression recognition and classification module.

[0028] The micro-expression recognition and classification module classifies the micro-expressions of the detected facial regions;

[0029] Based on the micro-expression classification results, the user's emotional state is inferred using a predefined emotion mapping table;

[0030] According to the user's emotional state, matching music is selected from the music database for recommendation.

[0031] Furthermore, a preferred method is proposed, wherein the method further includes user feedback collection and model optimization, specifically:

[0032] Integrate a feedback mechanism into the interface of the recommendation system, the feedback content includes the user's preference for the recommended music, changes in emotional state, and whether the music meets the user's expectations;

[0033] Record user feedback data in real time in the database, and use the feedback data to update the recommendation strategy and adjust the strategy parameters of the recommendation algorithm;

[0034] The reward value is adjusted according to user feedback, and the model parameters are updated in real time using the online gradient descent method.

[0035] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the related song push method based on facial micro-expression recognition as described in any one of the above items.

[0036] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for pushing related songs based on facial micro-expression recognition as described in any of the above items are executed.

[0037] The present invention is beneficial in that:

[0038] 1. The related song push system based on facial micro-expression recognition described in the present invention captures the user's facial expression through a camera, without the user actively providing information, thereby achieving a natural and non-intrusive interaction method.

[0039] 2. The related song push system based on facial micro-expression recognition described in the present invention adopts advanced computer vision technology and deep learning algorithms. The present invention can accurately identify subtle changes in users' facial expressions, thereby more accurately inferring the user's emotional state.

[0040] 3. The related song push system based on facial micro-expression recognition of the present invention can respond immediately when the user's emotions change, and dynamically update the recommended music list to better adapt to the user's emotional changes.

[0041] 4. The related song push system based on facial micro-expression recognition described in the present invention can recommend music in real time according to the user's current emotional state, and provide the user with a more personalized and situational music experience in combination with the user's historical habits. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the method for pushing related songs based on facial micro-expression recognition as described in Implementation Method Seven. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0044] Implementation method 1: The system for pushing related songs based on facial micro-expression recognition described in this implementation method comprises:

[0045] Data collection module, facial detection module, micro-expression recognition and classification module, emotional state inference module and music recommendation module;

[0046] The data acquisition module is used to collect facial video streams and transmit the facial video streams to the facial detection module, which is used to receive facial video streams and detect facial information in each frame of the facial video stream; the facial information is transmitted to the micro-expression recognition and classification module, which classifies micro-expressions of the detected facial area and inputs the classification results to the emotional state inference module for fusion judgment; the emotional state inference module is used to infer the emotional state of the user and input the emotional state of the user into the music recommendation module; the music recommendation module selects matching music from the music database for recommendation based on the emotional state of the user.

[0047] Traditional speech recognition or text analysis methods rely on the user's language expression to infer emotions, but these methods may be affected by factors such as the user's expressive ability, language style, and complexity of emotional expression, resulting in limited accuracy in emotion recognition. The system described in this embodiment uses facial micro-expression recognition, which can directly capture the slight emotional changes on the user's face, does not rely on the user's language expression, and can provide more direct and detailed emotional data, thereby improving the accuracy of emotion recognition.

[0048] Existing speech or text analysis methods often require active user input or long-term text data, which may not reflect instantaneous changes in user emotions in a timely manner. The system described in this embodiment can capture changes in facial emotions in real time through micro-expression recognition, adapt to the rapid fluctuations of user emotions, and enable the system to adjust music recommendations in a timely manner, providing a more real-time and dynamic user experience.

[0049] Prior art recommendation systems based on text or voice often fail to accurately capture the user's true emotional state, which may result in the recommended music not fully matching the user's current emotional state. The system described in this embodiment can more accurately infer the user's actual emotional state by comprehensively analyzing micro-expression data, thereby improving the personalization and matching of music recommendations.

[0050] Implementation method 2. This implementation method is a further limitation of the related song push system based on facial micro-expression recognition described in Implementation method 1. The frame rate of the facial video stream collected in the data acquisition module is 30 frames per second and the resolution is 640x480 pixels.

[0051] In this implementation, a frame rate of 30 frames per second is used to ensure the smoothness of the video stream, so that subtle changes in facial expressions can be accurately captured, avoiding the loss or omission of facial expression details caused by too low a frame rate. A sufficient frame rate can ensure that the system captures the user's emotional fluctuations in real time, providing timely data support for emotional state inference and music recommendation.

[0052] Furthermore, the use of a resolution of 640x480 pixels provides a good balance between image quality and computing resources, avoiding the computing burden of too high a resolution while still clearly capturing the key details of facial expressions. Compared to higher resolutions, 640x480 pixels can reduce the computing resources required for image processing and micro-expression recognition, improving the system's processing speed and response efficiency.

[0053] Implementation method three, this implementation method is a further limitation of the related song push system based on facial micro-expression recognition described in implementation method one, the facial detection module is formed by the facial detection model MTCNN, and the facial detection model MTCNN is composed of three sub-networks: P-Net, R-Net and O-Net; the P-Net receives an input image and generates candidate face regions, wherein the input image is an image of 640x480 pixels; the P-Net outputs the confidence of each region, and when the output confidence is higher than 0.9, the candidate region is retained and passed to the R-Net; the R-Net receives the candidate region transmitted by P-Net, the candidate region is cropped and scaled to 24x24 pixels, and face detection is performed in the candidate region. The R-Net outputs the confidence of each region and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face region, and the valid face region is transmitted to O-Net. The O-Net receives the refined face region output by R-Net and performs the final facial key point positioning. When the confidence output by O-Net is higher than 0.9, the bounding box coordinates of the face are obtained, and the key point positioning results are retained.

[0054] In this embodiment, by decomposing the facial detection task into three sub-networks (P-Net, R-Net, O-Net), the detection results can be gradually refined at each stage, thereby improving the accuracy of facial detection and key point positioning. Each sub-network focuses on different tasks when processing facial areas, from the generation of candidate areas to refinement to the precise positioning of key points, to achieve step-by-step optimization and reduce false detections and missed detections. By setting the confidence threshold, low-confidence areas can be effectively filtered out, improving processing efficiency and reducing computational burden.

[0055] Specifically, P-Net (Proposal Network) is responsible for generating candidate face regions and preliminarily evaluating the confidence of these regions. Through preliminary screening, the regions that need further processing can be quickly reduced, reducing the amount of computation. R-Net (Refine Network) performs more refined detection and bounding box fine-tuning on the candidate regions generated by P-Net to improve the accuracy of the regions and reduce false positives. O-Net (Output Network) further refines the bounding box and performs the final key point positioning to ensure accurate extraction of facial features. Setting the confidence threshold (>0.9) can ensure that the results of each step output have high reliability, thereby improving the accuracy of the final face detection and key point positioning. Through layer-by-layer processing and screening, the computational burden of subsequent networks can be reduced. P-Net and R-Net first screen and optimize the regions, and only the regions with high confidence are passed to O-Net, which can significantly reduce the number of regions that O-Net needs to process, thereby improving processing efficiency. O-Net is responsible for key point positioning in the final stage and can accurately obtain the location of facial features, which is crucial for micro-expression recognition, because small changes in facial expressions require accurate key point data for analysis.

[0056] Implementation method 4. This implementation method further limits the related song push system based on facial micro-expression recognition described in implementation method 1, and the micro-expression recognition and classification module is composed of a ResNet50 model and a yolov5-cls model.

[0057] The ResNet described in this embodiment introduces a residual learning framework to alleviate the degradation problem in deep networks through the design of residual blocks. Each residual block usually contains two convolutional layers, and there is a jump connection that bypasses these two convolutional layers and directly connects to the back, so that the network can learn residual mapping rather than original mapping. ResNet50 consists of multiple stages, each of which contains multiple residual blocks. The entire network can be divided into the following main parts:

[0058] The input layer is used to accept RGB images of size 224x224;

[0059] The initial convolutional layer is used to extract low-level features;

[0060] Four stages: Each stage contains multiple residual blocks. The residual blocks in these stages can be further divided into two types: the first residual block is at the beginning of each stage and usually changes the size of the feature map; the other residual blocks keep the feature map size unchanged.

[0061] A global average pooling layer is used to reduce the dimensionality;

[0062] Fully connected layer: used to output the final classification results. Note that the original ResNet50 has 1000 output nodes, and this fully connected layer is replaced in this implementation to have only 7 output nodes.

[0063] YOLOv5-cls consists of the following parts:

[0064] Backbone: The main feature extractor uses a structure similar to CSPNet to build the Backbone, which helps improve the efficiency and performance of the model.

[0065] Head structure: Maps the features extracted by Backbone to category labels, including a global average pooling layer to compress the feature map into a vector of fixed length, followed by a fully connected layer to output category probabilities.

[0066] Implementation mode 5: This implementation mode is a further limitation of the related song push system based on facial micro-expression recognition described in Implementation mode 1. The optimization process of the micro-expression recognition classification module is as follows:

[0067] We selected 2,000 images from each of the FER2013 dataset, which contains more than 35,000 grayscale images covering seven facial expressions: anger, disgust, fear, happiness, sadness, surprise, and calmness, and the RAF-DB dataset, which contains 30,000 color images with labels including age, gender, head posture, and lighting conditions, to obtain open source datasets.

[0068] Preprocess the open source dataset, trim the images in the dataset into 640x480 pixel labeled images, and scale them into 224x224 pixel square images, obtaining 1,400 expression data;

[0069] The 5400 images were divided into training set, validation set and test set in a ratio of 7:2:1;

[0070] When receiving face information, the ResNet50 model and the YOLOv5-CLS model are called simultaneously for prediction, and the confidence threshold of each model is set to 0.7;

[0071] If the prediction results of the two models are consistent, the results are directly passed to the emotional state inference module;

[0072] If the prediction results of the two models are inconsistent, the weighted average method is used to calculate the final decision, where the ResNet50 weight is 0.9 and the YOLOv5-CLS model weight is 0.85.

[0073] Calculate the weighted average of the two models’ predictions, re-compare the confidence levels, and make a final decision;

[0074] If only one model returns a prediction, the result is passed directly to the emotional state inference module.

[0075] In this implementation, the image datasets selected from FER2013 and RAF-DB cover a variety of emotions and environmental conditions. Through preprocessing, the images are standardized to a consistent size, which helps to reduce noise when training and validating the model and ensure the accuracy of emotion classification. ResNet50 and YOLOv5-CLS models are used for prediction at the same time, and confidence thresholds are set, which helps to comprehensively consider the advantages of different models and improve the reliability of the overall prediction.

[0076] Furthermore, when the prediction results of the two models are inconsistent, the final judgment is determined by weighted average method. This method allows the system to combine the prediction advantages of the two models, reduce the impact of the error of a single model on the final result, and improve the system's recognition stability of different emotional states.

[0077] When the prediction results of the two models are consistent, the system can reduce the calculation time and directly pass the results to the emotional state inference module, thereby improving the response speed of the system.

[0078] If the prediction results of the two models are inconsistent, the system can more objectively integrate the prediction information of the two through weighted averaging, thus avoiding the accuracy of overall emotion recognition affected by the misjudgment of a single model.

[0079] The image data is divided into training set, validation set and test set in a ratio of 7:2:1. Such a division can ensure the adequacy of the training process, the effectiveness of the validation process and the reliability of the test results.

[0080] In this implementation, by using large-scale and diverse datasets (such as FER2013 and RAF-DB) combined with advanced models (ResNet50 and YOLOv5-CLS), the system is able to capture more subtle micro-expression features, thereby improving the accuracy of emotion recognition. The weighted average method combines the advantages of the two models, reduces the negative impact that may be caused by inconsistent predictions of a single model, and improves the robustness of the system. For situations where the model prediction results are consistent and inconsistent, the system adopts different processing strategies, so that emotional states can be inferred efficiently and accurately in different situations. By directly transmitting the prediction results when the model results are consistent, the system reduces unnecessary computational burden and improves response speed.

[0081] Implementation 6: This implementation is a further limitation of the related song push system based on facial micro-expression recognition described in Implementation 1. The music recommendation module includes:

[0082] According to the user's emotional state, the user's emotional state is converted into a vector representation;

[0083] Label each piece of music in the music database with an emotional tag, and convert the emotional tag of the music into a vector representation;

[0084] The cosine similarity is used to calculate the similarity between the user's emotional state vector and the music emotion label vector;

[0085] Based on the calculated similarity, music with higher similarity is recommended to the user.

[0086] The high similarity mentioned in this implementation manner means that the calculated value of the cosine similarity is greater than 0.9, and the recommended tracks are sorted from high to low according to the similarity.

[0087] Implementation method 7: See Figure 1 The method for pushing related songs based on facial micro-expression recognition described in this embodiment is implemented based on the system described in the first embodiment, and the method includes:

[0088] Collect the user's facial video stream in real time through a camera or other video acquisition device;

[0089] The face detection module is used to detect the video stream and process it frame by frame. The image is converted into RGB format and passed into the face detection model to obtain the bounding box coordinates and key point information of all faces detected in each frame. The coordinate information is retained and the image in the coordinate box is extracted as a new file. After scaling and adjusting to a square, it is passed into the micro-expression recognition and classification module.

[0090] The micro-expression recognition and classification module classifies the micro-expressions of the detected facial regions;

[0091] Based on the micro-expression classification results, the user's emotional state is inferred using a predefined emotion mapping table;

[0092] According to the user's emotional state, matching music is selected from the music database for recommendation.

[0093] Implementation 8: This implementation is a further limitation of the method for pushing related songs based on facial micro-expression recognition described in Implementation 1. The method also includes user feedback collection and model optimization, specifically:

[0094] Integrate a feedback mechanism into the interface of the recommendation system, the feedback content includes the user's preference for the recommended music, changes in emotional state, and whether the music meets the user's expectations;

[0095] Record user feedback data in real time in the database, and use the feedback data to update the recommendation strategy and adjust the strategy parameters of the recommendation algorithm;

[0096] The reward value is adjusted according to user feedback, and the model parameters are updated in real time using the online gradient descent method.

[0097] Embodiment 9. A computer device described in this embodiment includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method for pushing related songs based on facial micro-expression recognition described in any one of embodiments 7 to 8.

[0098] Implementation method ten, a computer-readable storage medium described in this implementation method, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for pushing related songs based on facial micro-expression recognition as described in any one of implementation methods seven to eight are executed.

[0099] Implementation 11. This implementation is a specific example of the related song push system based on facial micro-expression recognition described in Implementation 1, and is also used to explain Implementation 2 to Implementation 6. Specifically:

[0100] Step 1: Data Collection

[0101] The user's facial video stream is collected in real time through a camera or other video acquisition device; the acquisition of the video stream mainly depends on the equipment provided by the actual deployment. During the research and development process, the video stream obtained based on the laptop camera is processed. The frame rate of video acquisition is set to 30 frames / second and the resolution is 640x480 pixels to ensure the quality and real-time performance of the video stream.

[0102] Step 2: Face detection and tracking

[0103] Use a pre-trained face detection model (such as MTCNN) to detect faces in each frame, and the face detection threshold is set to 0.9 to ensure high detection accuracy. The face detection model uses a multi-task cascade convolutional network (MTCNN), which consists of three sub-networks: P-Net, R-Net, and O-Net. Each network is responsible for different tasks and ultimately outputs the bounding box and key point positions of the face. By processing the video stream input in step 1 frame by frame, the image is converted into an RGB format suitable for MTCNN input. After passing it into the model, the bounding box coordinates and key point information of all faces detected in each frame of the image can be obtained, the coordinate information is retained, and the image in the coordinate box is extracted as a new file, which is scaled and adjusted to a square and passed to step 3.

[0104] Step 3: Micro-expression recognition and classification

[0105] Multiple deep learning models (such as ResNet50 and yolov5-cls) are used to classify micro-expressions in the detected facial areas, and the data results are fused and analyzed. The confidence threshold of the output of different models is set to 0.7 to filter out classification results with high uncertainty. Two deep learning models are used for micro-expression classification.

[0106] The ResNet50 network includes multiple residual modules, each of which uses residual connections to avoid the gradient vanishing problem, thereby achieving effective training of deep networks. In this embodiment, a pre-trained ResNet50 model is used as a basis, and the last fully connected layer of the model is replaced so that the number of nodes it outputs is the same as the number of expression categories.

[0107] The model is retrained to suit this embodiment. The classification judgment of expression recognition has extremely high requirements for the training data set. According to the international micro-expression field research, the emotion mapping table includes seven categories: 0 anger, 1 disgust, 2 fear, 3 happy, 4 sad, 5 surprise, and 6 calm. There are open source data sets (FER2013, ck+, etc.) in the world, and valuable images are screened. Taking FER2013 as an example, it contains more than 35,000 grayscale images, all of which are parts of human faces and contain seven basic expressions. This amount of data is very large, and the CSV file obtained from the open source platform is converted into an img file for inspection. It is known that most of the face data is seriously blurred, the amount of information is small, and most of them are Western faces, and even contain some comic-style images that are difficult to judge. The number of expressions in different categories varies greatly, which will affect the generalization ability of the final model. We manually screened these images and carefully selected 2,000 images with clear features and obvious contours. However, since FER2013 is a grayscale image, color images should be added to account for information loss. Therefore, we added the open source dataset RAF-DB for screening. In addition to expression labels, the images in RAF-DB also include additional annotations, such as age, gender, head posture, lighting conditions, occlusion (such as glasses, facial hair or self-occlusion), post-processing operations (such as various filters and special effects), etc., which can greatly enhance the generalization ability of the model. We screened 2,000 images from the nearly 30,000 images in the standard RAF-DB dataset, reduced the label information, and rewrote it into a form suitable for the two models.

[0108] At the same time, in order to achieve the best recognition effect, according to the above seven categories of expression classification standards, 1,000 high-pixel color face data were collected from the society, campus and the Internet, covering different ages, genders, races, head postures, etc., and using labeling tools, a self-made data set with customized labels was constructed. Since the self-made data set comes from natural pictures rather than intercepted face pictures, it will put training pressure on the model and even interfere with the training. Therefore, the obtained pictures are preprocessed, and the expression part is first trimmed into a standard image of 640x480, and then the expression part is selected with a square frame. When passing it into the model, the expression image is scaled to make it a standard square image of 224x224, and finally 1,400 expression data are obtained.

[0109] The above three types of data sets were mixed together, totaling more than 5,400 images, trying to ensure that the number of images in the seven types of labels was equal, and divided into training set, validation set and test set in a ratio of 7:2:1. Finally, training was performed and a highly accurate test model 1 was obtained.

[0110] Furthermore, the cls sub-model in the YOLOv5 model is also used for detection in this embodiment. It is noted that the YOLO model has a low demand for computing resources. In order to ensure the robustness of the entire system in different deployment environments, yolov5-cls is added as the classification basis. The above-prepared data set is still used for training, and a highly efficient test model 2 is obtained.

[0111] When the facial information is passed in step 2, the system calls two models in parallel for synchronous prediction. Each model will give its own confidence table, with 0.7 as the threshold. When the prediction confidence of a model for a certain expression is higher than 0.7, the prediction will be considered, which helps to filter out uncertain prediction results and improve the overall accuracy. If the judgment results given by the two models are consistent, they are directly passed to step 4; if they are different, the total table is queried upwards, and the weighted average method is adopted. The weight of model 1 is 0.9 and the weight of model 2 is 0.85. The confidence of the different judgments given by the two models is recalculated and compared, and the final judgment is made. If only one model returns a prediction result, only that result is passed to step 4.

[0112] Step 4: Emotional state inference

[0113] Based on the micro-expression classification results, the user's emotional state is inferred using a predefined emotion mapping table. The emotion mapping table contains seven categories: 0 angry, 1 disgust, 2 fear, 3 happy, 4 sad, 5 surprise, 6 neutral. Because it takes time for music playback to take effect, it is impossible to change the track immediately according to the target's expression changes. It is necessary to accumulate the target's expression change trend over a period of time, analyze the proportion of various expressions of the target within the time period, pay attention to the proportion of expressions other than 6 neutral, and pass the high proportion results to step 5.

[0114] Step 5: Music Recommendation

[0115] According to the user's emotional state, the most suitable music is selected from the music database for recommendation. The music recommendation algorithm is based on the similarity between the user's emotional state and the music emotion label. Cosine similarity is used as a metric to measure the similarity between the user's emotional state vector and the music emotion label vector. It should be noted that this part requires pre-labeling of the music database, and the copyright issues of the songs need to be paid attention to when using it. The actual needs of the usage scenario also need to be considered when writing the push logic.

[0116] Here is a set of preset solutions. The labels customized for music include the single emotional characteristics of the song itself and the emotional changes that may occur after listening. If you need to mobilize the emotions of the target of calmness, you can push labels 6→0 or 6→3 or 6→4, etc.; if you need to comfort, you can push labels 0→3 or 5→3, etc.; if you need to emphasize the atmosphere of horror or sadness, you can only push tracks 2 or 4. The final push logic also needs to be suitable for the actual deployment scenario.

[0117] Step 6: User feedback collection and model optimization

[0118] Collect user feedback on recommended music through the user interface and optimize the recommendation algorithm based on the feedback. User feedback includes preference, mood changes, etc. Use online learning methods, such as reinforcement learning or online gradient descent, to continuously adjust the parameters of the recommendation algorithm to improve the accuracy and personalization of recommendations.

[0119] In practical applications, the system described in this embodiment can be integrated into a smart retail solution. Such a system can help retailers better understand customers' emotional reactions, thereby improving customer experience and increasing sales conversion rates.

[0120] Specifically, let's assume that the goal is to develop an intelligent camera system that can analyze customer emotions. The system can be integrated into various entertainment or leisure environments, such as cafes, gyms, near fitting rooms, cash registers, etc., to collect customer micro-expression information and recommend music that suits the current mood to enhance the customer experience. At the same time, by analyzing this data, merchants can obtain customers' true feelings about the products and adopt corresponding marketing strategies.

[0121] Among them, the related song push system based on facial micro-expression recognition described in this embodiment adopts a smart camera, that is, a built-in high-performance embedded processor (such as NVIDIA Jetson series) to capture video streams. An integrated ambient light sensor ensures that high-quality images can be obtained under different lighting conditions. The music player is connected to the network and the cloud to play the pushed music.

[0122] The micro-expression recognition module is deployed in the embedded system inside the smart camera, responsible for real-time processing of video streams, identifying micro-expressions and making decisions on push notifications. The music recommendation engine recommends corresponding music lists based on emotional tags. Data processing and storage locally stores and processes the identified data to reduce the cost and latency of cloud services. The data analysis platform collects and analyzes data from various stores to generate reports and recommendations. The database contains music of various types and styles and their metadata. The user preference learning module is used to learn user preferences over a long period of time in order to provide more personalized recommendations.

[0123] The main uses of the present invention include but are not limited to:

[0124] Emotion monitoring and analysis: Through advanced computer vision technology and deep learning algorithms, the system can accurately capture and analyze the user's facial micro-expressions, thereby inferring the user's current emotional state.

[0125] Personalized music recommendation: Based on the user's emotional state, the system can select the music that best suits the current mood from the huge music library and provide users with a personalized music playlist.

[0126] Emotional regulation and improvement: By playing music that matches the user's emotional state, the system helps users regulate their emotions, relieve stress, and improve their mood.

[0127] The present invention can be directly applied to the following fields:

[0128] In the field of health care, the invention can be used to help individuals regulate their emotions by listening to music, and relieve anxiety or depression symptoms. In addition, it can also be used for mental health counseling and support, helping professionals monitor the emotional state of patients and identify potential mental health problems in a timely manner.

[0129] In the field of entertainment, the present invention can provide personalized music recommendation services, automatically adjust the playlist according to the user's emotional state, and provide the user with a more pleasant leisure experience.

[0130] In the field of smart homes, the present invention can be integrated into a smart audio system to automatically adjust the type of music played by analyzing the emotional state of the occupants to create a suitable family atmosphere.

[0131] In the field of education, the invention can be used in emotion management courses to help students better understand and manage their emotions. In addition, it can also be used to provide emotional feedback to help teachers understand students' engagement and emotional state.

[0132] In a remote working environment, the present invention can be used for remote conference emotion monitoring to help organizers better manage the conference atmosphere, and can be used for employee welfare programs to develop personalized welfare measures and improve employee satisfaction by monitoring the emotional state of employees.

[0133] Although the preferred embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

[0135] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A related song push system based on facial micro-expression recognition, characterized in that: The system comprises: Data collection module, facial detection module, micro-expression recognition and classification module, emotional state inference module and music recommendation module; The data acquisition module is used to collect facial video streams and transmit the facial video streams to the facial detection module, which is used to receive facial video streams and detect facial information in each frame of the facial video stream; the facial information is transmitted to the micro-expression recognition and classification module, which classifies micro-expressions of the detected facial area and inputs the classification results to the emotional state inference module for fusion judgment; the emotional state inference module is used to infer the emotional state of the user and input the emotional state of the user into the music recommendation module; the music recommendation module selects matching music from the music database for recommendation based on the emotional state of the user.

2. The related song push system based on facial micro-expression recognition according to claim 1 is characterized in that: The frame rate of the facial video stream collected in the data collection module is 30 frames per second and the resolution is 640x480 pixels.

3. The related song push system based on facial micro-expression recognition according to claim 1 is characterized in that: The face detection module is formed by a face detection model MTCNN, which is composed of three sub-networks: P-Net, R-Net and O-Net. The P-Net receives an input image and generates a candidate face region. The P-Net outputs the confidence of each region. When the output confidence is higher than 0.9, the candidate region is retained and passed to the R-Net. The R-Net receives the candidate region passed by the P-Net and performs face detection in the candidate region. The R-Net outputs the confidence of each region and fine-tunes the bounding box. When the output confidence is higher than 0.9, it is determined to be a valid face region and passed to the O-Net. The O-Net receives the refined face region output by the R-Net and performs final face key point positioning. When the confidence output by the O-Net is higher than 0.9, the bounding box coordinates of the face are obtained and the key point positioning results are retained.

4. The related song push system based on facial micro-expression recognition according to claim 1 is characterized in that: The micro-expression recognition and classification module consists of a ResNet50 model and a yolov5-cls model.

5. The related song push system based on facial micro-expression recognition according to claim 1 is characterized in that: The optimization process of the micro-expression recognition and classification module is as follows: We selected 2,000 images from each of the FER2013 dataset, which contains more than 35,000 grayscale images covering seven facial expressions: anger, disgust, fear, happiness, sadness, surprise, and calmness, and the RAF-DB dataset, which contains 30,000 color images with labels including age, gender, head posture, and lighting conditions, to obtain open source datasets. Preprocess the open source dataset, trim the images in the dataset into 640x480 pixel labeled images, and scale them into 224x224 pixel square images, obtaining 1,400 expression data; The 5400 images were divided into training set, validation set and test set in a ratio of 7:2:1; When receiving face information, the ResNet50 model and the YOLOv5-CLS model are called simultaneously for prediction, and the confidence threshold of each model is set to 0.7; If the prediction results of the two models are consistent, the results are directly passed to the emotional state inference module; If the prediction results of the two models are inconsistent, the weighted average method is used to calculate the final decision, where the ResNet50 weight is 0.9 and the YOLOv5-CLS model weight is 0.

85. Calculate the weighted average of the two models’ predictions, re-compare the confidence levels, and make a final decision; If only one model returns a prediction, the result is passed directly to the emotional state inference module.

6. The related song push system based on facial micro-expression recognition according to claim 1 is characterized in that: The music recommendation module includes: According to the user's emotional state, the user's emotional state is converted into a vector representation; Label each piece of music in the music database with an emotional tag, and convert the emotional tag of the music into a vector representation; The cosine similarity is used to calculate the similarity between the user's emotional state vector and the music emotion label vector; Based on the calculated similarity, music with higher similarity is recommended to the user.

7. A method for pushing related songs based on facial micro-expression recognition, characterized in that: The method is implemented based on the system according to claim 1, and the method comprises: Collect the user's facial video stream in real time through a camera or other video acquisition device; The face detection module is used to detect the video stream and process it frame by frame. The image is converted into RGB format and passed into the face detection model to obtain the bounding box coordinates and key point information of all faces detected in each frame. The coordinate information is retained and the image in the coordinate box is extracted as a new file. After scaling and adjusting to a square, it is passed into the micro-expression recognition and classification module. The micro-expression recognition and classification module classifies the micro-expressions of the detected facial regions; Based on the micro-expression classification results, the user's emotional state is inferred using a predefined emotion mapping table; According to the user's emotional state, matching music is selected from the music database for recommendation.

8. The method for pushing related songs based on facial micro-expression recognition according to claim 7, characterized in that: The method also includes user feedback collection and model optimization, specifically: Integrate a feedback mechanism into the interface of the recommendation system, the feedback content includes the user's preference for the recommended music, changes in emotional state, and whether the music meets the user's expectations; Record user feedback data in real time in the database, and use the feedback data to update the recommendation strategy and adjust the strategy parameters of the recommendation algorithm; The reward value is adjusted according to user feedback, and the model parameters are updated in real time using the online gradient descent method.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method for pushing related songs based on facial micro-expression recognition according to any one of claims 7-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for pushing related songs based on facial micro-expression recognition as described in any one of claims 7 to 8.

Images