Music recommendation method, system and equipment based on user emotion monitoring

Abstract

The invention relates to a music recommendation method, system and device based on user emotion monitoring, and relates to the technical field of music recommendation, and the method comprises the steps: monitoring a music playing instruction in response to the emotion of a user, and collecting a face image sequence of the user; key face image extraction is carried out, a key face image sequence and a non-key image sequence are obtained, and user basic emotion recognition is carried out according to the key face image sequence; performing ambient light analysis according to the non-key image sequence, and performing user emotion correction to obtain a corrected emotion level; and performing music recommendation according to the corrected emotion level. According to the method and the device, the technical problem that the music recommendation result cannot really match the user demand due to the fact that the emotional state of the user cannot be quickly and accurately recognized in the traditional method can be solved; the efficiency, accuracy and reliability of user emotion recognition can be remarkably improved, then music meeting the requirements of a user can be accurately recommended according to the emotion state of the user, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of music recommendation, and particularly to a music recommendation method, system and device based on user emotion monitoring. Background Art

[0002] With the continuous development of intelligent technologies, emotion recognition technology has been more widely applied in various fields. Especially in personalized recommendation systems, emotion recognition has become an important means to improve user experience.

[0003] Currently, traditional music recommendation systems mainly rely on static information such as users' historical behavior data, interest preferences, and social networks for recommendation. Although they can provide certain personalized services, they often ignore the impact of users' emotional fluctuations in different situations, and cannot quickly and accurately identify users' immediate emotional states. As a result, the recommended music often fails to match users' emotional needs, and cannot achieve precise emotional adjustment, thus affecting user experience. Summary of the Invention

[0004] Aiming at the technical problem in the traditional music recommendation method that the music recommendation result cannot truly match the user's needs due to the inability to quickly and accurately identify the user's emotional state, the present invention provides a music recommendation method, system and device based on user emotion monitoring to solve this problem.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: In a first aspect, the present invention provides a music recommendation method based on user emotion monitoring, including: in response to a user's emotion monitoring music playing instruction by a music playing device, collecting a sequence of facial images of the user; according to the sequence of facial images, extracting key facial images to obtain a sequence of key facial images, a sequence of key coefficients, a sequence of non-key images and a sequence of non-key coefficients, and based on the sequence of key facial images and the sequence of key coefficients, performing user basic emotion recognition to obtain a basic emotion level; performing ambient light analysis based on the sequence of non-key images and the sequence of non-key coefficients to obtain ambient light feature information, and performing user emotion correction to obtain a corrected emotion level; and based on the corrected emotion level, performing music recommendation to obtain a recommendation result.

[0006] Preferably, the music recommendation method based on user emotion monitoring further includes: monitoring by the music playing device whether the user triggers an emotion monitoring music playing instruction; if so, collecting the sequence of facial images of the user at a preset frequency, and if not, continuing to monitor.

[0007] Preferably, the music recommendation method based on user emotion monitoring further includes: performing face proportion recognition on multiple face images in the face image sequence to obtain a face proportion coefficient sequence; according to the face proportion coefficient sequence, screening a preset proportion of face images with the largest face proportion coefficient as key face images, arranging them in chronological order to obtain a key face image sequence, and generating a key coefficient sequence based on the face proportion coefficient corresponding to each key face image; obtaining other face images as non-key images, arranging them in chronological order to obtain a non-key image sequence, and generating a non-key coefficient sequence based on the face proportion coefficient corresponding to each non-key face image.

[0008] Preferably, the music recommendation method based on user emotion monitoring further includes: according to user sample data, collecting a sample face image set, annotating the face pixel proportion in each sample face image to obtain a sample face proportion coefficient set; using a convolutional neural network to construct a face proportion recognizer, and using the sample face image set and the sample face proportion coefficient set to perform supervised training and testing on the face proportion recognizer until the requirements are met; inputting multiple face images in the face image sequence into the face proportion recognizer, and identifying and outputting to obtain a face proportion coefficient sequence.

[0009] Preferably, the music recommendation method based on user emotion monitoring further includes: according to the user emotion recognition historical data, collecting a sample key face image set, identifying the emotion level in each sample key face image to obtain a sample basic emotion level set; using a convolutional neural network to construct a face emotion recognizer, and using the sample key face image set and the sample basic emotion level set to perform supervised training and testing on the face emotion recognizer until the requirements are met; inputting multiple key face images in the key face image sequence into the face emotion recognizer, and outputting to obtain a recognized emotion level sequence; according to multiple key coefficients in the key coefficient sequence, performing weighted calculation on multiple recognized emotion levels in the recognized emotion level sequence to obtain a basic emotion level.

[0010] Preferably, the music recommendation method based on user emotion monitoring further includes: performing grayscale processing on the non-key image sequence and calculating the average grayscale value to obtain an average grayscale value sequence; according to the non-key coefficient sequence, calculating to obtain an environment coefficient sequence, and using multiple environment coefficients in the environment coefficient sequence to perform weighted calculation on multiple average grayscale values in the average grayscale value sequence to obtain a weighted grayscale value as environmental light feature information; according to the environmental light feature information, classifying and indexing to obtain an emotion influence coefficient; using the emotion influence coefficient to correct the basic emotion level to obtain a corrected emotion level.

[0011] Preferably, the music recommendation method based on user emotion monitoring further includes: collecting a set of sample ambient light feature information according to the user emotion recognition data within a historical time period, and collecting the change range of the user emotion level under different sample ambient light feature information, which is marked as a set of sample emotion influence coefficients; using the set of sample ambient light feature information as an index header and the set of sample emotion influence coefficients as index values to construct an emotion influence index database; inputting the ambient light feature information into the emotion influence index database, and obtaining the emotion influence coefficient through index classification.

[0012] Preferably, the music recommendation method based on user emotion monitoring further includes: obtaining the historical music recommendation data of the user, collecting a set of sample corrected emotion levels, and the music selected by the user for playback when different sample corrected emotion levels appear, to obtain multiple sample music subsets; constructing the mapping relationship between the set of sample corrected emotion levels and the multiple sample music subsets to obtain a music recommendation database; inputting the corrected emotion level into the music recommendation database, mapping to obtain a music subset, and randomly recommending music within the music subset to obtain a recommendation result.

[0013] In a second aspect, the present invention provides a music recommendation system based on user emotion monitoring, including: a facial image acquisition module, configured to collect a sequence of facial images of a user through a music playback device in response to a user emotion monitoring music playback instruction; a basic emotion recognition module, configured to extract key facial images according to the sequence of facial images to obtain a sequence of key facial images, a sequence of key coefficients, a sequence of non-key images, and a sequence of non-key coefficients, and perform user basic emotion recognition according to the sequence of key facial images and the sequence of key coefficients to obtain a basic emotion level; a user emotion correction module, configured to perform ambient light analysis according to the sequence of non-key images and the sequence of non-key coefficients to obtain ambient light feature information, and perform user emotion correction to obtain a corrected emotion level; a music recommendation module, configured to perform music recommendation according to the corrected emotion level to obtain a recommendation result.

[0014] In a third aspect, the present invention further provides an electronic device, including: at least one processor; a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the method described in any item of the first aspect above.

[0015] The beneficial effects of the present invention are as follows: The music playback device monitors whether the user triggers an emotion monitoring music playback instruction. If so, a sequence of facial images of the user is collected at a preset frequency. Then, based on the sequence of facial images, key facial images are extracted to obtain a sequence of key coefficients, a sequence of key facial images, a sequence of non-key images, and a sequence of non-key coefficients. Next, based on the sequence of key facial images and the sequence of key coefficients, the basic emotion of the user is recognized to obtain a basic emotion level. On the other hand, based on the sequence of non-key images and the sequence of non-key coefficients, environmental light analysis is performed to obtain environmental light characteristic information, and the basic emotion level is corrected to obtain a corrected emotion level. Finally, based on the corrected emotion level, music recommendation is performed to obtain a recommendation result. That is to say, by combining intelligent recognition algorithms for user emotion recognition, and at the same time correcting the emotion recognition result according to environmental characteristics, and performing analysis and calculation based on the sequence of key coefficients and the sequence of non-key coefficients, the accuracy is higher, which can significantly improve the efficiency, accuracy, and reliability of user emotion recognition, and then can accurately recommend music that meets the user's needs according to the user's emotion state, improving the user experience. Description of the Drawings

[0016] Figure 1 It is a schematic flowchart of a music recommendation method based on user emotion monitoring provided by the present invention; Figure 2 It is a schematic structural diagram of a music recommendation system based on user emotion monitoring provided by the present invention; Figure 3 It is a schematic structural diagram of an electronic device provided by the present invention.

[0017] In the drawings, the components represented by each reference numeral are described as follows: Facial image acquisition module 11, basic emotion recognition module 12, user emotion correction module 13, music recommendation module 14, electronic device 500, memory 510, processor 520, first computer program 511. Detailed Embodiments

[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0019] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0020] In the description of the present invention, the term "for example" is used to mean "serving as an example, illustration, or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0021] Embodiment 1, as Figure 1 shown, the embodiment of the present invention provides a music recommendation method based on user emotion monitoring, which specifically includes the following steps: S10: Responding to a user's emotion monitoring music playing instruction through a music playing device, and collecting a sequence of the user's facial images.

[0022] Further, step S10 of the present invention further includes: S11: Monitoring by the music playing device whether the user triggers an emotion monitoring music playing instruction; S12: If so, collecting a sequence of the user's facial images at a preset frequency, and if not, continuing the monitoring.

[0023] Specifically, first, the music playback device monitors whether the user triggers an emotion monitoring music playback instruction, which can be achieved through various interaction methods (such as pressing the play button, voice command, gesture recognition, etc.). For example, by equipping sensors (such as touch sensors, voice recognition modules, etc.) to identify the user's instruction. Then, when it is detected that the user triggers the emotion monitoring music playback instruction, the built-in camera or an external camera device is used to collect the user's facial images at a preset frequency. The preset frequency can be dynamically adjusted according to requirements, such as collecting an image every second or every 3 seconds, to ensure the timely capture of emotional changes and obtain a sequence of facial images. If it is not detected that the user triggers the emotion monitoring music playback instruction, the facial image collection is not performed, and the monitoring continues. This process ensures that emotion monitoring can be carried out when the user actively requests to play music and does not excessively occupy system resources when the user does not trigger it.

[0024] S20: According to the sequence of facial images, key facial images are extracted to obtain a sequence of key facial images, a sequence of key coefficients, a sequence of non-key images, and a sequence of non-key coefficients. Based on the sequence of key facial images and the sequence of key coefficients, user basic emotion recognition is performed to obtain the basic emotion level.

[0025] Furthermore, step S20 of the present invention further includes: S21: Facial proportion recognition is performed on multiple facial images in the sequence of facial images to obtain a sequence of facial proportion coefficients.

[0026] Furthermore, step S21 of the present invention further includes: S211: According to user sample data, a set of sample facial images is collected, and the facial pixel proportion in each sample facial image is marked to obtain a set of sample facial proportion coefficients; S212: A convolutional neural network is used to construct a facial proportion recognizer, and the set of sample facial images and the set of sample facial proportion coefficients are used to perform supervised training and testing on the facial proportion recognizer until the requirements are met; S213: Multiple facial images in the sequence of facial images are input into the facial proportion recognizer, and the facial proportion coefficients are recognized and output to obtain a sequence of facial proportion coefficients.

[0027] Specifically, according to user sample data, a large number of user facial image samples are collected. These images cover different facial features and angles to obtain a set of sample facial images. Then, the facial pixel proportion in each sample facial image is marked. For example, image processing algorithms are used to identify and calculate the facial pixel proportion in each facial image, that is, the ratio of the number of facial pixel points in the facial image to the total number of pixel points in the image, and the facial pixel proportion is set as the sample facial proportion coefficient to obtain a set of sample facial proportion coefficients, where the sample facial images and the sample facial proportion coefficients correspond one by one.

[0028] Convolutional neural network is a classic deep learning model for processing image data, which is good at extracting local features from images and performing efficient classification or regression. Then, a facial proportion recognizer is constructed based on the convolutional neural network. The task of the facial proportion recognizer is to automatically identify the proportion coefficient of the facial area from facial images, including an input layer (the input data is facial images), a convolutional layer, a pooling layer, a fully connected layer, and an output layer (the output data is the facial proportion coefficient). The convolutional layer is used to extract spatial features in facial images. Convolutional operations can identify the contours and facial feature features of the face, which helps to calculate the facial proportion; the pooling layer is usually after the convolutional layer, which is used to reduce the size of the feature map and help extract more global features. Pooling operations can help the model adapt to facial images of different scales and angles; the fully connected layer is used to map the extracted facial features to the output of the proportion coefficient.

[0029] Furthermore, the sample facial image set and the sample facial proportion coefficient set are used as sample data, and the sample data is divided into a training set and a test set according to a predetermined ratio. For example, the training set accounts for 90% and is used to train the model to help the model learn the pattern of facial proportion; the test set accounts for 10% and is used to evaluate the performance of the trained model. Then, with the sample facial image as the input and the sample facial proportion coefficient as the supervision, the training set and the test set are used to perform supervised training and testing on the facial proportion recognizer. During the training process, first, each facial image in the training set is input into the convolutional neural network model. The features of the image are extracted through structures such as the convolutional layer and the pooling layer, and the output of the model, that is, the predicted value of the facial proportion, is generated through the fully connected layer. Then, the error between the output (predicted value) of the model and the actual facial proportion coefficient (true value) is calculated through a loss function (such as mean squared error). Then, the gradient of the loss function with respect to the model parameters is calculated through the backpropagation algorithm, and an optimization algorithm (such as SGD) is used to update the weights in the model to reduce the value of the loss function. The process of forward propagation and backpropagation is repeated, and the model is trained through multiple iterations. Each iteration will update the weights in the convolutional neural network until the value of the loss function converges to a small degree and the model can better fit the training data. During the testing process, the facial images in the test set are input into the trained convolutional neural network model, and the output of the model, that is, the predicted facial proportion coefficient, is calculated through forward propagation. Then, the predicted output of the model is compared with the true proportion coefficient to calculate the performance of the model on the test set. If the performance of the model on the test set is good (such as low MSE), it indicates that the model can effectively identify the facial proportion and has good generalization ability, and the training converges to obtain a trained facial proportion recognizer.

[0030] Finally, input multiple facial images in the facial image sequence into the facial proportion recognizer in sequence to output a facial proportion coefficient sequence. By constructing a facial proportion recognizer through a convolutional neural network, the facial proportion coefficient in the facial image can be accurately recognized, effectively improving the efficiency and accuracy of facial proportion recognition, and providing data support for subsequent emotion analysis and music recommendation.

[0031] S22: According to the facial proportion coefficient sequence, screen a preset proportion of facial images with the largest facial proportion coefficients as key facial images, arrange them in chronological order to obtain a key facial image sequence, and generate a key coefficient sequence based on the facial proportion coefficients corresponding to each key facial image; S23: Obtain other facial images as non-key images, arrange them in chronological order to obtain a non-key image sequence, and generate a non-key coefficient sequence based on the facial proportion coefficients corresponding to each non-key facial image.

[0032] Specifically, according to the facial proportion coefficient sequence and a preset proportion (such as 10%), screen out the facial images with the largest facial proportion coefficients as key facial images. For example, assume there are a total of 100 facial images and the preset proportion is 10%, then select the top 10 facial images with the largest proportion coefficients. These facial images represent the moments when the user's facial changes are more obvious and are the key sources of emotion information; then arrange the selected key facial images in chronological order to obtain a key facial image sequence, that is, these images can accurately reflect the changes in the user's emotional state at different time points.

[0033] Further, obtain the facial proportion coefficients corresponding to each key facial image in the key facial image sequence as key coefficients, and then obtain multiple key coefficients, arrange them in chronological order to obtain a key coefficient sequence.

[0034] The key coefficient reflects the facial proportion coefficient in the key facial image. The larger the key coefficient, the larger the proportion of the facial image, and the higher the accuracy of using this key facial image for user emotion recognition, avoiding the situation of inaccurate emotion recognition caused by incomplete facial image acquisition.

[0035] On the other hand, in the facial image sequence, remove those facial images that have been used as key images, and the remaining are non-key images; then arrange these non-key images in chronological order to obtain a non-key facial image sequence. These images contain fewer facial features. Although they are not as important as key images in emotion recognition, they still provide additional environmental or state information.

[0036] Further, obtain the facial proportion coefficient corresponding to each non-critical facial image as the non-critical coefficient, obtain multiple non-critical coefficients, and then arrange them in chronological order to obtain the non-critical facial image sequence. The non-critical coefficient reflects the proportion of the facial image within the non-critical facial image. When subsequently identifying the influence of environmental brightness on the user's emotion based on the non-critical image, the larger the proportion of the environmental image within the non-critical facial image with a smaller non-critical coefficient, the higher the accuracy of identifying the environmental brightness.

[0037] By dividing the critical images and non-critical images, the complexity of data processing can be reduced, and the accuracy and efficiency of emotion recognition can be effectively improved.

[0038] Furthermore, step S20 of the present invention further includes: S24: According to the historical data of user emotion recognition, collect a set of sample critical facial images, label the emotion level within each sample critical facial image to obtain a set of sample basic emotion levels; S25: Use a convolutional neural network to construct a facial emotion recognizer, and use the set of sample critical facial images and the set of sample basic emotion levels to perform supervised training and testing on the facial emotion recognizer until the requirements are met; S26: Input the multiple critical facial images within the critical facial image sequence into the facial emotion recognizer, and output to obtain a sequence of recognized emotion levels; S27: According to the multiple critical coefficients within the critical coefficient sequence, perform weighted calculation on the multiple recognized emotion levels within the sequence of recognized emotion levels to obtain the basic emotion level.

[0039] Specifically, obtain the historical data of user emotion recognition, that is, collect a large amount of user emotion marking data from the existing historical data of user emotion recognition. These data usually include the facial image sequence of the user in different emotion states and the corresponding emotion states (such as happy, angry, sad, calm, etc.). Then, process the emotion data of each user, extract the critical facial images related to each emotion state from them, that is, those facial images that can reflect the emotion change, to obtain a set of sample critical facial images; then label the emotion level within each sample critical facial image. The emotion level is usually based on a predefined emotion classification system. For example, it is divided into different emotion categories such as happy, angry, sad, anxious, calm, etc., such as 10 emotion levels from 1 to 10. The higher the level, the more positive the emotion, and the lower the level, the more negative the emotion. Each set of critical facial images will correspond to a specific emotion level label. For example, if the critical facial image shows a happy emotion state, then the critical facial image will be labeled with an emotion level such as happy; obtain a set of sample basic emotion levels, where the sample critical facial image sequence and the sample basic emotion levels correspond one by one.

[0040] Next, a facial emotion recognizer is constructed based on a convolutional neural network. The facial emotion recognizer is used to recognize the emotional state of a user through a sequence of facial images, and includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input data of the input layer is the key facial image, and the output data of the output layer is the basic emotion level. The convolutional layer is used to extract local features of the facial image through multiple convolutional kernels, such as key facial regions like eyes, mouth, eyebrows, etc. The pooling layer is used for dimensionality reduction processing to reduce the amount of computation and extract more abstract features. The fully connected layer is used to combine the features extracted by the convolutional layer and the pooling layer and output a vector for further classification. Then, the set of sample key facial images and the set of sample basic emotion levels are used as training data, and the training data is divided into a training set and a test set according to a predetermined ratio. For example, the training set accounts for 90% and the test set accounts for 10%.

[0041] Further, using the sample key facial images as input and the sample basic emotion levels as supervision, the facial emotion recognizer is supervised and trained and tested using the training set and the test set. During the training process, first, the input image is passed through the network to calculate the predicted value of the emotion category output by the network. Next, according to the network output and the actual emotion level label, the loss function (such as cross-entropy loss) is calculated to measure the gap between the prediction result and the true result. Then, through the backpropagation algorithm, the parameters of the convolutional neural network (such as the weights of the convolutional kernels) are adjusted to minimize the loss function and optimize the network performance. Through multiple training iterations, the accuracy of the model is gradually improved until the loss function tends to be stable and meets the required accuracy.

[0042] After the training is completed, the trained model is evaluated using the test set (i.e., the data in the sample data that did not participate in the training). By inputting the facial image sequence through the test set, the predicted emotion category of the recognizer is obtained and compared with the true label to evaluate indicators such as the accuracy, precision, and recall rate of the model. If the model performs well on the test set and reaches the preset performance indicators (such as the accuracy rate reaches more than 90%), it is considered that the training is completed, and a trained facial emotion recognizer is obtained.

[0043] Finally, multiple key facial images within the key facial image sequence are input into the facial emotion recognizer for analysis, and multiple recognized emotion levels are respectively output to form a recognized emotion level sequence.

[0044] Furthermore, multiple key coefficients within the key coefficient sequence are used to perform weighted calculation on multiple recognized emotion levels within the recognized emotion level sequence. For example, the ratio of each key coefficient to the sum of multiple key coefficients is calculated as the weight, and weighted calculation is performed on multiple recognized emotion levels to obtain the basic emotion level. Among them, the larger the key coefficient, the more the proportion of the facial image within the key facial image, and the higher the accuracy of the recognized emotion level, and the greater the weight.

[0045] By constructing a facial emotion recognizer based on a convolutional neural network, it is possible to automatically recognize the emotional state of a user through a sequence of facial images of the user, and perform weighted calculations according to the sequence of key coefficients to provide more accurate emotion classification results, which can significantly improve the intelligence and accuracy of user emotion recognition and provide reliable data support for subsequent user emotion analysis and music recommendation.

[0046] S30: Perform ambient light analysis based on the non-critical image sequence and the non-critical coefficient sequence to obtain ambient light feature information, perform user emotion correction, and obtain a corrected emotion level.

[0047] Furthermore, step S30 of the present invention further includes: S31: Grayscale the non-critical image sequence and calculate the average grayscale value to obtain an average grayscale value sequence.

[0048] Specifically, first, grayscale the non-critical image sequence, that is, grayscale each non-critical image. Grayscaling is the process of converting a color image into a grayscale image. Its purpose is to simplify image information, eliminate the interference of color information, and focus on brightness and contrast features. The grayscaling method is to convert the RGB (three color channels) values into grayscale values by weighted averaging. Each pixel in the grayscaled image will become a single grayscale value, representing the brightness of the image; after obtaining all the grayscale values in multiple non-critical images, calculate the mean value of all the pixel grayscale values in each non-critical image to obtain multiple average grayscale values, and obtain an average grayscale value sequence. The grayscale value reflects the illumination intensity in the environment and is used as the basis for subsequent emotion recognition correction because changes in ambient light may affect the user's emotions. For example, strong sunlight or dim light may affect the user's emotional expression.

[0049] S32: Calculate an environmental coefficient sequence according to the non-critical coefficient sequence, and use multiple environmental coefficients in the environmental coefficient sequence to perform weighted calculations on multiple average grayscale values in the average grayscale value sequence to obtain weighted grayscale values as ambient light feature information; S33: Classify and index according to the ambient light feature information to obtain an emotion influence coefficient.

[0050] In the embodiments of the present application, according to multiple non-critical coefficients in the non-critical coefficient sequence, an environmental coefficient is calculated, and the magnitude of the non-critical coefficient is negatively correlated with the magnitude of the environmental coefficient. For example, use 1 minus the non-critical coefficient as the environmental coefficient, and thus calculate the environmental coefficient sequence in this way. The environmental coefficient reflects the proportion of the environmental image in the non-critical facial image. The larger the environmental coefficient, the larger the proportion of the environmental image, and the higher the accuracy of the non-critical facial image in reflecting the environmental brightness.

[0051] Further, multiple environmental coefficients within the environmental coefficient sequence are used to perform weighted calculation on multiple average gray values within the average gray value sequence. For example, the ratio of each environmental coefficient to the sum of multiple environmental coefficients within the environmental coefficient sequence is calculated as the weight, and weighted calculation is performed on multiple average gray values within the average gray value sequence to obtain weighted gray values. The weighted gray values are used as environmental light feature information.

[0052] Further, classification indexing is performed based on the environmental light feature information to obtain the degree of influence of environmental light on emotions, that is, the emotion influence coefficient.

[0053] Further, step S33 of the present invention further includes: S331: According to the user emotion recognition data within the historical time, collect a set of sample environmental light feature information, and collect the change range of the user emotion level under different sample environmental light feature information, which is marked as a set of sample emotion influence coefficients; S332: Use the set of sample environmental light feature information as the index header and the set of sample emotion influence coefficients as the index value to construct an emotion influence index database; S333: Input the environmental light feature information into the emotion influence index database, and index classification to obtain the emotion influence coefficient.

[0054] Specifically, first, obtain the user emotion recognition data within a historical time period (such as within the most recent month), and collect the ambient light feature information within different time periods based on the user emotion recognition data as the sample ambient light feature information to construct a sample ambient light feature information set. Next, collect the change range of the user emotion level under different sample ambient light feature information, that is, for each group of ambient light feature information, record the user emotion level (such as an emotion score from 1 to 5), calculate the impact of the change in ambient light on the user emotion. For example, under certain lighting conditions, the user emotion may have large fluctuations, mark these change ranges as the emotion impact coefficients to reflect the degree of influence of ambient light on the user emotion, and construct a sample emotion impact coefficient set, where the sample ambient light feature information and the sample emotion impact coefficients correspond one by one. For example, when the lighting is weak, the user's emotion is low, and the sample emotion impact coefficient is less than 1 and the smaller it is, such as 0.6. Next, use multiple sample ambient light feature information in the sample ambient light feature information set as the index headers of the database, and use multiple sample emotion impact coefficients in the sample emotion impact coefficient set as the index values. Among them, each ambient light feature (that is, the average gray value of the grayscale image) is used as an index keyword, and each keyword will correspond to an emotion impact coefficient. These coefficients reflect the sensitivity of the user emotion change under specific ambient light conditions. Then, according to the corresponding relationship between the index headers and the index values, construct an emotion impact index database by mapping multiple index headers and multiple index values. In the emotion impact index database, an index relationship is formed between the ambient light feature information and the emotion impact coefficient. When a specific ambient light feature is input, the corresponding emotion impact coefficient can be quickly found.

[0055] Then input the ambient light feature information into the emotion impact index database for index classification. The database performs classification matching according to the input ambient light feature information to find the most relevant emotion impact coefficient. For example, the database contains a series of gray value ranges and the corresponding emotion impact coefficients. The input gray value will be classified into a certain range and the corresponding emotion impact coefficient will be returned. Obtain the emotion impact coefficient, and the emotion impact coefficient is used to correct the basic emotion level, thereby improving the accuracy of the emotion recognition result.

[0056] S34: Use the emotion impact coefficient to correct the basic emotion level to obtain a corrected emotion level.

[0057] Specifically, use the emotion impact coefficient to correct the basic emotion level, that is, adjust the basic emotion level up or down accordingly according to the emotion impact coefficient. For example, multiply the basic emotion level by the emotion impact coefficient, and use the product of the two as the corrected emotion level. The corrected emotion level represents the user's true emotion state considering the ambient light features and can more accurately reflect the user's actual emotion under the current environmental conditions.

[0058] By constructing an emotional impact index database and using the ambient light feature information index to obtain the emotional impact coefficient to correct the user's emotional level, the influence of ambient light on the user's emotions can be effectively considered, making the emotional recognition result more accurate, avoiding the emotional judgment error caused by the change of ambient light, and thus optimizing the accuracy of the overall music recommendation process and the user experience.

[0059] S40: According to the corrected emotional level, perform music recommendation to obtain a recommendation result.

[0060] Furthermore, step S40 of the present invention further includes: S41: Obtain the user's historical music recommendation data, collect a set of sample corrected emotional levels, and the music selected by the user for playback when different sample corrected emotional levels appear, to obtain multiple sample music subsets; S42: Construct a mapping relationship between the set of sample corrected emotional levels and the multiple sample music subsets to obtain a music recommendation database; S43: Input the corrected emotional level into the music recommendation database, map to obtain a music subset, and perform random music recommendation within the music subset to obtain a recommendation result.

[0061] Specifically, obtain the user's historical music recommendation data, which includes the user's previous emotional states and the music they selected for playback in that emotional state; then, according to the historical music recommendation data, collect multiple sample corrected emotional levels, that is, the corrected emotional levels of the user in different states (environmental light and other factors have been considered), construct a set of sample corrected emotional levels, and collect the music selected by the user for playback when different sample corrected emotional levels appear, to obtain multiple sample music subsets, where the sample corrected emotional level and the sample music subset are in one-to-one correspondence. Then, establish a mapping relationship between the sample corrected emotional level and the sample music subset, that is, the music type or style preferred by the user at a specific emotional level; then, based on the mapping relationship, construct a music recommendation database according to the set of sample corrected emotional levels and the multiple sample music subsets, and the music recommendation database is used to map the corrected emotional level to the corresponding music preference.

[0062] Then: Input the corrected emotional level into the music recommendation database for mapping and matching. By querying the database, the corresponding music subset can be mapped, that is, the music that the user is more likely to like in this emotional state; then, after obtaining the music subset, perform random recommendation within the subset to recommend one or more pieces of music that match the user's current emotional state to the user as the music recommendation result, and this recommendation result can best meet the user's emotional needs, thereby improving the user experience.

[0063] The music recommendation method based on user emotion monitoring provided by the embodiment of the present invention has at least the following technical effects: The music playback device monitors whether the user triggers an emotion monitoring music playback instruction. If so, a sequence of facial images of the user is collected at a preset frequency. Then, based on the sequence of facial images, key facial images are extracted to obtain a sequence of key coefficients, a sequence of key facial images, a sequence of non-key images, and a sequence of non-key coefficients. Next, based on the sequence of key facial images and the sequence of key coefficients, the basic emotion of the user is recognized to obtain a basic emotion level. On the other hand, based on the sequence of non-key images and the sequence of non-key coefficients, environmental light analysis is performed to obtain environmental light characteristic information, and the basic emotion level is corrected to obtain a corrected emotion level. Finally, based on the corrected emotion level, music recommendation is performed to obtain a recommendation result. That is to say, by combining intelligent recognition algorithms for user emotion recognition, and at the same time correcting the emotion recognition result according to environmental characteristics, and performing analysis and calculation based on the sequence of key coefficients and the sequence of non-key coefficients, the accuracy is higher, which can significantly improve the efficiency, accuracy, and reliability of user emotion recognition. Furthermore, it can accurately recommend music that meets the user's needs according to the user's emotion state, improving the user experience.

[0064] Embodiment 2, as Figure 2 shown, based on the same inventive concept as the music recommendation method based on user emotion monitoring provided in Embodiment 1, the embodiment of the present invention further provides a music recommendation system based on user emotion monitoring, including: a facial image acquisition module 11, configured to collect a sequence of facial images of the user through a music playback device in response to an emotion monitoring music playback instruction of the user; a basic emotion recognition module 12, configured to extract key facial images based on the sequence of facial images to obtain a sequence of key facial images, a sequence of key coefficients, a sequence of non-key images, and a sequence of non-key coefficients, and perform basic emotion recognition of the user based on the sequence of key facial images and the sequence of key coefficients to obtain a basic emotion level; a user emotion correction module 13, configured to perform environmental light analysis based on the sequence of non-key images and the sequence of non-key coefficients to obtain environmental light characteristic information, and perform user emotion correction to obtain a corrected emotion level; a music recommendation module 14, configured to perform music recommendation based on the corrected emotion level to obtain a recommendation result.

[0065] Further, the music recommendation system based on user emotion monitoring is further configured to: monitor through the music playback device whether the user triggers an emotion monitoring music playback instruction; if so, collect a sequence of facial images of the user at a preset frequency, and if not, continue to monitor.

[0066] Further, the music recommendation system based on user emotion monitoring is further configured to: perform face ratio recognition on multiple face images in the face image sequence to obtain a face ratio coefficient sequence; according to the face ratio coefficient sequence, screen a preset proportion of face images with the largest face ratio coefficient as key face images, arrange them in chronological order to obtain a key face image sequence, and generate a key coefficient sequence based on the face ratio coefficient corresponding to each key face image; obtain other face images as non-key images, arrange them in chronological order to obtain a non-key image sequence, and generate a non-key coefficient sequence based on the face ratio coefficient corresponding to each non-key face image.

[0067] Further, the music recommendation system based on user emotion monitoring is further configured to: collect a sample face image set according to user sample data, label the face pixel ratio in each sample face image to obtain a sample face ratio coefficient set; use a convolutional neural network to construct a face ratio recognizer, and use the sample face image set and the sample face ratio coefficient set to perform supervised training and testing on the face ratio recognizer until the requirements are met; input the multiple face images in the face image sequence into the face ratio recognizer, and identify and output to obtain a face ratio coefficient sequence.

[0068] Further, the music recommendation system based on user emotion monitoring is further configured to: collect a sample key face image set according to user emotion recognition historical data, identify the emotion level in each sample key face image to obtain a sample basic emotion level set; use a convolutional neural network to construct a face emotion recognizer, and use the sample key face image set and the sample basic emotion level set to perform supervised training and testing on the face emotion recognizer until the requirements are met; input the key face image sequence into the face emotion recognizer, and output to obtain a recognized emotion level sequence; according to the multiple key coefficients in the key coefficient sequence, perform weighted calculation on the multiple recognized emotion levels in the recognized emotion level sequence to obtain a basic emotion level.

[0069] Further, the music recommendation system based on user emotion monitoring is further configured to: perform grayscale processing on the non-key image sequence and calculate the average grayscale value to obtain an average grayscale value sequence; calculate to obtain an environmental coefficient sequence according to the non-key coefficient sequence, and use the multiple environmental coefficients in the environmental coefficient sequence to perform weighted calculation on the multiple average grayscale values in the average grayscale value sequence to obtain a weighted grayscale value as environmental light feature information; classify and index according to the environmental light feature information to obtain an emotion influence coefficient; use the emotion influence coefficient to correct the basic emotion level to obtain a corrected emotion level.

[0070] Further, the music recommendation system based on user emotion monitoring is further configured to: collect a set of sample ambient light feature information according to the user emotion recognition data within a historical time, and collect the change range of the user emotion level under different sample ambient light feature information, which is marked as a set of sample emotion influence coefficients; use the set of sample ambient light feature information as an index header, and use the set of sample emotion influence coefficients as index values to construct an emotion influence index database; input the ambient light feature information into the emotion influence index database, and obtain the emotion influence coefficient through index classification.

[0071] Further, the music recommendation system based on user emotion monitoring is further configured to: obtain the historical music recommendation data of the user, collect a set of sample corrected emotion levels, and the music selected by the user for playback when different sample corrected emotion levels appear, to obtain multiple sample music subsets; construct the mapping relationship between the set of sample corrected emotion levels and the multiple sample music subsets to obtain a music recommendation database; input the corrected emotion level into the music recommendation database, map to obtain a music subset, and perform random music recommendation within the music subset to obtain a recommendation result.

[0072] Example 3, please refer to Figure 3 , Figure 3 which is a schematic diagram of an embodiment of the electronic device provided by the embodiment of the present invention. As Figure 3 shown, the embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored on the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, the steps of the method in Embodiment 1 are implemented.

[0073] It should be noted that in the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0074] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more blocks.

[0076] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction means that implements the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more blocks.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows Figure 1 one or more flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more blocks.

[0078] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept.

[0079] Obviously, those skilled in the art can make various changes and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the present invention and its equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A music recommendation method based on user emotion monitoring, characterized in that: Methods include: The user's facial image sequence is collected by a music playing device in response to the user's emotion monitoring music playing instruction; Extract key facial images according to the facial image sequence to obtain a key facial image sequence, a key coefficient sequence, a non-key image sequence, and a non-key coefficient sequence; and identify basic emotions of the user according to the key facial image sequence and the key coefficient sequence to obtain a basic emotion level; Performing ambient light analysis according to the non-critical image sequence and non-critical coefficient sequence to obtain ambient light feature information, performing user emotion correction, and obtaining a corrected emotion level; Music recommendation is performed according to the modified emotion level to obtain a recommendation result.

2. The music recommendation method based on user emotion monitoring according to claim 1, characterized in that: The method includes: collecting a facial image sequence of a user in response to a user's emotion monitoring music playing instruction through a music playing device, including: Monitor whether the user triggers the emotion monitoring music playback instruction through the music playback device; If yes, the user's facial image sequence is collected at a preset frequency; if no, the monitoring continues.

3. The music recommendation method based on user emotion monitoring according to claim 1, characterized in that: According to the facial image sequence, key facial image extraction is performed to obtain a key coefficient sequence, a key facial image sequence, a non-key image sequence and a non-key coefficient sequence, including: Performing facial proportion recognition on a plurality of facial images in the facial image sequence to obtain a facial proportion coefficient sequence; According to the facial proportion coefficient sequence, facial images with a preset ratio of the largest facial proportion coefficient are selected as key facial images, and the facial images are arranged in chronological order to obtain a key facial image sequence, and a key coefficient sequence is generated based on the facial proportion coefficient corresponding to each key facial image; Other facial images are obtained as non-key images, and are arranged in chronological order to obtain a non-key image sequence, and a non-key coefficient sequence is generated based on the facial proportion coefficient corresponding to each non-key facial image.

4. The music recommendation method based on user emotion monitoring according to claim 3 is characterized in that: Performing facial proportion recognition on a plurality of facial images in the facial image sequence to obtain a facial proportion coefficient sequence includes: According to the user sample data, a sample facial image set is collected, and the facial pixel ratio in each sample facial image is marked to obtain a sample facial ratio coefficient set; A convolutional neural network is used to construct a facial proportion identifier, and the facial proportion identifier is supervised trained and tested by using the sample facial image set and the sample facial proportion coefficient set until the requirements are met; The multiple facial images in the facial image sequence are input into the facial proportion recognizer, and the facial proportion coefficient sequence is obtained by recognition output.

5. The music recommendation method based on user emotion monitoring according to claim 1, characterized in that: Performing user basic emotion recognition according to the key facial image sequence and the non-key coefficient sequence to obtain a basic emotion level includes: According to the historical data of user emotion recognition, a set of sample key facial images is collected, and the emotion level in each sample key facial image is marked to obtain a set of sample basic emotion levels; A convolutional neural network is used to construct a facial emotion recognizer, and the facial emotion recognizer is supervised trained and tested by using the sample key facial image set and the sample basic emotion level set until the requirements are met; Inputting a plurality of key facial images in the key facial image sequence into the facial emotion recognizer and outputting a sequence of recognized emotion levels; A weighted calculation is performed on a plurality of recognized emotion levels in the recognized emotion level sequence according to a plurality of key coefficients in the key coefficient sequence to obtain a basic emotion level.

6. The music recommendation method based on user emotion monitoring according to claim 1, characterized in that: Performing ambient light analysis according to the non-critical image sequence and the non-critical coefficient sequence to obtain ambient light feature information, performing user emotion correction, and obtaining a corrected emotion level, including: graying the non-critical image sequence, and calculating the average gray value to obtain an average gray value sequence; According to the non-critical coefficient sequence, an environmental coefficient sequence is calculated, and a plurality of environmental coefficients in the environmental coefficient sequence are used to perform weighted calculation on a plurality of average gray values ​​in the average gray value sequence to obtain weighted gray values ​​as ambient light feature information; According to the ambient light feature information, the emotion impact coefficient is obtained by classification indexing; The basic emotion level is modified using the emotion influence coefficient to obtain a modified emotion level.

7. The music recommendation method based on user emotion monitoring according to claim 6, characterized in that: According to the ambient light feature information, the emotion impact coefficient is obtained by classification indexing, including: According to the user emotion recognition data in the historical time, a set of sample ambient light feature information is collected, and the variation range of the user emotion level under different sample ambient light feature information is collected, and marked as a set of sample emotion influence coefficients; Using the sample ambient light feature information set as an index header and the sample emotion impact coefficient set as an index value to construct an emotion impact index database; The ambient light feature information is input into the emotion impact index database, and the emotion impact coefficient is obtained by index classification.

8. The music recommendation method based on user emotion monitoring according to claim 1, characterized in that: According to the modified emotion level, music recommendation is performed to obtain recommendation results, including: Obtaining historical music recommendation data of the user, collecting a set of sample-corrected emotion levels, and music selected by the user when different sample-corrected emotion levels appear, and obtaining multiple sample music subsets; Constructing a mapping relationship between the sample-corrected emotion level set and a plurality of sample music subsets to obtain a music recommendation database; The modified emotion level is input into the music recommendation database, mapped to obtain a music subset, and music is randomly recommended within the music subset to obtain a recommendation result.

9. A music recommendation system based on user emotion monitoring, characterized in that: The steps for implementing the music recommendation method based on user emotion monitoring according to any one of claims 1 to 8 include: A facial image acquisition module, for acquiring a facial image sequence of a user in response to a user's emotion monitoring music playback instruction through a music playback device; A basic emotion recognition module is used to extract key facial images according to the facial image sequence, obtain a key facial image sequence, a key coefficient sequence, a non-key image sequence and a non-key coefficient sequence, and perform user basic emotion recognition according to the key facial image sequence and the key coefficient sequence to obtain a basic emotion level; A user emotion correction module, used to perform ambient light analysis based on the non-critical image sequence and the non-critical coefficient sequence, obtain ambient light feature information, perform user emotion correction, and obtain a corrected emotion level; The music recommendation module is used to make music recommendations based on the modified emotion level and obtain recommendation results.

10. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the steps of the music recommendation method based on user emotion monitoring as described in any one of claims 1 to 8.

Images