Resource recommendation method and device, electronic equipment and computer readable storage medium
By acquiring and analyzing the user's facial images, identifying their emotional state, and recommending resources that match emotions, the problem of lack of real-time and accuracy of existing smart TV resource push methods is solved, and a better user experience is achieved.
Patent Information
- Application Number
- CN202411808296.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-15
AI Technical Summary
The existing smart TV resource push methods lack real-time and accuracy, and cannot dynamically recommend based on the user's real-time emotional state, resulting in poor user experience.
By obtaining the facial images of the target user, determining the expression feature information and expression timing feature information, using a two-way long and short-term memory network and multi-task learning model for emotion recognition, and recommending resources that match emotions.
It realizes real-time recommendation of resources that conform to user emotions based on the user's current facial image, improving the accuracy and personalization of the user experience.
Smart Images
Figure CN119835484A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of smart home technology, and specifically to a resource recommendation method, device, electronic device, and computer-readable storage medium. Background Art
[0002] With the continuous development of smart home devices, various smart home devices can adjust their own properties or parameters in real time according to user conditions to enhance the user experience.
[0003] In the prior art, resource push for smart TVs mainly relies on the user's historical viewing records and preference settings. However, this push method has a low degree of personalization in resource recommendation, and requires users to manually set preferences and search in advance. The interaction process is relatively cumbersome, resulting in the inability to make dynamic recommendations based on the user's real-time status.
[0004] Therefore, the existing resource push methods lack real-time and accuracy when pushing resources, do not consider the user's emotional state, and cannot provide users with a satisfactory viewing experience. Summary of the invention
[0005] The embodiments of the present application provide a resource recommendation method, device, electronic device, and computer-readable storage medium, which can recommend relevant resources that match the user's current emotion based on the user's current facial image.
[0006] In a first aspect, an embodiment of the present application provides a resource recommendation method, which is applied to a television device, and the method includes:
[0007] Obtain a facial image of the target user;
[0008] Determining expression feature information according to the facial image;
[0009] Determining expression time sequence feature information according to the expression feature information, wherein the expression time sequence feature information is used to characterize a temporal variation trend of the expression feature information;
[0010] Classify the temporal feature information of facial expressions to obtain emotion recognition results;
[0011] Recommend resources that match the emotion recognition result.
[0012] Optionally, in some embodiments of the present application, determining the expression timing feature information according to the expression feature information includes:
[0013] Processing the expression feature information through a forward processing module of a bidirectional long short-term memory network to determine a forward time series feature vector;
[0014] Processing the expression feature information through the backward processing module of the bidirectional long short-term memory network to determine a backward time series feature vector;
[0015] The expression temporal feature information is determined according to the forward temporal feature vector and the backward temporal feature vector.
[0016] Optionally, in some embodiments of the present application, the classifying the expression temporal feature information to obtain the emotion recognition result includes:
[0017] Determine the emotion classification result corresponding to the expression temporal feature information through the emotion classification module of the multi-task learning model;
[0018] Determining the emotion intensity result corresponding to the emotion classification result through the emotion intensity estimation module of the multi-task learning model;
[0019] The emotion recognition result is determined according to the emotion classification result and the emotion intensity result.
[0020] Optionally, in some embodiments of the present application, obtaining a facial image of a target user includes:
[0021] Acquiring a user image of the target user through a shooting module of the television device;
[0022] The user image is detected by a convolutional neural network to determine the facial image.
[0023] Optionally, in some embodiments of the present application, determining expression feature information according to the facial image includes:
[0024] Determining facial feature information based on the facial image using a facial detection model;
[0025] Determine target facial feature data from a preset facial feature database according to the facial feature information, wherein a plurality of preset facial feature data in the preset facial feature database respectively correspond to corresponding emotion tags;
[0026] Determining the emotion label of the target facial feature data;
[0027] The facial expression feature information is determined according to the emotion tag.
[0028] Optionally, in some embodiments of the present application, before determining the expression feature information according to the facial image, the method further includes:
[0029] Get a facial image dataset;
[0030] Performing recognition training on the facial image dataset through a convolutional neural network to generate the preset facial feature database;
[0031] The facial image data set includes multiple groups of images for facial expression recognition.
[0032] Optionally, in some embodiments of the present application, the recommending resources matching the emotion recognition result further includes:
[0033] Determining the age information of the target user according to the facial image;
[0034] Recommend resources that match the emotion recognition result and the age information.
[0035] In a second aspect, an embodiment of the present application further provides a resource recommendation device, which is applied to a television device, and the device includes:
[0036] An acquisition module, used to acquire a facial image of a target user;
[0037] A processing module, used for determining expression feature information according to the facial image;
[0038] The processing module is further used to determine expression time sequence feature information according to the expression feature information, wherein the expression time sequence feature information is used to characterize the change trend of the expression feature information over time;
[0039] The processing module is also used to classify the expression time sequence feature information to obtain the emotion recognition result;
[0040] The processing module is further used to recommend resources that match the emotion recognition result.
[0041] Optionally, in some embodiments of the present application, the processing module is used to:
[0042] Processing the expression feature information through a forward processing module of a bidirectional long short-term memory network to determine a forward time series feature vector;
[0043] Processing the expression feature information through the backward processing module of the bidirectional long short-term memory network to determine a backward time series feature vector;
[0044] The expression temporal feature information is determined according to the forward temporal feature vector and the backward temporal feature vector.
[0045] Optionally, in some embodiments of the present application, the processing module is further used to:
[0046] Determine the emotion classification result corresponding to the expression temporal feature information through the emotion classification module of the multi-task learning model;
[0047] Determining the emotion intensity result corresponding to the emotion classification result through the emotion intensity estimation module of the multi-task learning model;
[0048] The emotion recognition result is determined according to the emotion classification result and the emotion intensity result.
[0049] Optionally, in some embodiments of the present application, the acquisition module is used to:
[0050] Acquiring a user image of the target user through a shooting module of the television device;
[0051] The user image is detected by a convolutional neural network to determine the facial image.
[0052] Optionally, in some embodiments of the present application, the processing module is further used to:
[0053] Determining facial feature information based on the facial image using a facial detection model;
[0054] Determine target facial feature data from a preset facial feature database according to the facial feature information, wherein a plurality of preset facial feature data in the preset facial feature database respectively correspond to corresponding emotion tags;
[0055] Determining the emotion label of the target facial feature data;
[0056] The facial expression feature information is determined according to the emotion tag.
[0057] Optionally, in some embodiments of the present application, the processing module is further used to:
[0058] Get a facial image dataset;
[0059] Performing recognition training on the facial image dataset through a convolutional neural network to generate the preset facial feature database;
[0060] The facial image data set includes multiple groups of images for facial expression recognition.
[0061] Optionally, in some embodiments of the present application, the processing module is further used to:
[0062] Determining the age information of the target user according to the facial image;
[0063] Recommend resources that match the emotion recognition result and the age information.
[0064] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned resource recommendation method when executed by the processor.
[0065] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned resource recommendation method are implemented.
[0066] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.
[0067] In summary, the embodiments of the present application obtain a facial image of a target user; determine expression feature information based on the facial image; determine expression timing feature information based on the expression feature information, wherein the expression timing feature information is used to characterize the temporal change trend of the expression feature information; classify the expression timing feature information to obtain an emotion recognition result; and recommend a technical solution of resources that match the emotion recognition result. This can recommend relevant resources that match the user's current emotion based on the user's current facial image, thereby achieving real-time and accurate resource recommendations based on the user's emotional state, thereby improving the technical effect of user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0069] Figure 1 A schematic diagram of a scenario of a resource recommendation method provided in an embodiment of the present application;
[0070] Figure 2 is a schematic diagram of a scenario in which a smart device according to an embodiment of the present application executes the resource recommendation method;
[0071] Figure 3 is a schematic diagram of the resource recommendation system structure corresponding to the resource recommendation method provided in the embodiment of the present application;
[0072] Figure 4is a schematic diagram of a bidirectional long short-term memory network structure corresponding to the resource recommendation method provided in an embodiment of the present application;
[0073] Figure 5 is a schematic diagram of the multi-task learning model structure corresponding to the resource recommendation method provided in an embodiment of the present application;
[0074] Figure 6 is a schematic diagram of the structure of a resource recommendation device provided in an embodiment of the present application;
[0075] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0076] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0077] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the above features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0078] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.
[0079] First, the terms involved in this application are explained:
[0080] Facial Image Dataset: A set of facial images used to train and test facial expression recognition models, where each image is usually labeled to indicate a specific facial expression.
[0081] Deep Convolutional Neural Network Model: Convolutional Neural Network, CNN, a deep learning model for image processing and recognition, can be combined with facial image datasets to train viewer age and emotion recognition models.
[0082] Bidirectional Long Short-Term Memory (Bi-LSTM) is an enhanced LSTM (Long Short-Term Memory) structure that can simultaneously learn forward and backward information from sequence data. In this paper, it can simultaneously capture forward and backward time series features to improve the accuracy of emotion recognition.
[0083] Multi-Task Learning Model: Multi-Task Learning, MTL, is a machine learning method that aims to improve the performance of the model on each task by learning multiple related tasks simultaneously. In this paper, emotion classification and intensity estimation can be performed simultaneously to improve the generalization ability of the model.
[0084] The embodiment of the present application provides a resource recommendation method, device, electronic device and computer-readable storage medium. Specifically, the embodiment of the present application provides a resource recommendation device applicable to the resource recommendation method, the resource recommendation device includes a smart device and a main control device of the smart device, and the smart device can be a TV, a mobile phone, a tablet, a computer, etc.
[0085] In the prior art, resource recommendation strategies for smart devices are becoming more and more diverse. Many smart device manufacturers have improved traditional resource recommendation strategies. The improved resource recommendation strategies can more accurately recommend relevant resources to users based on their viewing history and preference settings.
[0086] For example, most existing resource recommendation methods determine the user identity based on the acquired user image, and recommend resources based on the viewing history, search history, and pre-set preferences or likes of the identity.
[0087] However, existing resource recommendation methods cannot capture users' emotional changes in real time, cannot make dynamic recommendations based on users' real-time status, and the degree of personalization of resource recommendations is low; users need to manually set preferences and search, and the interaction process is cumbersome.
[0088] Therefore, the resource recommendation method in the prior art has a low degree of personalization and requires users to manually set preferences and search in advance. The interaction process is relatively cumbersome, resulting in the technical problem of being unable to make dynamic recommendations based on the user's real-time status. When making resource recommendations, there is a lack of real-time and accuracy, and the user's emotional state is not taken into consideration, and it is impossible to provide users with a satisfactory viewing experience.
[0089] The embodiment of the present application provides a resource recommendation method, device, electronic device and computer-readable storage medium. The method adopts the following technical solutions: obtaining a facial image of a target user; determining expression feature information based on the facial image; determining expression time sequence feature information based on the expression feature information, the expression time sequence feature information is used to characterize the temporal change trend of the expression feature information; determining an emotion recognition result based on the emotion classification result corresponding to the expression time sequence feature information and the emotion intensity result corresponding to the emotion classification result; and recommending resources matching the emotion recognition result.
[0090] In summary, the resource recommendation method in the embodiment of the present application can recommend relevant resources that match the user's current emotion based on the user's current facial image, thereby achieving real-time and accurate resource recommendation based on the user's emotional state, thereby improving the technical effect of user experience.
[0091] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.
[0092] See also Figure 1 , Figure 1The scene diagram of the resource recommendation method provided in the embodiment of the present application is as follows. The resource recommendation system may include a smart device 100, a main control device 200 and a shooting module 300. The smart device 100, the main control device 200 and the shooting module 300 may be connected to each other by any means, including but not limited to signal communication through electronic circuits, communication through wireless signals, and the wireless signals may be computer network communications of the TCP / IP protocol suite (TCP / IP Protocol Suite, TCP / IP) and the User Datagram Protocol (UDP). The smart device 100 may receive control signals from a remote controller or a control panel, and the smart device 100 may also receive instruction information sent by the main control device 200. The smart device 100 may perform corresponding operations according to the corresponding instruction information. Accordingly, the shooting module 300 may also receive control signals from a remote controller or a control panel, and the shooting module 300 may also receive instruction information sent by the main control device 200. The shooting module 300 may perform corresponding operations according to the corresponding instruction information, such as the resource recommendation method in the present application.
[0093] In the embodiment of the present application, the smart device 100 includes but is not limited to a television, a mobile phone, a tablet, a computer, etc.
[0094] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer smart devices as shown in Figure 1 Only one smart device is shown, and no specific limitation is given here.
[0095] In addition, if Figure 1 As shown, the main control device 200 may include any hardware device capable of data processing and instruction sending, such as a CPU or a single-chip microcomputer embedded in the smart device, and the specific details are not limited here.
[0096] In addition, if Figure 1 As shown, the shooting module 300 may include any hardware device capable of acquiring images, such as a camera embedded in a smart device, and the specifics are not limited here.
[0097] It should be noted that Figure 1The scenario diagram of the resource recommendation system shown is only an example. The resource recommendation system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the resource recommendation system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.
[0098] For details, please refer to Figure 2 , Figure 2 : is a schematic diagram of a scenario in which a smart device according to an embodiment of the present application executes the resource recommendation method, wherein the specific execution process of the smart device executing the resource recommendation method is as follows:
[0099] S201: Acquire a facial image of a target user.
[0100] In the present application embodiment, Figure 3 The resource recommendation system structure diagram shown is a schematic diagram of a smart device that executes the resource recommendation method internally or externally. The resource recommendation system mainly includes: a face recognition module, an emotion recognition module and a resource push module.
[0101] It should be noted that smart devices include but are not limited to televisions, mobile phones, tablets, computers and other smart devices used by users to browse multiple resources. Smart devices can execute the resource recommendation method of the embodiment of the present application through internal or external processors, such as: a main control device of the smart device, a processor of the smart device, or a cloud processor or cloud server that communicates with the smart device.
[0102] As an optional embodiment, the facial recognition module is mainly used to control the camera to obtain user images, preprocess the user images, perform facial detection on the preprocessed user images to determine the user's facial image, and finally extract facial features.
[0103] Optionally, the resource recommendation method provided in the embodiment of the present application also includes: acquiring a user image of the target user through a shooting module of a television device; detecting the user image through a convolutional neural network to determine a facial image.
[0104] In the embodiment of the present application, taking the case where the smart device is a smart TV as an example, the video information or image information of the viewer is captured in real time through the shooting module of the smart TV, such as an infrared camera; a convolutional neural network (CNN) model is used to perform facial detection on the video information or image information captured above to determine the user's facial area, and then the Dlib model is used to extract facial features from the facial area, extract facial features such as eyes, mouth, and nose, and obtain a facial image.
[0105] As an optional embodiment, a convolutional neural network (CNN) model suitable for facial region detection is designed and constructed, and generally a network structure consisting of a convolutional layer, a pooling layer, a fully connected layer, etc. can be used. Alternatively, an existing pre-trained CNN model, such as ResNet, VGG, etc., can be used, and then the output layer of the model can be adjusted to suit the facial region detection task.
[0106] As an optional embodiment, facial features are extracted from the facial region using the Dlib model, mainly including: loading an image and graying it, using an image processing library (such as OpenCV) to load the image to be processed, and converting it into a grayscale image. Facial feature extraction is usually performed on a grayscale image. And run the face detection function, use the face detection model of Dlib to locate the face in the grayscale image, and obtain the location information of one or more faces. Further, run facial feature extraction, for each detected face, use the pre-trained model of Dlib to extract facial feature points from the image, and the feature points can be used to describe the shape and structure of the face (such as eyes, nose, mouth and other parts). Finally, using the extracted features, the extracted facial feature points can be used to perform various tasks, such as facial expression analysis, facial key point detection, facial recognition, etc. According to the required tasks, various machine learning methods or models can be used to further process and analyze the above features.
[0107] It should be noted that the Dlib model contains library functions and algorithms for machine learning and computer vision. The Dlib model refers to the machine learning model and image processing model trained or used by the Dlib library. It can be widely used in face recognition, facial expression analysis, facial key point detection, real-time object detection, posture estimation and other fields.
[0108] As an optional embodiment, after the video information or image information of the viewer is captured in real time by the shooting module of the smart TV, each frame of the image needs to be pre-processed, for example: noise reduction processing, grayscale processing, etc.
[0109] S202: Determine expression feature information according to the facial image.
[0110] In the present application embodiment, Figure 3 As shown, after extracting the user's facial features to obtain the target user's facial image, the emotion recognition module performs facial recognition processing to determine the user's facial expression feature information.
[0111] As an optional embodiment, facial recognition processing can be performed using a face data model (preset facial feature database) trained by a convolutional neural network (CNN) and a facial image dataset (FER-2013) to identify the age of the viewer (i.e., the target user) and the facial expression corresponding to the viewer in the current frame or within a preset time period.
[0112] It should be noted that the Facial Image Dataset (FER-2013) is a widely used facial image dataset that contains multiple sets of images for facial expression recognition. The dataset contains 35,887 grayscale images of size 48x48 pixels, covering 7 different facial expression categories: Angry, Disgust, Fear, Happy, Sad, Surprise, and Neutral.
[0113] Before determining expression feature information based on the facial image, optionally, the resource recommendation method provided in the embodiment of the present application also includes: obtaining a facial image dataset; performing recognition training on the facial image dataset through a convolutional neural network to generate a preset facial feature database.
[0114] In an embodiment of the present application, a preset facial feature database is trained using a convolutional neural network CNN and a facial image dataset FER-2013, and the extracted facial image is compared with the trained preset facial feature database to identify the facial expression features of the viewer.
[0115] It should be noted that the facial image dataset includes multiple groups of images for facial expression recognition.
[0116] Optionally, the resource recommendation method provided in the embodiment of the present application also includes: determining facial feature information based on the facial image through a facial detection model; determining target facial feature data from a preset facial feature database based on the facial feature information; determining an emotion label of the target facial feature data; and determining expression feature information based on the emotion label.
[0117] In the embodiment of the present application, the facial feature information may be the facial features of a plurality of consecutive images within a preset time period, or may be the facial features corresponding to a single image. The target facial feature data is the preset facial feature data in the preset facial feature database that matches the facial feature information, and the plurality of preset facial feature data in the preset facial feature database respectively correspond to corresponding emotion tags.
[0118] As an optional embodiment, after a preset facial feature database is trained using a convolutional neural network CNN and a facial image dataset FER-2013, multiple preset facial feature data in the preset facial feature database are used to perform comparison processing with the facial images respectively.
[0119] It should be noted that the comparison result of the comparison processing can be expressed in various forms. For example, in the embodiment of the present application, it can be expressed by a similarity value.
[0120] As an optional embodiment, based on the similarity value after comparison, the preset facial feature data corresponding to the above-mentioned facial image in the preset facial feature database is determined, the target facial feature data is obtained, and the emotion label of the target facial feature data is determined; thereby, the emotion label is determined to be the emotion label corresponding to the facial image, and the expression feature corresponding to the emotion label is used as the expression feature information.
[0121] S203: Determine expression temporal feature information according to the expression feature information.
[0122] In the embodiment of the present application, the target time series feature vector is used to characterize the temporal change trend of the expression feature information. The emotional state judgment algorithm composed of the Bi-LSTM model and the multi-task learning model is combined to analyze the continuous changes in facial expressions and judge the emotional state of the viewer.
[0123] It should be noted that in the embodiment of the present application, the facial image of the target user is continuously acquired, and therefore, the expression feature information obtained may be a continuous sequence of facial expression images.
[0124] As an optional embodiment, the bidirectional long short-term memory network Bi-LSTM can capture both forward and backward time series features, and can fuse the information of the current time step with the previous and next contexts by processing the forward and backward directions of the sequence at the same time. The model can capture both past and future information in the sequence at the same time, thereby better understanding the intrinsic relationship of the sequence. For tasks such as face detection, language translation, and sentiment analysis, Bi-LSTM can achieve better performance and predictive capabilities by modeling the context.
[0125] Optionally, the resource recommendation method provided in the embodiment of the present application also includes: processing expression feature information through a forward processing module of a bidirectional long short-term memory network to determine a forward time series feature vector; processing expression feature information through a backward processing module of a bidirectional long short-term memory network to determine a backward time series feature vector; determining the expression time series feature information based on the forward time series feature vector and the backward time series feature vector.
[0126] In the present application embodiment, Figure 4The schematic diagram of the bidirectional long short-term memory network structure shown in Figure 1 shows that in the bidirectional long short-term memory network Bi-LSTM, the input sequence is passed to the forward LSTM layer and the backward LSTM layer (i.e., the forward processing module and the backward processing module) respectively. The memory cells and gating mechanisms in each LSTM layer update and pass information through learning weights. The outputs of the forward LSTM layer and the backward LSTM layer are connected together to form a comprehensive representation that can be used for downstream tasks such as classification, label prediction, sequence generation, etc.
[0127] As an optional embodiment, a continuous sequence of facial expression images is input into the facial recognition module, and in the facial recognition step of the emotion recognition module, a convolutional neural network (CNN) is used to extract features from each frame of the image, generate a feature vector sequence, and input the feature vector sequence into a bidirectional long short-term memory network Bi-LSTM to extract temporal features. The bidirectional long short-term memory network Bi-LSTM can capture the forward and backward time series features at the same time, merge and generate a temporal feature vector, i.e., the above-mentioned expression temporal feature information, and output it.
[0128] As an optional embodiment, after the expression temporal feature information is generated, the emotion recognition result is determined based on the generated temporal feature vector and the multi-task learning model.
[0129] S204: Classify the expression temporal feature information to obtain emotion recognition results.
[0130] In the present application embodiment, Figure 5 The schematic diagram of the multi-task learning model structure shown in the figure takes a time series feature vector as input, which can be a sequence data containing multiple time steps. The feature vector of each time step can contain emotion-related sound, text, or other perceptual features. A shared representation layer can be used to extract a shared representation of the input features. This can be a recurrent neural network (such as LSTM or GRU) or a convolutional neural network (such as a 1D convolutional network). The shared representation layer converts the time series feature vector into a fixed-length feature representation.
[0131] Optionally, the resource recommendation method provided in the embodiment of the present application also includes: determining the emotion classification result corresponding to the expression timing feature information through the emotion classification module of the multi-task learning model; determining the emotion intensity result corresponding to the emotion classification result through the emotion intensity estimation module of the multi-task learning model; and determining the emotion recognition result based on the emotion classification result and the emotion intensity result.
[0132] As an optional embodiment, after the shared representation layer, a specific task 1 output layer is added for emotion classification. It can be a fully connected layer that maps the shared representation to the categories of emotion classification. Common emotion classification tasks include positive / negative classification or multi-category emotion classification. In addition, a specific task 2 output layer is added for intensity estimation. It can be a fully connected layer that maps the shared representation to a continuous value of emotion intensity. The intensity estimation task can be a regression problem, predicting the intensity or degree of emotion.
[0133] Optionally, for the emotion classification task, the cross entropy loss function can be used to calculate the classification loss. For the intensity estimation task, the mean square error loss function can be used to calculate the regression loss. The final loss function can be a weighted sum of the two task loss functions. In addition, the labels of multiple emotion classification and intensity estimation tasks can be used to update the parameters of the model through joint training. An optimization algorithm such as gradient descent can be used to minimize the overall loss function.
[0134] In the embodiment of the present application, by inputting the time series feature vector into the MTL model, the emotion classification and intensity estimation tasks can be learned simultaneously, thereby improving the generalization ability and effect of the model. The MTL model can share feature representations and optimize multiple tasks through joint training, thereby better understanding and processing emotion-related time series data.
[0135] S205: Recommending resources that match the emotion recognition result.
[0136] In an embodiment of the present application, after the emotional state of the viewer is identified, a matching film resource is selected from a film resource library, and the matching film resource is pushed to the smart TV for playback.
[0137] As an optional embodiment, the film resource library can be a resource library stored in the smart device itself, or a resource library stored in the cloud, and cloud resources are obtained by establishing communication with the cloud.
[0138] Optionally, the resource recommendation method provided in the embodiment of the present application also includes: determining the age information of the target user based on the facial image; and recommending resources whose emotion recognition results match the age information.
[0139] In the embodiment of the present application, the age information of the target user can also be determined based on the facial image, and resource recommendations can be made by combining the age information with the emotion recognition results.
[0140] As an optional embodiment, with the knowledge and consent of the target user, computer vision technology, such as face detection and face recognition, is used to extract the user's facial features from the facial image, which may include features such as facial contour, eyes, and mouth. Using the facial features, a machine learning or deep learning model can be used to estimate the user's age.
[0141] It should be noted that in the embodiment of the present application, the recommendation can be personalized according to the user's feedback and behavior. For example, if the user is not interested in or does not like certain resources, the recommendation can be adjusted according to the user's feedback to provide resources that are more in line with the user's preferences.
[0142] In the embodiment of the present application, Dlib and Bi-LSTM models can more accurately extract facial features and temporal features, and improve the accuracy of face recognition and emotion recognition. Combined with high-performance processors and graphics processing systems, it can process and identify the user's age and emotional state in real time, and push matching movie resources in real time according to the user's age and emotional state. The system automatically recommends and plays resources without the user having to manually search and select movies, thereby improving the user's viewing experience.
[0143] Through the embodiments of the present application, a technical solution is adopted for acquiring a facial image of a target user; determining expression feature information based on the facial image; determining expression timing feature information based on the expression feature information, wherein the expression timing feature information is used to characterize the temporal change trend of the expression feature information; classifying the expression timing feature information to obtain an emotion recognition result; and recommending resources that match the emotion recognition result. This can recommend relevant resources that match the user's current emotion based on the user's current facial image, thereby achieving real-time and accurate resource recommendations based on the user's emotional state, thereby improving the technical effect of user experience.
[0144] In order to better implement the resource recommendation method of the present application, the present application also provides a resource recommendation device based on the resource recommendation method. The meanings of the terms are the same as those in the resource recommendation method, and the specific implementation details can refer to the description in the method embodiment.
[0145] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of a resource recommendation device provided in an embodiment of the present application, wherein the resource recommendation device 600 is applied to a television device, and specifically can be as follows:
[0146] An acquisition module 601 is used to acquire a facial image of a target user;
[0147] Processing module 602, used to determine expression feature information according to the facial image;
[0148] The processing module 602 is further used to determine expression time sequence feature information according to the expression feature information, where the expression time sequence feature information is used to characterize the change trend of the expression feature information over time;
[0149] The processing module 602 is also used to classify the expression time sequence feature information to obtain the emotion recognition result;
[0150] The processing module 602 is also used to recommend resources that match the emotion recognition result.
[0151] Optionally, in some embodiments of the present application, the processing module 602 is used to:
[0152] The forward processing module of the bidirectional long short-term memory network processes the expression feature information and determines the forward time series feature vector;
[0153] The expression feature information is processed through the backward processing module of the bidirectional long short-term memory network to determine the backward time series feature vector;
[0154] The expression temporal feature information is determined according to the forward temporal feature vector and the backward temporal feature vector.
[0155] Optionally, in some embodiments of the present application, the processing module 602 is further configured to:
[0156] Determine the emotion classification result corresponding to the expression temporal feature information through the emotion classification module of the multi-task learning model;
[0157] Determine the emotion intensity result corresponding to the emotion classification result through the emotion intensity estimation module of the multi-task learning model;
[0158] The emotion recognition result is determined based on the emotion classification result and the emotion intensity result.
[0159] Optionally, in some embodiments of the present application, the acquisition module 601 is used to:
[0160] Acquire a user image of a target user through a camera module of a television device;
[0161] The user image is detected through a convolutional neural network to determine the facial image.
[0162] Optionally, in some embodiments of the present application, the processing module 602 is further configured to:
[0163] Determine facial feature information based on the facial image using a facial detection model;
[0164] Determine target facial feature data from a preset facial feature database according to the facial feature information, wherein a plurality of preset facial feature data in the preset facial feature database respectively correspond to corresponding emotion tags;
[0165] Determine the emotion label of the target facial feature data;
[0166] Determine facial expression feature information based on emotion labels.
[0167] Optionally, in some embodiments of the present application, the processing module 602 is further configured to:
[0168] Get a facial image dataset;
[0169] The facial image dataset is trained through convolutional neural network to generate a preset facial feature database;
[0170] Among them, the facial image dataset includes multiple groups of images for facial expression recognition.
[0171] Optionally, in some embodiments of the present application, the processing module 602 is further configured to:
[0172] Determine the target user's age information based on the facial image;
[0173] Recommend resources that match emotion recognition results with age information.
[0174] In the embodiment of the present application, the acquisition module 601 first acquires the facial image of the target user, then the processing module 602 determines the expression feature information based on the facial image, and then the processing module 602 determines the expression timing feature information based on the expression feature information. The expression timing feature information is used to characterize the temporal change trend of the expression feature information, and then the processing module 602 classifies the expression timing feature information to obtain an emotion recognition result. Finally, the processing module 602 recommends resources that match the emotion recognition result.
[0175] Among them, the embodiment of the present application obtains the facial image of the target user; determines the expression feature information based on the facial image; determines the expression timing feature information based on the expression feature information, and the expression timing feature information is used to characterize the temporal change trend of the expression feature information; classifies the expression timing feature information to obtain emotion recognition results; and recommends resources that match the emotion recognition results. The technical solution can recommend relevant resources that match the user's current emotions based on the user's current facial image, thereby achieving real-time and accurate resource recommendations based on the user's emotional state, thereby improving the technical effect of user experience.
[0176] In addition, the present application also provides an electronic device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:
[0177] The electronic device may include components such as a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a power supply 703, and an input unit 704. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0178] The processor 701 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 702, and calling data stored in the memory 702, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 701.
[0179] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.
[0180] The electronic device also includes a power supply 703 for supplying power to each component. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 703 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.
[0181] The electronic device may further include an input unit 704, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0182] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 701 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 will run the application programs stored in the memory 702, thereby implementing the steps in any resource recommendation method provided in the embodiments of the present application.
[0183] The embodiment of the present application obtains a facial image of a target user; determines expression feature information based on the facial image; determines expression timing feature information based on the expression feature information, wherein the expression timing feature information is used to characterize a temporal change trend of the expression feature information; classifies the expression timing feature information to obtain an emotion recognition result; and recommends a technical solution of resources that match the emotion recognition result. The technical solution can recommend relevant resources that match the user's current emotion based on the user's current facial image, thereby achieving real-time and accurate resource recommendations based on the user's emotional state, thereby improving the technical effect of user experience.
[0184] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0185] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0186] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any resource recommendation method provided in the present application.
[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0188] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0189] Since the instructions stored in the computer-readable storage medium can execute the steps in any resource recommendation method provided in the present application, the beneficial effects that can be achieved by any resource recommendation method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0190] The above is a detailed introduction to a resource recommendation method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A resource recommendation method, characterized in that: Applied to a television device, the method comprises: Obtain a facial image of the target user; Determining expression feature information according to the facial image; Determining expression time sequence feature information according to the expression feature information, wherein the expression time sequence feature information is used to characterize a temporal variation trend of the expression feature information; Classifying the expression temporal feature information to obtain an emotion recognition result; Recommend resources that match the emotion recognition result.
2. The method according to claim 1, characterized in that The step of determining the expression time sequence feature information according to the expression feature information comprises: Processing the expression feature information through a forward processing module of a bidirectional long short-term memory network to determine a forward time series feature vector; Processing the expression feature information through the backward processing module of the bidirectional long short-term memory network to determine a backward time series feature vector; The expression temporal feature information is determined according to the forward temporal feature vector and the backward temporal feature vector.
3. The method according to claim 1, characterized in that The step of classifying the expression temporal feature information to obtain an emotion recognition result includes: Determine the emotion classification result corresponding to the expression temporal feature information through the emotion classification module of the multi-task learning model; Determining the emotion intensity result corresponding to the emotion classification result through the emotion intensity estimation module of the multi-task learning model; The emotion recognition result is determined according to the emotion classification result and the emotion intensity result.
4. The method according to claim 1, characterized in that The step of obtaining the facial image of the target user comprises: Acquiring a user image of the target user through a shooting module of the television device; The user image is detected by a convolutional neural network to determine the facial image.
5. The method according to claim 4, characterized in that The step of determining the expression feature information according to the facial image comprises: Determining facial feature information based on the facial image using a facial detection model; Determine target facial feature data from a preset facial feature database according to the facial feature information, wherein a plurality of preset facial feature data in the preset facial feature database respectively correspond to corresponding emotion tags; Determining the emotion label of the target facial feature data; The facial expression feature information is determined according to the emotion tag.
6. The method according to claim 5, characterized in that Before determining the expression feature information according to the facial image, the method further includes: Get a facial image dataset; Performing recognition training on the facial image dataset through a convolutional neural network to generate the preset facial feature database; The facial image data set includes multiple groups of images for facial expression recognition.
7. The method according to claim 1, characterized in that The recommending of resources matching the emotion recognition result further includes: Determining the age information of the target user according to the facial image; Recommend resources that match the emotion recognition result and the age information.
8. A resource recommendation device, characterized in that: Applied to television equipment, the device comprises: An acquisition module, used to acquire a facial image of a target user; A processing module, used for determining expression feature information according to the facial image; The processing module is further used to determine expression time sequence feature information according to the expression feature information, wherein the expression time sequence feature information is used to characterize the change trend of the expression feature information over time; The processing module is further used to determine an emotion recognition result according to an emotion classification result corresponding to the expression temporal feature information and an emotion intensity result corresponding to the emotion classification result; The processing module is further used to recommend resources that match the emotion recognition result.
9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the resource recommendation method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the resource recommendation method according to any one of claims 1 to 7 are implemented.