Method, device, electronic device and storage medium for identifying emotion of object in video

Abstract

The present application provides a method, device, electronic device and storage medium for object emotion recognition in a video, the method comprising: obtaining at least one facial image of a target object from a video, inputting the facial image into a pre-trained emotion feature extraction model, thereby obtaining an emotion vector corresponding to the facial image and containing multi-dimensional emotion feature values, and determining the emotion of the target object based on the emotion vector corresponding to at least one facial image. The present solution extracts features from multiple dimensions of the facial image of the target object, thereby obtaining a multi-dimensional emotion vector. The emotion determined by using the multi-dimensional emotion vector can more finely represent the emotion of the target object, making the interpreted emotion more accurate and detailed.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition, and in particular to a method, device, electronic device and storage medium for object emotion recognition in a video. Background Art

[0002] Films and TV shows usually tell stories through the language of the camera, and character emotions are an important part of the story. By identifying the emotions of characters in the video, we can analyze the character's ability to express emotions, which helps us understand the character's acting skills.

[0003] Currently, when identifying people's emotions, face recognition technology is usually used to identify the expressions of people in video images and determine the emotions of people based on the expressions. This method can only identify a few simple emotions such as "laughing", "crying", "happy", "angry", and "sad", which is not enough to fully express the emotional characteristics of the characters and has low detail. Summary of the invention

[0004] In order to solve the technical problem of low precision in character emotion recognition in the prior art, the present application provides a method, device, electronic device and storage medium for object emotion recognition in a video.

[0005] In a first aspect, the present application provides a method for identifying emotion of an object in a video, comprising:

[0006] Acquire at least one facial image frame of a target object from a video;

[0007] Inputting the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image;

[0008] Generating an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image;

[0009] The emotion of the target object is determined according to an emotion vector of at least one frame of the facial image.

[0010] As a possible implementation manner, determining the emotion of the target object according to the emotion vector of at least one frame of the facial image includes:

[0011] Determine a target emotion vector from emotion vectors of at least one frame of the facial image;

[0012] The target emotion vector is input into a preset emotion interpretation model to obtain emotion interpretation information corresponding to the target emotion vector and used to represent the emotion of the target object. The emotion interpretation model includes multiple emotion interpretation information corresponding to multiple emotion vectors.

[0013] As a possible implementation manner, determining the target emotion vector from the emotion vector of at least one frame of the facial image includes:

[0014] If a facial image of the target object is obtained from the video, the emotion vector of the facial image is determined as the target emotion vector;

[0015] If multiple frames of facial images of the target object are obtained from the video, a time sequence of emotion vectors is generated according to the order of the playback time of the multiple frames of facial images from first to last, wherein the time sequence includes the emotion vectors of the multiple frames of facial images;

[0016] Dividing the time interval corresponding to the time series into a plurality of candidate time intervals;

[0017] For each of the candidate time intervals, determining the density of each emotion vector contained in the candidate time interval;

[0018] From the plurality of candidate time intervals, a candidate time interval including emotion vectors having densities meeting preset requirements is selected as a target time interval;

[0019] The emotion vector with the highest density in the target time interval is determined as the target emotion vector.

[0020] As a possible implementation manner, determining the density of each emotion vector included in the candidate time interval includes:

[0021] Calculating the edit distance between the emotion vectors included in the candidate time interval;

[0022] For each emotion vector in the candidate time interval, other emotion vectors in the candidate time interval whose edit distance with the emotion vector is less than a preset distance threshold are used as similar vectors of the emotion vector;

[0023] For each emotion vector in the candidate time interval, the density of the emotion vector is determined according to the emotion vector and a similarity vector of the emotion vector.

[0024] As a possible implementation manner, determining the density of the emotion vector according to the emotion vector and the similarity vector of the emotion vector includes:

[0025] Performing weighted summation on the frequency of occurrence of the emotion vector in the candidate time interval and the frequency of occurrence of the similarity vector in the candidate time interval to obtain a sum value;

[0026] The density of the emotion vector is obtained by dividing the sum by the time span of the candidate time interval.

[0027] As a possible implementation manner, selecting, from the plurality of candidate time intervals, a candidate time interval in which the density of each emotion vector meets a preset requirement as the target time interval includes:

[0028] For each of the candidate time intervals, determining the density level to which the density of each emotion vector contained in the candidate time interval belongs according to a preset density level classification rule;

[0029] For each emotion vector, comparing the density of the emotion vector with a density threshold corresponding to the density level to which the density belongs;

[0030] If the density of each emotion vector included in the candidate time interval is greater than the corresponding density threshold, it is determined that the candidate time interval meets the preset requirements, and the candidate time interval is used as the target time interval.

[0031] As a possible implementation, the emotion feature extraction model includes multiple image regression models, wherein the image regression model takes a facial image as input and an emotion feature value as output, and the dimensions of the emotion feature values ​​output by different image regression models are different.

[0032] As a possible implementation manner, generating the emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image includes:

[0033] Discretizing the emotion feature values ​​of multiple dimensions of the facial image respectively;

[0034] The discretized emotion feature values ​​of multiple dimensions are combined into an emotion vector of the facial image.

[0035] In a second aspect, an embodiment of the present application further provides a device for identifying emotions of an object in a video, comprising:

[0036] An acquisition module, used for acquiring at least one frame of facial image of a target object from a video;

[0037] A feature extraction module, used for inputting the facial image into a pre-trained emotion feature extraction model, so that the emotion feature value extraction model extracts emotion feature values ​​of multiple dimensions of the facial image;

[0038] A vector generation module, used to generate an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions;

[0039] The emotion determination module is used to determine the emotion of the target object according to the emotion vector of at least one frame of the facial image.

[0040] In a third aspect, an embodiment of the present invention further provides a computer device, comprising: a processor and a memory, wherein the processor is used to execute a method program for identifying object emotions in a video stored in the memory to implement the method for identifying object emotions in a video described in the first aspect.

[0041] In a fourth aspect, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for object emotion recognition in video described in the first aspect.

[0042] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0043] The embodiment of the present application provides a method for identifying emotions of an object in a video, which obtains at least one facial image of a target object from a video, inputs the facial image into a pre-trained emotion feature extraction model, thereby obtaining an emotion vector corresponding to the facial image and containing multi-dimensional emotion feature values, and determines the emotion of the target object based on the emotion vector corresponding to at least one facial image. This solution extracts features from multiple dimensions of the facial image of the target object, thereby obtaining a multi-dimensional emotion vector. The emotion determined by using the multi-dimensional emotion vector can more finely represent the emotion of the target object, making the interpreted emotion more accurate and detailed.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 is a flow chart of a method for identifying object emotions in a video according to an exemplary embodiment;

[0047] Figure 2 is a flow chart of a method for training an emotion feature extraction model according to an exemplary embodiment;

[0048] Figure 3 is a schematic diagram of an emotion feature extraction model according to an exemplary embodiment;

[0049] Figure 4 is a flow chart of a method for identifying object emotions in a video according to another exemplary embodiment;

[0050] Figure 5is a flow chart of a method for determining a target emotion vector according to an exemplary embodiment;

[0051] Figure 6 is a block diagram of a device for identifying object emotions in a video according to an exemplary embodiment;

[0052] Figure 7 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] See also Figure 1 , is a flow chart of a method for identifying object emotions in a video provided by an embodiment of the present application. Figure 1 As shown, the method for identifying object emotions in a video provided by this embodiment may include:

[0055] S11. Acquire at least one frame of facial image of the target object from the video.

[0056] In an embodiment of the present invention, a video to be subjected to emotion recognition is obtained, and then an object (such as a person or an animal) to be subjected to emotion recognition is determined, and the object is used as a target object, and then at least one frame of image containing the face of the target object is extracted from the video. The target object can be selected by a user (such as an operator), or an object can be randomly selected from the video as the target object.

[0057] As an embodiment, the following steps may be used to obtain a facial image of a target object from a video:

[0058] Step 1: Sample the video frames to obtain a set of video frames.

[0059] In the embodiment of the present invention, the video can be frame sampled according to a preset sampling frequency, and the sampling frequency can be set in advance according to actual needs. For example, if the original video is 25 frames per second, the sampling frequency can be set to 8 frames per second. It can be seen that the sampling frequency can be set dynamically. Sampling the video frames according to the sampling frequency can reduce the subsequent image processing amount on the basis of ensuring accurate recognition.

[0060] In practical applications, the Premiere software can be used to perform frame extraction processing on the video according to a preset sampling frequency, thereby obtaining a set of video frames.

[0061] Step 2: Perform facial recognition on each image in the video frame set to determine at least one image that contains the face of the target object.

[0062] In an embodiment of the present invention, an image containing a clear face of a target object can be obtained, and then a face recognition algorithm is used based on the image to perform facial recognition on each image in a video frame set, thereby identifying an image containing the face of the target object from the video frame set, and using the image containing the face of the target object as the facial image of the target object.

[0063] Furthermore, if multiple images containing the target object's face are determined in the above step 1, in order to improve the accuracy of emotion recognition, the multiple images can be screened to select images with better facial quality as the facial images of the target object. Specifically, the images can be screened through step 3.

[0064] Step 3: Extract the facial quality of the image containing the target object's face, and select the image whose facial quality meets the preset quality condition as the facial image of the target object.

[0065] In the embodiment of the present invention, facial quality may include multiple facial quality parameters (for example, parameters such as facial clarity, side face angle and / or facial size).

[0066] As an embodiment, for each image containing the face of the target object, the facial quality parameters can be compared with the thresholds corresponding to the parameters, and whether the facial quality of the image meets the preset quality conditions can be determined based on the comparison results. For example, if the facial quality parameters include facial clarity, side face angle, and facial size, the facial clarity in the facial quality can be compared with a preset clarity threshold, the side face angle can be compared with a preset angle threshold, and the facial size can be compared with a preset size threshold. If it is determined that the clarity is lower than the clarity threshold, or the side face angle is greater than the angle threshold, or the face size is less than the size threshold, it is determined that the image does not meet the preset quality conditions. If it is determined that the clarity is not lower than the clarity threshold, and the side face angle is not greater than the angle threshold, and the face size is not less than the size threshold, it is determined that the image meets the preset quality conditions.

[0067] In an embodiment of the present invention, the above method can filter out low-quality noise images such as those with too low clarity, too large side face angles, and too small facial sizes, and reduce the noise of the images to obtain higher-quality facial images, making the emotion recognition results determined based on facial images more accurate.

[0068] S12. Input the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image.

[0069] In an embodiment of the present invention, the emotion feature extraction model is a pre-trained and stored model, which takes a facial image as input and outputs a multi-dimensional emotion feature value of an object in the facial image. Each dimension represents an emotion feature at an abstract semantic level, and different dimensions have different emotion features to be compiled. For example, the emotion feature extraction model can extract features from multiple dimensions of a facial image, such as joy-sadness, anticipation-surprise, anger-fear, and disgust-trust.

[0070] Furthermore, under the same dimension, different emotion feature values ​​can reflect different emotions.

[0071] As for how the emotion feature extraction model is trained, we will discuss it in detail below. Figure 2 This will be explained and will not be described in detail here.

[0072] S13. Generate an emotion vector of the facial image based on the emotion feature values ​​of multiple dimensions of the facial image.

[0073] In an embodiment of the present invention, the emotion feature values ​​of multiple dimensions of the facial image can be spliced ​​in a preset arrangement order to generate an emotion vector of the facial image, wherein the arrangement order can be set according to actual needs. For example, if the dimensions corresponding to the obtained multi-dimensional emotion feature values ​​are joy-sadness, anticipation-surprise, anger-fear, and disgust-trust, the preset arrangement order can be joy-sadness, anticipation-surprise, anger-fear, and disgust-trust.

[0074] S14. Determine the emotion of the target object based on the emotion vector of at least one frame of facial image.

[0075] In the embodiment of the present invention, different emotion vectors may correspond to different emotions. After the emotion vector is obtained, the emotion corresponding to the emotion vector may be determined according to the preset correspondence between the emotion vector and the emotion, and used as the emotion of the target object.

[0076] The larger the dimension of the emotional feature, the more types of emotional vectors can be generated, and the more types of corresponding emotions. It can be seen that the present application can identify more types of emotions with higher detail.

[0077] The embodiment of the present application provides a method for identifying emotions of an object in a video, which obtains at least one facial image of a target object from a video, inputs the facial image into a pre-trained emotion feature extraction model, thereby obtaining an emotion vector corresponding to the facial image and containing multi-dimensional emotion feature values, determines a target emotion vector based on the emotion vector corresponding to at least one facial image, and inputs the target emotion vector into a preset emotion interpretation model to obtain corresponding emotion interpretation information for representing the emotion of the target object. The method extracts features from multiple dimensions of the facial image of the target object, thereby obtaining a multi-dimensional emotion vector, and the emotion interpretation information determined by using the multi-dimensional emotion vector can more finely represent the emotion of the target object, making the interpreted emotion more accurate and detailed.

[0078] join Figure 2 , which is a flow chart of a method for training an emotion feature extraction model provided by an embodiment of the present invention. Figure 2 As shown, the following steps may be included:

[0079] S21. Determine the initial model.

[0080] In an embodiment of the present invention, the initial model may include multiple image regression models, and different image regression models correspond to different dimensions. For example, the initial model may include four image regression models, and the dimensions corresponding to the four image regression models are joy-sadness, anticipation-surprise, anger-fear, and disgust-trust, respectively.

[0081] It should be noted that the number of image regression models included in the initial model can be set according to actual needs and is not specifically limited in this embodiment.

[0082] S22. Train each image regression model separately.

[0083] In an embodiment of the present invention, the training of the image regression model may include the following steps:

[0084] Step 1: Get the training dataset.

[0085] As an embodiment, the training data set may include multiple frames (e.g., 100,000 frames) of facial images annotated with emotion labels, wherein the emotion labels represent the emotional feature values ​​of the objects in the facial images on the target dimension, wherein the target dimension is the dimension corresponding to the currently trained image regression model. For example, when training an image regression model corresponding to the dimension of joy-sadness, the target dimension is joy-sadness, and the emotion labels in the obtained training data are emotion labels representing joy, sadness, or no emotion, wherein the emotion labels can be represented by -1, 0, or 1, wherein -1 represents joy, 0 represents no emotion, and 1 represents sadness.

[0086] As an embodiment, in order to obtain sufficient training data and ensure the annotation quality of the training data, a training data set may be constructed by cutting frames of classic clips of film and television works, which may specifically include:

[0087] Manually select classic clips (generally less than five minutes in length) with significant features (features are 1 or -1) on the target dimension corresponding to the image regression model to be trained, capture the key frames of objects in this clip that show emotions corresponding to the target dimension, and uniformly add labels (0, 1 or -1) to the captured key frames. Then, store the labeled key frames to a preset location. When training the image regression model, directly obtain the stored images from the preset location as the training data set.

[0088] Furthermore, when constructing a training data set for each image regression model, a balanced selection method can be used to select the number of classic clips. For example, do not select too many works of one actor. The actors should include both male and female, young and old, and should wear ancient costumes, Republican costumes, and modern costumes. The number of objects and works, as well as the distribution of factors that do not directly determine the characteristics of the emotional dimension, such as the actor, gender, and age of the character itself, are controlled to ensure that the training data can reasonably reflect the emotional characteristics of this dimension.

[0089] Step 2: Use the training data set to train the image regression model to obtain a trained image regression model.

[0090] In an embodiment of the present invention, the image regression model may be a deep convolutional neural network, and thus the image regression model may be trained using an existing deep convolutional neural network training method, which will not be described in detail in this embodiment.

[0091] S23. Combine the trained multiple image regression models into an emotion feature extraction model.

[0092] Take the 4-dimensional emotion feature extraction model as an example, Figure 3As shown, the emotion feature model includes 4 image regression models, and the 4 image regression models share one input port. Different image regression models correspond to different output ports. In this way, the emotion feature model may include 1 input port and 4 output ports. When applied, a facial image can be input to output the emotion feature values ​​of the 4 dimensions corresponding to the facial image.

[0093] The emotion feature extraction model trained in the above manner can input only one facial image and then obtain emotion feature values ​​in multiple dimensions, which can facilitate more realistic and detailed reflection of the object's emotions.

[0094] When the emotion feature extraction model trained in the above manner is used to extract emotion features, since the image regression model usually outputs a value between 0 and 1, the emotion feature value output by the emotion feature extraction model is a value between 0 and 1, and usually one dimension can correspond to three emotions, such as joy-sadness, the corresponding emotions can be joy, no emotion, or sadness. In this way, in the same dimension, different emotion feature values ​​may correspond to the same emotion, such as joy-sadness, any emotion feature value between 0 and 0.2 represents joy, any emotion feature value between 0.2 and 0.8 represents no emotion, and any emotion feature value between 0.8 and 1 represents sadness.

[0095] Based on the above content, in another embodiment of the present invention, before generating the emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image, the emotion feature values ​​of each dimension can be discretized first, and then the discretized emotion feature values ​​can be combined into an emotion vector.

[0096] As an embodiment, for the emotional feature value of each dimension, the emotional feature value can be discretized into -1, 0 or 1 according to the upper threshold and lower threshold corresponding to the dimension, where -1, 0 or 1 respectively represent different emotions. For example, in the dimension of joy-sadness, -1 can represent joy, 0 can represent no emotion, and 1 can represent sadness.

[0097] Specifically, an upper threshold and a lower threshold may be set for each dimension, and then for each dimension, the emotional feature value may be discretized according to the upper threshold and the lower threshold of the dimension, so that the emotional feature value is discretized into one of the three polarities of -1, 0, and 1. Taking the dimension of joy-sadness as an example, the corresponding upper threshold may be 0.80, and the lower threshold may be 0.20. If the emotional feature value of the dimension is 0.15, the emotional feature value will be discretized to -1, if the emotional feature value is 0.45, it will be discretized to 0, and if the output result is 0.90, it will be discretized to 1.

[0098] Among them, the upper threshold and lower threshold corresponding to different dimensions may be different, and the upper and lower thresholds of each dimension may be determined by experiments. Specifically, the upper and lower thresholds may be determined in the following manner for each dimension: obtain a sample set containing multiple sample images, label the sample images in the sample set with emotion labels, and regress the image model corresponding to the output dimension of the sample set to obtain the output result, and then for the upper and lower thresholds, equally spaced threshold values ​​may be taken for exhaustive enumeration, and the penalty function f = Σ5e2 + e1 is calculated (e2 represents the model output result of -1 is manually labeled as +1, or the model output result of +1 is manually labeled as -1, e1 represents the model output result of 0 is manually labeled as +1 or -1, or the model output result of +1 or -1 is manually labeled as 0), and the upper and lower thresholds that minimize f are obtained as the upper and lower thresholds corresponding to the dimension.

[0099] Through the embodiments of the present invention, the characteristic data values ​​are discretized and then the emotion vector is generated, and the emotion characteristic values ​​of each dimension are mapped to three polarities of -1, 0, and 1, so that different emotion characteristic values ​​representing the same emotion are represented by the same numerical value, ensuring that each emotion vector finally obtained can represent different emotions, thereby reducing the complexity of subsequent processing.

[0100] See also Figure 4 A flowchart of a method for identifying object emotions in a video provided by another embodiment of the present invention is shown in FIG. Figure 4 As shown, the method may include:

[0101] S41. Acquire at least one frame of facial image of the target object from the video.

[0102] S42. Input the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image.

[0103] S43. Generate an emotion vector of the facial image based on the emotion feature values ​​of multiple dimensions of the facial image.

[0104] For the specific implementation of S41-S43, please refer to the detailed description of S11-S13, which will not be repeated here.

[0105] S44. Determine a target emotion vector from emotion vectors of at least one frame of facial image.

[0106] In an embodiment of the present invention, if a facial image of the target object is obtained from the video in S41, the emotion vector of the facial image is determined as the target emotion vector; if multiple facial images of the target object are obtained from the video in S41, the emotion vector corresponding to the facial image with significant emotions can be selected from the multiple facial images as the target emotion vector.

[0107] As for how to select the emotion vector corresponding to the facial image with significant emotion from multiple frames of facial images as the target emotion vector, we will combine the following with the attached Figure 5 This will be explained and will not be described in detail here.

[0108] S45. Input the target emotion vector into a preset emotion interpretation model to obtain emotion interpretation information corresponding to the target emotion vector and used to represent the emotion of the target object.

[0109] In an embodiment of the present invention, the emotion interpretation information may be words used to express emotions, such as anxiety, worry, etc. The emotion interpretation model includes multiple emotion interpretation information corresponding to multiple emotion vectors. For example, if the emotion feature extraction model can output emotion feature values ​​of four dimensions, then according to the emotion feature values ​​output by the emotion feature extraction model, 3 to the fourth power, i.e., 81 different emotion vectors, can be generated in theory. Different emotion vectors can represent different emotional states of the object, for example, 81 subdivided emotions such as anxiety, worry, expectation, jealousy, etc. can be represented. Then, the emotion interpretation model can include 81 emotion interpretation information corresponding to 81 emotion vectors.

[0110] As an embodiment, the emotion interpretation model can also provide mutual semantic associations between the 81 kinds of emotion interpretation information. For example, alertness and doubt can be set as similar emotion interpretation information, because both contain the psychological activity of "what happened in the outside world is unexpected", and hopefulness and cynicism can be set as opposite emotion interpretation information, because hopefulness means "accepting" the facts of the external environment, while cynicism means "sneering" at the external environment, which are completely opposite. By setting semantic associations, the emotions of the target object can be interpreted more accurately and meticulously based on the emotion interpretation information associated with the interpreted emotion interpretation information.

[0111] Furthermore, after obtaining the emotion interpretation information corresponding to the target emotion vector, an emotion interpretation sentence containing the emotion interpretation information may be generated.

[0112] As an embodiment, a specific sentence pattern may be set in advance according to actual needs, and then a position for inserting the emotion interpretation information may be preset in the specific sentence pattern, so that after obtaining the emotion interpretation information corresponding to the target emotion vector, a preset function may be called to insert the obtained emotion interpretation information into the corresponding position, thereby generating an emotion interpretation sentence containing the emotion interpretation information. For example, the specific sentence pattern may be "the target object feels ()", where () is the position set for inserting the emotion interpretation information, and if the emotion interpretation information is annoyed, the generated emotion interpretation sentence is "the target object feels annoyed".

[0113] As another embodiment, a position for inserting a target object identifier (such as name, variety, etc.) can also be preset in a specific sentence pattern, and then after determining the emotion interpretation information, the target object identifier and the emotion interpretation information are respectively inserted into the corresponding positions of the specific sentence pattern to generate a corresponding emotion interpretation sentence. For example, the preset sentence pattern can be "() feels ()", where the first () is set to fill in the target object identifier, and the second () is set to fill in the emotion interpretation information. If the target object's name is A and the emotion interpretation information is annoyed, the generated emotion interpretation sentence is "A feels annoyed". The target object identifier can be input from the outside, or a corresponding relationship table between the object image and the object identifier can be pre-set, and then obtained by table lookup.

[0114] By using the above method, generating emotion interpretation sentences can more intuitively understand the emotions of the object.

[0115] In the embodiment of the present invention, when multiple frames of facial images are acquired, by screening the emotion vectors, the amount of calculation can be reduced and the work efficiency can be improved while ensuring the accuracy of the recognition result.

[0116] See also Figure 5 , which is a flow chart of a method for determining a target emotion vector provided by an embodiment of the present invention, such as Figure 5 As shown, the following steps may be included:

[0117] S51. Generate a time series of emotion vectors according to the order of the playing time of the multiple frames of facial images from first to last, wherein the time series includes the emotion vectors of the multiple frames of facial images.

[0118] As an embodiment, the playback time corresponding to each facial image can be determined, and then the corresponding playback time label can be added to the emotion vector of each facial image to identify the playback time corresponding to the emotion vector. Finally, the emotion vectors are sorted in order of playback time from first to last, thereby obtaining a time series.

[0119] S52. Divide the time interval corresponding to the time series into multiple candidate time intervals.

[0120] In an embodiment of the present invention, the time interval corresponding to the time series can be divided by a sliding window of a fixed length, wherein the length of the sliding window can be set according to actual needs, for example, it can be 5 minutes. By sliding the sliding window in the time series, the time interval of the time series can be divided into multiple continuous sub-time intervals, that is, candidate time intervals. For example, if the total length of the time interval of the time series is 20 minutes, a sliding window with a length of 5 minutes can be used to divide the time interval of the time series into 4 candidate time intervals with a length of 5 minutes.

[0121] S53. For each candidate time interval, determine the density of each emotion vector included in the candidate time interval.

[0122] In an embodiment of the present invention, the density of the emotion vector in the candidate time interval can reflect the frequency of the emotion vector appearing in the candidate time. The greater the density, the greater the frequency of the emotion vector appearing, which means that the emotion corresponding to the emotion vector is more obvious in the candidate time interval.

[0123] S54. From the plurality of candidate time intervals, select a candidate time interval in which the density of each emotion vector meets the preset requirements as the target time interval.

[0124] In the embodiment of the present invention, for each candidate time interval, the following steps may be used to determine whether it meets the preset requirements:

[0125] Step 1: Determine the density level to which the density of each emotion vector contained in the candidate time interval belongs according to the preset density level division rule.

[0126] The density level division rule may include multiple density levels and density values ​​corresponding to the multiple density levels. Therefore, the density level corresponding to each density may be determined according to the density level division rule.

[0127] Step 2: For each emotion vector, the density of the emotion vector is compared with the density threshold corresponding to the density level to which the density belongs.

[0128] As an example, the density threshold may be determined in the following manner:

[0129] Artificially annotate several emotionally charged segments of the subjects in the film and television clips, use the algorithm to derive the values ​​of each density level, take the lowest value of these segments for each density level, and round down the lowest value (e.g., if the lowest value is 2.24, take 2) as the density threshold.

[0130] Step 3: If the density of each emotion vector included in the candidate time interval is greater than the corresponding density threshold, it is determined that the candidate time interval meets the preset requirements, otherwise it is determined that the candidate time interval does not meet the preset requirements.

[0131] In the above manner, facial images containing full subject emotions can be filtered out.

[0132] S55. Determine the emotion vector with the highest density in the target time interval as the target emotion vector.

[0133] The following is an explanation of the method of determining the density of each emotion vector included in the candidate time interval in S53:

[0134] In the embodiment of the present invention, for any candidate time interval, the following steps can be used to determine the density of each emotion vector contained therein:

[0135] Step 1: Calculate the edit distance between each emotion vector in the candidate time interval.

[0136] In this embodiment of the present invention, the definition of the edit distance is: the edit distance between two emotion vectors v1 and v2 = Σ1000 (when i1 = -i2), or 1 (when i1*i2 = 0), or 0 (when i1 = i2), where i1 is the value of dimension i in v1, and i2 is the value of dimension i in v2.

[0137] Step 2: For each emotion vector in the candidate time interval, other emotion vectors in the candidate time interval whose edit distance with the emotion vector is less than a preset distance threshold are used as similar vectors of the emotion vector.

[0138] In the embodiment of the present invention, the distance threshold may be set according to actual needs, for example, may be 0 or 1.

[0139] Step 3: For each emotion vector in the candidate time interval, determine the density of the emotion vector according to the emotion vector and the similarity vector of the emotion vector.

[0140] In the embodiment of the present invention, determining the density of the emotion vector according to the emotion vector and the similarity vector of the emotion vector may include:

[0141] The frequency of the emotion vector appearing in the candidate time interval and the frequency of the similar vector of the emotion vector appearing in the candidate time interval are weighted summed to obtain the sum value, and the sum value is divided by the time span of the candidate time interval to obtain the density of the emotion vector, and the unit of density is frames per second.

[0142] Among them, the frequency of the emotion vector appearing in the candidate time interval is the number of frames of the facial image corresponding to the emotion vector in the candidate time interval, and the frequency of the similar vector is the same. The weights in the weighted summation can be set according to the needs. The time span of the candidate time interval is the difference between the start and end time of the candidate time interval. For example, if the start time of the candidate time interval is 00:01 and the end time is 05:01, then the time span of the candidate time interval is 5 minutes, which is 300 seconds.

[0143] The embodiment of the present application provides a method of screening facial images according to the density of emotion vectors, screening out facial images containing full object emotions, and then determining the target object's emotions based on the emotion vectors of the facial images containing full object emotions, thereby ensuring the accuracy of emotion recognition.

[0144] In another embodiment of the present invention, the method for identifying object emotions in a video may further include: obtaining text-related information related to the target object from the video, and then using a bag-of-words model to extract keywords from the text (adjectives that directly describe a person's emotional state or verbs that indirectly express a person's emotional state), and determining the emotion of the target object based on relevant theoretical research on the emotion roulette model.

[0145] See also Figure 6 , is another embodiment of the present invention, providing a device for recognizing emotions of objects in a video, such as Figure 6 As shown, the device may include:

[0146] The acquisition module 601 is used to acquire at least one frame of facial image of the target object from the video.

[0147] The feature extraction module 602 is used to input the facial image into a pre-trained emotion feature extraction model so that the emotion feature value extraction model can extract emotion feature values ​​of multiple dimensions of the facial image.

[0148] The vector generation module 603 is used to generate an emotion vector of a facial image according to emotion feature values ​​of multiple dimensions.

[0149] The emotion determination module 604 is used to determine the emotion of the target object according to the emotion vector of at least one frame of facial image.

[0150] As an embodiment, the emotion determination module 604 is specifically configured to:

[0151] Determine a target emotion vector from emotion vectors of at least one frame of facial image;

[0152] The target emotion vector is input into a preset emotion interpretation model to obtain emotion interpretation information corresponding to the target emotion vector and used to represent the emotion of the target object. The emotion interpretation model includes multiple emotion interpretation information corresponding to multiple emotion vectors.

[0153] As an embodiment, determining a target emotion vector from emotion vectors of at least one frame of the facial image includes:

[0154] If a facial image of the target object is obtained from the video, the emotion vector of the facial image is determined as the target emotion vector;

[0155] If multiple facial images of the target object are obtained from the video, a time series of emotion vectors is generated according to the order of the playback time of the multiple facial images from first to last, and the time series includes the emotion vectors of the multiple facial images;

[0156] Divide the time interval corresponding to the time series into multiple candidate time intervals;

[0157] For each of the candidate time intervals, determining the density of each emotion vector contained in the candidate time interval;

[0158] From multiple candidate time intervals, select a candidate time interval whose density of each emotion vector meets preset requirements as a target time interval;

[0159] The emotion vector with the highest density in the target time interval is determined as the target emotion vector.

[0160] As an embodiment, determining the density of each emotion vector contained in the candidate time interval includes:

[0161] Calculate the edit distance between the emotion vectors contained in the candidate time interval;

[0162] For each emotion vector in the candidate time interval, other emotion vectors in the candidate time interval whose edit distance with the emotion vector is less than a preset distance threshold are used as similar vectors of the emotion vector;

[0163] For each emotion vector in the candidate time interval, the density of the emotion vector is determined according to the emotion vector and the similarity vector of the emotion vector.

[0164] As an embodiment, determining the density of the emotion vector according to the emotion vector and the similarity vector of the emotion vector includes:

[0165] The frequency of the emotion vector appearing in the candidate time interval and the frequency of the similar vector appearing in the candidate time interval are weighted and summed to obtain the sum value;

[0166] The density of the emotion vector is obtained by dividing the sum by the time span of the candidate time interval.

[0167] As an embodiment, selecting a candidate time interval including emotion vectors having densities meeting preset requirements from multiple candidate time intervals as a target time interval includes:

[0168] For each candidate time interval, determine the density level to which the density of each emotion vector contained in the candidate time interval belongs according to a preset density level classification rule;

[0169] For each emotion vector, the density of the emotion vector is compared with a density threshold corresponding to the density level to which the density belongs;

[0170] If the density of each emotion vector included in the candidate time interval is greater than the corresponding density threshold, it is determined that the candidate time interval meets the preset requirements, and the candidate time interval is used as the target time interval.

[0171] As an embodiment, the emotion feature extraction model includes multiple image regression models. The image regression model takes a facial image as input and an emotion feature value as output. Different image regression models output emotion feature values ​​with different dimensions.

[0172] As an embodiment, the vector generation module 603 is specifically used for:

[0173] Discretizing the emotion feature values ​​of multiple dimensions of the facial image respectively;

[0174] The discretized emotion feature values ​​of multiple dimensions are combined into an emotion vector of the facial image.

[0175] See also Figure 7 , Figure 7 It is a structural schematic diagram of an electronic device provided by another embodiment of the present application.

[0176] like Figure 7 As shown, the electronic device provided in this embodiment includes: at least one processor 701, a memory 702, at least one network interface 703 and other user interfaces 704. The various components in the electronic device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus systems 705 in Figure X.

[0177] The user interface 704 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0178] It can be understood that the memory 702 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0179] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 7021 and a second application 7022 .

[0180] The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The second application 7022 includes various second application programs, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The program for implementing the method of the embodiment of the present invention can be included in the second application 7022.

[0181] In the embodiment of the present invention, by calling the program or instruction stored in the memory 702, specifically, the program or instruction stored in the second application 7022, the processor 701 is used to execute the method steps provided by each method embodiment, for example, including:

[0182] Acquire at least one facial image frame of a target object from a video;

[0183] Inputting the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image;

[0184] Generating an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image;

[0185] The emotion of the target object is determined according to an emotion vector of at least one frame of the facial image.

[0186] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with its hardware.

[0187] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application, or a combination thereof.

[0188] For software implementation, the technology of this article can be implemented by a unit that performs the functions of this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0189] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0190] When one or more programs in the storage medium can be executed by one or more processors, the method for object emotion recognition in a video executed on the electronic device side can be implemented.

[0191] The processor is used to execute the object emotion recognition method program in the video stored in the memory to implement the following steps of the object emotion recognition method in the video executed on the electronic device side:

[0192] Acquire at least one facial image frame of a target object from a video;

[0193] Inputting the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image;

[0194] Generating an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image;

[0195] The emotion of the target object is determined according to an emotion vector of at least one frame of the facial image.

[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0198] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0199] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0200] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0201] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0202] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0203] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0204] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0205] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for identifying object emotions in a video, characterized in that: include: Acquire at least one facial image frame of a target object from a video; Inputting the facial image into a pre-trained emotion feature extraction model so that the emotion feature extraction model extracts emotion feature values ​​of multiple dimensions of the facial image; Generating an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image; Determining the emotion of the target object according to the emotion vector of at least one frame of the facial image, comprising: determining a target emotion vector from the emotion vector of at least one frame of the facial image; inputting the target emotion vector into a preset emotion interpretation model to obtain emotion interpretation information corresponding to the target emotion vector and used to represent the emotion of the target object, wherein the emotion interpretation model includes a plurality of emotion interpretation information corresponding to a plurality of emotion vectors; Wherein, determining the target emotion vector from the emotion vector of at least one frame of the facial image comprises: if multiple frames of facial images of the target object are obtained from the video, generating a time sequence of emotion vectors according to the order of the playback time of the multiple frames of the facial images from first to last, wherein the time sequence includes the emotion vectors of the multiple frames of the facial images; Dividing the time interval corresponding to the time series into a plurality of candidate time intervals; For each of the candidate time intervals, determining the density of each emotion vector contained in the candidate time interval; From the plurality of candidate time intervals, a candidate time interval including emotion vectors having densities meeting preset requirements is selected as a target time interval; The emotion vector with the highest density in the target time interval is determined as the target emotion vector.

2. The method according to claim 1, characterized in that The step of determining a target emotion vector from the emotion vector of at least one frame of the facial image further comprises: If a facial image of the target object is obtained from the video, the emotion vector of the facial image is determined to be the target emotion vector.

3. The method according to claim 2, characterized in that The determining the density of each emotion vector included in the candidate time interval includes: Calculating the edit distance between the emotion vectors included in the candidate time interval; For each emotion vector in the candidate time interval, other emotion vectors in the candidate time interval whose edit distance with the emotion vector is less than a preset distance threshold are used as similar vectors of the emotion vector; For each emotion vector in the candidate time interval, the density of the emotion vector is determined according to the emotion vector and a similarity vector of the emotion vector.

4. The method according to claim 3, characterized in that The determining the density of the emotion vector according to the emotion vector and the similarity vector of the emotion vector comprises: Performing weighted summation on the frequency of occurrence of the emotion vector in the candidate time interval and the frequency of occurrence of the similarity vector in the candidate time interval to obtain a sum value; The density of the emotion vector is obtained by dividing the sum by the time span of the candidate time interval.

5. The method according to claim 2, characterized in that: The step of selecting, from the plurality of candidate time intervals, a candidate time interval whose density of each emotion vector meets a preset requirement as a target time interval includes: For each of the candidate time intervals, determining the density level to which the density of each emotion vector contained in the candidate time interval belongs according to a preset density level classification rule; For each emotion vector, comparing the density of the emotion vector with a density threshold corresponding to the density level to which the density belongs; If the density of each emotion vector included in the candidate time interval is greater than the corresponding density threshold, it is determined that the candidate time interval meets the preset requirements, and the candidate time interval is used as the target time interval.

6. The method according to any one of claims 1 to 5, characterized in that: The emotion feature extraction model includes a plurality of image regression models, wherein the image regression model takes a facial image as input and an emotion feature value as output, and the dimensions of the emotion feature values ​​output by different image regression models are different.

7. The method according to claim 6, characterized in that Generating the emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions of the facial image comprises: Discretizing the emotion feature values ​​of multiple dimensions of the facial image respectively; The discretized emotion feature values ​​of multiple dimensions are combined into an emotion vector of the facial image.

8. A device for identifying emotions of objects in a video, characterized in that: include: An acquisition module, used for acquiring at least one frame of facial image of a target object from a video; A feature extraction module, used for inputting the facial image into a pre-trained emotion feature extraction model, so that the emotion feature value extraction model extracts emotion feature values ​​of multiple dimensions of the facial image; A vector generation module, used to generate an emotion vector of the facial image according to the emotion feature values ​​of multiple dimensions; The emotion determination module is used to determine the emotion of the target object according to the emotion vector of at least one frame of the facial image, including: determining a target emotion vector from the emotion vector of at least one frame of the facial image; inputting the target emotion vector into a preset emotion interpretation model to obtain emotion interpretation information corresponding to the target emotion vector and used to represent the emotion of the target object, wherein the emotion interpretation model includes multiple emotion interpretation information corresponding to multiple emotion vectors; Wherein, determining the target emotion vector from the emotion vector of at least one frame of the facial image comprises: if multiple frames of facial images of the target object are obtained from the video, generating a time sequence of emotion vectors according to the order of the playback time of the multiple frames of the facial images from first to last, wherein the time sequence includes the emotion vectors of the multiple frames of the facial images; Dividing the time interval corresponding to the time series into a plurality of candidate time intervals; For each of the candidate time intervals, determining the density of each emotion vector contained in the candidate time interval; From the plurality of candidate time intervals, a candidate time interval including emotion vectors having densities meeting preset requirements is selected as a target time interval; The emotion vector with the highest density in the target time interval is determined as the target emotion vector.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute the object emotion recognition method program in the video stored in the memory to implement the object emotion recognition method in the video described in any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method for object emotion recognition in a video as described in any one of claims 1-7.

Images