Video emotion recognition method and device and computer equipment

By combining facial and background features in video emotion recognition and using neural network models for emotion analysis, the problem of low accuracy of face image emotion analysis is solved, and the accuracy of emotion recognition is improved.

CN118982852APending Publication Date: 2024-11-19CHINA TELECOM BESTPAY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410992532.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the prior art, the accuracy of emotion analysis based on face images is not high, especially when the face part is not clear, it is difficult to accurately identify human emotions.

Method used

By obtaining the video to be identified and inputting it into the pre-trained neural network model, feature extraction of facial and background videos is performed, combining facial features and background features, using neural network models to determine the emotion of the object, using three-dimensional convolution and attention mechanisms, etc. Technology improves the accuracy of emotion recognition.

Benefits of technology

The recognition of face parts and background parts in the video is achieved comprehensively considering the recognition of face parts and background parts, which improves the accuracy of emotional recognition, and solves the problem of low accuracy of face images based on face images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982852A_ABST
    Figure CN118982852A_ABST
Patent Text Reader

Abstract

The invention discloses a video emotion recognition method and device and computer equipment. The method comprises the following steps: acquiring a to-be-identified video; the to-be-recognized video is input into a pre-trained neural network model, the neural network model outputs emotion of an object in the to-be-recognized video, the neural network model carries out feature extraction on a face video in the to-be-recognized video and carries out feature extraction on background videos except the face video in the to-be-recognized video, and the emotion of the object in the to-be-recognized video is output; and determining the emotion of the object according to the facial features extracted from the facial video and the background features extracted from the background video. According to the invention, the technical problem of low accuracy of sentiment analysis based on a face image in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a video emotion recognition method, device and computer equipment. Background Art

[0002] Recognizing human emotions from image content has broad application prospects in real life, such as in the fields of healthcare and human-computer interaction systems. Research on sentiment analysis using ordinary deep networks mainly focuses on the perception of facial expressions. The most discriminative method is the sentiment analysis system based on face images, but when the face is not clear, the accuracy of sentiment judgment is not high.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a video emotion recognition method, apparatus and computer device to at least solve the technical problem of low accuracy of emotion analysis based on facial images in the related art.

[0005] According to one aspect of an embodiment of the present invention, a video emotion recognition method is provided, comprising: obtaining a video to be recognized; inputting the video to be recognized into a pre-trained neural network model, and having the neural network model output the emotion of an object in the video to be recognized, wherein the neural network model performs feature extraction on facial video in the video to be recognized, and performs feature extraction on background video other than the facial video in the video to be recognized, and determines the emotion of the object based on facial features extracted from the facial video and background features extracted from the background video.

[0006] Optionally, the video to be identified is input into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, including: inputting the video to be identified into a segmentation unit, and the segmentation unit segments the video to be identified into a facial video and a background video, wherein the neural network model includes the segmentation unit; inputting the facial video into a first three-dimensional convolution unit, and the first three-dimensional convolution unit performs feature extraction on the facial video to obtain facial features, wherein the neural network model includes the first three-dimensional convolution unit; inputting the background video into a feature extraction module, and the feature extraction module performs feature extraction on the background video to obtain background features, wherein the neural network model includes the feature extraction module; splicing the facial features and the background features, and inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object, wherein the neural network model includes the classification unit.

[0007] Optionally, the background video is input into a feature extraction module, and the feature extraction module performs feature extraction on the background video to obtain background features, including: inputting the background video into a second three-dimensional convolution unit, and the second three-dimensional convolution unit performs feature extraction on the background video to obtain initial background features, wherein the feature extraction module includes the second three-dimensional convolution unit; inputting the initial background features into an attention mechanism unit, and the attention mechanism unit determines the weight of the initial background features, wherein the feature extraction module includes the attention mechanism unit; and multiplying the weight of the initial background features by the initial background features to obtain background features.

[0008] Optionally, the facial video is input into a first three-dimensional convolution unit, and the first three-dimensional convolution unit extracts features from the facial video to obtain facial features, including: using a spatial convolution layer to convolve the facial video to obtain spatial features of the facial video, wherein the first three-dimensional convolution unit includes the spatial convolution layer; using a temporal convolution layer to convolve the spatial features of the facial video to obtain facial features, wherein the first three-dimensional convolution unit includes the temporal convolution layer.

[0009] Optionally, facial features and background features are spliced, and the spliced ​​features are input into a classification unit, and the classification unit outputs the emotion of the object, including: sampling facial features using a first average pooling layer, and sampling background features using a second average pooling layer; splicing the sampled facial features and the sampled background features to obtain spliced ​​features; inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object.

[0010] Optionally, the neural network model is obtained in the following manner: obtaining training samples, wherein the training samples include a sample video, and an emotion type corresponding to the sample video; inputting the sample video and the corresponding emotion type into the original neural network model, having the original neural network model extract features of the sample video, and determining an initial emotion type based on the extracted features; adjusting the parameters included in the original neural network model based on the initial emotion type and the emotion type corresponding to the sample video, to obtain the neural network model.

[0011] According to another aspect of an embodiment of the present invention, a video emotion recognition device is also provided, including: an acquisition module for acquiring a video to be recognized; a recognition module for inputting the video to be recognized into a pre-trained neural network model, and the neural network model outputs the emotion of an object in the video to be recognized, wherein the neural network model extracts features of facial videos in the video to be recognized, and extracts features of background videos other than facial videos in the video to be recognized, and determines the emotion of the object based on facial features extracted from the facial video and background features extracted from the background video.

[0012] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned video emotion recognition methods.

[0013] According to another aspect of the embodiments of the present invention, a computer device is provided. The computer device includes a processor, and the processor is used to run a program. When the program is run, any one of the above-mentioned video emotion recognition methods is executed.

[0014] According to yet another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, any one of the above-mentioned video emotion recognition methods is implemented.

[0015] In an embodiment of the present invention, a video to be identified is obtained; the video to be identified is input into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, wherein the neural network model performs feature extraction on the facial video in the video to be identified, and performs feature extraction on the background video other than the facial video in the video to be identified, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video, thereby achieving the purpose of comprehensively considering the facial part and the background part in the video to identify the emotion of the object, thereby achieving the technical effect of improving the accuracy of identifying emotions, and further solving the technical problem of low accuracy of emotion analysis based on facial images in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a computer terminal for implementing a video emotion recognition method is shown;

[0018] Figure 2 is a flow chart of a video emotion recognition method provided according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the structure of a neural network model provided according to an optional embodiment of the present invention;

[0020] Figure 4 It is a structural block diagram of a video emotion recognition device provided according to an optional embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] According to an embodiment of the present invention, a method embodiment of video emotion recognition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a video emotion recognition method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1Different configurations are shown.

[0025] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0026] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video emotion recognition method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the video emotion recognition method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0028] Figure 2 FIG. 1 is a flow chart of a video emotion recognition method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0029] Step S202, obtaining the video to be identified.

[0030] In this step, a video can be selected from a video library or a real-time captured video as input, and the video contains the face of the object to be identified and the surrounding background information.

[0031] Step S204, inputting the video to be identified into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, wherein the neural network model extracts features of the facial video in the video to be identified, and extracts features of the background video other than the facial video in the video to be identified, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video.

[0032] In this step, the video to be identified can be input into a neural network model that has been trained with a large amount of data for emotion recognition. The neural network model will extract features of the face and background in the video, and comprehensively consider the facial features extracted from the facial video and the background features extracted from the background video to determine the emotion of the object. This method of comprehensively considering facial and background information can improve the accuracy of emotion recognition. Usually, people can express their emotions not only from their faces, but also based on the surrounding situations, such as behavior, interaction with others, etc. Therefore, background information will also affect the expression of emotions.

[0033] Through the above steps, the purpose of comprehensively considering the facial part and the background part of the video to identify the emotion of the object can be achieved, thereby achieving the technical effect of improving the accuracy of emotion recognition, and further solving the technical problem of low accuracy of emotion analysis based on facial images in related technologies.

[0034] As an optional embodiment, the video to be identified is input into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, including: inputting the video to be identified into a segmentation unit, and the segmentation unit segments the video to be identified into a facial video and a background video, wherein the neural network model includes the segmentation unit; inputting the facial video into a first three-dimensional convolution unit, and the first three-dimensional convolution unit performs feature extraction on the facial video to obtain facial features, wherein the neural network model includes the first three-dimensional convolution unit; inputting the background video into a feature extraction module, and the feature extraction module performs feature extraction on the background video to obtain background features, wherein the neural network model includes the feature extraction module; splicing the facial features and the background features, and inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object, wherein the neural network model includes the classification unit.

[0035] Optionally, first, the video to be identified can be input into a segmentation unit, which is used to segment the video into a facial video and a background video. Specifically, a pre-trained face detection model or a deep learning-based model such as a multi-task convolutional network can be used to identify the face area in the image and return the location of the face. The specific method can be to preliminarily distinguish the face based on the skin color threshold, and then perform more detailed segmentation based on the key points of the face.

[0036] Secondly, the facial video can be input into the first three-dimensional convolution unit, which will extract features from the facial video to obtain facial features. Through the convolution operation and activation function, key features in the facial video can be extracted for emotion recognition. At the same time, the background video can be input into the feature extraction module, which will extract features from the background video to obtain background features. By extracting features from the background video, some important information about the background can be extracted for emotion recognition.

[0037] Finally, the facial features and background features can be concatenated together to form a complete feature vector, which is then input into the classification unit, which outputs the object's emotion. By comprehensively considering the feature information of the face and the background, the accuracy of emotion recognition can be improved.

[0038] As an optional embodiment, the background video is input into a feature extraction module, and the feature extraction module performs feature extraction on the background video to obtain background features, including: inputting the background video into a second three-dimensional convolution unit, and the second three-dimensional convolution unit performs feature extraction on the background video to obtain initial background features, wherein the feature extraction module includes the second three-dimensional convolution unit; inputting the initial background features into an attention mechanism unit, and the attention mechanism unit determines the weight of the initial background features, wherein the feature extraction module includes the attention mechanism unit; multiplying the weight of the initial background features by the initial background features to obtain background features.

[0039] Optionally, the feature extraction module for extracting features from the background video includes a three-dimensional convolution unit and an attention mechanism unit, which can better assist the performance of the emotion recognition task. First, the background video can be input into a feature extraction module, in which the background video is first further feature extracted by a second three-dimensional convolution unit, which can help extract richer and more abstract background features so as to better understand the content in the background video. Secondly, in the feature extraction module, an attention mechanism unit is also included, which is used to determine the importance of each feature in the entire feature vector according to the initial background features input, thereby improving the model's attention to background information. Finally, the weight of the background feature is multiplied by the initial background feature to obtain the final background feature. This step can help highlight important background features, reduce the interference of irrelevant information on emotion recognition, and improve the accuracy and effect of the model. The solution provided by this optional embodiment can enable the model to understand the information in the background video more deeply, thereby better assisting the performance of the emotion recognition task. In particular, the introduction of the attention mechanism can help the model focus on important features and improve the performance and effect of the model.

[0040] As an optional embodiment, a facial video is input into a first three-dimensional convolution unit, and the first three-dimensional convolution unit extracts features of the facial video to obtain facial features, including: using a spatial convolution layer to convolve the facial video to obtain spatial features of the facial video, wherein the first three-dimensional convolution unit includes the spatial convolution layer; using a temporal convolution layer to convolve the spatial features of the facial video to obtain facial features, wherein the first three-dimensional convolution unit includes the temporal convolution layer.

[0041] Optionally, the facial video can be input into the spatial convolution layer in the first three-dimensional convolution unit, which performs convolution operations on the facial video and extracts the spatial features of the facial video by sliding the convolution kernel. These spatial features can capture the pattern and structural information of different positions in the facial video. After the spatial features of the facial video are extracted by the spatial convolution layer, these features are input into the temporal convolution layer in the first three-dimensional convolution unit. The temporal convolution layer performs convolution operations on the spatial features of the facial video to capture the time series information in the facial video. This can better understand the dynamic changes and emotional expressions in the facial video.

[0042] The 3D convolution operation obtains the corresponding continuous information from both the time and space dimensions during the convolution stage. The method is to construct a 3D convolution kernel and convolve it with the action composed of multiple continuous frames, so that the feature map of the convolution layer is connected to multiple continuous frames on the previous layer, thus capturing the motion information. Specifically, facial convolution feature extraction can use 5 3×3×3 convolution layers, followed by activation functions and 5 maximum pooling layers. The pooling kernel scale of the first pooling layer is 1×2×2, in order to avoid premature merging of temporal information. The remaining pooling layers are all maximum pooling layers of size 2×2×2. The number of kernels of the five convolution layers are 32, 64, 128, 256, and 256 respectively. Finally, the output features are spatially averaged through an average pooling layer to obtain the final output result. Although using 3D convolution as a feature extraction method can simultaneously incorporate information from both spatial and temporal aspects, its computational cost and the generated model are too large. Therefore, this optional embodiment performs convolution in the spatial dimension and convolution in the temporal dimension respectively to replace the traditional 3D convolution, and replaces the original 3×3×3 convolution kernel with 1×3×3 and 3×1×1 respectively. The former is used to extract spatial features first, and the latter is used to obtain features in the temporal dimension. At the same time, a 1×1 convolution layer is added before and after the 3×3 convolution layer to reduce and restore the dimension of the feature map, respectively, which greatly reduces the amount of calculation and improves the computing speed.

[0043] As an optional embodiment, facial features and background features are spliced, and the spliced ​​features are input into a classification unit, and the classification unit outputs the emotion of the object, including: using a first average pooling layer to sample facial features, and using a second average pooling layer to sample background features; splicing the sampled facial features and the sampled background features to obtain spliced ​​features; inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object.

[0044] Optionally, after facial feature extraction is completed, the first average pooling layer can be used to sample facial features. Average pooling is a commonly used downsampling operation that reduces the size of the feature map by calculating the average value of each region in the input feature map. This can retain the main features and reduce the amount of calculation, which helps to improve the generalization ability of the model. Similarly, background features are sampled and downsampled using the second average pooling layer. By sampling the average pooling layer, the main information of the background features can be obtained, and redundant features can be reduced to prepare for subsequent feature fusion.

[0045] By concatenating the facial features and background features sampled by the average pooling layer, a comprehensive feature vector can be obtained. Such feature fusion can combine the information of the face and the background to more comprehensively represent the emotional state of the object. Finally, the concatenated feature vector can be input into the classification unit. The classification unit can be a fully connected neural network layer or other classifier, which is used to perform emotion classification on the comprehensive features and output the emotional state of the object. By integrating the facial and background information and processing it through the classification unit, the emotion of the object in the video can be more accurately identified.

[0046] As an optional embodiment, the neural network model is obtained in the following manner: obtaining training samples, wherein the training samples include a sample video and an emotion type corresponding to the sample video; inputting the sample video and the corresponding emotion type into the original neural network model, and having the original neural network model extract features from the sample video, and determine an initial emotion type based on the extracted features; and adjusting the parameters included in the original neural network model based on the initial emotion type and the emotion type corresponding to the sample video, thereby obtaining the neural network model.

[0047] Optionally, when training the neural network model, it is first necessary to prepare training samples, which include video data and corresponding emotion type labels. Each sample video has an emotion type, such as happiness, sadness, anger, etc., which is used to train the neural network model for emotion recognition. The sample video and the corresponding emotion type are input into the original neural network model. The original neural network model will extract features of the sample video and extract key information from the video for subsequent emotion classification. Based on the features extracted by the original neural network model, the original neural network can preliminarily determine the initial emotion type of the sample video. This initial emotion type is obtained by extracting and processing the features of the sample video based on the model's own existing knowledge, and may be different from the real emotion type corresponding to the sample video. According to the difference between the real emotion type corresponding to the sample video and the initial emotion type, the parameters in the original neural network model can be adjusted. Through optimization methods such as back propagation algorithm and gradient descent, the model parameters are adjusted so that the performance of the model on the training set is closer to the real emotion type, thereby improving the generalization ability of the model. After multiple rounds of training and parameter adjustment, the final neural network model is obtained. This model can more accurately perform emotion recognition on video data. By inputting video data, the corresponding emotion type can be output to realize the function of emotion recognition.

[0048] As a specific embodiment, Figure 3 is a schematic diagram of the structure of a neural network model provided according to an optional embodiment of the present invention, such as Figure 3 As shown, the neural network model of this embodiment can adopt a dual-stream channel network structure. First, the face of the input frame-level image is cropped, and then the face image (facial video) is sent to the face encoding module (first three-dimensional convolution unit), and the partial features of the face are extracted through 3D convolution and pooling. At the same time, the 3D convolution is optimized to reduce the number of parameters and improve the operation rate, and then the facial expression features are obtained through spatial average pooling; on the other hand, the remaining background video in the image is separately taken out as the input of the context encoding stream (feature extraction module), and the same 3D convolution is used. Two convolutions are also used to improve the operation rate to obtain the extracted background context features, and then the background context features are sent to an attention inference module. Through this module, the context cataloging can focus on different contexts, that is, the background context part that is helpful for sentiment analysis is extracted. Specifically, the background context features are input into the attention mechanism to obtain the spatial attention weight, and then combined with the context features obtained by the previous convolution (specifically, they can be multiplied) to obtain the background context features, and finally the outputs of the dual-stream channels are combined for the final classification.

[0049] Through the above steps, we can combine facial information and background context information, use the attention mechanism to focus on the body movements of the character background, and assist with the character's body language. At the same time, we use the optimized 3D convolution method to better extract the time dimension information in the video, reduce the amount and complexity of calculations, and the calculation rate, and ultimately improve the model's accuracy in identifying the overall emotional tendency of the character.

[0050] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that the video emotion recognition method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0052] According to an embodiment of the present invention, a video emotion recognition device for implementing the above-mentioned video emotion recognition method is also provided. Figure 4 is a structural block diagram of a video emotion recognition device provided according to an embodiment of the present invention. Figure 4 As shown, the video emotion recognition device includes: an acquisition module 41 and a recognition module 42. The video emotion recognition device is described below.

[0053] The acquisition module 41 is used to acquire the video to be identified.

[0054] The recognition module 42 is connected to the acquisition module 41, and is used to input the video to be recognized into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be recognized, wherein the neural network model extracts features of the facial video in the video to be recognized, and extracts features of the background video other than the facial video in the video to be recognized, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video.

[0055] It should be noted that the acquisition module 41 and the identification module 42 correspond to steps S202 to S206 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0056] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0057] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video emotion recognition method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned video emotion recognition method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The processor can call the information and application programs stored in the memory through the transmission device to execute the following steps: obtain the video to be identified; input the video to be identified into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, wherein the neural network model extracts features of the facial video in the video to be identified, and extracts features of the background video other than the facial video in the video to be identified, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video.

[0059] Optionally, the processor may also execute the program code of the following steps: inputting the video to be identified into a pre-trained neural network model, and having the neural network model output the emotion of the object in the video to be identified, including: inputting the video to be identified into a segmentation unit, and having the segmentation unit segment the video to be identified into a facial video and a background video, wherein the neural network model includes the segmentation unit; inputting the facial video into a first three-dimensional convolution unit, and having the first three-dimensional convolution unit perform feature extraction on the facial video to obtain facial features, wherein the neural network model includes the first three-dimensional convolution unit; inputting the background video into a feature extraction module, and having the feature extraction module perform feature extraction on the background video to obtain background features, wherein the neural network model includes the feature extraction module; splicing the facial features and the background features, and inputting the spliced ​​features into a classification unit, and having the classification unit output the emotion of the object, wherein the neural network model includes the classification unit.

[0060] Optionally, the processor may also execute the following program code: inputting the background video into a feature extraction module, and having the feature extraction module perform feature extraction on the background video to obtain background features, including: inputting the background video into a second three-dimensional convolution unit, and having the second three-dimensional convolution unit perform feature extraction on the background video to obtain initial background features, wherein the feature extraction module includes a second three-dimensional convolution unit; inputting the initial background features into an attention mechanism unit, and having the attention mechanism unit determine the weight of the initial background features, wherein the feature extraction module includes an attention mechanism unit; and multiplying the weight of the initial background features by the initial background features to obtain background features.

[0061] Optionally, the processor may also execute the program code of the following steps: inputting the facial video into a first three-dimensional convolution unit, and extracting features of the facial video by the first three-dimensional convolution unit to obtain facial features, including: convolving the facial video using a spatial convolution layer to obtain spatial features of the facial video, wherein the first three-dimensional convolution unit includes a spatial convolution layer; and convolving the spatial features of the facial video using a temporal convolution layer to obtain facial features, wherein the first three-dimensional convolution unit includes a temporal convolution layer.

[0062] Optionally, the processor may also execute the program code of the following steps: splicing facial features and background features, and inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object, including: sampling facial features using a first average pooling layer, and sampling background features using a second average pooling layer; splicing the sampled facial features and the sampled background features to obtain spliced ​​features; inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object.

[0063] Optionally, the processor may also execute the program code of the following steps: the neural network model is obtained in the following manner: obtaining a training sample, wherein the training sample includes a sample video and an emotion type corresponding to the sample video; inputting the sample video and the corresponding emotion type into the original neural network model, and having the original neural network model extract features from the sample video, and determining an initial emotion type based on the extracted features, and adjusting the parameters included in the original neural network model based on the initial emotion type and the emotion type corresponding to the sample video, to obtain the neural network model.

[0064] The embodiment of the present invention provides a solution for video emotion recognition. The video to be recognized is obtained; the video to be recognized is input into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be recognized, wherein the neural network model extracts features of the facial video in the video to be recognized, and extracts features of the background video other than the facial video in the video to be recognized, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video, thereby achieving the purpose of comprehensively considering the facial part and the background part in the video to recognize the emotion of the object, thereby achieving the technical effect of improving the accuracy of emotion recognition, and further solving the technical problem of low accuracy of emotion analysis based on facial images in the related art.

[0065] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0066] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the video emotion recognition method provided by the above embodiment.

[0067] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0068] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: obtaining a video to be identified; inputting the video to be identified into a pre-trained neural network model, and the neural network model outputs the emotion of an object in the video to be identified, wherein the neural network model performs feature extraction on facial videos in the video to be identified, and performs feature extraction on background videos other than facial videos in the video to be identified, and determines the emotion of the object based on facial features extracted from the facial video and background features extracted from the background video.

[0069] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting a video to be identified into a pre-trained neural network model, and having the neural network model output the emotion of an object in the video to be identified, including: inputting the video to be identified into a segmentation unit, and having the segmentation unit segment the video to be identified into a facial video and a background video, wherein the neural network model includes the segmentation unit; inputting the facial video into a first three-dimensional convolution unit, and having the first three-dimensional convolution unit perform feature extraction on the facial video to obtain facial features, wherein the neural network model includes the first three-dimensional convolution unit; inputting the background video into a feature extraction module, and having the feature extraction module perform feature extraction on the background video to obtain background features, wherein the neural network model includes the feature extraction module; splicing the facial features and the background features, and inputting the spliced ​​features into a classification unit, and having the classification unit output the emotion of the object, wherein the neural network model includes the classification unit.

[0070] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting the background video into a feature extraction module, and the feature extraction module performs feature extraction on the background video to obtain background features, including: inputting the background video into a second three-dimensional convolution unit, and the second three-dimensional convolution unit performs feature extraction on the background video to obtain initial background features, wherein the feature extraction module includes a second three-dimensional convolution unit; inputting the initial background features into an attention mechanism unit, and the attention mechanism unit determines the weight of the initial background features, wherein the feature extraction module includes an attention mechanism unit; multiplying the weight of the initial background features by the initial background features to obtain background features.

[0071] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting a facial video into a first three-dimensional convolution unit, and extracting features from the facial video by the first three-dimensional convolution unit to obtain facial features, including: convolving the facial video using a spatial convolution layer to obtain spatial features of the facial video, wherein the first three-dimensional convolution unit includes a spatial convolution layer; and convolving the spatial features of the facial video using a temporal convolution layer to obtain facial features, wherein the first three-dimensional convolution unit includes a temporal convolution layer.

[0072] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: splicing facial features and background features, and inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object, including: sampling facial features using a first average pooling layer, and sampling background features using a second average pooling layer; splicing the sampled facial features and the sampled background features to obtain spliced ​​features; inputting the spliced ​​features into a classification unit, and the classification unit outputs the emotion of the object.

[0073] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: the neural network model is obtained in the following manner: obtaining a training sample, wherein the training sample includes a sample video, and an emotion type corresponding to the sample video; inputting the sample video and the corresponding emotion type into the original neural network model, and the original neural network model extracts features of the sample video, and determines an initial emotion type based on the extracted features, and adjusts the parameters included in the original neural network model based on the initial emotion type and the emotion type corresponding to the sample video to obtain the neural network model.

[0074] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video emotion recognition method in each embodiment of the present application.

[0075] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0076] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0079] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0081] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video emotion recognition method, characterized in that: include: Get the video to be identified; The video to be identified is input into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be identified, wherein the neural network model extracts features of the facial video in the video to be identified, and extracts features of the background video other than the facial video in the video to be identified, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video.

2. The method according to claim 1, characterized in that The step of inputting the video to be identified into a pre-trained neural network model, and outputting the emotion of the object in the video to be identified by the neural network model, comprises: Inputting the video to be identified into a segmentation unit, wherein the segmentation unit segments the video to be identified into the facial video and the background video, wherein the neural network model includes the segmentation unit; Inputting the facial video into a first three-dimensional convolution unit, and extracting features from the facial video by the first three-dimensional convolution unit to obtain the facial features, wherein the neural network model includes the first three-dimensional convolution unit; Inputting the background video into a feature extraction module, and extracting features from the background video by the feature extraction module to obtain the background features, wherein the neural network model includes the feature extraction module; The facial features and the background features are spliced ​​together, and the spliced ​​features are input into a classification unit, and the classification unit outputs the emotion of the object, wherein the neural network model includes the classification unit.

3. The method according to claim 2, characterized in that The step of inputting the background video into a feature extraction module, and extracting features from the background video by the feature extraction module to obtain the background features, comprises: Inputting the background video into a second three-dimensional convolution unit, and extracting features from the background video by the second three-dimensional convolution unit to obtain initial background features, wherein the feature extraction module includes the second three-dimensional convolution unit; Input the initial background features into an attention mechanism unit, and the attention mechanism unit determines the weight of the initial background features, wherein the feature extraction module includes the attention mechanism unit; The weight of the initial background feature is multiplied by the initial background feature to obtain the background feature.

4. The method according to claim 2, characterized in that: The step of inputting the facial video into a first three-dimensional convolution unit, and extracting features from the facial video by the first three-dimensional convolution unit to obtain the facial features, comprises: Convolving the facial video using a spatial convolution layer to obtain spatial features of the facial video, wherein the first three-dimensional convolution unit includes the spatial convolution layer; A temporal convolution layer is used to convolve the spatial features of the facial video to obtain the facial features, wherein the first three-dimensional convolution unit includes the temporal convolution layer.

5. The method according to claim 2, characterized in that: The step of splicing the facial features and the background features, and inputting the spliced ​​features into a classification unit, wherein the classification unit outputs the emotion of the object, comprises: The facial features are sampled using a first average pooling layer, and the background features are sampled using a second average pooling layer; splicing the sampled facial features and the sampled background features to obtain the spliced ​​features; The concatenated features are input into the classification unit, and the classification unit outputs the emotion of the object.

6. The method according to any one of claims 1 to 5, characterized in that The neural network model is obtained in the following way: Acquire a training sample, wherein the training sample includes a sample video and an emotion type corresponding to the sample video; The sample video and the corresponding emotion type are input into the original neural network model, and the original neural network model extracts features from the sample video and determines the initial emotion type based on the extracted features. According to the initial emotion type and the emotion type corresponding to the sample video, the parameters included in the original neural network model are adjusted to obtain the neural network model.

7. A video emotion recognition device, characterized in that: include: An acquisition module, used to acquire the video to be identified; A recognition module is used to input the video to be recognized into a pre-trained neural network model, and the neural network model outputs the emotion of the object in the video to be recognized, wherein the neural network model extracts features of the facial video in the video to be recognized, and extracts features of the background video other than the facial video in the video to be recognized, and determines the emotion of the object based on the facial features extracted from the facial video and the background features extracted from the background video.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the video emotion recognition method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the video emotion recognition method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: The computer instructions are executed by the processor to implement the video emotion recognition method described in any one of claims 1 to 6.