Emotion prediction method and device, equipment, storage medium and computer program product

Abstract

The invention provides an emotion prediction method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: performing emotion scoring on a first video frame in a first video to obtain a first score of the first video frame; screening out a second video frame from the first video frame based on the first score; extracting a first emotion feature of the second video frame; and performing emotion prediction on the second video based on the first emotion feature to obtain a first probability of the second video, the first probability being used for determining the emotion of the second video. According to the invention, the emotion of the second video can be accurately predicted based on the first video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an emotion prediction method, apparatus, device, storage medium, and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, sentiment analysis, as an important branch of artificial intelligence and natural language processing, has gradually become a research hotspot. It is committed to extracting and modeling emotional information in data so that machines can understand, analyze and even predict human emotional states. In the field of video sentiment analysis, video, as a form of multimodal data, contains a rich source of emotional information. The comprehensive analysis of these modalities can not only improve the accuracy of emotion recognition, but also play an important role in personalized recommendations, user experience optimization and emotional interaction. Summary of the Invention

[0003] Embodiments of the present application provide an emotion prediction method, apparatus, device, storage medium, and computer program product that can accurately predict the emotion of a second video based on a first video.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a method for predicting emotion, the method comprising:

[0006] Performing an emotion score on a first video frame in a first video to obtain a first score for the first video frame;

[0007] Based on the first score, filtering out a second video frame from the first video frame;

[0008] extracting a first emotion feature of the second video frame;

[0009] Based on the first emotion feature, emotion prediction is performed on the second video to obtain a first probability of the second video, where the first probability is used to determine the emotion of the second video.

[0010] The present invention provides an emotion prediction device, comprising:

[0011] An emotion scoring module is used to perform an emotion scoring on a first video frame in a first video to obtain a first score of the first video frame;

[0012] a video frame screening module, configured to screen out a second video frame from the first video frame based on the first score;

[0013] A feature extraction module, configured to extract a first emotion feature of the second video frame;

[0014] The emotion prediction module is used to perform emotion prediction on the second video based on the first emotion feature to obtain a first probability of the second video, where the first probability is used to determine the emotion of the second video.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is used to implement the emotion prediction method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the emotion prediction method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the emotion prediction method provided by the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] An emotion score is performed on the first video frame in the first video to obtain a first score for the first video frame, and then based on the first score, the second video frame is screened out from the first video frame, thereby realizing the screening of the second video frame in the first video. On this basis, the first emotion feature of the second video frame is extracted, and based on the first emotion feature, the emotion prediction of the second video is performed to obtain a first probability for determining the emotion of the second video. In this way, an accurate prediction of the emotion of the second video based on the first video is realized. In addition, compared with the scheme of predicting the emotion of the second video through the first video frame, the scheme of predicting the emotion of the second video through the screened second video frame can reduce the computing resources used for prediction, thereby realizing efficient prediction of the emotion of the second video based on the first video. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the structure of the emotion prediction system architecture provided by the embodiment of the present application;

[0023] Figure 2 Schematic diagram of the structure of an electronic device for emotion prediction provided by an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of the first flow chart of the emotion prediction method provided in an embodiment of the present application;

[0025] Figure 4 This is a second flow chart of the emotion prediction method provided in an embodiment of the present application;

[0026] Figure 5 3 is a schematic diagram of a third flow chart of the emotion prediction method provided in an embodiment of the present application;

[0027] Figure 6 4 is a schematic diagram of a fourth flow chart of the emotion prediction method provided in an embodiment of the present application;

[0028] Figure 7 This is a fifth flow chart of the emotion prediction method provided in the embodiment of the present application;

[0029] Figure 8 is a sixth flow chart of the emotion prediction method provided in an embodiment of the present application;

[0030] Figure 9 This is a seventh flow chart of the emotion prediction method provided in an embodiment of the present application;

[0031] Figure 10 This is the first principle of the emotion prediction method provided by the embodiment of the present application;

[0032] Figure 11 This is the second principle of the emotion prediction method provided in the embodiment of this application.

[0033] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0039] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Video frames: In the context of multimodal video analysis, video frames are comprehensive information carriers that integrate rich information from visual, audio, and textual modalities. The visual modality provides visual elements such as objects, scenes, colors, and textures in the picture, which is the basis for understanding the video content. The audio modality enhances the understanding of the scene atmosphere and character emotions through sound signals synchronized with the picture. Textual modalities, such as subtitles or comments, further supplement semantic information and help to more accurately grasp the meaning of the video.

[0042] 2) Frame rate (FPS): The frame rate refers to the number of video frames displayed per second. For example, 30 FPS means 30 frames per second. The frame rate determines the smoothness of the video. A higher frame rate (such as 60 FPS or higher) makes the video appear smoother, while a lower frame rate (such as 10 FPS) may cause the video to appear jerky.

[0043] 3) Color Histogram: A color histogram is a statistical tool used to represent the distribution of the number of pixels of each color in an image. It is usually a one-dimensional or three-dimensional array that records the frequency of each color in the image. It can provide global information about the color distribution in the image without considering the spatial position of the pixels.

[0044] 4) Motion Intensity: Motion intensity refers to the degree of pixel change between video frames. It is often used to measure the amount of motion in a video and can be determined by calculating pixel differences or optical flow between adjacent frames. Motion intensity can be used for motion detection, motion segmentation, and motion estimation in video analysis.

[0045] 5) Neural Network Model: A neural network model is a computational model inspired by biological neural networks. It consists of a large number of interconnected nodes (neurons) and learns the patterns and features of input data to perform various tasks, such as classification, regression, and feature extraction. Neural network models are widely used in machine learning and artificial intelligence, capable of handling complex pattern recognition and prediction tasks. Common neural network models include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers.

[0046] 6) Loss Function: The loss function is a function that measures the difference between the model's predicted value and the true value. It is a key part of training a neural network. By minimizing the loss function, the model can gradually optimize its parameters to better fit the training data.

[0047] Before explaining the emotion prediction method provided in the embodiments of the present application, the method in the related art is first described.

[0048] In related technologies, video data includes information of visual modalities, audio modalities and text modalities, and the information of different modalities is highly heterogeneous. Among them, the information of visual modalities is mainly pixel features, the information of audio modalities is mainly frequency domain features, and the information of text modalities is mainly semantic features of natural language. This heterogeneity between modalities brings difficulties to the fusion modeling of information between modalities; in addition, related technologies generally process the entire frame of the video, but since there is usually a large amount of redundant information in the video content, and emotion-related content is often concentrated in specific time points or key frames, not only is the computational cost high, but the efficiency and accuracy of sentiment analysis are also low.

[0049] Based on at least one of the above-mentioned problems existing in the methods in the related art, the embodiments of the present application provide an emotion prediction method, device, electronic device, computer-readable storage medium and computer program product, which can accurately predict the emotion of the second video based on the first video.

[0050] See also Figure 1 , Figure 1 It is a structural diagram of the emotion prediction system architecture provided in the embodiment of the present application. In the emotion prediction system 10 provided in the embodiment of the present application, in order to support an emotion prediction application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0051] Terminal 400 may be used to obtain an emotion prediction request carrying the first video.

[0052] In some embodiments, an emotion prediction plug-in may be implanted in the client running in the terminal 400 to implement the emotion prediction method locally on the client. For example, the terminal 400 calls the emotion prediction plug-in to implement the emotion prediction method, performs an emotion score on the first video frame in the first video, obtains a first score for the first video frame, filters out the second video frame from the first video frame based on the first score, extracts the first emotion feature of the second video frame, and performs emotion prediction on the second video based on the first emotion feature to obtain a first probability for determining the emotion of the second video.

[0053] In some embodiments, after the terminal 400 obtains the emotion prediction request carrying the first video, it calls the emotion prediction interface of the server 200 (which can be provided in the form of a cloud service). The server 200 implements the emotion prediction method through the emotion prediction plug-in, performs an emotion prediction on the first video frame in the first video, obtains a first score for the first video frame, filters out the second video frame from the first video frame based on the first score, extracts the first emotion feature of the second video frame, and performs emotion prediction on the second video based on the first emotion feature to obtain a first probability for determining the emotion of the second video, and returns the first probability to the terminal 400.

[0054] The emotion prediction method provided by the embodiments of the present application can be applied to various scenarios that require emotion prediction. For example, in an application scenario of user experience optimization, the robot terminal 400 can collect a first video of the user who is interacting with it, and the robot terminal 400 calls the emotion prediction plug-in to implement the emotion prediction method to obtain a first probability for determining the emotion of the user who is interacting with it at a future moment; or, after the robot terminal 400 obtains the emotion prediction request carrying the first video, it calls the emotion prediction interface of the server 200, and the server 200 implements the emotion prediction method through the emotion prediction plug-in to obtain a first probability for determining the emotion of the user who is interacting with it at a future moment, and returns the first probability to the terminal 400.

[0055] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0056] The terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, intelligent voice interaction device, smart home appliance, vehicle-mounted terminal, aircraft, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0057] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for emotion prediction provided by an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 The terminal 400 or server 200 in the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0058] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0059] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0060] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard drive, an optical drive, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0061] The memory 550 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0062] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0063] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0064] A network communication module 552 for reaching other computing devices via one or more (wired or wireless) network interfaces 520 , exemplary network interfaces 520 including Bluetooth, WiFi, and USB;

[0065] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0066] The input processing module 554 is configured to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

[0067] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 Emotion prediction device 555 stored in memory 550 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: emotion scoring module 5551, video frame screening module 5552, feature extraction module 5553, and emotion prediction module 5554. These modules are logically connected and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0068] In some embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the emotion prediction method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0069] As mentioned above, the electronic device that implements the emotion prediction method of the embodiment of the present application can be a terminal, a server, or a combination of the two, so the execution subject of each step will not be repeated below, see Figure 3 , Figure 3 This is a first flow chart of the emotion prediction method provided by the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0070] In step 101, a first video frame in a first video is subjected to an emotion score to obtain a first score of the first video frame.

[0071] Among them, the first video is an existing video, and the first video frame can be obtained by one of the following methods: filtering out the first video frame from the first video based on a preset filtering rule; randomly extracting a preset number of first video frames from the first video; directly determining the video frame of the first video as the first video frame.

[0072] In some embodiments, the preset filtering rules include rules for filtering based on the motion intensity of the video frame and rules for filtering based on the difference between the color histogram vectors between frames. "Filtering out the first video frame from the first video based on the preset filtering rules" can be achieved by the following processing: for each video frame included in the first video, determine the motion intensity and the first color histogram vector of the video frame, and determine the second color histogram vector of the video frame adjacent to the video frame in the first video; for each video frame included in the first video, if the motion intensity is greater than a first set threshold, or the distance between the first color histogram vector and the second color histogram vector is greater than a second set threshold, determine the video frame as the first video frame.

[0073] As an example, when the preset filtering rules include rules for filtering based on the motion intensity of video frames, the motion intensity of the video frame is determined for each video frame included in the first video. When the motion intensity of the video frame is greater than a first set threshold, the video frame is determined as the first video frame. Methods for determining the motion intensity of the video frame include optical flow method, pixel difference method, and background difference method, etc. The first set threshold and the second set threshold can be thresholds preset by engineering and technical personnel.

[0074] Taking the optical flow method as an example, the motion intensity M(i) of the i-th video frame of the first video is determined by the motion intensity determination formula (1.1). When the motion intensity M(i) of the i-th video frame is greater than the first set threshold, it indicates that there is a significant motion change in the i-th video frame, and the i-th video frame is determined as the first video frame. The motion intensity determination formula (1.1) is as follows:

[0075]

[0076] Where M(i) is the motion intensity of the i-th video frame of the first video, P is the set of pixels in the i-th video frame, |P| is the number of pixels included in the i-th video frame, and V(p) is the optical flow vector of any pixel point p in the i-th video frame. The optical flow vector is used to measure the degree of change of the video frame.

[0077] When the preset filtering rules include rules for filtering based on the difference between color histogram vectors between frames, for each video frame included in the first video, the first color histogram vector of the video frame and the second color histogram vector of the previous video frame adjacent to it are determined, and when the vector distance between the first color histogram vector and the second color histogram vector is greater than a second set threshold, the video frame is determined as the first video frame, wherein the vector distance between the first color histogram vector and the second color histogram vector may include cosine distance, Euclidean distance, Manhattan distance, etc.

[0078] Taking the Euclidean distance as an example, based on the Euclidean distance determination formula (1.2), the Euclidean distance between the first color histogram vector of the i-th video frame of the first video and the second color histogram vector of the i-1-th video frame of the first video (the previous video frame adjacent to the i-th video frame) is determined. When the Euclidean distance between the first color histogram vector and the second color histogram vector is greater than the second set threshold, it indicates that the i-th video frame of the first video is a scene switching point in the first video, and the i-th video frame of the first video is determined as the first video frame. The Euclidean distance determination formula (1.2) is as follows:

[0079] H(i)=||H(f i )-H(f i-1)||(1.2)

[0080] Among them, H(f i ) is the first color histogram vector of the i-th video frame of the first video, H(f i-1 ) is the second color histogram vector of the i-1th video frame of the first video, and H(i) is the Euclidean distance between the first color histogram vector and the second color histogram vector.

[0081] In an embodiment of the present application, by determining the motion intensity and color histogram vector of the video frame, and combining the first preset threshold and the second preset threshold, effective screening of the first video frame in the first video is achieved, and video frames in the first video with significant motion changes or scene switching points can be accurately identified.

[0082] As an example, emotion scoring of the first video frame refers to scoring the degree to which the first video frame is relevant to the emotion. The first score of the first video frame is used to characterize the degree to which the first video frame is relevant to the emotion. The higher the first score of the first video frame, the higher the degree to which the first video frame is relevant to the emotion, and vice versa. For example, the first video frame A includes an image of a smiling child, and the first video frame B includes an image of a woman drinking water. The degree to which the first video frame A is relevant to the emotion is higher than the degree to which the first video frame B is relevant to the emotion. Then the first score of the first video frame A is higher than the first score of the first video frame B.

[0083] In some embodiments, see Figure 4 , Figure 4 This is a second flow chart of the emotion prediction method provided in the embodiment of the present application. Figure 3 The illustrated step 101 can be implemented by following the steps 1011 to 1012 , which are described in detail below.

[0084] In step 1011, for each first modal information included in the first video frame, the first modal information is encoded to obtain encoding features of the first modal information, and based on the encoding features, the first modal information is emotionally scored to obtain a second score of the first modal information.

[0085] The first video frame may include first modality information of a visual modality and first modality information of an audio modality.

[0086] As an example, in a case where the first video frame includes first modal information of a visual modality, image encoding is performed on the first modal information of the visual modality to obtain encoding features of the first modal information of the visual modality, and based on the encoding features of the first modal information of the visual modality, an emotion score is performed on the first modal information of the visual modality to obtain a second score of the first modal information of the visual modality, wherein image encoding of the first modal information of the visual modality is implemented by a neural network for image encoding, which may be a deep convolutional neural network (ResNet-101), a vision transformer (ViT, VisionTransformer) and a residual network (ResNet, Residual Network), which are not limited here.

[0087] In a case where the first video frame includes first modal information of an audio modality, audio encoding is performed on the first modal information of the audio modality to obtain encoding features of the first modal information of the audio modality, and based on the encoding features of the first modal information of the audio modality, an emotion score is performed on the first modal information of the audio modality to obtain a second score of the first modal information of the audio modality, wherein audio encoding of the first modal information of the audio modality is implemented by a neural network for audio encoding, which may be a waveform encoder (Wav2Vec2.0 encoder).

[0088] In some embodiments, see Figure 5 , Figure 5 This is a third flow chart of the emotion prediction method provided in the embodiment of the present application. Figure 4 The step 1011 shown as "performing a sentiment score on the first modal information based on the encoding features to obtain a second score of the first modal information" can be implemented by following the steps 10111 to 10112, which are described in detail below.

[0089] In step 10111, sentiment classification processing is performed on the encoding features of the first modal information to obtain multiple first probabilities corresponding to the first modal information.

[0090] The first probability is used to represent the possibility that the first modal information belongs to a first emotion category. The first emotion category is any one of multiple emotion categories, and the first emotion category includes positive emotion, neutral emotion, and negative emotion.

[0091] As an example, a linear transformation is performed on the coding features of the first modal information to obtain the transformed coding features, the transformed coding features are mapped to obtain the mapped coding features, and the mapped coding features are normalized to obtain the emotion probability vector corresponding to the first modal information. The emotion probability vector corresponding to the first modal information includes multiple first probabilities, wherein the mapping process of the transformed coding features can be implemented based on an activation function, and the activation function can include a nonlinear activation function, a logarithmic activation function, and a hyperbolic tangent activation function, etc., which are not limited here.

[0092] In order to facilitate the explanation of the scheme, the above “performing sentiment classification processing on the encoding features of the first modal information to obtain multiple first probabilities corresponding to the first modal information” is implemented through a fully connected layer. In some embodiments, “performing sentiment classification processing on the encoding features of the first modal information to obtain multiple first probabilities corresponding to the first modal information” can also be implemented based on N cascaded fully connected layers. “Based on N cascaded fully connected layers, performing sentiment classification processing on the encoding features of the first modal information to obtain multiple first probabilities corresponding to the first modal information” can be implemented by the following steps: Based on In the first fully connected layer, the encoded features are fully connected to obtain the output features of the first fully connected layer; based on the nth fully connected layer, the output features of the n-1th fully connected layer are fully connected to obtain the output features of the nth fully connected layer; based on the Nth fully connected layer, the output features of the N-1th fully connected layer are fully connected to obtain the output features of the Nth fully connected layer, and the output features of the Nth fully connected layer are normalized to obtain multiple first probabilities corresponding to the first modal information, wherein n and N are positive integers, 1<n<N, and n increases successively.

[0093] In step 10112, a preset operation is performed on the multiple first probabilities to obtain a second score of the first modal information.

[0094] The multiple first probabilities include a first sub-probability belonging to a positive emotion, a second sub-probability belonging to a neutral emotion, and a third sub-probability belonging to a negative emotion.

[0095] For example, the emotion probability vector corresponding to the first modal information can be represented as [0.5, 0.2, 0.3]. The three elements in the emotion probability vector are three first probabilities. The three first probabilities are: the first sub-probability belonging to positive emotion is 0.5, the second sub-probability belonging to neutral emotion is 0.2, and the third sub-probability belonging to negative emotion is 0.3.

[0096] In some embodiments, the first probability includes a first sub-probability belonging to a positive emotion, a second sub-probability belonging to a neutral emotion, and a third sub-probability belonging to a negative emotion. Figure 5Step 10112 shown can be implemented by the following processing: determining the difference between the first sub-probability and the third sub-probability, and determining the maximum probability from the first sub-probability, the second sub-probability and the third sub-probability; and determining the product of the difference and the maximum probability as the second score.

[0097] As an example, the difference between the first sub-probability and the third sub-probability, and the maximum probability among the first sub-probability, the second sub-probability, and the third sub-probability are determined, and based on the preset operation formula (1.3), the product of the difference and the maximum probability is determined as the second score of the first modal information, where the preset operation formula (1.3) is as follows:

[0098] S=max(P positive ,P neutral ,P negative )*(P positive -P negative ) (1.3)

[0099] Among them, S is the second score, P positive is the first sub-probability of positive sentiment, P neutral is the second sub-probability of belonging to neutral emotion, P negative is the third sub-probability of negative sentiment, max(P positive ,P neuyral ,P negative ) is the maximum probability among the first sub-probability, the second sub-probability and the third sub-probability.

[0100] In an embodiment of the present application, by calculating the difference between the first sub-probability belonging to positive emotion and the third sub-probability belonging to negative emotion, the polarity of the emotion of the first modal information is quantified, and the product of the first sub-probability and the maximum probability is determined as the second score, thereby providing a comprehensive quantitative indicator for the emotional tendency of the first modal information, improving the interpretability of the emotion prediction, and achieving accurate scoring of the second score of the first modal information.

[0101] In step 1012, based on preset weight coefficients of the multiple first modal information, the second scores of the multiple first modal information are fused to obtain the first score.

[0102] The preset weight coefficient may be a weight coefficient preset by engineering technicians.

[0103] As an example, when the first video frame includes the first modal information of the visual modality and the first modal information of the audio modality, the second score of the first modal information of the visual modality and the second score of the first modal information of the audio modality are weighted and summed based on the preset weight coefficient of the first modal information of the visual modality and the preset weight coefficient of the first modal information of the audio modality by formula (1.4), to obtain the first score of the first video frame. Formula (1.4) is as follows:

[0104] S total (i) = w img ·S img (i)+w audio ·S audio (i)

[0105] w img +w audio =1 (1.4)

[0106] Among them, S total (i) is the first score of the i-th first video frame, S img (i) is the second score of the first modal information of the visual modality, w img is the preset weight coefficient of the first modal information of the visual modality, S audio (i) is the second score of the first modal information of the audio modality, w audio is a preset weight coefficient of the first modal information of the audio modality, and the sum of the preset weight coefficients of multiple first modal information is 1.

[0107] In the implementation of this application, multiple first modal information are separately encoded and sentimentally scored, and the second scores of multiple first modal information are weightedly fused based on preset weight coefficients, thereby fully utilizing the complementarity between multiple first modal information of the first video frame, and being able to more comprehensively and accurately reflect the emotion-related first scores of the first video frame, thereby improving the accuracy and reliability of the sentiment score.

[0108] In step 102, a second video frame is selected from the first video frame based on the first score.

[0109] As an example, the first video can be divided into multiple video segments, and based on the first score, the third video frame of each video segment is determined, and the first video frame with the largest first score, the first video frame with the smallest first score, and the third video frame of each video segment are determined as the second video frame.

[0110] It is also possible to filter out the fourth video frame whose first score is greater than the preset score threshold from the first video frame based on the preset score threshold, and determine the first video frame with the largest first score, the first video frame with the smallest first score, and the fourth video frame as the second video frame, which is not limited here.

[0111] In some embodiments, Figure 3 Step 102 shown can be implemented by the following processing: dividing the first video into multiple video segments; for each video segment, clustering the first video frames included in the video segment based on the first scores of the first video frames included in the video segment to obtain cluster clusters, and determining the third video frame from the cluster clusters; determining the first video frame with the largest first score, the first video frame with the smallest first score, and the third video frame of each video segment as the second video frame.

[0112] As an example, based on the frame rate of the first video, the first video is divided into multiple video segments. The frame rate of the first video is positively correlated with the minimum number of second video frames contained in each video segment. For example, when the frame rate of the first video is 30FPS, the minimum number of second video frames contained in each video segment is 1. When the frame rate of the first video is 60FPS, the minimum number of second video frames contained in each video segment is 2, and so on.

[0113] The following processing is performed on each video clip: based on the first score of the first video frame included in the video clip, the first video frames included in the video clip are clustered; when the difference between the first scores of any two video frames in each cluster is less than a preset difference threshold, the clustering ends and at least one cluster is obtained; when the number of clusters is 1, the lowest number of any first video frames in the cluster is determined as the third video frame; when the number of clusters is greater than 1, on the premise of ensuring that the number of third video frames is greater than or equal to the lowest number, any first video frame in each cluster is determined as the third video frame.

[0114] The first video frame with the largest first score, the first video frame with the smallest first score, and the third video frame of each video segment are determined as second video frames. The second video frames can be represented as a video frame set K. in, is the i-th second video frame in the video frame set K, where i and I are positive integers, 1<i<I.

[0115] For example, when the length of the first video is 10 seconds and the frame rate is 30 FPS, the first video is divided into 10 video segments, each video segment is 1 second. Taking the first video segment (0-1 second) as an example, the first scores of the 10 first video frames included in the first video frame are distributed as follows: [0.2, 0.21, 0.22, 0.5, 0.51, 0.52, 0.7, 0.71, 0.72, 0.9]. Based on the preset difference threshold of 0.1, the 10 first video frames included in the first video frame are clustered to obtain 4 cluster clusters: [0.2, 0.21, 0.22], [0. .5, 0.51, 0.52], [0.7, 0.71, 0.72] and [0.9], select the video frame closest to the cluster center from each cluster as the third video frame (or select any video frame, which is not limited here), that is, the video frames with first scores of 0.21, 0.51, 0.71, and 0.9 respectively are determined as the third video frames, and the processing shown above is performed on the remaining 9 video clips of the first video in turn, and finally the first video frame with the largest first score, the first video frame with the smallest first score, and the third video frames corresponding to the 10 video clips are determined as the second video frames.

[0116] In an embodiment of the present application, clustering is performed based on the first score of the first video frame to obtain at least one clustering cluster, and the third video frame is determined based on the number of clustering clusters and the minimum number of third video frames. Finally, the first video frames with the largest and smallest first scores and the third video frame of each video clip are determined as the second video frames. This not only retains the two video frames with the largest difference in first scores in the first video frames, but also extracts the most representative third video frame in each video clip through segmented clustering, thereby more comprehensively reflecting the overall emotional changes of the first video and improving the accuracy and efficiency of subsequent emotional predictions.

[0117] In step 103, a first emotion feature of the second video frame is extracted.

[0118] In some embodiments, see Figure 6 , Figure 6 This is a fourth flow chart of the emotion prediction method provided in the embodiment of the present application. Figure 3 The illustrated step 103 can be implemented by following the steps 1031 to 1032 , which are described in detail below.

[0119] In step 1031, for each second modal information included in the second video frame, a second emotional feature of the second modal information is extracted.

[0120] As an example, the second video frame may include second modal information of visual modality, second modal information of audio modality and second modal information of text modality, and the second emotional features of the second modal information of visual modality, the second emotional features of the second modal information of audio modality and the second emotional features of the second modal information of text modality of the second video frame are extracted respectively, wherein the second modal information of text modality may be obtained by text extraction of the second modal information of visual modality, or may be obtained by speech recognition of the second modal information of audio modality, which is not limited here.

[0121] For example, for the second video frame set The i-th second video frame in Extract the second video frame The second emotional feature of each second modal information can be represented as V m (i), where m is any modality of the second video frame.

[0122] In the case where the second video frame includes the second modal information of the visual modality, that is, m is the visual modality, the second modal information of the visual modality is subjected to emotional feature extraction by a neural network for extracting emotional features of the visual modality, and the second emotional features of the second modal information of the visual modality are obtained, which can be represented as V img (i) = f vision (f i ), where f i The second video frame The second modality information of the visual modality, V m (i) can be represented as V img (i), V img (i) is the second emotional feature of the second modal information of the visual modality, f vision The neural network used to extract emotional features of the visual modality can be a vision transformer (ViT), a residual network (ResNet), and a CLIP model, which are not limited here.

[0123] In the case where the second video frame includes the second modal information of the audio modality, that is, m is the audio modality, the second modal information of the audio modality is subjected to emotional feature extraction by a neural network for extracting emotional features of the audio modality, and the second emotional features of the second modal information of the audio modality are obtained, which can be represented as V audio (i) = g audio (A i ), where A i The second video frame The second modal information of the audio mode, V m(i) can be represented as V audio (i), V audio (i) is the second emotional feature of the second modal information of the audio modality, g audio A neural network for extracting emotional features from audio modalities, including but not limited to the Wav2Vec2.0 model.

[0124] In the case where the second video frame includes the second modal information of the text modality, that is, m is the text modality, the second modal information of the text modality is subjected to emotional feature extraction by a neural network for extracting emotional features of the text modality, and the second emotional features of the second modal information of the text modality are obtained, which can be represented as V text (i) = h text (T i ), where T i The second video frame The second modal information of the text modal, V m (i) can be represented as V text (i), V text (i) is the second emotional feature of the second modal information of the text modality, h text Neural networks for extracting sentiment features from text modalities, including but not limited to the dual-headed attention model (BERT model).

[0125] In step 1032, the second emotional features of multiple second modal information are fused to obtain the first emotional features of the second video frame.

[0126] In some embodiments, see Figure 7 , Figure 7 This is a fifth flow chart of the emotion prediction method provided in the embodiment of the present application. Figure 6 The illustrated step 1032 can be implemented by following the steps 10321 to 10323, which are described in detail below.

[0127] In step 10321, attention processing is performed on the second emotional features of multiple second modal information to obtain a weight coefficient for each second modal information.

[0128] As an example, for the second video frame set The i-th second video frame in Based on formula (1.5), for the second video frame The second emotional feature V of each second modal information m (i) Perform attention processing to obtain the weight coefficient of each second modal information. Formula (1.5) is as follows:

[0129]

[0130] Among them, m is any mode among visual mode (img), audio mode (audio) and text mode (text), α m (i) is the weight coefficient of the second modal information of mode m, V m (i) The second emotional feature of the second modal information of modality m, W m is the weight coefficient of mode m, b m is the bias term coefficient of modality m, exp is the exponential operation function, and m′ is the modality index used to traverse the visual modality, audio modality, and text modality. For example, when m is the visual modality, α m (i) is the weight coefficient of the second modality information of the visual modality, b m is the bias coefficient of the visual modality, V m (i) is the second emotional feature V of the second modal information of the visual modality im g (i).

[0131] In step 10322, based on the weight coefficient of each second modal information, the second emotional features of multiple second modal information are fused to obtain the third emotional feature.

[0132] Continuing with the above example, through formula (1.6), based on the weight coefficients corresponding to the second modal information of the visual modality, the second modal information of the audio modality, and the second modal information of the text modality, the second emotional features of the above three modalities are integrated to obtain the third emotional feature. Formula (1.6) is as follows:

[0133] V modality (i)=∑ m∈{img,audio,text} α m (i) V m (i) (1.6)

[0134] Among them, V modality (i) is the third emotional feature, α m (i) is the weight coefficient of the second modal information of mode m, V m (i) is the second emotional feature of the second modal information of modality m, and m is any modality among visual modality (img), audio modality (audio) and text modality (text).

[0135] In step 10323, feature extraction is performed on the third emotional feature to obtain the first emotional feature.

[0136] Continuing from the above example, the third emotional feature V is analyzed by using a neural network that is used to capture the deep interactive relationship between multimodal information. modality (i) Perform feature extraction to obtain the second video The first emotional feature of the image is obtained, wherein the neural network used to capture the deep interactive relationship between multimodal information can be a multi-layer Transformer model, a GPT3 model, etc., which is not limited here.

[0137] Taking the multi-layer Transformer model as an example, the third emotional feature V modality (i) Perform feature extraction to obtain the second video The first emotional feature can be represented as V emotion (i)=Transformer(V modality (i)), where V emotion (i) is the second video The first emotional feature of , Transformer is a multi-layer Transformer model.

[0138] In an embodiment of the present application, by performing attention processing, weighted fusion and feature extraction on the second emotional features of multiple second modal information, while fully considering the deep interactive relationship between multimodal information, the first emotional features of the second video frame are accurately extracted.

[0139] In some embodiments, Figure 6 The shown step 1032 can also be implemented by the following processing: for each second modal information included in the second video frame, the second emotional feature of the second modal information is fully connected to obtain the second emotional feature after full connection processing; the second emotional features after full connection processing of multiple second modal information are integrated to obtain the first emotional feature.

[0140] Continuing with the above example, for the second video frame The second emotional features of each second modal information are fully connected to obtain the second emotional features after the full connection processing, the second emotional features after the full connection processing of multiple second modal information are spliced ​​to obtain the spliced ​​features, and the spliced ​​features are further fused through the feedforward neural network for feature fusion to obtain the second video frame The first emotional characteristic can be represented as Among them, V fusion (i) is the second video The first emotional characteristic, is a feedforward neural network, V concat (i) is the splicing feature, W fusion is the trainable weight coefficient of the feedforward neural network, b fusion is the trainable bias coefficient of the feedforward neural network.

[0141] In step 104, emotion prediction is performed on the second video based on the first emotion feature to obtain a first probability of the second video.

[0142] The second video is a continuation of the first video or a video related to the first video, such as a video to be generated after the first video, and the first probability is used to determine the emotion of the second video.

[0143] In some embodiments, see Figure 8 , Figure 8 This is a sixth flow chart of the emotion prediction method provided in the embodiment of the present application. Figure 3 The illustrated step 104 can be implemented by following the steps 1041 to 1042 , which are described in detail below.

[0144] In step 1041 , a fourth emotional feature of the second video is predicted based on the first emotional feature.

[0145] As an example, the first emotional feature of the second video frame is predicted by the neural network for emotion prediction to obtain the fourth emotional feature of the second video, which can be represented as H t =TransformerAttention(V emotion (1),…,V emotion (i),…,V emotion (I)), where H t is the fourth emotional feature of the second video, V emotion (i) is the second video frame set The i-th second video frame in The first emotional feature, TransformerAttention is a neural network used for emotion prediction, which can be a large model of the attention mechanism Transformer.

[0146] In step 1042, emotion recognition is performed on the fourth emotion feature to obtain a first probability of the second video.

[0147] As an example, by using formula (1.7), emotion recognition is performed on the fourth emotion feature to obtain the emotion probability vector of the second video, and based on the emotion probability vector of the second video, the first probability of the second video is determined. Formula (1.7) is as follows:

[0148] Y t =softmax(W cls ·H t +b cls ) (1.7)

[0149] Among them, Y t is the emotion probability vector of the second video, H tis the fourth emotional feature of the second video, W cls is the weight coefficient for linear transformation and mapping of the fourth emotional feature, b cls is the bias term coefficient for performing linear transformation and mapping processing on the fourth emotional feature, and softmax is the operation identifier for normalization processing.

[0150] In some embodiments, Figure 8 Step 1042 shown can be implemented by the following processing: performing linear transformation on the fourth emotional feature to obtain the transformed fourth emotional feature; performing mapping processing on the transformed fourth emotional feature to obtain the mapped fourth emotional feature; performing normalization processing on the mapped fourth emotional feature to obtain the first probability of the second video.

[0151] The emotion probability vector of the second video includes multiple second probabilities, and the second probabilities are used to represent the possibility that the second video belongs to the second emotion category.

[0152] As an example, the mapped fourth emotion feature is normalized to obtain the emotion probability vector of the second video, and the emotion corresponding to the maximum probability among the multiple second probabilities included in the emotion probability vector of the second video is determined as the emotion of the second video. The specific processing of determining the emotion probability vector of the second video can be referred to step 10111 above and will not be repeated here.

[0153] Multiple second probabilities may include the probability that the emotion of the second video belongs to any emotion category. For example, multiple second probabilities may include the probability that the emotion of the second video belongs to a positive emotion, the probability that it belongs to a neutral emotion, and the probability that it belongs to a negative emotion. Multiple second probabilities may also include the probability that the emotion of the second video is any emotion type. For example, the probability distribution of the emotion of the second video may include the probability that the emotion of the second video is happy, sad, angry, etc.

[0154] In an embodiment of the present application, emotion prediction is performed based on the first emotion feature to obtain the fourth emotion feature of the second video, and emotion recognition is performed on the fourth emotion feature of the second video, thereby achieving accurate prediction of the emotion of the second video. At the same time, it supports probability calculation of multiple emotion categories (such as positive, neutral, negative, and more detailed emotions such as happiness, sadness, anger, etc.), greatly improving the flexibility and accuracy of emotion prediction.

[0155] In some embodiments, the emotion prediction method of the embodiment of the present application is implemented by a neural network model, see Figure 9 , Figure 9 This is the seventh flow chart of the emotion prediction method provided in the embodiment of the present application. The training process of the neural network model can be implemented by following steps 201 to 204, which are described in detail below.

[0156] In step 201, emotion prediction is performed on an existing first video frame sample using a neural network model to obtain a first emotion of a second video frame sample.

[0157] As an example, for each third modal information included in the first video frame sample, the emotional features of the third modal information are extracted, and the emotional features of multiple third modal information are fused to obtain the fused emotional features of the first video frame sample. The fused emotional features of the first video frame sample are emotionally predicted to obtain the first emotion of the second video frame sample. For the specific implementation, please refer to steps 103 and 104 shown above.

[0158] In step 202 , a first loss value is determined based on the emotion label of the second video frame sample and the first emotion.

[0159] As an example, based on the difference between the emotion label of the second video frame sample and the first emotion, a loss function is constructed, and the loss value of the loss function is determined as the first loss value, wherein the loss function may include a cross-entropy loss function (Cross-Entropy Loss), an L2 loss function, and a mean squared error loss function (MSE, Mean Squared Error), etc., which are not limited here.

[0160] In step 203, probability statistics are performed on the emotion labels of the second video frame samples to obtain the mean and variance of the emotion labels, and the first loss value is adjusted based on the mean and variance to obtain the second loss value.

[0161] As an example, the emotion label of the second video frame sample is subjected to probability statistics to obtain the Gaussian distribution of the emotion label. Based on the difference between the mean of the Gaussian distribution of the first emotion and the emotion label and the variance of the Gaussian distribution, a Gaussian loss function (1.8) is constructed. Formula (1.8) is as follows:

[0162]

[0163] Among them, L Gaussion is the loss value of the Gaussian loss function, Y i is the first emotion of the second video frame sample, μ is the mean of the Gaussian distribution of the emotion label, σ is the standard deviation of the Gaussian distribution of the emotion label, and exp is the exponential operation function.

[0164] When the first loss value is the loss value of the cross entropy loss function, the loss value of the Gaussian loss function and the first loss value are fused through formula (1.9) to obtain the second loss value. Formula (1.9) is as follows:

[0165] L cls =L ce +λLGaussion (1.9)

[0166] Among them, L cls is the second loss value, L ce is the first loss value of the cross entropy loss function, L Gaussion is the loss value of the Gaussian loss function, and λ is the preset hyperparameter.

[0167] In step 204, the neural network model is trained based on the second loss value.

[0168] As an example, by minimizing the second loss value, backpropagation is performed on the neural network model, and the parameters of the neural network model are updated to obtain a trained neural network model.

[0169] In steps 201 to 204 shown in the embodiment of the present application, the neural network model is trained by combining the traditional loss function with the Gaussian loss function, thereby enhancing the neural network model's ability to model the uncertainty of emotional labels, thereby improving the robustness and accuracy of the neural network model's emotional prediction.

[0170] In some embodiments, Figure 9 The following processing can be performed before the shown step 201: using the initial neural network model, predicting the emotional features of the existing third video frame sample to obtain the predicted emotional features of the fourth video frame sample after the third video frame sample, and extracting the real emotional features of the fourth video frame sample through the initial neural network model; determining the fourth loss value based on the predicted emotional features and the real emotional features of the fourth video frame sample; and updating the parameters of the initial neural network model based on the fourth loss value to obtain the neural network model.

[0171] As an example, “using the initial neural network model to predict the emotional features of the existing third video frame sample to obtain the second emotion of the fourth video frame sample after the third video frame sample” can refer to step 104 shown above, and “extracting the true emotional features of the fourth video frame sample” can refer to step 103 shown above, which will not be repeated here.

[0172] Based on the difference between the predicted emotional features and the actual emotional features of the fourth video frame sample, a loss function is constructed, and the loss value of the loss function is determined as the fourth loss value. By minimizing the fourth loss value, the initial neural network model is backpropagated, and the parameters of the initial neural network model are updated to obtain the neural network model.

[0173] Among them, the loss function may include a cross-entropy loss function (Cross-Entropy Loss), an L2 loss function, and a mean squared error loss function (MSE), etc., which are not limited here. When the loss function is the L2 loss function, the third loss value is determined based on the difference between the predicted emotional features and the true emotional features of the fourth video frame sample through formula (1.10). Formula (1.10) is as follows:

[0174]

[0175] Among them, L prediction is the fourth loss value, H t is the predicted emotional feature of the fourth video frame sample, V emotion is the true emotional feature of the fourth video frame sample, and t is the time when the fourth video frame sample is located.

[0176] In an embodiment of the present application, by introducing a pre-training stage, the initial neural network model is used to predict the emotional features of the third video frame sample, and the model parameters are updated based on the difference between the predicted emotional features and the actual emotional features, thereby optimizing the training process of the neural network model, helping the model to better learn the representation of emotional features, and improving the performance and convergence speed of the model in the subsequent formal training stage.

[0177] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0178] With the rapid development of artificial intelligence (AI) technology, affective computing has become a hot research topic in academia and industry. Sentiment analysis, a core area of ​​affective computing, aims to extract and model emotional information from data, enabling machines to understand, analyze, and even predict human emotions. In the field of video sentiment analysis, video, as a multimodal data form, contains a rich source of emotional information, including visual modalities (such as facial expressions and body movements), audio modalities (such as voice intonation and rhythm), and textual modalities (such as semantic content and discourse structure).

[0179] However, in related technologies, information of different modalities in video data presents a high degree of heterogeneity (the visual modality is mainly based on pixel features, the audio modality is expressed as frequency domain features, and the text modality involves the semantic features of natural language). This heterogeneity brings difficulties to the fusion modeling between modalities. In addition, video content usually contains a large amount of redundant information, and emotion-related content is often concentrated in specific time points or key frames. Related technologies generally process the entire video frame, resulting in high computational costs and low analysis efficiency.

[0180] In order to solve the above problems, the embodiment of the present application proposes an emotion prediction method, which significantly improves data processing efficiency through key frame capture technology, and uses multimodal feature fusion to improve the accuracy of emotion recognition. Specifically, the embodiment of the present application proposes a multimodal large model emotion analysis framework based on key frame capture in video, which can capture key frames in video and use a multimodal large model to perform emotion analysis on the emotions expressed in the input video, and the computational cost is low; the embodiment of the present application proposes a multimodal large model emotion analysis algorithm based on key frame capture in video, which can significantly improve data processing efficiency through key frame capture technology, use multimodal feature fusion to improve the accuracy of emotion recognition, and combine the learning and expression capabilities of the large model to accurately analyze the emotions expressed in the video.

[0181] See also Figure 10 , Figure 10 This is the first principle of the emotion prediction method provided by the embodiment of the present application. The emotion prediction method proposed in the embodiment of the present application is divided into four modules: a key frame extraction module 901, which is used to extract the key frames with the most significant emotional information in the video, screen out redundant data in the video, and thus improve the efficiency of emotion analysis; a multimodal feature extraction module 902, which is used to extract emotion-related modal features from the image modality (i.e., the visual modality mentioned above), speech modality (i.e., the audio modality mentioned above) and text modality of the key frames, and provide multi-dimensional information support for emotion analysis; a multimodal feature fusion module 903, which is used to fuse multimodal features into a unified emotion feature representation through a cross-modal attention mechanism and alignment method; a sentiment analysis module 904 based on a multimodal large model, which uses a multimodal large model to model the fused features, classify the emotions of the video, and track its dynamic changes, and finally output the emotion analysis results.

[0182] The following will describe the workflow of the emotion prediction method provided in the embodiment of the present application in combination with the above four modules.

[0183] 1. Key frame extraction module.

[0184] The key frame extraction module selects the frames with the most emotional information from the video (i.e., the first video above) by detecting significant changes and emotional features between video frames.

[0185] First, the motion intensity of each frame is calculated by the optical flow method Among them, M(i) represents the motion intensity of the i-th frame, P is the set of pixels in the frame, |P| is the total number of pixels, and V(p) is the optical flow vector of pixel p, which is used to measure the degree of change of the frame. When M(i)>T m (manually set threshold, i.e., the first set threshold mentioned above), it is considered that the frame has significant motion changes and can be used as a key frame for preliminary screening.

[0186] In addition, the scene switching in the video is determined by the color histogram difference H(i), H(i) = || H(f i )-H(f i-1 )||, where H(f i ) represents frame f i The color histogram of (that is, the color histogram vector above), ||·|| represents the L2 norm of the vector, when H(i)>T h (ie, the second threshold value set above), it is considered that the frame f i It is a scene switching point, which is also the key frame that is initially screened out.

[0187] Then, the key frames preliminarily screened out above are determined as candidate frames (i.e., the first video frame mentioned above). The image modality information (i.e., the first modality information of the visual modality mentioned above) and the speech modality information (i.e., the first modality information of the audio modality mentioned above) of the candidate frames are scored for sentiment relevance respectively through the pre-trained sentiment scoring model. The obtained sentiment score of the image modality (i.e., the second score mentioned above) is expressed as S img (i), the sentiment score of the speech modality is denoted as S audio (i), and determine the comprehensive emotion score of the candidate frame (i.e., the first score above) by the formula (1.4) shown above, where w img and w audio are the weights of image modality and speech modality respectively, satisfying w img +w audio =1.

[0188] Subsequently, the frames with the largest and smallest absolute values ​​of the comprehensive emotion scores are retained, and then the frames with similar comprehensive emotion scores in the initially screened set of candidate frames are removed (the threshold of the score difference is set to within 0.1 for frames with similar scores). Combined with the time interval constraint, it is ensured that when the video frame rate is 30FPS, one frame is retained per second to ensure that the key frames are evenly distributed on the timeline of the video.

[0189] Finally, output the keyframe set Among them, i and I are positive integers, 1<i<I, each key frame Represents the moments with the strongest emotions and the most significant changes in the video, providing accurate input data for subsequent sentiment analysis.

[0190] The sentiment scoring model is introduced below.

[0191] The sentiment scoring model consists of an image coding model, a speech coding model, and a classification head for predicting sentiment scores. The image coding model can be a CNN model (e.g., ResNet101), represented by f emotion, the speech coding model can be the Wav2Vec2.0 model, represented by g emotion The output of the sentiment scoring model is a normalized (softmax) classification output. The sentiment categories are set as positive, neutral, and negative. Different emotions correspond to different score intervals, which can quickly distinguish emotions. The output of the sentiment scoring model can be represented as P emotion , P emotion ={P positive ,P neutral ,P negative},in,[ positive is the probability that the emotion corresponding to the current frame is a positive emotion (i.e., the first sub-probability above), P neutral is the probability that the emotion corresponding to the current frame is neutral (i.e., the second sub-probability above), P negative is the probability that the emotion corresponding to the current frame is a negative emotion (i.e., the third sub-probability above).

[0192] Based on the formula (1.3) shown above, the emotion relevance score of the current frame is output, where max([ positive ,P neutral ,P negative ) represents the maximum probability of the emotion corresponding to the current frame, for example, a strong positive emotion, P positive -P negative It can not only express the polarity of emotion, but also reflect the intensity of a certain emotion. If the probability of positive emotion is the same as that of negative emotion, the emotion relevance score of the current frame is 0. The closer the emotion corresponding to the previous frame is to the positive emotion, the greater the emotion relevance score. The closer the emotion corresponding to the previous frame is to the negative emotion, the smaller the emotion relevance score.

[0193] 2. Multimodal feature extraction module.

[0194] The multimodal feature extraction module extracts the emotional features of image modality, speech modality and text modality (i.e. the second emotional feature mentioned above) from key frames, providing multi-dimensional information support for subsequent sentiment analysis.

[0195] First, for the image modality information (i.e., the second modality information of the visual modality above), a pre-trained visual model (e.g., ViT model or CLIP model) is used to extract the key frame f i The visual feature vector V img (i), V img (i) = f vision (f i ), where f vision It is a visual emotion feature extraction function that can recognize emotional features such as facial expressions and scene emotions.

[0196] Secondly, for the information of the speech mode (ie, the second modality information of the audio mode above), from the audio segment A corresponding to the frame i Extract the speech feature vector V audio (i), V audio (i) = g audio (A i ), where g audio It is a pre-trained speech emotion model (e.g., Wav2Vec model) that is used to capture the emotional information of intonation, timbre, and speaking rate.

[0197] Finally, for the text modality information (i.e. the second modality information of the text modality above), the audio is first transcribed into text T using automatic speech recognition (ASR). i , and then use the language sentiment model (for example, BERT model) to extract the text feature vector V text (i), V text (i) = h text (T i ), where h text It is the text sentiment feature extraction function.

[0198] Finally, the output multimodal feature set V(i) = {V img (i),V img (i),V text (i)}, providing joint sentiment information including visual modality, speech modality and text modality for sentiment analysis.

[0199] 3. Multimodal feature fusion module.

[0200] The multimodal feature fusion module uses a neural network to perform nonlinear fusion of visual feature vectors, speech feature vectors, and text feature vectors to generate a unified emotional feature representation. Multimodal feature fusion can use a pre-trained open source multimodal fusion neural network (such as the omnimodal understanding open source model Megrez-3B-Omni), or you can design your own neural network to achieve feature fusion.

[0201] For example, first we extract the multimodal features {V img (i),V i,g (i),V text (i)} Each modality is subjected to dimensionality reduction and nonlinear mapping through an independent fully connected layer to extract the deep features of each modality. Then, the deep features of the modalities after dimensionality reduction are concatenated into a fusion feature vector V concat (i) to capture the correlation information between modalities. Finally, the concatenated features are further fused through a fusion network (such as a multi-layer perceptron, MLP) to output the multimodal joint sentiment feature V fusion(i) (i.e., the first emotional feature above), the multimodal joint emotional feature expresses the comprehensive emotional information of the key frame fi, which can effectively combine multimodal features to provide support for subsequent emotional modeling. For the converged network, W fusion is the weight coefficient of the fusion network, b fusion is the bias coefficient of the fusion network.

[0202] 4. Sentiment analysis module based on multimodal large model.

[0203] Based on keyframe collection With the multimodal feature set V(i)={V img (i),V img (i),V text (i)}, use a multimodal large model sentiment analysis algorithm based on key frame capture in the video to output the sentiment analysis results of the video.

[0204] The following is an introduction to the multimodal large-model sentiment analysis algorithm based on key frame capture in videos.

[0205] The input of the algorithm is: a set of key frames With the multimodal feature set V(i)={V img (i),V img (i),V text (i)}, i.e., the multimodal features (image modality, speech modality, and text modality) of each keyframe.

[0206] The output of the algorithm is: emotional dynamic trend, that is, the continuous emotional fluctuation curve function of the video's emotion over time. The emotion of the video at any moment can be queried by inputting time t.

[0207] See also Figure 11 , Figure 11 This is the second principle of the emotion prediction method provided by the embodiment of the present application. Figure 11 Steps 1 to 4 shown in the figure illustrate the implementation of a multimodal large-model sentiment analysis algorithm based on key frame capture in a video.

[0208] Step 1: Multimodal deep fusion driven by a multimodal large model.

[0209] The multimodal joint sentiment feature V output by the multimodal feature fusion module can be directly fusion (i) As the output of this step, the fused modal features can also be determined through a multimodal deep fusion module driven by a multimodal large model, which is not limited here.

[0210] In order to solve the problem of different contributions of different modalities to emotional expression, a dynamic modal weight mechanism is introduced to determine the weight corresponding to each modality based on the attention mechanism through the formula (1.5) shown above, that is, the input key frame Multimodal features V img (i),V img (i),V text (i) The corresponding weights, where α m (i) is a key frame The dynamic weight of mode m, which represents the contribution ratio of each mode, W m is the weight matrix of mode m, which is used to learn the importance of modal features, b m is the bias term of mode m, used to adjust the weight distribution, V m (i) is the eigenvector of mode m.

[0211] According to the formula (1.6) shown above, based on the weight corresponding to each modality, the multimodal feature V img (i),v img (i),V text (i) Perform weighted fusion to obtain the fused modal feature V modality (i) (i.e. the third emotional feature above), and the fusion feature V modality (i) Input a pre-trained large model (e.g., multi-layer Transformer, GPT3, etc.) to further capture the deep interaction relationship between multiple modalities and obtain key frames The emotional characteristics (i.e. the first emotional characteristics above) V emotion (i), V emotion (i)=Transformer(V modality (i)).

[0212] This step utilizes the expressive power of the large multimodal model and comprehensively considers the relationships between multimodal features through a multi-head attention mechanism (multi-layer Transformer), thereby enhancing the emotional expressiveness of the emotional features of key frames and significantly improving the capture effect of complex emotional signals.

[0213] Step 2: Emotional dynamic modeling and prediction driven by a multimodal large model.

[0214] The emotional features V of all key frames emotion (1),…,V emotion (i),…,V emotion (I) Input the Transformer model based on the attention mechanism in chronological order to obtain the emotional state representation H at time t t , Ht =TransformerAttention(V emotion (1),…,V emotion (i),…,V emotion (I)), where H t is the emotional state representation at time t (the emotional feature at time t, i.e., the fourth emotional feature of the second video above), which is based on the emotional features V of all key frames emotion (i) Prediction of the emotional state representation at time t (the key frame is the frame of the video before time t).

[0215] When training a multimodal large model, based on the steps shown above, determine the emotional feature V of the frame corresponding to time t emotion (t), H obtained by comparing the prediction t With the actual V emotion (t), construct the loss constraint shown in formula (1.10) above, and obtain the pre-training loss value L prediction (ie the fourth loss value above), by minimizing L prediction , pre-training of large multimodal models.

[0216] This step can combine the information of historical and current frames to comprehensively model the dynamic changes of emotions.

[0217] Step 3: Sentiment classification based on Gaussian distribution perception.

[0218] Referring to formula (1.7) shown above, the emotional state at time t is represented by H t Input the classification head to generate sentiment labels, where the classification head uses a fully connected layer and a normalization layer (software), Y t represents the sentiment classification label predicted by the model at time t (such as "positive", "negative", "neutral", i.e. the sentiment of the second video above), W cls represents the weight matrix of the classification head, which is used to map the sentiment features to the classification space, b cls Represents the bias term of the classification head, which is used to adjust the classification results.

[0219] When training the classification head in the multimodal large model, the classification head is connected to the pre-trained multimodal large model, and the emotional feature V is generated through the classification head. emotion (i) The sentiment label Y i , Y i =softmax(W cls ·V emotion (i)+b cls ), where Y i Keyframes predicted for large models Sentiment classification label, Wcls represents the weight matrix of the classification head, which is used to map the sentiment features to the classification space, b cls Represents the bias term of the classification head, which is used to adjust the classification results.

[0220] Y i ′ is a key frame The true emotional label, using cross entropy loss L ce (Y i ,Y i ′) constrains the classification accuracy of the classification head of the multimodal large model.

[0221] Refer to formula (1.8) shown above. In order to improve the robustness of the classification results, a Gaussian distribution-aware regularization mechanism is introduced. Based on the sentiment classification label Y i The probability distribution P(Y i ′), impose constraints on the classification results, where L Gaussion is Gaussian loss, μ is the probability distribution P(Y i ′), σ is the mean of the probability distribution P(Y i ′), the two can be obtained by the sentiment classification label Y i The probability distribution of ′ is obtained by statistical analysis.

[0222] The total loss of the final classification head is shown in formula (1.9) above, where L cls is the total loss of the classification head (i.e. the second loss value above), L ce is the cross entropy loss (i.e. the first loss value above), L Gaussion is the Gaussian loss (i.e. the loss value of the Gaussian loss function above), and λ is an artificially set hyperparameter.

[0223] This step uses normalization processing to map emotional features to multi-category probability distributions to meet the requirements of multi-classification tasks. At the same time, regularization constraints can effectively suppress abnormal emotional classification results, ensuring that emotional predictions are more stable and reliable.

[0224] Step 4: Output the results.

[0225] Based on step 3, the emotional features of each key frame in the key frame sequence are classified, and the emotional category corresponding to each key frame is output. At the same time, the emotional features H of the video at time t can be t This is input into the classification head in step 3 to further predict the emotion category at time t, obtaining the emotion category of the video at time t. Through the above operations, the emotion characteristics of the video frames before time t can be used to predict and classify the emotion characteristics of the video at time t, thus achieving the prediction of the emotion category at future moments.

[0226] In summary, the beneficial effects of the emotion prediction method provided by the embodiments of the present application include:

[0227] 1) A multimodal large-model sentiment analysis framework based on key-frame capture in videos is proposed. This framework can capture key-frames in videos and use a multimodal large-model for sentiment analysis, accurately analyzing the emotions expressed in the input videos with low computational cost.

[0228] 2) A multimodal large-model sentiment analysis algorithm based on key-frame capture in videos is proposed. This algorithm can significantly improve data processing efficiency through key-frame capture technology, improve the accuracy of sentiment recognition by using multimodal feature fusion, and combine the cross-modal modeling capabilities and temporal sentiment analysis capabilities of the large model to accurately analyze the emotions expressed in the video and predict the emotions at any moment in the video.

[0229] The following continues to describe the exemplary structure of the emotion prediction device 555 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the emotion prediction device 555 of the memory 550 may include:

[0230] The emotion scoring module 5551 is used to perform emotion scoring on the first video frame in the first video to obtain a first score of the first video frame.

[0231] The video frame filtering module 5552 is configured to filter out the second video frame from the first video frame based on the first score.

[0232] The feature extraction module 5553 is used to extract the first emotion feature of the second video frame.

[0233] The emotion prediction module 5554 is used to perform emotion prediction on the second video based on the first emotion feature to obtain a first probability of the second video, and the first probability is used to determine the emotion of the second video.

[0234] In some embodiments, the emotion scoring module 5551 is also used to encode each first modal information included in the first video frame to obtain the encoding features of the first modal information, and perform emotion scoring on the first modal information based on the encoding features to obtain the second score of the first modal information; based on the preset weight coefficients of multiple first modal information, the second scores of the multiple first modal information are integrated to obtain the first score.

[0235] In some embodiments, the sentiment scoring module 5551 is also used to perform sentiment classification processing on the encoded features of the first modal information to obtain multiple first probabilities corresponding to the first modal information, where the first probability is used to indicate the possibility that the first modal information belongs to a first sentiment category, and the first sentiment category is any one of multiple sentiment categories; a preset operation is performed on the multiple first probabilities to obtain a second score of the first modal information.

[0236] In some embodiments, the multiple first probabilities include a first sub-probability belonging to a positive emotion, a second sub-probability belonging to a neutral emotion, and a third sub-probability belonging to a negative emotion. The emotion scoring module 5551 is also used to determine the difference between the first sub-probability and the third sub-probability, and determine the maximum probability from the first sub-probability, the second sub-probability and the third sub-probability; and determine the product of the difference and the maximum probability as the second score.

[0237] In some embodiments, the video frame screening module 5552 is further used to divide the first video into multiple video segments; for each video segment, based on the first score of the first video frame included in the video segment, the first video frames included in the video segment are clustered to obtain a cluster cluster, and a third video frame is determined from the cluster cluster; the first video frame with the largest first score, the first video frame with the smallest first score, and the third video frame of each video segment are determined as the second video frame.

[0238] In some embodiments, the feature extraction module 5553 is also used to extract the second emotional feature of each second modal information included in the second video frame; and fuse the second emotional features of multiple second modal information to obtain the first emotional feature of the second video frame.

[0239] In some embodiments, the feature extraction module 5553 is also used to perform attention processing on multiple second emotional features of the second modal information to obtain a weight coefficient for each second modal information; based on the weight coefficient of each second modal information, fuse multiple second emotional features of the second modal information to obtain a third emotional feature; and perform feature extraction on the third emotional feature to obtain the first emotional feature.

[0240] In some embodiments, the emotion prediction module 5554 is further used to predict a fourth emotion feature of the second video based on the first emotion feature; perform emotion recognition on the fourth emotion feature to obtain a first probability of the second video.

[0241] In some embodiments, the emotion prediction module 5554 is also used to perform a linear transformation on the fourth emotion feature to obtain a transformed fourth emotion feature; perform mapping processing on the transformed fourth emotion feature to obtain a mapped fourth emotion feature; and perform normalization processing on the mapped fourth emotion feature to obtain a first probability of the second video.

[0242] The emotion prediction device 555 provided in the embodiment of the present application may further include a model training module. In some embodiments, the emotion prediction method is implemented through a neural network model. The model training module is used to perform emotion prediction on the existing first video frame sample through the neural network model to obtain the first emotion of the second video frame sample; determine a first loss value based on the emotion label of the second video frame sample and the first emotion; perform probability statistics on the emotion label of the second video frame sample to obtain the mean and variance of the emotion label, and adjust the first loss value based on the mean and the variance to obtain a second loss value; and train the neural network model based on the second loss value.

[0243] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the emotion prediction method described in the present invention.

[0244] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the emotion prediction method provided by the embodiment of the present application.

[0245] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0246] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0247] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0248] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0249] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A sentiment prediction method, characterized in that: The method comprises: Performing an emotion score on a first video frame in a first video to obtain a first score for the first video frame; Based on the first score, filtering out a second video frame from the first video frame; extracting a first emotion feature of the second video frame; Based on the first emotion feature, emotion prediction is performed on the second video to obtain a first probability of the second video, where the first probability is used to determine the emotion of the second video.

2. The method according to claim 1, characterized in that The performing emotion scoring on the first video frame in the first video to obtain a first score of the first video frame includes: For each piece of first modal information included in the first video frame, encode the first modal information to obtain encoding features of the first modal information, and perform a sentiment score on the first modal information based on the encoding features to obtain a second score of the first modal information; Based on preset weight coefficients of multiple first modal information, the second scores of the multiple first modal information are fused to obtain the first score.

3. The method according to claim 2, characterized in that The step of performing a sentiment score on the first modal information based on the encoding feature to obtain a second score of the first modal information includes: Performing sentiment classification processing on the encoded features of the first modal information to obtain a plurality of first probabilities corresponding to the first modal information, the first probabilities being used to indicate a likelihood that the first modal information belongs to a first sentiment category, where the first sentiment category is any one of a plurality of sentiment categories; A preset operation is performed on the multiple first probabilities to obtain a second score of the first modal information.

4. The method according to claim 3, characterized in that The plurality of first probabilities include a first sub-probability belonging to a positive emotion, a second sub-probability belonging to a neutral emotion, and a third sub-probability belonging to a negative emotion; The performing a preset operation on the plurality of first probabilities to obtain a second score of the first modal information includes: determining a difference between the first sub-probability and the third sub-probability, and determining a maximum probability from the first sub-probability, the second sub-probability, and the third sub-probability; The product of the difference and the maximum probability is determined as the second score.

5. The method according to claim 1, wherein The filtering out the second video frame from the first video frame based on the first score includes: Dividing the first video into a plurality of video segments; For each video segment, clustering the first video frames included in the video segment based on the first scores of the first video frames included in the video segment to obtain clusters, and determining a third video frame from the clusters; The first video frame with the largest first score, the first video frame with the smallest first score, and the third video frame of each video segment are determined as the second video frame.

6. The method according to claim 1, characterized in that The extracting the first emotion feature of the second video frame includes: For each second modal information included in the second video frame, extracting a second emotional feature of the second modal information; The second emotion features of the plurality of second modal information are fused to obtain the first emotion feature of the second video frame.

7. The method according to claim 6, characterized in that The fusing of the second emotion features of the plurality of second modal information to obtain the first emotion feature of the second video frame includes: performing attention processing on the second emotional features of the plurality of second modal information to obtain a weight coefficient for each piece of second modal information; Based on the weight coefficient of each second modal information, fusing the second emotional features of the plurality of second modal information to obtain a third emotional feature; Feature extraction is performed on the third emotion feature to obtain the first emotion feature.

8. The method according to claim 1, characterized in that The performing emotion prediction on the second video based on the first emotion feature to obtain a first probability of the second video includes: Predicting a fourth emotional feature of the second video based on the first emotional feature; Emotion recognition is performed on the fourth emotion feature to obtain a first probability of the second video.

9. The method according to claim 8, characterized in that The performing emotion recognition on the fourth emotion feature to obtain a first probability of the second video includes: Performing a linear transformation on the fourth emotional feature to obtain a transformed fourth emotional feature; performing mapping processing on the transformed fourth emotional feature to obtain a mapped fourth emotional feature; Normalization is performed on the mapped fourth emotion feature to obtain a first probability of the second video.

10. The method according to any one of claims 1 to 9, characterized in that The emotion prediction method is implemented through a neural network model; the training process of the neural network model includes: Using the neural network model, emotion prediction is performed on the existing first video frame sample to obtain the first emotion of the second video frame sample; Determining a first loss value based on the emotion label of the second video frame sample and the first emotion; Performing probability statistics on the emotion labels of the second video frame samples to obtain a mean and a variance of the emotion labels, and adjusting the first loss value based on the mean and the variance to obtain a second loss value; Based on the second loss value, the neural network model is trained.

11. An emotion prediction device, characterized in that: The device comprises: An emotion scoring module is used to perform an emotion scoring on a first video frame in a first video to obtain a first score of the first video frame; a video frame screening module, configured to screen out a second video frame from the first video frame based on the first score; A feature extraction module, configured to extract a first emotion feature of the second video frame; The emotion prediction module is used to perform emotion prediction on the second video based on the first emotion feature to obtain a first probability of the second video, where the first probability is used to determine the emotion of the second video.

12. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the emotion prediction method according to any one of claims 1 to 10 when executing the computer executable instructions or computer program stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the emotion prediction method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the emotion prediction method according to any one of claims 1 to 10 is implemented.

Images