Online video emotion recognition method based on dynamic migration model

Abstract

The invention provides an online video emotion recognition method based on a dynamic migration model. The method comprises the following steps: segmenting an online video into segments; and carrying out dynamic transfer learning and identification on the fragments according to a time sequence, wherein key frame extraction, depth feature extraction, optimal historical paradigm selection, historical paradigm set updating and emotion label and standard image output are included. According to the method, a dynamic transfer learning method is adopted, the current clip is learned and recognized based on the reliable paradigm in the preorder video clip, the difference between the current image and the reliable paradigm is focused on, the method can adapt to the difference of emotional expression modes under different culture and different character features, and the recognition performance can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to an online video emotion recognition method based on a dynamic migration model. Background Art

[0002] Emotion recognition is key to enabling advanced human-computer interaction (HCI), virtual reality (VR), augmented reality (AR), and vehicle networking technologies. For example, in vehicle networking, the driver's emotional state is crucial to driving safety. Accurately identifying their emotions helps the system take timely measures to ensure safety. In recent years, the application of dynamic emotion recognition technology based on video, voice, and other data has become increasingly widespread, primarily using transfer learning and deep learning strategies.

[0003] The prior art discloses a method for dynamic facial emotion recognition based on deep learning, comprising the following steps: S1, obtaining a facial image sequence; S2, extracting image features of each image in the facial image sequence using a VGG convolutional neural network; S3, identifying facial emotions using an LSTM recurrent neural network in combination with the image features extracted in S2; S4, repeatedly training the network using a loss function, optimizing network parameters, and constructing a complete dynamic facial emotion recognition model. The facial image sequence is processed by combining a VGG convolutional neural network with an LSTM recurrent neural network, significantly improving the accuracy of emotion recognition. The method has the following drawbacks: the combination of VGG+LSTM requires a large amount of computing resources and is difficult to meet real-time requirements (such as instant feedback in video surveillance); different cultures have different ways of expressing emotions (such as the magnitude of a smile), and a single model may not be universally applicable.

[0004] Various facial emotion recognition methods in the existing technology still have some problems: poor adaptability to dynamic data, such as CNN and LSTM, which have difficulty processing the continuous changes in facial expressions in online video streams, resulting in insufficient real-time performance; insufficient application of transfer learning, and failure to fully utilize the reliable labels of historical data for dynamic migration, resulting in low incremental learning efficiency; dynamic clustering algorithms such as AP and EEM have high time complexity and cannot meet the real-time processing requirements of online videos.

[0005] Therefore, there is a need to improve the existing technology. Summary of the Invention

[0006] The present invention provides an online video emotion recognition method based on a dynamic migration model to solve the above problems.

[0007] A technical solution adopted by the present invention is to provide an online video emotion recognition method based on a dynamic migration model, comprising the following steps:

[0008] S1, obtain online video and split it into segments according to time windows;

[0009] S2. Dynamic transfer learning and recognition of segments in chronological order:

[0010] S21, extracting key frames from the current segment, obtaining a facial region image sequence, and using a GoogLeNet convolutional neural network to extract deep features of each image in the facial region image sequence;

[0011] S22, calculating the similarity probability between the current image and the historical example set, and selecting the example with the highest similarity probability in the historical example set as the best historical example;

[0012] S23, when the similarity between the best historical example and the current image is less than a preset value, adding the current image to the current example set;

[0013] S24, repeating steps S22 and S23 until the similarity between each image and its best historical example is determined;

[0014] S25, calculating the KL distance between the current example set and the historical example set, wherein the current example set includes examples in the historical example set;

[0015] S26, when the KL distance reaches the updated value, the historical example set is updated with the current example set for the next segment learning;

[0016] S27: Output the emotion label of the current segment and the corresponding standard image.

[0017] Furthermore, the method for obtaining the initial set of historical examples is as follows:

[0018] SA1, taking the first segment after the online video is split as the source segment;

[0019] SA2, extract the key frames of the source segment and obtain the facial region image sequence of the source segment;

[0020] SA3, using the GoogLeNet convolutional neural network to extract deep features of each image in the source segment facial region image sequence;

[0021] SA4. Use AP clustering algorithm to obtain the initial historical example set.

[0022] Furthermore, the number of key frames of the extracted source segments is twice or more than the number of key frames of the remaining segments, making the initial historical example set more effective.

[0023] Furthermore, in S22:

[0024] The formula for calculating similarity probability is Among them, x i 、xj Represents the two feature vectors whose similarity needs to be calculated, s(x i ,x j ) represents x i and x j The similarity score of , parameter σ is the standard deviation;

[0025] The formula for determining the best historical example is Where t is the fragment number, The feature vector representing the i-th image of the t-th segment, E (t-1) A collection of representative historical examples, An example representing a collection of historical examples.

[0026] Furthermore, in S21, PCA is used to reduce the dimensionality of the depth features of each image.

[0027] The beneficial effects of the online video emotion recognition method based on the dynamic migration model of the present invention are:

[0028] 1. Dynamic transfer learning is used to learn and recognize the current video clip based on reliable examples in the previous video clip, focusing on the differences between the current image and reliable examples. This method can adapt to the differences in emotional expression in different cultures and personality traits, and can effectively improve recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of an online video emotion recognition method based on a dynamic migration model according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0030] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0031] See also Figure 1 The first embodiment of the present invention provides an online video emotion recognition method based on a dynamic migration model, which mainly includes two steps:

[0032] 1. Obtain an initial set of historical examples based on the first clip of the online video:

[0033] Get the online video and split it into 30-second segments. The first segment is called the "source segment" and the subsequent segments are called "drift segments."

[0034] Extract key frames of the source segment at a rate of 22 frames per second to obtain a facial region image sequence of the source segment (660 frames), where the size of each image is 266*340 pixels;

[0035] Perform normalization processing;

[0036] Use the GoogLeNet convolutional neural network to extract the deep features of each image in the source segment facial region image sequence;

[0037] PCA is used to reduce the dimensionality of the deep features of each image;

[0038] The AP clustering algorithm is used to obtain the initial historical example set.

[0039] 2. Dynamic transfer learning and identification of drift fragments based on historical example sets:

[0040] Extract key frames at a rate of 11 frames per second to obtain a facial region image sequence (330 frames) with an image size of 266*340 pixels;

[0041] Perform normalization processing;

[0042] Use the GoogLeNet convolutional neural network to extract the deep features of each image in the facial region image sequence;

[0043] PCA is used to reduce the dimensionality of the deep features of each image;

[0044] The similarity probability between the current image and the historical example set is calculated according to formula (1):

[0045] Among them, x i 、x j Represents the two feature vectors whose similarity needs to be calculated, s(x i ,x j ) represents x i and x j The similarity score of , parameter σ is the standard deviation;

[0046] According to formula (2), the example with the highest similarity probability in the historical example set is selected as the best historical example:

[0047] Where t is the fragment number, The feature vector representing the i-th image of the t-th segment, E (t-1) A collection of representative historical examples, An example representing a collection of historical examples.

[0048] When the similarity between the best historical example and the current image is less than a preset value, the current image is added to the current example set;

[0049] Complete the selection and similarity determination of the best historical examples for each image, so that images that are slightly different from the historical example set are added to the current example set, where the current example set contains examples from the historical example set;

[0050] Calculate the KL distance between the current example set and the historical example set. Update the historical example set only when there is a sufficient difference between the two, that is, when the KL distance reaches the updated value.

[0051] Output the emotion label of the current segment and the corresponding standard image.

[0052] The beneficial effects of the online video emotion recognition method based on the dynamic migration model of the present invention are:

[0053] 1. Using dynamic transfer learning, the system learns and recognizes the current video clip based on reliable examples in the previous video clip, focusing on the differences between the current image and the reliable examples. This method can adapt to the differences in emotional expression in different cultures and personality traits, effectively improving recognition performance.

[0054] 2. This method can be applied to multiple fields such as intelligent driving, medical care, and security to judge the user's emotional state in real time and respond promptly.

[0055] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An online video emotion recognition method based on a dynamic migration model, characterized in that: The steps include: S1, obtain online video and split it into segments according to time windows; S2. Dynamic transfer learning and recognition of segments in chronological order: S21, extracting key frames from the current segment, obtaining a facial region image sequence, and using a GoogLeNet convolutional neural network to extract deep features of each image in the facial region image sequence; S22, calculating the similarity probability between the current image and the historical example set, and selecting the example with the highest similarity probability in the historical example set as the best historical example; S23, when the similarity between the best historical example and the current image is less than a preset value, adding the current image to the current example set; S24, repeating steps S22 and S23 until the similarity between each image and its best historical example is determined; S25, calculating the KL distance between the current example set and the historical example set, wherein the current example set includes examples in the historical example set; S26, when the KL distance reaches the updated value, the historical example set is updated with the current example set for the next segment learning; S27: Output the emotion label of the current segment and the corresponding standard image.

2. The online video emotion recognition method based on the dynamic migration model according to claim 1 is characterized in that: Method for obtaining the initial historical example set: SA1, taking the first segment after segmenting the online video as the source segment; SA2, extract the key frames of the source segment and obtain the facial region image sequence of the source segment; SA3, using the GoogLeNet convolutional neural network to extract deep features of each image in the source segment facial region image sequence; SA4. Use AP clustering algorithm to obtain the initial historical example set.

3. The online video emotion recognition method based on the dynamic migration model according to claim 2 is characterized in that: The number of key frames of the extracted source segments is twice or more than the number of key frames of the remaining segments, making the initial history example set more effective.

4. The online video emotion recognition method based on a dynamic migration model according to any one of claims 1 to 3, characterized in that: In S22: The formula for calculating the similarity probability is, where xi and xj represent the two feature vectors whose similarity needs to be calculated, s(xi,xj) represents the similarity score of xi and xj, and the parameter σ is the standard deviation; the formula for determining the best historical example is, where t is the segment number, represents the feature vector of the i-th image of the t-th segment, represents the historical example set, and represents the example in the historical example set.

5. The online video emotion recognition method based on a dynamic migration model according to any one of claim 4, characterized in that: In S21, PCA is also used to reduce the dimensionality of the deep features of each image.

Images