Micro-expression recognition model training method, micro-expression recognition method, and related device

Abstract

The application discloses a micro-expression recognition model training method, a micro-expression recognition method and related equipment. The training method comprises the following steps: acquiring a complete macro-expression sequence and a micro-expression sequence, and screening a part of the macro-expression sequence which is consistent with the emotion label of the micro-expression sequence from the complete macro-expression sequence; constructing a teacher model and a student model, the teacher model and the student model having the same neural network structure; inputting the complete macro-expression sequence into the teacher model for pre-training; taking the part of the macro-expression sequence and the micro-expression sequence as training data, and performing model training on the student model under the guidance of the pre-trained teacher model until the student model meets a preset convergence condition, and then completing the model training. Through the combination of the micro-expression and the macro-expression, and the model training of the micro-expression recognition task performed by the teacher-student model in the knowledge distillation framework, the problem of insufficient micro-expression samples and low feature intensity can be effectively made up, so that the accuracy of the micro-expression recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and in particular to a training method for a micro-expression recognition model, a micro-expression recognition method, and related equipment. Background Technology

[0002] Emotional interaction is a highly anticipated area in human-computer interaction, and a key element in making computers more human-like. Vision-based emotional interaction primarily focuses on facial expression recognition. While recognition technology for common facial expressions (macro-expressions) is quite mature, because these expressions can be artificially controlled, macro-expression recognition may not accurately reflect a person's inner feelings. For example, in financial transactions, there may be fraudulent activities such as identity theft, document forgery, and information concealment. Due to deliberate human control, macro-expression recognition cannot accurately detect these anomalies. Therefore, micro-expression recognition, which reveals a person's true feelings and emotions, is receiving increasing attention. Micro-expression recognition technology helps identify unusual psychological states in traders, thereby preventing fraud and improving security.

[0003] In the field of micro-expression recognition, the micro-expression feature intensity is very low, making it difficult for even professionally trained people to accurately identify micro-expressions. Furthermore, there are too few publicly available micro-expression datasets, and the sample distribution is extremely uneven, increasing the difficulty of model training. This makes it difficult for the accuracy of micro-expression recognition to meet application requirements. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a training method, a micro-expression recognition method and related equipment for micro-expression recognition models that can be applied to financial technology or other related fields, with the main purpose of improving the accuracy of micro-expression recognition.

[0005] The technical solution of the present invention is as follows:

[0006] The first aspect of this invention provides a method for training a micro-expression recognition model, comprising:

[0007] Obtain complete macro-expression sequences and micro-expression sequences, and filter out the macro-expression sequences that match the emotion tags of the micro-expression sequences from the complete macro-expression sequences;

[0008] Construct a teacher model and a student model, which have the same neural network structure;

[0009] The complete macro-expression sequence is input into the teacher model for pre-training.

[0010] The macro-expression sequence and micro-expression sequence are used as training data. Under the guidance of the pre-trained teacher model, the student model is trained until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0011] In one embodiment, after obtaining the complete macro-expression sequence and micro-expression sequence, the method further includes:

[0012] The complete macro-expression sequence and micro-expression sequence are subjected to face cropping and alignment processing.

[0013] In one embodiment, inputting the complete macro-expression sequence into the teacher model for pre-training includes:

[0014] The complete macro-expression sequence is input into the feature extraction network of the teacher model to extract the features of each frame of the image;

[0015] The features of each frame of the image are input into the time series network of the teacher model, and the output features with temporal sequence are generated.

[0016] The temporal features are input into the recognition network of the teacher model, and the expression recognition result is output.

[0017] The loss function value is obtained based on the expression recognition result and the emotion label of the complete macro expression sequence;

[0018] The model parameters of the teacher model are updated based on the loss function value until the pre-training of the teacher model is completed.

[0019] In one embodiment, the step of using the partial macro-expression sequence and micro-expression sequence as training data to train the student model under the guidance of a pre-trained teacher model until the student model meets a preset convergence condition, thus completing model training and obtaining a micro-expression recognition model, includes:

[0020] The partial macro expression sequence is input into the student model, and the student model is trained in the first stage under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model.

[0021] Freeze the parameters of the student model, excluding the recognition network, after the first phase of training.

[0022] The micro-expression sequence is input into a student model with some parameters frozen for the second stage of training. The parameters of the recognition network of the student model are adjusted individually until the student model meets the preset convergence condition, thus obtaining the micro-expression recognition model.

[0023] In one embodiment, the partial macro expression sequence is the data from the beginning of the complete macro expression sequence.

[0024] A second aspect of the present invention provides a micro-expression recognition method, comprising:

[0025] Obtain the image sequence to be identified;

[0026] The image sequence to be identified is input into a pre-trained micro-expression recognition model for micro-expression recognition, and the micro-expression recognition result of the image sequence is output.

[0027] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0028] A third aspect of the present invention provides a training apparatus for a micro-expression recognition model, comprising:

[0029] The data acquisition module is used to acquire complete macro expression sequences and micro expression sequences, and to filter out the portion of the macro expression sequences that are consistent with the emotion tags of the micro expression sequences from the complete macro expression sequences;

[0030] The model building module is used to build teacher and student models, which have the same neural network structure.

[0031] The first training module is used to input the complete macro expression sequence into the teacher model for pre-training;

[0032] The second training module is used to use the partial macro-expression sequence and micro-expression sequence as training data to train the student model under the guidance of the pre-trained teacher model until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0033] A fourth aspect of the present invention provides a micro-expression recognition device, comprising:

[0034] The image acquisition module is used to acquire the image sequence to be identified;

[0035] The recognition module is used to input the image sequence to be recognized into a pre-trained micro-expression recognition model for micro-expression recognition, and output the micro-expression recognition result of the image sequence;

[0036] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0037] A fifth aspect of the present invention provides an electronic device comprising at least one processor; and,

[0038] A memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the training method of the micro-expression recognition model or to perform the micro-expression recognition method described above.

[0040] A sixth aspect of the present invention provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the training method of the micro-expression recognition model or to perform the micro-expression recognition method described above.

[0041] Beneficial Effects: This invention discloses a training method for a micro-expression recognition model, a micro-expression recognition method, and related equipment. Compared with existing technologies, this invention obtains complete macro-expression sequences and micro-expression sequences, filters out a portion of the macro-expression sequences that match the emotion labels of the micro-expression sequences from the complete macro-expression sequences, constructs a teacher model and a student model with the same neural network structure, inputs the complete macro-expression sequences into the teacher model for pre-training, and uses the partial macro-expression sequences and micro-expression sequences as training data to train the student model under the guidance of the pre-trained teacher model until the student model meets the preset convergence condition, thus completing model training. By combining micro-expressions and macro-expressions and using the teacher-student model in the knowledge distillation framework for model training in the micro-expression recognition task, the problems of insufficient micro-expression samples and low feature strength can be effectively compensated for, thereby improving the accuracy of micro-expression recognition. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0043] Figure 1 A flowchart illustrating the training method for the micro-expression recognition model provided in an embodiment of the present invention;

[0044] Figure 2 A flowchart of step S300 in the training method of the micro-expression recognition model provided in the embodiment of the present invention;

[0045] Figure 3 A flowchart of step S400 in the training method of the micro-expression recognition model provided in the embodiment of the present invention;

[0046] Figure 4 This is an architecture diagram of the teacher model and student model in the training method of the micro-expression recognition model provided in the embodiments of the present invention;

[0047] Figure 5 A flowchart of a micro-expression recognition method provided in an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the functional modules of the training device for the micro-expression recognition model provided in an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of the functional modules of the micro-expression recognition device provided in an embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0052] Visual-based emotional interaction primarily focuses on facial expression recognition. While recognition technology for common facial expressions (macro-expressions) is quite mature, it cannot accurately reflect a person's inner feelings because these expressions can be manipulated. For example, in financial transactions, fraudulent activities such as identity theft, document forgery, and information concealment may occur. Due to deliberate human control, macro-expression recognition cannot accurately detect these anomalies. Therefore, micro-expression recognition, which reveals a person's true feelings and emotions, is receiving increasing attention. Micro-expression recognition technology helps identify unusual psychological states in traders, thereby preventing fraud and improving security.

[0053] In the field of micro-expression recognition, the micro-expression feature intensity is very low, making it difficult for even professionally trained people to accurately identify micro-expressions. Furthermore, there are too few publicly available micro-expression datasets, and the sample distribution is extremely uneven, increasing the difficulty of model training. This makes it difficult for the accuracy of micro-expression recognition to meet application requirements.

[0054] To address the aforementioned problems, this invention proposes a training method for a micro-expression recognition model. Please refer to [link / reference]. Figure 1 , Figure 1This is a flowchart of an embodiment of the training method for the micro-expression recognition model provided by the present invention. The training method for the micro-expression recognition model provided in this embodiment is applied to a system comprising a terminal device, a network, and a server. The network serves as a medium for providing a communication link between the terminal device and the server, and may include various connection types, such as wired, wireless communication links, or fiber optic cables. The operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), Android system, or other operating systems. The terminal device connects to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data. Specifically, it may be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, portable computers, and desktop servers. Figure 1 As shown, the method specifically includes the following steps:

[0055] S100. Obtain the complete macro expression sequence and micro expression sequence, and select the part of the macro expression sequence that is consistent with the emotion label of the micro expression sequence from the complete macro expression sequence.

[0056] In this embodiment, datasets of complete macro-expression sequences and micro-expression sequences are obtained. Furthermore, samples consistent with the emotional expressions in the micro-expression sequences are selected from the complete macro-expression sequences, i.e., partial macro-expression sequences. These datasets, composed of three different expression sequences, serve as training data for the micro-expression recognition model. Specifically, an expression sequence refers to multiple frames of facial images arranged in a time sequence. Expressions that flash by quickly, lasting less than 0.5 seconds, are called micro-expressions. In contrast, expressions with a slightly longer duration, between 1 and 5 seconds, are called macro-expressions. Therefore, a complete macro-expression sequence includes a transition from an expressionless face image to a face image with a complete macro-expression, and a micro-expression sequence includes a transition from an expressionless face image to a face image with micro-expressions. It is evident that both micro-expressions and macro-expressions are continuous processes starting from an expressionless state. If micro-expressions are considered as the preliminary process of macro-expressions, i.e., the process controlled by the user... If micro-expressions are unconsciously made before creating macro-expressions, then the complete macro-expression sequence includes face images ranging from expressionless faces to face images with micro-expressions, and finally to face images with complete macro-expressions. In other words, micro-expressions can be considered as incomplete macro-expressions. Therefore, macro-expressions and micro-expressions can be organically combined. A portion of the macro-expression sequence can be selected from the complete macro-expression sequence, which consists of face images with micro-expression labels. This portion of the macro-expression sequence is usually the data from the beginning of the complete macro-expression sequence, such as the first 20% or 30% of the data. This is to compensate for the problem of insufficient micro-expression data and uneven sample distribution.

[0057] Preferably, after obtaining the complete macro-expression sequence and micro-expression sequence, the method further includes:

[0058] The complete macro-expression sequence and micro-expression sequence are subjected to face cropping and alignment processing.

[0059] In this embodiment, after preparing a complete dataset of macro-expression sequences and micro-expression sequences, preprocessing of face images in the dataset is performed on face cropping and alignment. This involves extracting various faces from images of different sizes in the dataset, correcting tilted faces to normal poses, and ensuring that features such as the eyes are in the same position in each extracted face, thereby improving the accuracy of subsequent micro-expression recognition tasks. Specific alignment methods can include, for example, feature-point based methods. This method marks key feature points on the face, such as eyes, nose, and mouth, and then aligns based on these feature points. This method is simple and intuitive, but its performance is poor for low-resolution or blurry images. Alternatively, deep learning-based methods can be used. These methods utilize deep neural networks for face alignment, such as DeepID and DeepFace, which are based on convolutional neural networks and automatically complete alignment by learning facial features. The specific method can be flexibly chosen according to actual needs, and this embodiment does not limit the choice.

[0060] S200. Construct a teacher model and a student model, wherein the teacher model and the student model have the same neural network structure.

[0061] In this embodiment, a teacher-student model within a knowledge distillation framework is constructed for learning and training the micro-expression recognition task. The teacher and student models have the same neural network structure, such as... Figure 4 As shown, both models include feature extraction networks, actual sequence networks, and recognition networks to classify and recognize input facial expression sequences. Teacher-student models typically refer to a deep learning approach where a well-trained and performing model (the teacher model) guides the training of a new model (the student model). This involves training the student model through knowledge distillation to mimic the behavior of the teacher model. The student model's generalization ability can be improved by utilizing the teacher model's internal representations. When applied to micro-expression recognition tasks with low feature strength and difficulty in feature extraction, a well-trained teacher model can effectively enhance the student model's feature extraction capabilities and recognition accuracy for micro-expressions.

[0062] S300. Input the complete macro expression sequence into the teacher model for pre-training.

[0063] In this embodiment, before training the micro-expression recognition task, the teacher model is pre-trained using complete macro-expression sequences. The complete macro-expression sequences and corresponding emotion labels enable the teacher model to learn the changes in facial features from expressionless to complete macro-expressions under different expression sequences, so that the pre-trained teacher model can accurately classify and recognize expressions from complete macro-expression sequences.

[0064] In one embodiment, such as Figure 2 As shown, step S300 includes:

[0065] S301. Input the complete macro expression sequence into the feature extraction network of the teacher model to extract the features of each frame of the image;

[0066] S302. Input the features of each frame image into the time series network of the teacher model, and output features with temporal sequence.

[0067] S303. Input the temporal features into the recognition network of the teacher model and output the facial expression recognition result;

[0068] S304. Obtain the loss function value based on the expression recognition result and the emotion label of the complete macro expression sequence;

[0069] S305. Update the model parameters of the teacher model according to the loss function value until the pre-training of the teacher model is completed.

[0070] In this embodiment, when pre-training the teacher model using the complete macro-expression sequence, the complete macro-expression sequence is first input into the feature extraction network to extract the features of each frame of the image. Then, the extracted features of each frame of the image are input into the time series network LSTM to output features with temporal sequence characteristics. LSTM neural network is a deep learning model commonly used for modeling sequential data; its full name is Long Short-Term Memory Network. Compared to traditional recurrent neural networks (RNNs), LSTM networks (Memory Networks) have better long-term dependency and memory capabilities, thus enabling them to accurately and effectively process time-series data. They can better extract corresponding temporal features based on image feature changes in complete macro-expression sequences, allowing teacher models to learn facial features of complete expressions. Finally, these temporal features are input into the recognition network for classification and recognition, outputting expression recognition results. Specifically, the output shows the probability distribution of different expression categories. This probability distribution reflects the relative likelihood and boundary ambiguity between different categories. Since complete macro-expression sequences also contain some micro-expression images, the probability distribution of different expression categories not only predicts which category is most likely correct but also provides probability values ​​for all categories, including macro-expressions and micro-expressions. This provides more information for the subsequent training of student models, improving their capabilities.

[0071] Based on the facial expression recognition results of the teacher model, the recognition accuracy of the teacher model is evaluated by a loss function such as the cross-entropy function. The model parameters of the teacher model are updated by backpropagation based on the loss function value until the pre-training of the teacher model is completed. Specifically, the pre-training can end when the loss function value is less than a preset value, or when the training period reaches a preset period, etc., so as to obtain a teacher model with good performance on the complete macro facial expression sequence recognition task, providing a reliable training foundation for the subsequent training of student models.

[0072] S400. Using the macro-expression sequence and micro-expression sequence as training data, the student model is trained under the guidance of the pre-trained teacher model until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0073] In this embodiment, under the guidance of the pre-trained teacher model, partial macro-expression sequences and micro-expression sequences with micro-expression features are used as training data and input into the learning model for model training. This not only effectively alleviates the problem of insufficient micro-expression datasets and uneven sample distribution by extracting partial macro-expression sequences with micro-expression features from complete macro-expression sequences, but also allows the teacher model to transfer its learned knowledge on how to extract and recognize expression features in sequences to the student model during training. The student model then attempts to imitate the behavior of the teacher model, learning how to more effectively express the inherent rules of the data within a limited parameter space. This enables the student model to exhibit better generalization ability and robustness in the micro-expression recognition task. Thus, by learning the behavior of the teacher model, the learning model can improve its own feature extraction and recognition capabilities. After the model converges, the trained student model is obtained and used as the final micro-expression recognition model, effectively compensating for the problems of insufficient micro-expression samples and low feature strength, thereby improving the accuracy of micro-expression recognition.

[0074] In one embodiment, such as Figure 3 As shown, step S400 includes:

[0075] S401. Input the partial macro expression sequence into the student model, and perform the first stage of training on the student model under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model;

[0076] S402. Freeze the parameters of the other layers in the student model after the first stage of training, except for the recognition network.

[0077] S403. Input the micro-expression sequence into the student model with some parameters frozen for the second stage of training. Adjust the parameters of the recognition network of the student model individually until the student model meets the preset convergence condition to obtain the micro-expression recognition model.

[0078] In this embodiment, the training of the student model is divided into two stages. In the first stage, the extracted partial macro-expression sequences with micro-expression features are input into the student model. Under the guidance of the pre-trained teacher model, early facial expression recognition training is performed. In this first stage, the parameters of all layers in the student model are adjusted based on the output results and loss function values ​​of the learning model. Specifically, since the partial macro-expression training extracts early images from the complete macro-expression training, the teacher model can already extract and recognize the expression probability distribution. At this point, the teacher model can guide the learning model using methods such as soft labeling and feature matching, thereby transferring the knowledge from the teacher model to the student model.

[0079] The soft-label method refers to a teacher model generating probability distributions for the input data instead of hard category labels. These probability distributions contain more information, reflecting the relative likelihood and ambiguity of different categories. The student model learns based on these soft labels, thus mimicking the teacher model's decision-making process. During knowledge distillation, the student model doesn't just imitate the highest-probability category; instead, it learns the entire "softened" probability distribution of the teacher model. For example, by increasing the temperature scaling parameter to smooth the distribution, the maximum probability value decreases while the probabilities of other categories increase accordingly. This allows the student model to not only focus on the most likely category but also learn the relative differences between different categories, thereby fully capturing the subtle differences between different micro-expression categories from the input partial macro-expression sequences, improving the model's ability to extract micro-expression features. The feature matching method means that the student model not only matches the true data labels but also tries to make its internal layer feature representations as close as possible to the teacher model's feature representations under the same input. In other words, when the student model adjusts its parameters through backpropagation, it needs to replicate the feature space structure extracted in the intermediate layer as closely as possible to the feature space of the teacher model. In this way, the student model can leverage the key feature information extracted by the teacher model to achieve miniaturization and acceleration of the model while maintaining high accuracy.

[0080] After completing the first stage of training under the guidance of the teacher model, the parameters of all layers in the student model except for the recognition network are frozen. This retains the feature vectors of the intermediate layers learned from the teacher model. In the second stage of training, the recognition network in the student model is trained separately using micro-expression sequences. The parameters of the recognition network are further precisely adjusted until the student model meets preset convergence conditions, such as reaching a preset number of training iterations or the loss function value being less than a preset value. The converged student model is then used as the final micro-expression recognition model. This embodiment utilizes the complementary data of micro-expressions and macro-expressions, and trains the micro-expression recognition model based on the teacher-student model within the knowledge distillation framework. Furthermore, the parameter adjustments at different stages during the student model training allow for more precise adjustment of the model parameters, enabling the final trained micro-expression recognition model to accurately and efficiently recognize micro-expressions in images. This effectively compensates for the problems of insufficient micro-expression samples and low feature strength, thereby improving the accuracy of micro-expression recognition.

[0081] Another embodiment of the present invention provides a micro-expression recognition method, such as... Figure 5 As shown, the method includes the following steps:

[0082] S500: Obtain the image sequence to be identified;

[0083] S600 inputs the image sequence to be identified into a pre-trained micro-expression recognition model to perform micro-expression recognition, and outputs the micro-expression recognition result of the image sequence;

[0084] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0085] In this embodiment, after training the micro-expression recognition model using the training method described above, end-to-end micro-expression recognition can be achieved in various application scenarios. For example, in scenarios such as financial transactions and remote video interviews, the user's video stream can be collected in real time, and the image sequence extracted from the video stream can be input into the micro-expression recognition model for micro-expression recognition. The corresponding micro-expression recognition results can then be output. Through accurate micro-expression recognition, reliable reference information can be provided for various application scenarios such as transaction security detection and dialogue strategy adjustment.

[0086] Another embodiment of the present invention provides a training device for a micro-expression recognition model, such as... Figure 6 As shown, the training device 1 includes:

[0087] Data acquisition module 11 is used to acquire complete macro expression sequences and micro expression sequences, and to filter out a portion of the macro expression sequences that are consistent with the emotion tags of the micro expression sequences from the complete macro expression sequences;

[0088] Model building module 12 is used to build a teacher model and a student model, which have the same neural network structure;

[0089] The first training module 13 is used to input the complete macro expression sequence into the teacher model for pre-training;

[0090] The second training module 14 is used to use the partial macro-expression sequence and micro-expression sequence as training data, and to train the student model under the guidance of the pre-trained teacher model until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0091] The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the training and execution process of a micro-expression recognition model. For specific implementation methods of each module, please refer to the corresponding method embodiments described above, which will not be repeated here.

[0092] In one embodiment, the training device 1 further includes:

[0093] The preprocessing module is used to perform face cropping and alignment processing on the complete macro expression sequence and micro expression sequence.

[0094] In one embodiment, the first training module 13 includes:

[0095] The first input unit is used to input the complete macro expression sequence into the feature extraction network of the teacher model to extract the features of each frame of the image;

[0096] The first input unit is used to input the features of each frame of image into the time series network of the teacher model and output features with temporal sequence.

[0097] The third input unit is used to input the temporal features into the recognition network of the teacher model and output the expression recognition result.

[0098] The loss calculation unit is used to obtain the loss function value based on the expression recognition result and the emotion label of the complete macro expression sequence;

[0099] The parameter update unit is used to update the model parameters of the teacher model according to the loss function value until the pre-training of the teacher model is completed.

[0100] In one embodiment, the second training module 14 includes:

[0101] The first training unit is used to input the partial macro expression sequence into the student model, and to perform the first stage of training on the student model under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model.

[0102] The parameter freezing unit is used to freeze the parameters of other layers in the student model, excluding the recognition network, after the first stage of training.

[0103] The second training unit is used to input the micro-expression sequence into a student model with some parameters frozen for the second stage of training, and to adjust the parameters of the recognition network of the student model individually until the student model meets the preset convergence condition to obtain a micro-expression recognition model.

[0104] In one embodiment, the partial macro expression sequence is the data from the beginning of the complete macro expression sequence.

[0105] Another embodiment of the present invention provides a micro-expression recognition device, such as... Figure 7 As shown, device 2 includes:

[0106] Image acquisition module 21 is used to acquire the image sequence to be identified;

[0107] The recognition module 22 is used to input the image sequence to be recognized into a pre-trained micro-expression recognition model for micro-expression recognition, and output the micro-expression recognition result of the image sequence;

[0108] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0109] The image acquisition module 21 and the recognition module 22 are connected. The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the micro-expression recognition execution process. For the specific implementation of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0110] Another embodiment of the present invention provides an electronic device, such as... Figure 8 As shown, the electronic device 10 includes:

[0111] One or more processors 110 and memory 120, Figure 8 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0112] Processor 110 is used to perform various control logics of electronic device 10. It can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0113] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the training method of the micro-expression recognition model in the embodiments of the present invention. The processor 110 executes various functional applications and data processing of the electronic device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, that is, implementing the training method of the micro-expression recognition model or the micro-expression recognition method in the above method embodiments.

[0114] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the electronic device 10 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0115] One or more units are stored in memory 120. When executed by one or more processors 110, they perform the training method of the micro-expression recognition model in any of the above method embodiments, for example, performing the following steps:

[0116] Obtain complete macro-expression sequences and micro-expression sequences, and filter out the macro-expression sequences that match the emotion tags of the micro-expression sequences from the complete macro-expression sequences;

[0117] Construct a teacher model and a student model, which have the same neural network structure;

[0118] The complete macro-expression sequence is input into the teacher model for pre-training.

[0119] The macro-expression sequence and micro-expression sequence are used as training data. Under the guidance of the pre-trained teacher model, the student model is trained until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0120] In one embodiment, after obtaining the complete macro-expression sequence and micro-expression sequence, the method further includes:

[0121] The complete macro-expression sequence and micro-expression sequence are subjected to face cropping and alignment processing.

[0122] In one embodiment, inputting the complete macro-expression sequence into the teacher model for pre-training includes:

[0123] The complete macro-expression sequence is input into the feature extraction network of the teacher model to extract the features of each frame of the image;

[0124] The features of each frame of the image are input into the time series network of the teacher model, and the output features with temporal sequence are generated.

[0125] The temporal features are input into the recognition network of the teacher model, and the expression recognition result is output.

[0126] The loss function value is obtained based on the expression recognition result and the emotion label of the complete macro expression sequence;

[0127] The model parameters of the teacher model are updated based on the loss function value until the pre-training of the teacher model is completed.

[0128] In one embodiment, the step of using the partial macro-expression sequence and micro-expression sequence as training data to train the student model under the guidance of a pre-trained teacher model until the student model meets a preset convergence condition, thus completing model training and obtaining a micro-expression recognition model, includes:

[0129] The partial macro expression sequence is input into the student model, and the student model is trained in the first stage under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model.

[0130] Freeze the parameters of the student model, excluding the recognition network, after the first phase of training.

[0131] The micro-expression sequence is input into a student model with some parameters frozen for the second stage of training. The parameters of the recognition network of the student model are adjusted individually until the student model meets the preset convergence condition, thus obtaining the micro-expression recognition model.

[0132] In one embodiment, the partial macro expression sequence is the data from the beginning of the complete macro expression sequence.

[0133] Alternatively, the micro-expression recognition method in any of the above method embodiments can be executed, for example, by performing the following steps:

[0134] Obtain the image sequence to be identified;

[0135] The image sequence to be identified is input into a pre-trained micro-expression recognition model for micro-expression recognition, and the micro-expression recognition result of the image sequence is output.

[0136] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0137] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they perform the training method for the micro-expression recognition model in any of the above method embodiments, for example, performing the following steps:

[0138] Obtain complete macro-expression sequences and micro-expression sequences, and filter out the macro-expression sequences that match the emotion tags of the micro-expression sequences from the complete macro-expression sequences;

[0139] Construct a teacher model and a student model, which have the same neural network structure;

[0140] The complete macro-expression sequence is input into the teacher model for pre-training.

[0141] The macro-expression sequence and micro-expression sequence are used as training data. Under the guidance of the pre-trained teacher model, the student model is trained until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model.

[0142] In one embodiment, after obtaining the complete macro-expression sequence and micro-expression sequence, the method further includes:

[0143] The complete macro-expression sequence and micro-expression sequence are subjected to face cropping and alignment processing.

[0144] In one embodiment, inputting the complete macro-expression sequence into the teacher model for pre-training includes:

[0145] The complete macro-expression sequence is input into the feature extraction network of the teacher model to extract the features of each frame of the image;

[0146] The features of each frame of the image are input into the time series network of the teacher model, and the output features with temporal sequence are generated.

[0147] The temporal features are input into the recognition network of the teacher model, and the expression recognition result is output.

[0148] The loss function value is obtained based on the expression recognition result and the emotion label of the complete macro expression sequence;

[0149] The model parameters of the teacher model are updated based on the loss function value until the pre-training of the teacher model is completed.

[0150] In one embodiment, the step of using the partial macro-expression sequence and micro-expression sequence as training data to train the student model under the guidance of a pre-trained teacher model until the student model meets a preset convergence condition, thus completing model training and obtaining a micro-expression recognition model, includes:

[0151] The partial macro expression sequence is input into the student model, and the student model is trained in the first stage under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model.

[0152] Freeze the parameters of the student model, excluding the recognition network, after the first phase of training.

[0153] The micro-expression sequence is input into a student model with some parameters frozen for the second stage of training. The parameters of the recognition network of the student model are adjusted individually until the student model meets the preset convergence condition, thus obtaining the micro-expression recognition model.

[0154] In one embodiment, the partial macro expression sequence is the data from the beginning of the complete macro expression sequence.

[0155] Alternatively, the micro-expression recognition method in any of the above method embodiments can be executed, for example, by performing the following steps:

[0156] Obtain the image sequence to be identified;

[0157] The image sequence to be identified is input into a pre-trained micro-expression recognition model for micro-expression recognition, and the micro-expression recognition result of the image sequence is output.

[0158] The pre-trained micro-expression recognition model is obtained using the training method described above.

[0159] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0160] In summary, the micro-expression recognition model training method, micro-expression recognition method, and related equipment disclosed in this invention include the following training method: acquiring complete macro-expression sequences and micro-expression sequences; selecting a portion of the macro-expression sequences from the complete macro-expression sequences that match the emotion labels of the micro-expression sequences; constructing a teacher model and a student model, which have the same neural network structure; inputting the complete macro-expression sequences into the teacher model for pre-training; and using the portion of the macro-expression sequences and micro-expression sequences as training data to train the student model under the guidance of the pre-trained teacher model until the student model meets a preset convergence condition, thus completing the model training. By combining micro-expressions and macro-expressions and using a teacher-student model within a knowledge distillation framework for model training in the micro-expression recognition task, the problems of insufficient micro-expression samples and low feature strength can be effectively compensated for, thereby improving the accuracy of micro-expression recognition.

[0161] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0162] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A training method for a micro-expression recognition model, characterized in that, include: Obtain complete macro-expression sequences and micro-expression sequences, and filter out the macro-expression sequences that match the emotion tags of the micro-expression sequences from the complete macro-expression sequences; The partial macro expression sequence is the data from the beginning of the complete macro expression sequence, arranged in chronological order. Construct a teacher model and a student model, which have the same neural network structure; The complete macro-expression sequence is input into the teacher model for pre-training. The macro-expression sequence and micro-expression sequence are used as training data. Under the guidance of the pre-trained teacher model, the student model is trained until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model. The process involves using the partial macro-expression sequence and micro-expression sequence as training data, and training the student model under the guidance of the pre-trained teacher model until the student model meets a preset convergence condition, thus completing the model training and obtaining a micro-expression recognition model. This includes: The aforementioned macro-expression sequence is input into the student model. Under the guidance of the pre-trained teacher model, the student model undergoes a first-stage training, adjusting the parameters of all layers in the student model. Specifically, the student model learns using soft labels generated by the teacher model, and feature matching is used to make the feature representation of the intermediate layers of the student model similar to the feature representation of the teacher model under the same input. Freeze the parameters of the student model, excluding the recognition network, after the first phase of training. The micro-expression sequence is input into a student model with some parameters frozen for the second stage of training. The parameters of the recognition network of the student model are adjusted individually until the student model meets the preset convergence condition, thus obtaining the micro-expression recognition model.

2. The training method for the micro-expression recognition model according to claim 1, characterized in that, After obtaining the complete macro-expression sequence and micro-expression sequence, the method further includes: The complete macro-expression sequence and micro-expression sequence are subjected to face cropping and alignment processing.

3. The training method for the micro-expression recognition model according to claim 1, characterized in that, The step of inputting the complete macro-expression sequence into the teacher model for pre-training includes: The complete macro-expression sequence is input into the feature extraction network of the teacher model to extract the features of each frame of the image; The features of each frame of the image are input into the time series network of the teacher model, and the output features with temporal sequence are generated. The temporal features are input into the recognition network of the teacher model, and the expression recognition result is output. The loss function value is obtained based on the expression recognition result and the emotion label of the complete macro expression sequence; The model parameters of the teacher model are updated based on the loss function value until the pre-training of the teacher model is completed.

4. A micro-expression recognition method, characterized in that, include: Obtain the image sequence to be identified; The image sequence to be identified is input into a pre-trained micro-expression recognition model for micro-expression recognition, and the micro-expression recognition result of the image sequence is output. The pre-trained micro-expression recognition model is obtained using the training method described in any one of claims 1-3.

5. A training device for a micro-expression recognition model, characterized in that, include: The data acquisition module is used to acquire complete macro-expression sequences and micro-expression sequences, and to filter out a portion of the macro-expression sequences that are consistent with the emotion tags of the micro-expression sequences from the complete macro-expression sequences; the portion of the macro-expression sequences is the first part of the data arranged in chronological order from the complete macro-expression sequences. The model building module is used to build teacher and student models, which have the same neural network structure. The first training module is used to input the complete macro expression sequence into the teacher model for pre-training; The second training module is used to use the partial macro-expression sequence and micro-expression sequence as training data, and train the student model under the guidance of the pre-trained teacher model until the student model meets the preset convergence condition, thus completing the model training and obtaining the micro-expression recognition model. The second training module includes: The first training unit is used to input the partial macro expression sequence into the student model, and to perform the first stage of training on the student model under the guidance of the pre-trained teacher model, adjusting the parameters of all layers in the student model; specifically, the student model learns with the soft labels generated by the teacher model, and uses feature matching to make the feature representation of the intermediate layers of the student model similar to the feature representation of the teacher model under the same input. The parameter freezing unit is used to freeze the parameters of other layers in the student model, excluding the recognition network, after the first stage of training. The second training unit is used to input the micro-expression sequence into a student model with some parameters frozen for the second stage of training, and to adjust the parameters of the recognition network of the student model individually until the student model meets the preset convergence condition to obtain a micro-expression recognition model.

6. A micro-expression recognition device, characterized in that, include: The image acquisition module is used to acquire the image sequence to be identified; The recognition module is used to input the image sequence to be recognized into a pre-trained micro-expression recognition model for micro-expression recognition, and output the micro-expression recognition result of the image sequence; The pre-trained micro-expression recognition model is obtained using the training method described in any one of claims 1-3.

7. An electronic device, characterized in that, Includes at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the training method of the micro-expression recognition model according to any one of claims 1-3, or to perform the micro-expression recognition method according to claim 4.

8. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the training method of the micro-expression recognition model according to any one of claims 1-3, or to perform the micro-expression recognition method according to claim 4.

Images