Micro-expression recognition model construction method, micro-expression recognition method and related device

By constructing a micro-expression recognition model combining feature extraction and image generation, the problem of low micro-expression recognition accuracy in the financial and medical fields is solved, and a higher accuracy of micro-expression recognition and analysis capabilities are achieved.

CN118918626APending Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411061154.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the fields of finance and medical care, it is difficult to build a micro-expression recognition model that can achieve higher accuracy in the existing technology, mainly due to the difficulty in collecting micro-expression data sets and the short and small amplitude of micro-expression.

Method used

By obtaining the micro-expression sample frame sequence and expression analysis network, feature extraction and image generation training are carried out, and combined with image feature extraction network, micro-expression amplification network and expression analysis network, a micro-expression recognition model is built, and amplified image generation module and intensity control module are used to build a magnified image generation module and a strength control module. Optimize micro-expression amplification and recognition.

Benefits of technology

It realizes higher accuracy of micro-expression recognition in the fields of finance and medical care, improves the recognition and amplification capabilities of the micro-expression recognition model, and can more accurately capture and analyze subtle facial expression changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918626A_ABST
    Figure CN118918626A_ABST
Patent Text Reader

Abstract

The invention relates to the financial field, the medical field and the artificial intelligence technology field, and especially relates to a micro-expression recognition model construction method, a micro-expression recognition method and a related device. According to the micro-expression recognition model construction method, a micro-expression sample frame sequence and an expression analysis network need to be acquired firstly; inputting the neutral frame into an image feature extraction network subjected to feature extraction training for feature extraction to obtain a sample neutral feature; performing image generation training on the original micro-expression amplification network based on the sample neutral features and the top frame to obtain a micro-expression amplification network subjected to image generation training; and coupling the image feature extraction network, the micro-expression amplification network and the expression analysis network to obtain a micro-expression recognition model. Therefore, the micro-expression recognition model can achieve higher accuracy in actual application scenes such as finance and medical treatment, and on the basis, the micro-expression recognition model is applied to processing the micro-expression recognition frame, so that a micro-expression recognition result with higher accuracy can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of finance, medicine and artificial intelligence, and in particular to a micro-expression recognition model construction method, a micro-expression recognition method and related devices. Background Art

[0002] The application of micro-expression recognition in the financial and medical fields has significant potential and practical value. In the financial field, micro-expression amplification technology can be used as an auxiliary tool in business negotiations or contract signing. In the medical field, medical staff can use this technology to identify subtle changes in the subject's expression that are not easy to detect, as these changes can be an important indicator of changes in psychological state.

[0003] In related technologies, in order to achieve better performance in practical application scenarios such as finance and medicine, it is necessary to pre-train the micro-expression recognition model. In this process, since the micro-expressions in practical application scenarios are short-lived and small in amplitude, it is very difficult to pre-collect a reliable micro-expression data set for training the micro-expression recognition model. Due to this limitation, how to build a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medicine is still a problem that needs to be solved in the industry. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a micro-expression recognition model construction method, a micro-expression recognition method and a related device, which can train a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medical care.

[0005] A method for constructing a micro-expression recognition model according to a first aspect of the present invention includes:

[0006] Acquire a micro-expression sample frame sequence and an expression parsing network; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame, and the expression parsing network is used to extract micro-expression information of the image;

[0007] Inputting the neutral frame into the image feature extraction network that has been trained for feature extraction to extract features, thereby obtaining sample neutral features;

[0008] Performing image generation training on the original micro-expression amplification network based on the sample neutral features and the top frame to obtain the micro-expression amplification network that has undergone image generation training;

[0009] The image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression analysis network are coupled to construct the micro-expression recognition model.

[0010] The micro-expression recognition model construction method according to the embodiment of the present invention has at least the following beneficial effects:

[0011] The method for constructing a micro-expression recognition model of an embodiment of the present invention requires first obtaining a micro-expression sample frame sequence and an expression parsing network; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame; the neutral frame is input into an image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features; based on the sample neutral features and the top frame, the original micro-expression magnification network is trained for image generation to obtain a micro-expression magnification network that has been trained for image generation; the image feature extraction network that has been trained for feature extraction, the micro-expression magnification network that has been trained for image generation, and the expression parsing network are coupled to construct a micro-expression recognition model. In this way, a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medical care can be trained.

[0012] According to some embodiments of the present invention, the micro-expression magnification network includes an enlarged image generation module, and the image generation training is performed on the original micro-expression magnification network based on the sample neutral features and the top frame to obtain the micro-expression magnification network after image generation training, including:

[0013] Obtaining the magnification code corresponding to the neutral frame and the original micro-expression magnification network;

[0014] Inputting the sample neutral feature and the magnification code into an amplified image generation module to generate an image, thereby obtaining a neutral frame amplified image;

[0015] Based on the top frame, the enlarged image of the neutral frame is subjected to micro-expression identification to obtain a micro-expression identification result and a micro-expression deviation parameter;

[0016] The model parameters of the magnified image generation module are adjusted based on the micro-expression deviation parameters, and the sample neutral features and the magnification factor encoding are input into the magnified image generation module for image generation, until the micro-expression magnification network meets the first preset condition, thereby obtaining the micro-expression magnification network trained for image generation.

[0017] According to some embodiments of the present invention, the micro-expression amplification network further includes an intensity control module, and the obtaining of the amplification factor code corresponding to the neutral frame includes:

[0018] Get the magnification factor;

[0019] The magnification factor is input into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor.

[0020] According to some embodiments of the present invention, before inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor, the method further includes:

[0021] Get random noise;

[0022] The step of inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor comprises:

[0023] The random noise and the amplification factor are input into the intensity control module, so that the intensity control module determines the amplification code according to the amplification factor and the random noise.

[0024] According to some embodiments of the present invention, the adjusting the model parameters of the magnified image generation module based on the micro-expression deviation parameter, returning to execute inputting the sample neutral feature and the magnification factor encoding into the magnified image generation module for image generation, until the micro-expression magnification network meets a first preset condition, and obtaining the micro-expression magnification network trained for image generation, includes:

[0025] Performing a first adjustment on the model parameters of the enlarged image generation module based on the micro-expression deviation parameter, and performing a second adjustment on the model parameters of the intensity control module;

[0026] After the first adjustment and the second adjustment, the step of returning to the step of inputting the sample neutral feature and the magnification factor encoding into the enlarged image generation module for image generation is performed until the micro-expression amplification network meets a first preset condition, thereby obtaining the micro-expression amplification network trained for image generation.

[0027] According to some embodiments of the present invention, the micro-expression sample frame sequence includes a plurality of neutral frames, and the step of obtaining the magnification factor includes:

[0028] Acquire the magnification factor matched to each of the neutral frames;

[0029] The step of inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor comprises:

[0030] The magnification factor corresponding to each of the neutral frames is sequentially input into the intensity control module, so that the intensity control module determines the magnification code corresponding to each of the neutral frames according to the magnification factor corresponding to each of the neutral frames.

[0031] According to some embodiments of the present invention, before inputting the neutral frame into the image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features, it also includes performing feature extraction training on the image feature extraction network, specifically including:

[0032] Acquire a feature extraction training image and the original image feature extraction network;

[0033] Inputting the feature extraction training image into the image feature extraction network to extract image features to obtain image training features;

[0034] Inputting the feature extraction training image into a pre-trained image coding network to extract prior features and obtain image prior features;

[0035] Based on the image prior features, the image training features are subjected to feature distribution identification to obtain feature distribution identification results and feature deviation parameters;

[0036] The model parameters of the image feature extraction network are adjusted based on the feature deviation parameters, and the execution returns to input the feature extraction training image into the image feature extraction network for image feature extraction until the image feature extraction network meets the second preset condition, and the parameters of the image feature extraction network are frozen to obtain the image feature extraction network that has been trained for feature extraction.

[0037] A micro-expression recognition method according to a second aspect of the present invention includes:

[0038] Get micro-expression recognition frames;

[0039] Inputting the micro-expression recognition frame into a micro-expression recognition model, wherein the micro-expression recognition model is formed by coupling the image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression analysis network; wherein the micro-expression recognition model is constructed by the micro-expression recognition model construction method described in the first aspect of the embodiment;

[0040] Extracting features from the micro-expression recognition frame using the image feature extraction network that has been trained for feature extraction, to obtain micro-expression recognition features;

[0041] Inputting the micro-expression recognition features into the micro-expression magnification network trained by image generation to perform micro-expression magnification processing to obtain a micro-expression magnified image;

[0042] The expression analysis is performed on the micro-expression magnified image based on the expression analysis network to obtain a micro-expression recognition result.

[0043] The micro-expression recognition method according to the embodiment of the present invention has at least the following beneficial effects:

[0044] The micro-expression recognition method of the embodiment of the present application needs to first obtain a micro-expression recognition frame; input the micro-expression recognition frame into a micro-expression recognition model, and the micro-expression recognition model is formed by coupling an image feature extraction network trained for feature extraction, a micro-expression magnification network trained for image generation, and an expression analysis network; wherein the micro-expression recognition model is constructed by the micro-expression recognition model construction method of the first embodiment; the micro-expression recognition frame is subjected to feature extraction by the image feature extraction network trained for feature extraction to obtain micro-expression recognition features; the micro-expression recognition features are input into the micro-expression magnification network trained for image generation to perform micro-expression magnification processing to obtain a micro-expression magnified image; the micro-expression magnified image is subjected to expression analysis based on the expression analysis network to obtain a micro-expression recognition result. Since the micro-expression recognition model is constructed by the micro-expression recognition model construction method of the first embodiment, the micro-expression recognition model can achieve higher accuracy in practical application scenarios such as finance and medical care. On this basis, the micro-expression recognition model is applied to process micro-expression recognition frames to obtain micro-expression recognition results with higher accuracy.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the micro-expression recognition method as described in any one of the embodiments of the first aspect of the present invention, or the micro-expression recognition method as described in the embodiments of the second aspect of the present invention.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement a micro-expression recognition method as described in any one of the embodiments of the first aspect of the present invention, or a micro-expression recognition method as described in the embodiments of the second aspect of the present invention.

[0047] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0049] Figure 1 A schematic diagram of a process for constructing a micro-expression recognition model provided by an embodiment of the present invention;

[0050] Figure 2 Another schematic diagram of a process for constructing a micro-expression recognition model provided by an embodiment of the present invention;

[0051] Figure 3 For the embodiment of the present invention Figure 1 Schematic diagram of the process of step S103;

[0052] Figure 4 For the embodiment of the present invention Figure 3 Schematic diagram of the process of step S301;

[0053] Figure 5 For the embodiment of the present invention Figure 4 Schematic diagram of the process of step S401;

[0054] Figure 6 For the embodiment of the present invention Figure 4 Schematic diagram of the process of step S402;

[0055] Figure 7 For the embodiment of the present invention Figure 3 Schematic diagram of the process of step S304;

[0056] Figure 8 A schematic diagram of a process flow of a micro-expression recognition method provided by an embodiment of the present invention;

[0057] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0059] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0060] In the description of the present invention, it should be understood that descriptions involving orientation, such as up, down, left, right, front, back, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0061] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0062] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution. In addition, the identification of specific steps below does not represent a limitation on the order of steps and execution logic, and the execution order and execution logic between each step should be understood and inferred with reference to the contents described in the embodiment.

[0063] Micro-expressions refer to the brief and subtle changes in facial muscles when people's emotions change. Micro-expressions are a non-verbal expression of emotions that can reflect people's true inner feelings and psychological state, and help improve communication effectiveness and trust. However, because micro-expressions occur quickly, with small amplitudes and are difficult to detect, people often find it difficult to accurately identify and understand each other's micro-expressions.

[0064] The application of micro-expression recognition in the financial and medical fields has significant potential and practical value. In the financial field, micro-expression amplification technology can be used as an auxiliary tool in business negotiations or contract signing processes. Since people may show subtle emotional changes when negotiating or making decisions, these changes may have a significant impact on the negotiation results or credit assessment. By identifying and amplifying these micro-expressions, financial professionals can more accurately capture customers' abnormal emotions or potential dishonest behavior, thereby improving the accuracy of risk assessment and the quality of decision-making. In addition, this technology may also be used for investor sentiment analysis to help financial institutions better understand market dynamics and investor behavior.

[0065] In the medical field, the application of micro-expression amplification technology is also extensive. In mental health testing and face-to-face consultations, medical staff can use this technology to identify subtle changes in the subject's expression that are not easy to detect. These changes can be important indicators of changes in mental state. For example, for subjects with depression, anxiety or other emotional disorders, subtle changes in micro-expressions may reveal their true feelings, helping medical staff to make more accurate diagnoses and develop personalized treatment plans. In addition, in doctor-patient communication, medical staff can better understand the needs and emotional state of subjects by identifying their micro-expressions, thereby establishing more effective communication and trust relationships and improving the efficiency and quality of treatment.

[0066] In related technologies, in order to achieve better performance in practical application scenarios such as finance and medicine, it is necessary to pre-train the micro-expression recognition model. In this process, since the micro-expressions in practical application scenarios are short-lived and small in amplitude, it is very difficult to pre-collect a reliable micro-expression data set for training the micro-expression recognition model. Due to this limitation, how to build a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medicine is still a problem that needs to be solved in the industry.

[0067] Some researchers use micro-expression amplification as a starting point to increase recognition rate. Most of these methods only amplify features to help the network classify micro-expressions. These methods are limited to the simulation of laboratory scenes and are still not very effective in practical applications.

[0068] Later, some researchers directly magnified the micro-expression images into macro-expression images with corresponding emotion labels, and achieved ideal results in outdoor scene tests. However, due to the setting of double-frame (initial frame and top frame of micro-expression sequence) input in this method, if the test object is magnified, the effect often depends on the difference between the two frames. Since expressions are subtle and complex, there is no unified paradigm for the amplitude of micro-expressions under different emotions and different samples, so some samples will be magnified, resulting in poor results.

[0069] The present invention proposes a micro-expression recognition model construction method, a micro-expression recognition method and related devices, which can improve the recognition and amplification capabilities of micro-expressions, and train a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medicine, thereby providing a new perspective and tool in the financial and medical fields, helping professionals to have a deeper understanding of human emotions and behaviors, and thus make more wise and effective decisions in their respective fields.

[0070] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a micro-expression recognition model construction method, a micro-expression recognition method and a related device, which can train a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medical care.

[0071] Reference Figure 1 The micro-expression recognition model construction method provided by the embodiment of the present invention may include but is not limited to:

[0072] Step S101, obtaining a micro-expression sample frame sequence and an expression parsing network; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame, and the expression parsing network is used to extract micro-expression information of the image;

[0073] Step S102, inputting the neutral frame into an image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features;

[0074] Step S103, performing image generation training on the original micro-expression amplification network based on the sample neutral features and the top frame to obtain a micro-expression amplification network that has undergone image generation training;

[0075] Step S104, coupling the image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression analysis network to construct a micro-expression recognition model.

[0076] The method for constructing a micro-expression recognition model of an embodiment of the present invention requires first obtaining a micro-expression sample frame sequence and an expression parsing network; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame; the neutral frame is input into an image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features; based on the sample neutral features and the top frame, the original micro-expression magnification network is trained for image generation to obtain a micro-expression magnification network that has been trained for image generation; the image feature extraction network that has been trained for feature extraction, the micro-expression magnification network that has been trained for image generation, and the expression parsing network are coupled to construct a micro-expression recognition model. In this way, a micro-expression recognition model that can achieve higher accuracy in practical application scenarios such as finance and medical care can be trained.

[0077] In step S101 of some embodiments, a micro-expression sample frame sequence and an expression parsing network are obtained; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame. It should be noted that the micro-expression sample frame sequence refers to a series of images of facial expression changes captured in a specific situation, which record the continuous change from the neutral state of the expression to the vertex state of the expression. In this process, the neutral frame represents a certain moment in the transition process of the expression from the initial state to the vertex state. In the expression sequence, the neutral frame may capture the moment when the expression intensity gradually increases, but has not yet reached the vertex, while the top frame is the end point of the expression change, showing the maximum degree of expression, such as obvious emotional expressions such as smiles, surprise, anger, etc. Acquiring these micro-expression sample frame sequences is a prerequisite for subsequent feature extraction and image generation training. The expression parsing network is a deep learning model specially designed to analyze and recognize facial expressions, and extract corresponding micro-expression information for the images input therein. This network can be obtained through a large amount of training and can recognize and classify different facial expressions, including micro-expressions. Expression parsing networks can be acquired by training existing deep learning models, or by using pre-trained models and fine-tuning them to suit specific application requirements.

[0078] In some embodiments, in order to obtain a micro-expression sample frame sequence, a high-precision camera device may be used to record in a controlled or natural environment. During the recording process, the participants whose micro-expressions are being recorded may be asked to watch specific videos or pictures to induce them to produce micro-expressions. These recorded samples need to be screened and pre-processed to ensure that each frame is clear, accurate, and can represent different micro-expression changes.

[0079] In some embodiments, the micro-expression sample frame sequence may further include an initial frame, which is the first frame in a video or animation sequence, marking the beginning of the sequence. In expression analysis, the initial frame may be a neutral expression or a state without expression, serving as the starting point of expression changes.

[0080] Reference Figure 2 According to some embodiments of the present invention, before step S102 inputs the neutral frame into the image feature extraction network that has been trained for feature extraction to extract features and obtain the sample neutral features, it also includes training the image feature extraction network for feature extraction, specifically including:

[0081] Step S201, obtaining a feature extraction training image and an original image feature extraction network;

[0082] Step S202, inputting the feature extraction training image into the image feature extraction network to extract image features and obtain image training features;

[0083] Step S203, inputting the feature extraction training image into a pre-trained image coding network to extract prior features and obtain image prior features;

[0084] Step S204, performing feature distribution identification on the image training features based on the image prior features to obtain feature distribution identification results and feature deviation parameters;

[0085] Step S205, adjust the model parameters of the image feature extraction network based on the feature deviation parameters, return to execute the feature extraction training image input into the image feature extraction network for image feature extraction, until the image feature extraction network meets the second preset condition, freeze the parameters of the image feature extraction network, and obtain the image feature extraction network that has been trained for feature extraction.

[0086] In step S201 of some embodiments, the embodiments of the present application need to obtain image data for training the image feature extraction network. These image data, i.e., feature extraction training images, contain facial images of various expression states, usually including micro-expressions and macro-expressions. At the same time, an original image feature extraction network is required. This network is untrained and will be used as a basic model for subsequent training and optimization.

[0087] In step S202 of some embodiments, the feature extraction training images are input into an image feature extraction network to extract image training features. The image training features are learned from the feature extraction training images and can represent the expression information in the image.

[0088] In step S203 of some embodiments, in order to improve the accuracy and robustness of feature extraction, it is also necessary to input the feature extraction training image into a pre-trained image coding network to extract image prior features. This pre-trained network already has a certain feature extraction capability and can provide image prior features. The image prior features can be used as a reference to help optimize and guide the subsequent feature extraction process.

[0089] In some more specific embodiments, the pre-trained image encoding network may be a pre-trained CLIP image encoder.

[0090] In step S204 of some embodiments, based on the image prior features obtained from the pre-trained image coding network, feature distribution identification is performed on the image training features. The purpose of this step is to evaluate and compare the difference between the image training features and the image prior features to obtain a feature distribution identification result. Through this comparison, the deviation between the training features and the prior features can be identified, and the feature deviation parameters can be calculated. Among them, the feature deviation parameters reflect the distribution of the image training features in the feature space, which provides a basis for the subsequent optimization of the image coding network.

[0091] In step S205 of some embodiments, the model parameters of the image feature extraction network are adjusted using feature deviation parameters. This adjustment process involves an optimization algorithm, such as gradient descent, to minimize feature deviations, so that the image training features are closer to the image prior features. Through repeated parameter adjustment and feature extraction training, until the image feature extraction network meets the second preset condition, that is, reaches a certain performance standard. Once the image feature extraction network meets these conditions, the parameters of the image feature extraction network can be frozen to obtain an image feature extraction network that has been trained for feature extraction. This image feature extraction network that has been trained for feature extraction can more accurately extract expression features in the image, providing strong support for subsequent micro-expression recognition and analysis.

[0092] In step S102 of some embodiments, the neutral frame is input into an image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features. It should be noted that step S102 is a key link in the process of building a micro-expression recognition model, and its main purpose is to extract key features from the acquired micro-expression sample frame sequence. The core of this step is to use an image feature extraction network that has been trained for feature extraction, input the neutral frame into the network, and thus obtain sample neutral features corresponding to the neutral frame. In this process, it is first necessary to ensure that the image feature extraction network has been fully trained and can accurately identify and extract key features in the image. This network can use deep learning technology, such as convolutional neural network (CNN), which can learn hierarchical feature representation of images. Since the image feature extraction network has been trained for feature extraction, the image feature extraction network can identify details and patterns in the image and encode this information into a set of feature vectors.

[0093] After the neutral frame is input into this trained image feature extraction network, the network will output a set of features that capture the expression characteristics and facial structure in the neutral frame, namely the sample neutral features. These sample neutral features are the basis for image generation and micro-expression amplification in subsequent steps.

[0094] In some more specific embodiments, in order to ensure the accuracy and reliability of feature extraction, the image feature extraction network may also be carefully adjusted and optimized, including selecting a suitable network architecture, adjusting network parameters, using data enhancement technology, etc.

[0095] In addition, the accuracy of feature extraction directly affects the performance of the micro-expression recognition model. Therefore, in some embodiments, the performance of the feature extraction network needs to be carefully evaluated, for example, by cross-validation, test set evaluation, etc. to ensure that the extracted features can effectively represent the changes in micro-expressions. In some cases, the embodiments of the present application can also further train or fine-tune the feature extraction network to adapt to specific application scenarios or improve its robustness in complex environments.

[0096] In summary, step S102 is a basic step in the construction of the micro-expression recognition model, which provides necessary feature information for subsequent image generation and micro-expression amplification. Therefore, this step can ensure that the micro-expression recognition model has higher accuracy and reliability in subsequent recognition and analysis.

[0097] In step S103 of some embodiments, the original micro-expression magnification network is trained for image generation based on the sample neutral features and the top frame to obtain a micro-expression magnification network that has been trained for image generation. It should be noted that step S103 is the image generation training stage in the process of building the micro-expression recognition model. The purpose of this step is to use the extracted neutral frame features and the top frame to train the micro-expression magnification network, thereby obtaining a micro-expression magnification network that has been trained for image generation. In this process, it is first necessary to perform image generation training on the original micro-expression magnification network with the top frame in step S101 and the sample neutral features obtained in step S102, so as to guide the micro-expression magnification network to learn how to generate the magnified micro-expression state from the neutral expression state.

[0098] In some embodiments, the micro-expression amplification network can be a deep learning model, which can adopt a generative adversarial network (GAN) or other generative model architecture. The micro-expression amplification network needs to learn the feature amplification rules of micro-expressions through a large amount of training data. During the training process of image generation training of the original micro-expression amplification network based on sample neutral features and top frames, the micro-expression amplification network will try to generate images, which should be as close as possible to the real micro-expression amplification effect in visual terms. At the same time, the discriminator or similar evaluation mechanism will evaluate the quality of the generated image and provide feedback to guide the generation network to make corresponding adjustments.

[0099] In order to achieve effective training, the original micro-expression magnification network needs to be continuously optimized iteratively. This includes adjusting the parameters of the network, optimizing the loss function, using a suitable optimization algorithm, etc. The loss function can include the pixel-level difference between the generated image and the top frame, the similarity at the feature level, or other indicators reflecting the image quality. In this way, the micro-expression magnification network will gradually learn how to accurately magnify the micro-expression features while maintaining the naturalness and realism of the image until the micro-expression magnification network has been fully trained, and a micro-expression magnification network trained by image generation can be obtained. The micro-expression magnification network trained by image generation can generate high-quality magnified images based on the features and magnification requirements of the neutral frame. These images will provide support for the final expression parsing and recognition. Through this step, the micro-expression recognition model will have higher accuracy and reliability when dealing with subtle expression changes.

[0100] In some embodiments, in order to improve the stability of training and the diversity of generated images, random noise can be introduced as part of the network input, or methods such as Cyclic Consistency Loss can be used to maintain the diversity and detail richness of the generated images.

[0101] Reference Figure 3 According to some embodiments of the present invention, the micro-expression magnification network includes a magnified image generation module. Step S103 performs image generation training on the original micro-expression magnification network based on the sample neutral features and the top frame to obtain a micro-expression magnification network that has been trained for image generation, which may include but is not limited to:

[0102] Step S301, obtaining the magnification code corresponding to the neutral frame and the original micro-expression magnification network;

[0103] Step S302, inputting the sample neutral feature and the magnification code into the magnified image generation module to generate an image, thereby obtaining a neutral frame magnified image;

[0104] Step S303, performing micro-expression identification on the enlarged image of the neutral frame based on the top frame to obtain a micro-expression identification result and a micro-expression deviation parameter;

[0105] Step S304, adjusting the model parameters of the magnified image generation module based on the micro-expression deviation parameter, returning to execute the encoding of the sample neutral features and the magnification factor into the magnified image generation module for image generation, until the micro-expression magnification network meets the first preset condition, and obtaining the micro-expression magnification network trained for image generation.

[0106] In step S301 of some embodiments, it is necessary to obtain the magnification codes corresponding to the neutral frame, which determine the degree of micro-expression magnification. In addition, the original micro-expression magnification network refers to an untrained basic model, which will be used for subsequent image generation training.

[0107] Reference Figure 4 According to some embodiments of the present invention, the micro-expression magnification network further includes an intensity control module. Step S301 obtains the magnification factor encoding corresponding to the neutral frame, which may include but is not limited to:

[0108] Step S401, obtaining a magnification factor;

[0109] Step S402: input the magnification factor into the intensity control module, so that the intensity control module determines the magnification code according to the magnification factor.

[0110] In some embodiments of the present invention, the construction of the micro-expression amplification network not only includes an amplified image generation module, but also integrates an intensity control module to achieve fine control of the amplification process. The introduction of this module enables the micro-expression amplification network to dynamically adjust the amplification intensity according to different needs and conditions, thereby generating micro-expression images that are more in line with expectations.

[0111] In step S401 of some embodiments, a magnification factor is obtained. The magnification factor is a key parameter that determines the degree of micro-expression magnification. The magnification factor can be obtained based on a variety of methods, for example, determined by user input, pre-set according to the needs of a specific application scenario, or dynamically calculated by analyzing the characteristics of micro-expressions through algorithms. The selection of the magnification factor will affect the detail clarity and expression intensity of the final magnified image.

[0112] In step S402 of some embodiments, after obtaining the magnification factor, it is input into the intensity control module. The intensity control module is responsible for determining the final magnification code according to the magnification factor. The intensity control module may use a series of algorithms and models, such as nonlinear mapping functions or neural networks, to convert the magnification factor into a specific magnification code. The magnification code is a set of instructions or parameters that instruct the magnified image generation module on how to magnify the input neutral frame.

[0113] Through the coordinated work of step S401 and step S402, the micro-expression amplification network can achieve precise control of the amplification process, thereby improving the flexibility and effectiveness of micro-expression recognition technology. The introduction of this intensity control mechanism enables the micro-expression amplification network to adapt to various application scenarios and meet the different needs of different users and different situations for micro-expression amplification.

[0114] Reference Figure 5 According to some embodiments of the present invention, the micro-expression sample frame sequence includes multiple neutral frames, and step S401 obtains the magnification factor, which may include but is not limited to:

[0115] Step S501, obtaining a magnification factor matched to each neutral frame;

[0116] Step S402: input the magnification factor into the intensity control module, so that the intensity control module determines the magnification code according to the magnification factor, which may include but is not limited to:

[0117] Step S502 , inputting the magnification factor corresponding to each neutral frame into the intensity control module in sequence, so that the intensity control module determines the magnification code corresponding to each neutral frame according to the magnification factor corresponding to each neutral frame.

[0118] In some embodiments of the present invention, the micro-expression sample frame sequence is composed of a plurality of neutral frames, which capture the facial expressions of individuals in specific situations, usually showing a change from a state without obvious emotional changes to a state with gradually increasing expression intensity. For these neutral frames, each appropriate magnification factor can be determined so that the characteristics of the micro-expressions can be appropriately enhanced in the subsequent magnification process.

[0119] In step S501 of some embodiments, a magnification factor matching each neutral frame is obtained. This step involves analyzing the features of each neutral frame and determining a suitable magnification factor based on these features. The magnification factor reflects the magnification range from the neutral frame to the top frame (i.e., the maximum degree of expression change), and can be automatically calculated by an algorithm based on pre-set standards, or manually set based on expert judgment. For example, if the expression changes in a neutral frame are more subtle, a larger magnification factor may be required to ensure that the enlarged image can clearly show these changes.

[0120] In step S502 of some embodiments, in order to ensure that each neutral frame can be properly magnified, the magnification factor corresponding to each neutral frame is sequentially input into the intensity control module. In this way, the intensity control module can determine the corresponding magnification code according to the value of each magnification factor. This process allows each neutral frame to be personalized for magnification, ensuring that the generated magnified image meets the magnification requirements while maintaining the naturalness and accuracy of the image.

[0121] By processing each neutral frame one by one, the intensity control module can generate a customized magnification code for each neutral frame in the micro-expression sample frame sequence. These codes are then used to guide the magnified image generation module to generate a series of magnified neutral frame magnified images, which visually show the characteristics of micro-expressions more clearly, thereby providing richer information for the recognition and analysis of micro-expressions. This personalized magnification strategy based on the characteristics of each frame improves the flexibility and effectiveness of the micro-expression magnification network, enabling it to adapt to different micro-expression samples and analysis requirements.

[0122] In some embodiments, the intensity control module not only responds to the input of the magnification factor, but also considers other factors, such as the complexity of the image content, the expected visual effect, etc., to generate the magnification code that best suits the current situation. The generated magnification code is then used to control the magnified image generation module to ensure that the generated magnified image meets the magnification requirements while maintaining the naturalness and accuracy of the image.

[0123] Reference Figure 6 According to some embodiments of the present invention, before inputting the magnification factor into the intensity control module in step S402 so that the intensity control module determines the magnification code according to the magnification factor, the method further includes:

[0124] Step S601, obtaining random noise;

[0125] Step S402: input the magnification factor into the intensity control module, so that the intensity control module determines the magnification code according to the magnification factor, which may include but is not limited to:

[0126] Step S602: inputting the random noise and the amplification factor into the intensity control module, so that the intensity control module determines the amplification code according to the amplification factor and the random noise.

[0127] In some embodiments of the present invention, the intensity control module of the micro-expression amplification network can not only determine the amplification factor encoding based on the amplification factor, but also introduce random noise to increase the diversity and naturalness of the generation process.

[0128] In step S601 of some embodiments, before inputting the magnification factor into the intensity control module, random noise is first acquired. Random noise can simulate small changes and uncertainties in the real world, which helps to avoid excessive smoothness or pattern repetition of the generated image. The acquisition of random noise can be achieved in many ways, such as using a pseudo-random number generator to generate a set of noise data with specific distribution characteristics.

[0129] In step S602 of some embodiments, in order to further improve the controllability of the amplification process and the naturalness of the result, the random noise previously obtained is also inputted when the amplification factor is inputted into the strength control module. In this way, when the strength control module determines the amplification code, it not only considers the demand for amplification, but also incorporates the random factor. This method of combining the amplification factor and the random noise enables the strength control module to adjust the amplification code more flexibly, and the generated amplification code not only reflects the demand for amplification, but also includes the subtle changes caused by the random noise.

[0130] In this way, the intensity control module can output richer and more realistic magnification codes, which are then used to guide the magnified image generation module to generate the final magnified image. This method of combining magnification factors and random noise improves the generation quality and applicability of the micro-expression magnification network, allowing the network to adapt to more complex and varied micro-expression magnification tasks and generate more accurate and natural micro-expression images.

[0131] In some more specific embodiments, the random noise can also be controlled to change randomly within a certain value range, for example, between 0-1, and the random noise between 0-1 can be used to represent a random variable of the expression magnification factor varying between minimum and maximum, so as to simulate various magnification factors that may be required in actual application scenarios.

[0132] In step S302 of some embodiments, the sample neutral features and the magnification code are input to the magnified image generation module. The magnified image generation module is a key component of the micro-expression magnification network, which is responsible for generating the magnified image according to the sample neutral features and the magnification specified by the magnification code. Through this step, a neutral frame magnified image can be obtained. Among them, the neutral frame magnified image can visually show the characteristics of micro-expressions more clearly.

[0133] In step S303 of some embodiments, the top frame is used as a reference to perform micro-expression identification on the generated neutral frame enlarged image. This step involves evaluating the difference between the enlarged image and the real micro-expression top frame to obtain a micro-expression identification result. In addition, by performing micro-expression identification on the neutral frame enlarged image through the top frame, a micro-expression deviation parameter can also be calculated. The micro-expression deviation parameter quantifies the difference between the neutral frame enlarged image and the top frame, providing a basis for further optimizing the enlarged image generation module in subsequent steps.

[0134] In step S304 of some embodiments, the model parameters of the magnified image generation module are adjusted based on the micro-expression deviation parameter. This adjustment process may involve using gradient descent or other optimization algorithms to minimize the deviation, thereby improving the quality of the magnified image. By continuously iterating and updating the model parameters of the magnified image generation module, the execution of steps S302 and S303 is returned until the micro-expression magnification network meets the first preset condition, which means that the micro-expression magnification network reaches the expected performance standard. Once these conditions are met, the training of the micro-expression magnification network is completed, and the micro-expression magnification network trained for image generation is obtained.

[0135] It should be pointed out that the micro-expression magnification network trained by image generation can generate high-quality neutral frame magnification images according to the sample neutral features and the magnification requirements corresponding to the magnification factor encoding. These images are closer to the real top frame in terms of details and expression intensity. In this way, the micro-expression magnification network provides strong support for the recognition and analysis of micro-expressions, and enhances the recognizability and practicality of micro-expressions in practical applications.

[0136] Reference Figure 7 , according to some embodiments of the present invention, step S304 adjusts the model parameters of the magnified image generation module based on the micro-expression deviation parameter, returns to execute the sample neutral feature and the magnification encoding input into the magnified image generation module for image generation, until the micro-expression magnification network meets the first preset condition, and obtains the micro-expression magnification network trained for image generation, which may include but is not limited to;

[0137] Step S701, performing a first adjustment on the model parameters of the enlarged image generation module based on the micro-expression deviation parameter, and performing a second adjustment on the model parameters of the intensity control module;

[0138] Step S702, after the first adjustment and the second adjustment, returns to execute inputting the sample neutral features and the magnification factor encoding into the enlarged image generation module for image generation, until the micro-expression magnification network meets the first preset condition, and obtains the micro-expression magnification network trained for image generation.

[0139] In some embodiments of the present invention, the training process of the micro-expression magnification network involves careful parameter adjustment to ensure that the generated magnified images meet the expected quality standards. This process not only focuses on the optimization of the magnified image generation module, but also involves the adjustment of the intensity control module to achieve a more refined and natural micro-expression magnification effect.

[0140] In step S701 of some embodiments, a first adjustment is made to the model parameters of the magnified image generation module based on the micro-expression deviation parameters. The micro-expression deviation parameters are obtained from the micro-expression identification results, and they quantify the difference between the generated magnified image and the real micro-expression top frame. Through these parameters, it is possible to identify which aspects of the magnified image need to be improved, such as the clarity of details, the intensity of expression, or other visual features. In addition, the embodiments of the present application also require a second adjustment to the model parameters of the intensity control module. The intensity control module is responsible for determining the magnification coding based on the magnification factor and random noise, and its optimization directly affects the accuracy and reliability of the magnification coding. By adjusting the intensity control module, the generation process of the magnification coding can be improved, thereby improving the overall quality of the magnified image.

[0141] In step S702 of some embodiments, after completing the first adjustment and the second adjustment, the sample neutral feature and the magnification code are input into the enlarged image generation module for image generation, and the sample neutral feature and the magnification code are input into the enlarged image generation module again to perform a new round of image generation. This process is iterative, and the performance of the enlarged image generation module and the intensity control module are gradually optimized through continuous cycles. Each iteration will adjust the parameters according to the micro-expression deviation parameters and regenerate the enlarged image until the micro-expression magnification network meets the first preset condition. Among them, the first preset condition may include an index of image quality, the accuracy of expression recognition, or other performance evaluation criteria.

[0142] The micro-expression amplification network of the embodiment of the present application is adjusted through the above-mentioned iterative training, and finally obtains a micro-expression amplification network trained for image generation. It should be pointed out that the micro-expression amplification network trained for image generation can generate high-quality and expected amplified images according to the characteristics of the neutral frame and the amplification requirements. These images are richer in details and closer to the real micro-expression top frame in terms of expression intensity, thereby improving the accuracy and reliability of micro-expression recognition. This process reflects the complexity and sophistication of the micro-expression amplification network training, ensuring the high performance and practicality of the final model in practical applications.

[0143] In step S104 of some embodiments, the image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression parsing network are coupled to construct a micro-expression recognition model. It should be noted that step S104 is the final integration stage of constructing the micro-expression recognition model. This step involves effectively coupling the previously trained independent components - the image feature extraction network, the micro-expression amplification network, and the expression parsing network - to form a coordinated micro-expression recognition system.

[0144] First, the image feature extraction network is responsible for extracting sample neutral features from the input neutral frame. The sample neutral features capture the subtle changes in facial expressions and important information about facial structure. The image feature extraction network has undergone feature extraction training and can accurately identify and extract image features.

[0145] Next, the micro-expression magnification network undergoes image generation training in step S103, and can generate magnified micro-expression images based on the neutral features of the samples. The micro-expression magnification network learns how to enhance the visibility of micro-expressions, making their features more obvious, and facilitating subsequent expression analysis.

[0146] The expression parsing network is responsible for analyzing the generated magnified image and identifying the expression type and emotional state in the image. The expression parsing network can perform in-depth analysis or classification of the magnified micro-expression image.

[0147] In step S104, the image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression parsing network need to be coupled into a unified framework to form a micro-expression recognition model.

[0148] In some embodiments, the interfaces between the networks need to be adjusted during the coupling process to ensure that the feature data can be smoothly transmitted between the networks and to optimize the performance of the micro-expression recognition model to achieve a higher accuracy. During the coupling process, some additional training or fine-tuning can also be performed to ensure that the various components can work together after integration and improve the accuracy of the micro-expression recognition model in recognizing micro-expressions.

[0149] In practical applications, micro-expression recognition often requires a quick response, so the processing speed and resource consumption of the entire system are also important considerations. Therefore, in some embodiments, in order to consider the real-time and efficiency of the micro-expression recognition model during the integration process, it may be necessary to streamline and optimize the micro-expression recognition model to meet the needs of actual deployment.

[0150] Through steps S101 to S104 of the embodiment of the present application, a micro-expression recognition model with high recognition accuracy can be obtained, which combines the capabilities of feature extraction, image magnification and expression analysis, and can automatically detect and analyze micro-expressions from videos, providing strong support for the fields of medical care, security, human-computer interaction, etc. The micro-expression recognition model can not only recognize subtle changes in expression, but also maintain high accuracy and robustness in complex environments.

[0151] Reference Figure 8 According to the micro-expression recognition method of the embodiment of the present invention, the method may include but is not limited to:

[0152] Step S801, obtaining a micro-expression recognition frame;

[0153] Step S802, inputting the micro-expression recognition frame into a micro-expression recognition model, wherein the micro-expression recognition model is coupled by an image feature extraction network trained for feature extraction, a micro-expression amplification network trained for image generation, and an expression analysis network; wherein the micro-expression recognition model is constructed by the micro-expression recognition model construction method of the first aspect embodiment;

[0154] Step S803, extracting features from the micro-expression recognition frame using an image feature extraction network that has been trained for feature extraction, to obtain micro-expression recognition features;

[0155] Step S804, inputting the micro-expression recognition features into the micro-expression magnification network trained by image generation to perform micro-expression magnification processing to obtain a micro-expression magnified image;

[0156] Step S805, performing expression analysis on the micro-expression magnified image based on the expression analysis network to obtain a micro-expression recognition result.

[0157] The micro-expression recognition method of the embodiment of the present application needs to first obtain a micro-expression recognition frame; input the micro-expression recognition frame into a micro-expression recognition model, and the micro-expression recognition model is formed by coupling an image feature extraction network trained for feature extraction, a micro-expression magnification network trained for image generation, and an expression analysis network; wherein the micro-expression recognition model is constructed by the micro-expression recognition model construction method of the first embodiment; the micro-expression recognition frame is subjected to feature extraction by the image feature extraction network trained for feature extraction to obtain micro-expression recognition features; the micro-expression recognition features are input into the micro-expression magnification network trained for image generation to perform micro-expression magnification processing to obtain a micro-expression magnified image; the micro-expression magnified image is subjected to expression analysis based on the expression analysis network to obtain a micro-expression recognition result. Since the micro-expression recognition model is constructed by the micro-expression recognition model construction method of the first embodiment, the micro-expression recognition model can achieve higher accuracy in practical application scenarios such as finance and medical care. On this basis, the micro-expression recognition model is applied to process micro-expression recognition frames to obtain micro-expression recognition results with higher accuracy.

[0158] In some embodiments of the present invention, the recognition and analysis of micro-expressions are achieved through a comprehensive micro-expression recognition model, which combines multiple trained networks to accurately capture and analyze the characteristics of micro-expressions.

[0159] In step S801 of some embodiments, micro-expression recognition frames are obtained. Micro-expression recognition frames refer to specific frames containing micro-expressions extracted from a video or image sequence. These frames contain subtle facial expression changes that need to be recognized and analyzed.

[0160] In step S802 of some embodiments, after obtaining the micro-expression recognition frame, it is input into the micro-expression recognition model. This model consists of three main parts: an image feature extraction network trained for feature extraction, a micro-expression amplification network trained for image generation, and an expression parsing network. These three are coupled together to form a complete micro-expression recognition model. The micro-expression recognition model is constructed according to the micro-expression recognition model construction method described in the embodiments of the present application, ensuring the accuracy and effectiveness of the micro-expression recognition model.

[0161] In step S803 of some embodiments, in the micro-expression recognition model, the input micro-expression recognition frame is firstly subjected to feature extraction by the image feature extraction network. In this step, the image feature extraction network uses the knowledge obtained from its training to extract key features from the micro-expression recognition frame. These features can represent the core information of the micro-expression and provide a basis for subsequent magnification and analysis.

[0162] In step S804 of some embodiments, the extracted micro-expression recognition features are then input into a micro-expression magnification network. The micro-expression magnification network magnifies the features based on its training results to generate a micro-expression magnified image. In this process, the micro-expression magnification network enhances the features of micro-expressions, making the originally small expression changes more obvious and easier to analyze in the image.

[0163] In step S805 of some embodiments, the expression analysis network is used to analyze the magnified micro-expression image. This step is to analyze the magnified image in depth through the expression analysis network, identify the expression type and emotional state in the image, and finally obtain the micro-expression recognition result.

[0164] Through this series of coherent steps, the micro-expression recognition model can extract features from the micro-expression recognition frame, perform amplification processing, and parse the specific emotional information of the micro-expression. This process not only improves the accuracy of micro-expression recognition, but also makes micro-expression analysis technology more efficient and practical, providing strong technical support for practical application scenarios such as finance and medical care that require accurate emotion recognition.

[0165] Reference Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0166] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0167] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the micro-expression recognition model construction method or micro-expression recognition method of the embodiment of this application;

[0168] Input / output interface 903, used to implement information input and output;

[0169] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0170] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0171] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0172] The embodiment of the present application further provides a computer program product, which includes a computer program. The processor of the computer device reads and executes the computer program, so that the computer device executes the above-mentioned micro-expression recognition model construction method or the above-mentioned micro-expression recognition method.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, which may include but is not limited to any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0176] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, which may include but is not limited to a number of instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0180] It should also be understood that the various implementations provided in the embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0181] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art may make various equivalent modifications or substitutions without violating the spirit of the present disclosure, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for constructing a micro-expression recognition model, characterized in that: include: Obtaining a micro-expression sample frame sequence and an expression parsing network; wherein the micro-expression sample frame sequence includes a neutral frame and a top frame, and the expression parsing network is used to extract micro-expression information of the image; Inputting the neutral frame into the image feature extraction network that has been trained for feature extraction to perform feature extraction to obtain sample neutral features; Performing image generation training on the original micro-expression amplification network based on the sample neutral features and the top frame to obtain the micro-expression amplification network that has undergone image generation training; The image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression parsing network are coupled to construct the micro-expression recognition model.

2. The method according to claim 1, characterized in that The micro-expression amplification network includes an amplified image generation module. The image generation training is performed on the original micro-expression amplification network based on the sample neutral features and the top frame to obtain the micro-expression amplification network that has undergone image generation training, including: Obtaining the magnification code corresponding to the neutral frame and the original micro-expression amplification network; Inputting the sample neutral feature and the magnification code into an amplified image generation module to generate an image, thereby obtaining a neutral frame amplified image; performing micro-expression identification on the amplified image of the neutral frame based on the top frame to obtain a micro-expression identification result and a micro-expression deviation parameter; The model parameters of the magnified image generation module are adjusted based on the micro-expression deviation parameter, and the sample neutral feature and the magnification factor encoding are input into the magnified image generation module for image generation until the micro-expression magnification network meets the first preset condition, thereby obtaining the micro-expression magnification network trained for image generation.

3. The method according to claim 2, characterized in that The micro-expression amplification network further includes an intensity control module, and obtaining the amplification factor code corresponding to the neutral frame includes: Get the magnification factor; The magnification factor is input into the intensity control module, so that the intensity control module determines the magnification code according to the magnification factor.

4. The method according to claim 3, characterized in that Before inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor, the method further includes: Get random noise; Inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor includes: The random noise and the amplification factor are input into the intensity control module, so that the intensity control module determines the amplification code according to the amplification factor and the random noise.

5. The method according to claim 3 or 4, characterized in that The method further comprises adjusting the model parameters of the magnified image generation module based on the micro-expression deviation parameter, returning to execute inputting the sample neutral feature and the magnification factor encoding into the magnified image generation module for image generation until the micro-expression magnification network satisfies a first preset condition, and obtaining the micro-expression magnification network trained for image generation, including: Performing a first adjustment on the model parameters of the magnified image generation module based on the micro-expression deviation parameter, and performing a second adjustment on the model parameters of the intensity control module; After the first adjustment and the second adjustment, the sample neutral feature and the magnification factor encoding are input into the magnified image generation module for image generation until the micro-expression amplification network meets the first preset condition, thereby obtaining the micro-expression amplification network trained for image generation.

6. The method according to claim 3, characterized in that The micro-expression sample frame sequence includes a plurality of neutral frames, and the obtaining of the magnification factor includes: obtaining the magnification factor matched to each of the neutral frames; Inputting the magnification factor into the intensity control module so that the intensity control module determines the magnification code according to the magnification factor includes: The magnification factor corresponding to each of the neutral frames is sequentially input into the intensity control module, so that the intensity control module determines the magnification code corresponding to each of the neutral frames according to the magnification factor corresponding to each of the neutral frames.

7. The method according to claim 1, characterized in that Before inputting the neutral frame into the image feature extraction network that has been trained for feature extraction to extract features and obtain sample neutral features, the method further includes performing feature extraction training on the image feature extraction network, specifically including: Obtaining a feature extraction training image and the original image feature extraction network; Inputting the feature extraction training image into the image feature extraction network to extract image features to obtain image training features; Inputting the feature extraction training image into a pre-trained image coding network to extract prior features and obtain image prior features; Performing feature distribution identification on the image training features based on the image prior features to obtain feature distribution identification results and feature deviation parameters; The model parameters of the image feature extraction network are adjusted based on the feature deviation parameters, and the execution returns to inputting the feature extraction training image into the image feature extraction network for image feature extraction until the image feature extraction network meets the second preset condition, and the parameters of the image feature extraction network are frozen to obtain the image feature extraction network that has been trained for feature extraction.

8. A micro-expression recognition method, characterized in that: include: Get micro-expression recognition frames; Inputting the micro-expression recognition frame into a micro-expression recognition model, wherein the micro-expression recognition model is coupled by the image feature extraction network trained for feature extraction, the micro-expression amplification network trained for image generation, and the expression parsing network; wherein the micro-expression recognition model is constructed by the micro-expression recognition model construction method according to any one of claims 1 to 7; Performing feature extraction on the micro-expression recognition frame using the image feature extraction network that has been trained for feature extraction to obtain micro-expression recognition features; Inputting the micro-expression recognition features into the micro-expression magnification network trained by image generation to perform micro-expression magnification processing to obtain a micro-expression magnified image; The expression analysis is performed on the micro-expression magnified image based on the expression analysis network to obtain a micro-expression recognition result.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the micro-expression recognition model construction method according to any one of claims 1 to 7, or the micro-expression recognition method according to claim 8 is implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the micro-expression recognition model construction method according to any one of claims 1 to 7, or the micro-expression recognition method according to claim 8.

Citation Information

Patent Citations

  • Micro-expression classification method based on action amplification and self-adaptive attention region selection

    CN112200065A