Micro-expression detection method and system based on multi-modal large model, medium and equipment

Abstract

The invention relates to the field of artificial intelligence and emotion recognition, and discloses a micro-expression detection method and system based on a multi-modal large model, a medium and equipment, and the method comprises the steps: carrying out the face detection of each frame of video, extracting a face region in the video through a deep learning face detection model, and carrying out the unified adjustment of the image size after cutting; a face AU regression network is adopted to extract a face AU feature value of each frame of image so as to represent the motion intensity of face muscles, and face dynamic change information is constructed in combination with a time sequence; and combining the AU features of each frame of image with the image data to generate a language prompt suitable for a multi-modal large language model, training and optimizing the multi-modal large language model to perform emotion reasoning, and performing quantitative evaluation on an emotion reasoning result. According to the method, the accuracy problem caused by tiny change of facial muscles in micro-expression detection is solved; and the reasoning capability and accuracy of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and emotion recognition technology, in particular to a micro-expression detection method, system, medium and equipment based on a multi-modal large model. BACKGROUND

[0002] Emotion state recognition is an important research direction in the field of artificial intelligence and computer vision. With the accumulation of multi-modal data and the enhancement of large language model capabilities, deep learning-based expression recognition methods have made significant progress. Especially in the standard expression recognition task, the application of convolutional neural networks, attention mechanisms, and time modeling techniques has greatly improved the accuracy of models on controlled datasets.

[0003] However, when dealing with micro-expression recognition problems under uncontrolled conditions, the following main challenges still exist: (1) Traditional image or short sequence-based feature extraction methods are difficult to effectively capture the instantaneous change characteristics of micro-expressions. Micro-expressions usually appear and disappear within 0.2 seconds, with small changes in amplitude and extremely small facial muscle displacement. Conventional visual models have difficulty extracting sufficient significant feature signals, resulting in unstable recognition performance.

[0004] (2) Micro-expression data has the problems of sample scarcity and high labeling cost. Existing public datasets have limited sample sizes, and most labels rely on professional FACS coders for frame-by-frame labeling, which is highly subjective and has poor reusability. This limits the generalization ability of deep models in micro-expression tasks.

[0005] (3) The physiological basis of micro-expression comes from facial action units (AUs), but most existing recognition models do not fully utilize the dynamic evolution characteristics of AUs, only treating them as static additional features, and lack effective modeling of the causal relationship between muscle action sequences and emotional logic.

[0006] (4) Existing recognition systems are mostly "black box" structures, only outputting classification results, and lack of explainable reasoning chains. In high-risk application scenarios (such as interrogation, psychological assessment, etc.), it is difficult to be trusted or used by end users, limiting its deployment and landing.

[0007] (5) There is a lack of unified chain language explanation quality evaluation system. Even if some methods introduce language output modules, there is no perfect explanation consistency index system, making it difficult to evaluate the credibility of model output, restricting the self-optimization ability of models in open scenarios.

[0008] In summary, the above analysis, research a kind of can fuse facial action unit dynamic characteristics and multi-modal large language model semantic reasoning ability, with high recognition precision and strong explainability simultaneously micro-expression detection method, become the important direction of current technology development. SUMMARY

[0009] To solve the above problems, the purpose of the present application is to provide a micro-expression detection method, system, medium and equipment based on multi-modal large model, which can overcome the limitations of traditional methods in micro-expression recognition, such as insufficient modeling of short-time dynamics, uninterpretable emotion reasoning, high label cost, poor model generalization, etc.

[0010] To achieve the above purpose, in the first aspect, the technical scheme adopted by the present application is: a micro-expression detection method based on multi-modal large model, comprising: performing face detection on each frame of video, extracting the face region in the video using a deep learning face detection model, and uniformly adjusting the image size after cropping; using a face AU regression network to extract the face AU feature value of each frame of image to represent the movement intensity of facial muscles, and combining time series to construct facial dynamic change information; combining the AU feature of each frame of image with image data to generate a language prompt suitable for a multi-modal large language model, and after training and optimizing the multi-modal large language model, performing emotion reasoning, and quantitatively evaluating the emotion reasoning result.

[0011] Further, the training of the multi-modal large language model uses a semi-supervised learning framework, uses public data sets to obtain standard labeled data sets and unlabeled data sets, uses labeled data for supervised learning, uses a teacher-student model mechanism to generate pseudo-labels from unlabeled data, and further optimizes the model.

[0012] Further, a soft zoom loss function is used to optimize the emotion classification task during model training, and the soft zoom loss function is For:

[0013] In the formula, P is the probability of the predicted class, is a class balance coefficient, is a focusing factor, which adjusts the weight of difficult-to-classify samples.

[0014] Further, the multi-modal large language model uses a Qwen-2.5VL model, which performs emotion reasoning by processing multi-modal input and combining face AU features and image information.

[0015] Further, the emotion reasoning includes a three-stage chain reasoning structure: The action recognition layer identifies key facial action units (AUs) whose changes in intensity exceed a preset threshold by analyzing the changes in facial muscle action unit intensity at the current moment. The emotional cue layer infers possible emotion types based on the changing trends and historical data of AU; The decision-making level concludes by outputting a final sentiment judgment based on the reasoning chain, along with an explanation of the causal reasoning chain, to ensure the transparency and credibility of the reasoning results.

[0016] Furthermore, a temporal modeling method based on long short-term memory is employed to capture long-term dependencies in facial muscle activity sequences.

[0017] Furthermore, the results of emotion reasoning are quantitatively evaluated, specifically by using a chain consistency scoring mechanism to quantitatively evaluate the linguistic fluency of the reasoning chain, the coherence of causal logic, and the consistency between the AU and image features. Chain Consistency Scoring The formula is:

[0018] In the formula, Score for language fluency To score the coherence of the causal chain, The consistency score between the language interpretation and the facial AU sequence is used. The weighting coefficients for the language fluency score. The weighting coefficients for the causal chain coherence score. The weighting coefficients for the consistency scores between language interpretation and facial AU sequences.

[0019] Secondly, the technical solution adopted by this invention is as follows: a micro-expression detection method based on a multimodal large-scale model, comprising: a data processing module, which performs facial detection on each frame of video, extracts the facial region in the video using a deep learning face detection model, and uniformly adjusts the image size after cropping; a facial AU extraction module, which uses a facial AU regression network to extract facial AU feature values ​​of each frame of image to characterize the movement intensity of facial muscles, and constructs facial dynamic change information by combining time series; and an emotion reasoning module, which combines the AU features of each frame of image with image data to generate language prompts suitable for a multimodal large-scale language model, trains and optimizes the multimodal large-scale language model to perform emotion reasoning, and quantitatively evaluates the emotion reasoning results.

[0020] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0021] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0022] The present invention has the following advantages due to the adoption of the above technical solutions: 1. This invention can effectively address the problem of scarce micro-expression samples. Through automatic label generation and chain interpretation mechanism, it reduces the reliance on expert annotation and improves the system's generalization and transferability.

[0023] 2. This invention proposes a chain-based explanation framework, which enables the model to generate a causal reasoning chain while providing identification results, thereby helping to enhance end users' trust and understanding of the results.

[0024] 3. This invention achieves a closed-loop system from data generation and inference generation to credibility assessment through a joint optimization mechanism of CCS scoring and loss function, effectively improving the robustness of the system.

[0025] 4. This invention can be combined with any standard multimodal large language model as the reasoning basis, has strong adaptability, and can be flexibly deployed on a variety of vision-language platforms.

[0026] 5. The present invention has a clear overall system structure and high execution efficiency. It can achieve stable performance in edge devices with limited hardware resources and high-concurrency server environments, thereby improving the actual deployment value of the system.

[0027] In summary, this invention comprehensively utilizes multiple innovative technical solutions, including dynamic modeling of facial action units, three-segment chain interpretation, automatic label generation, and closed-loop consistency evaluation, which significantly improves the micro-expression detection task in terms of recognition accuracy, inference transparency, and generalization ability. It provides an efficient, reliable, and scalable technical approach to address the shortcomings of existing technologies. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the micro-expression recognition method based on a multimodal large model in an embodiment of the present invention. Figure 2 This is a schematic diagram of the overall framework of the micro-expression recognition method based on a multimodal large model in this embodiment of the invention; Figure 3 This is a schematic diagram of the three-stage chain-based language reasoning structure proposed in this invention. Detailed Implementation

[0029] To effectively address the shortcomings of current methods in areas such as data scarcity, uninterpretable reasoning, and weak temporal modeling capabilities, this invention employs a novel micro-expression detection and interpretation method. Specifically, it provides a micro-expression detection method, system, medium, and device based on a multimodal large-scale model, including data preprocessing, facial action unit (AU) extraction, and emotion inference steps. By combining facial AU features with multimodal input from image data, this invention can efficiently classify emotions, overcoming the limitations of traditional methods in capturing short-term dynamic changes in micro-expressions. By adaptively adjusting the model of facial muscle dynamic features, it solves the accuracy problem caused by subtle changes in facial muscles in micro-expression detection. During implementation, training with a multimodal large-scale language model further enhances the model's inference ability and accuracy. This invention can be effectively applied to fields such as psychological research, sentiment analysis, and intelligent security, possessing broad application prospects and strong practicality.

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0032] In one embodiment of the present invention, a micro-expression detection method based on a multimodal large model is provided. The core idea is to dynamically model facial action units (AUs), fuse multimodal information, and generate a chain-like language mechanism. An efficient reasoning method for micro-expression recognition tasks is proposed, combining structured action signals and image data, and leveraging a large language model to generate interpretable reasoning chains with causal logic, thereby improving recognition accuracy and system reliability. Specifically, in this embodiment, as... Figure 1 As shown, the method includes the following steps: 1) Perform face detection on each frame of the video, use a deep learning face detection model to extract the face region in the video, and then crop and adjust the image size uniformly; the deep learning face detection model can be MTCNN or HaarCascades, etc.

[0033] 2) A facial AU regression network is used to extract facial AU feature values ​​from each frame of the image to characterize the motion intensity of facial muscles. The dynamic change information of the face is constructed by combining the time series and then an AU matrix is ​​generated for subsequent analysis.

[0034] 3) Combine the AU features of each frame with the image data to generate language prompts suitable for the multimodal large language model (LLM). After training and optimizing the multimodal large language model, perform emotion reasoning and quantitatively evaluate the emotion reasoning results.

[0035] In this embodiment, the AU features of each frame image are combined with the image data to generate a multimodal input that includes AU change trends, time series, and facial muscle movement patterns.

[0036] In step 1) above, a face detection algorithm is used to extract the facial region from the video, and each frame of the image is cropped and uniformly adjusted to a size of 224×224 to ensure the consistency of the input data.

[0037] In step 2) above, the facial AU regression network improves its ability to capture micro-expression details through multi-scale feature fusion and enhances the model's ability to recognize complex emotions.

[0038] In this embodiment, the dynamic changes of facial muscles are optimized by an image feature extraction module based on a convolutional neural network (CNN) to enhance the real-time detection capability of micro-expressions.

[0039] In step 3) above, the training of the multimodal large language model adopts a semi-supervised learning framework. Standard labeled datasets and unlabeled datasets are obtained using public datasets. The labeled data is used for supervised learning, and pseudo-labels are generated from the unlabeled data using a teacher-student model mechanism to further optimize the model.

[0040] Public datasets such as CASME II and SAMM are used. A teacher-student model mechanism is employed to generate pseudo-labels from unlabeled data. The teacher model generates pseudo-labels using existing labeled data, and the student model is trained based on these pseudo-labels.

[0041] In this embodiment, to optimize the emotion classification task, the SoftVariFocal Loss function is used during model training. for:

[0042] In the formula, To predict the probability of a category, This is the category balance coefficient. As a focusing factor, adjust the weights of difficult-to-classify samples.

[0043] In step 3) above, the multimodal large language model adopts the Qwen-2.5VL model. The Qwen-2.5VL model processes multimodal inputs and combines facial AU features and image information to perform emotion inference. Using Qwen-2.5VL as the inference framework can combine image data and language input to improve recognition accuracy while generating emotion inference chains.

[0044] In this embodiment, the generated language prompts are input into a multimodal large language model for emotion inference. Emotion inference includes a three-stage chain-like reasoning structure: The action recognition layer identifies key facial action units (AUs) whose changes in intensity exceed a preset threshold by analyzing the changes in facial muscle action unit intensity at the current moment.

[0045] The emotional cue layer, based on the changing trends and historical data of AU, infers possible emotion types, such as anger, sadness, joy, etc.

[0046] The decision-making level concludes by outputting a final sentiment judgment based on the reasoning chain, along with an explanation of the causal reasoning chain, to ensure the transparency and credibility of the reasoning results.

[0047] Among them, a temporal modeling method based on long short-term memory (LSTM) is used to capture long-term dependencies in facial muscle activity sequences to improve the temporal accuracy of emotion reasoning.

[0048] In step 3) above, to ensure that the generated emotion reasoning chain has high quality and consistency, the emotion reasoning results are quantitatively evaluated. Specifically, the chain consistency scoring (CCS) mechanism is used to quantitatively evaluate the language fluency of the reasoning chain, the coherence of causal logic, and the consistency between AU and image features, thereby optimizing the model's reasoning ability.

[0049] Chain Consistency Scoring The formula is:

[0050] In the formula, Score for language fluency To score the coherence of the causal chain, The consistency score between the language interpretation and the facial AU sequence is used. The weighting coefficients for the language fluency score. The weighting coefficients for the causal chain coherence score. The weighting coefficients for the consistency scores between language interpretation and facial AU sequences.

[0051] In step 3) above, the interpretability of the causal reasoning chain generated by emotion inference is further analyzed using the Feature Importance Score (FIS) to provide the degree of influence of facial muscles in emotion judgment. The emotion category and reasoning chain explanation in the emotion inference results are further optimized using the backpropagation algorithm, thereby improving the reliability and accuracy of the reasoning process.

[0052] In the above embodiments, during training, this invention adaptively selects frames with significant changes as training samples by calculating the rate of change of facial action units (AUs), thereby improving training efficiency and the model's generalization ability. This method can effectively improve the training effect of micro-expression detection tasks, reduce sample noise, and enhance the model's expressiveness.

[0053] In summary, this invention can be applied not only to standard vision-language frameworks but also integrated with existing vision-language models. It can be flexibly applied to other object detection network frameworks, improving the accuracy and generalization ability of micro-expression detection through integration with existing vision-language models. This invention can adaptively adjust the selection of positive and negative samples based on the dynamic features of facial muscles, improving training efficiency through dynamic sample selection. It is adaptable to various hardware platforms and computing environments, supporting applications on edge computing devices and large-scale server clusters, ensuring efficient and real-time emotion analysis capabilities. This invention can be applied to multiple fields such as intelligent security, psychological research, and human-computer interaction, and has particularly high application value in real-time emotion monitoring and evaluation.

[0054] In one embodiment of the present invention, a micro-expression detection method based on a multimodal large model is provided, comprising: The data processing module performs face detection on each frame of video, uses a deep learning face detection model to extract the face region in the video, and then adjusts the image size uniformly after cropping. The facial AU extraction module uses a facial AU regression network to extract facial AU feature values ​​from each frame of the image to characterize the motion intensity of facial muscles, and combines time series to construct facial dynamic change information. The emotion reasoning module combines the AU features of each frame with the image data to generate language prompts suitable for a multimodal large language model. After training and optimizing the multimodal large language model, emotion reasoning is performed, and the emotion reasoning results are quantitatively evaluated.

[0055] In the above embodiments, the training of the multimodal large language model adopts a semi-supervised learning framework. Standard labeled datasets and unlabeled datasets are obtained using public datasets. The labeled data is used for supervised learning, and pseudo-labels are generated from the unlabeled data using a teacher-student model mechanism to further optimize the model.

[0056] In the above embodiments, a soft zoom loss function is used to optimize the emotion classification task during model training. for:

[0057] In the formula, To predict the probability of a category, This is the category balance coefficient. As a focusing factor, adjust the weights of difficult-to-classify samples.

[0058] In the above embodiments, the multimodal large language model adopts the Qwen-2.5VL model. The Qwen-2.5VL model performs emotion inference by processing multimodal inputs and combining facial AU features and image information.

[0059] In the above embodiments, emotion reasoning includes a three-stage chain reasoning structure: The action recognition layer identifies key facial action units (AUs) whose changes in intensity exceed a preset threshold by analyzing the changes in facial muscle action unit intensity at the current moment. The emotional cue layer infers possible emotion types based on the changing trends and historical data of AU; The decision-making level concludes by outputting a final sentiment judgment based on the reasoning chain, along with an explanation of the causal reasoning chain, to ensure the transparency and credibility of the reasoning results.

[0060] In the above embodiments, a temporal modeling method based on long short-term memory is used to capture long-term dependencies in facial muscle activity sequences.

[0061] In the above embodiments, the emotion reasoning results are quantitatively evaluated by: using a chain consistency scoring mechanism to quantitatively evaluate the language fluency of the reasoning chain, the coherence of causal logic, and the consistency between the AU and image features. Chain Consistency Scoring The formula is:

[0062] In the formula, Score for language fluency To score the coherence of the causal chain, The consistency score between the language interpretation and the facial AU sequence is used. The weighting coefficients for the language fluency score. The weighting coefficients for the causal chain coherence score. The weighting coefficients for the consistency scores between language interpretation and facial AU sequences.

[0063] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0064] This embodiment illustrates the application of the method in a terminal device. It is understood that the method is also applicable to deployment on a server, and can be used to build a micro-expression recognition system that integrates terminal and server collaboration, achieving collaborative processing between multiple modules through network communication. Figure 2 As shown, the method in this embodiment includes the following steps: (1) Data collection and preprocessing. Download and prepare labeled and unlabeled datasets (CASMEⅡ, SAMM). Use labeled data for supervised learning and unlabeled data to generate pseudo-labels. Perform face detection on video frames. Use the face detection model MTCNN to extract the facial regions in each frame, crop and uniformly adjust them to a size of 224×224 to ensure the consistency of input data.

[0065] (2) Facial Action Unit (AU) Extraction. Facial action unit (AU) features are extracted from each frame of the image using the Facial Action Unit Regression Network AU-Net. These AU features represent the intensity of muscle activity in different areas of the face and are a key element in micro-expression detection. AU features are related to changes in facial expressions and can effectively capture subtle features that change instantaneously, helping the model distinguish micro-expressions of different emotions.

[0066] (3) Generating a Prompt. The extracted AU features are combined with the facial image data of each frame to generate a prompt suitable for the Qwen-2.5VL model. This prompt includes not only the intensity value of the facial AU, but also the time-series changes of the AU features, helping the model to better understand the dynamic trends of emotion changes. The Qwen-2.5VL model receives the fused input of image data and AU features, and performs emotion inference through multimodal learning.

[0067] (4) Emotion Inference and Multimodal Learning. The Qwen-2.5VL model performs emotion inference by processing multimodal inputs and combining facial AU features and image information. The model infers possible emotion categories, such as anger, joy, and sadness, based on the dynamic changes of facial action units. At the same time, the inference process adopts a three-stage chain inference structure: first, the action recognition layer analyzes the main changes of the current facial muscles; then, the emotion cue layer infers the possible emotion type based on AU combinations and historical evolution trends; finally, the decision layer outputs the final emotion judgment, along with an explanation of the causal inference chain, to ensure the interpretability of the inference result.

[0068] (5) Model Training and Optimization. This invention employs a semi-supervised learning framework. During training, labeled data is used for supervised learning, and unlabeled data is used to generate pseudo-labels for further model training. Pseudo-label generation adopts a teacher-student model mechanism. Under this framework, the teacher model generates pseudo-labels through inference from existing data, and the student model uses these pseudo-labels for training. In this way, the model can learn from limited labeled data while expanding the training set using unlabeled data. The loss function is optimized using the Soft VariFocal Loss function, which can reduce the noise impact of pseudo-labels and enhance the model's ability to learn from negative samples.

[0069] (6) After model training is completed, the present invention uses a test set to evaluate the model. During the inference phase, the input test image is processed by face detection and AU extraction before entering the Qwen-2.5VL model for emotion inference. The model outputs emotion classification results and inference chain explanations. Evaluation metrics include accuracy, F1-score, etc., to evaluate the model's performance. During the inference process, in addition to emotion judgment, the model also generates detailed causal inference chains to enhance the interpretability of the results.

[0070] Through the detailed implementation steps described above, this invention successfully combines facial AU features with the multimodal learning capabilities of the Qwen-2.5VL model, not only improving the accuracy of micro-expression recognition but also ensuring the interpretability of the inference results. This method has significant application value in the fields of micro-expression detection and emotion analysis, especially in areas such as psychological research, affective computing, and human-computer interaction.

[0071] To further verify the effectiveness and practicality of this invention, an example of micro-expression recognition is given below. Table 1 shows the recognition results for 7-class and 3-class tasks, with the metrics being UF1 (Unweighted F1), UAR (Unweighted Average Recall), and ACC (Accuracy).

[0072] Table 1 shows the validation results of the examples on the dataset.

[0073] As shown in Table 1, the proposed method significantly outperforms the comparison model in both the 7-class and 3-class tasks (* indicates fine-tuning on the CoT dataset). In the 7-class task, UF1 improved from 0.1924 to 0.2236, UAR from 0.1927 to 0.2229, and ACC from 0.2281 to 0.2694; in the 3-class task, UF1 improved from 0.3958 to 0.4371, UAR from 0.3520 to 0.3861, and ACC from 0.4747 to 0.4812. These results validate the effectiveness and significant performance improvement of this invention in micro-expression recognition.

[0074] Experimental results on multiple publicly available micro-expression datasets demonstrate that the proposed method outperforms traditional models across various evaluation metrics. Particularly when processing micro-expression data with extremely short time spans and subtle amplitudes, the introduction of action unit sequence modeling and multimodal language inference mechanisms significantly improves the model's recognition accuracy, inference consistency, and result interpretability.

[0075] In addition, such as Figure 3 As shown, the chain-based language interpretation structure and chain-based consistency scoring mechanism proposed in this invention not only improve the controllability and reliability of the model, but also have good transferability and scalability. They are applicable to other micro-expression recognition frameworks and temporal visual analysis tasks, demonstrating high adaptability and practical value.

[0076] In one embodiment of the present invention, a computing device is provided. This computing device can be a terminal and may include a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. When the computer programs are executed by the processor, they implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0077] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0079] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0080] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-expression detection method based on a multimodal large model, characterized in that, include: Face detection is performed on each frame of the video. A deep learning face detection model is used to extract the face regions in the video. After cropping, the image size is adjusted uniformly. A facial AU regression network is used to extract facial AU feature values ​​for each frame of the image to characterize the motion intensity of facial muscles, and combined with time series to construct facial dynamic change information. The AU features of each frame are combined with the image data to generate language prompts suitable for a multimodal large language model. After training and optimizing the multimodal large language model, emotion inference is performed, and the emotion inference results are quantitatively evaluated.

2. The micro-expression detection method based on a multimodal large model as described in claim 1, characterized in that, The training of the multimodal large language model adopts a semi-supervised learning framework. It uses public datasets to obtain standard labeled datasets and unlabeled datasets. The labeled data is used for supervised learning, and a teacher-student model mechanism is used to generate pseudo-labels from the unlabeled data to further optimize the model.

3. The micro-expression detection method based on a multimodal large model as described in claim 2, characterized in that, The model training process employs a soft zoom loss function to optimize the emotion classification task. for: In the formula, To predict the probability of a category, This is the category balance coefficient. As a focusing factor, adjust the weights of difficult-to-classify samples.

4. The micro-expression detection method based on a multimodal large model as described in claim 1, characterized in that, The multimodal large language model adopts the Qwen-2.5VL model, which performs emotion inference by processing multimodal inputs and combining facial AU features and image information.

5. The micro-expression detection method based on a multimodal large model as described in claim 1, characterized in that, Emotional reasoning includes a three-part chain reasoning structure: The action recognition layer identifies key facial action units (AUs) whose changes in intensity exceed a preset threshold by analyzing the changes in facial muscle action unit intensity at the current moment. The emotional cue layer infers possible emotion types based on the changing trends and historical data of AU; The decision-making level concludes by outputting a final sentiment judgment based on the reasoning chain, along with an explanation of the causal reasoning chain, to ensure the transparency and credibility of the reasoning results.

6. The micro-expression detection method based on a multimodal large model as described in claim 5, characterized in that, A temporal modeling method based on long short-term memory is used to capture long-term dependencies in facial muscle activity sequences.

7. The micro-expression detection method based on a multimodal large model as described in claim 1, characterized in that, The results of emotion reasoning are quantitatively evaluated, specifically by using a chain consistency scoring mechanism to quantitatively evaluate the linguistic fluency of the reasoning chain, the coherence of causal logic, and the consistency between the AU and image features. Chain Consistency Scoring The formula is: In the formula, Score for fluency of speech To score the coherence of the causal chain, The consistency score between the language interpretation and the facial AU sequence is used. The weighting coefficients for the language fluency score. The weighting coefficients for the causal chain coherence score. The weighting coefficients for the consistency scores between language interpretation and facial AU sequences.

8. A micro-expression detection method based on a multimodal large model, characterized in that, include: The data processing module performs face detection on each frame of video, uses a deep learning face detection model to extract the face region in the video, and then adjusts the image size uniformly after cropping. The facial AU extraction module uses a facial AU regression network to extract facial AU feature values ​​from each frame of the image to characterize the intensity of facial muscle movement, and combines time series to construct facial dynamic change information. The emotion reasoning module combines the AU features of each frame with the image data to generate language prompts suitable for a multimodal large language model. After training and optimizing the multimodal large language model, emotion reasoning is performed, and the emotion reasoning results are quantitatively evaluated.

9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 7.

10. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 7.

Images