Micro-expression recognition method, device and equipment based on artificial intelligence and storage medium

Abstract

The invention belongs to the field of artificial intelligence and the field of financial science and technology, and relates to an artificial intelligence-based micro-expression recognition method, which comprises the steps of collecting multi-modal data of a user; respectively preprocessing the light image and the infrared image to obtain a target light image and a target infrared image; preprocessing the voice data to obtain target voice; performing feature extraction on the target light image, the target infrared image and the target voice to obtain a corresponding first image feature, a corresponding second image feature and a corresponding voice feature; performing feature fusion processing on the first image feature, the second image feature and the voice feature to obtain a fused feature; and calling a micro-expression recognition model to recognize the fusion features to generate a micro-expression recognition result. The invention further provides a micro-expression recognition device based on artificial intelligence, computer equipment and a storage medium. The micro-expression recognition model constructed based on the teacher-student network and the knowledge distillation mechanism is used for recognizing the fusion features, and the recognition precision of micro-expression recognition can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence development technology and the field of financial technology, and in particular to an artificial intelligence-based micro-expression recognition method, device, computer equipment and storage medium. Background Art

[0002] Micro-expressions refer to short and uncontrollable facial expressions that humans reveal when they try to suppress or hide their true emotions. In many fields, such as intelligent customer service, financial transactions, judicial interrogations, and clinical medicine, micro-expression detection has received widespread attention, and it is necessary to identify the micro-expressions of the subjects to explore their true emotions and intentions. Since micro-expressions usually last for a short time and are less intense than normal expressions, we generally need to analyze each frame of the test video of the subject separately to detect and identify micro-expressions.

[0003] In the prior art, the current micro-expression recognition technology is mainly based on single-modal image data for recognition. Such a recognition processing method has the problem of low recognition accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a micro-expression recognition method, device, computer equipment and storage medium based on artificial intelligence to solve the technical problem that the existing micro-expression recognition technology is mainly based on single modal data of images, and such recognition processing method has low recognition accuracy.

[0005] In order to solve the above technical problems, the present application embodiment provides a micro-expression recognition method based on artificial intelligence, which adopts the following technical solutions:

[0006] Collecting multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data;

[0007] Preprocessing the light image and the infrared image respectively based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image;

[0008] Preprocessing the speech data based on a preset second processing strategy to obtain a corresponding target speech;

[0009] Extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features;

[0010] Performing feature fusion processing on the first image feature, the second image feature, and the speech feature to obtain corresponding fusion features;

[0011] Calling a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism;

[0012] The fused features are recognized and processed based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0013] Furthermore, before the step of calling the pre-trained micro-expression recognition model, the method further includes:

[0014] Get the pre-built initial teacher network and initial student network;

[0015] Acquire pre-collected multimodal micro-expression data;

[0016] Training the initial teacher model based on the multimodal micro-expression data to obtain a trained teacher network;

[0017] The trained teacher network is used as the teacher model, and the initial student network is used as the student of the teacher model for training by minimizing the KL divergence loss function to obtain a trained student model;

[0018] Fine-tune and optimize the trained student model to obtain a corresponding target student model;

[0019] The target student model is used as the micro-expression recognition model.

[0020] Furthermore, the step of collecting multimodal data corresponding to the user specifically includes:

[0021] Capturing a light image corresponding to the user's face based on a preset camera;

[0022] collecting a facial thermal distribution image corresponding to the user's face based on a preset thermal imager, and using the facial thermal distribution image as the infrared image;

[0023] The voice data corresponding to the user is collected based on a preset recording device.

[0024] Furthermore, the step of preprocessing the light image and the infrared image based on the preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image specifically includes:

[0025] Performing image cropping processing on the light image and the infrared image respectively to obtain a corresponding first light image and a first infrared image;

[0026] performing illumination normalization processing on the first light image and the first infrared image respectively to obtain a corresponding second light image and a second infrared image;

[0027] performing scale normalization processing on the second light image and the second infrared image respectively to obtain a corresponding third light image and a third infrared image;

[0028] performing mean-variance normalization processing on the third light image and the third infrared image respectively to obtain a corresponding fourth light image and fourth infrared image;

[0029] The fourth light image is used as the target light image, and the fourth infrared image is used as the target infrared image.

[0030] Furthermore, the step of preprocessing the speech data based on the preset second processing strategy to obtain the corresponding target speech specifically includes:

[0031] Performing a mute removal process on the voice data to obtain a corresponding first voice;

[0032] Performing denoising on the first speech to obtain a corresponding second speech;

[0033] Performing frame processing on the second speech to obtain a corresponding third speech;

[0034] Performing spectrum extraction processing on the third speech to obtain a corresponding fourth speech;

[0035] The fourth speech is used as the target speech.

[0036] Further, the steps of extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features specifically include:

[0037] Extracting features of the target light image based on a preset convolutional neural network to obtain the first image features;

[0038] Extracting features of the target infrared image based on a preset convolutional stacked autoencoder to obtain the second image features;

[0039] The target speech is subjected to feature extraction based on a preset speech feature learning network to obtain the speech feature.

[0040] Furthermore, the step of performing feature fusion processing on the first image feature, the second image feature and the voice feature to obtain corresponding fusion features specifically includes:

[0041] Call the preset cross-modal converter;

[0042] Inputting the first image feature, the second image feature, and the speech feature into the cross-modal converter;

[0043] Performing feature conversion processing on the first image feature, the second image feature, and the speech feature through the cross-modal converter to obtain corresponding conversion features;

[0044] The converted feature is used as the fused feature.

[0045] In order to solve the above technical problems, the embodiment of the present application also provides a micro-expression recognition device based on artificial intelligence, which adopts the following technical solution:

[0046] A collection module, used to collect multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data;

[0047] A first preprocessing module, configured to preprocess the light image and the infrared image based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image;

[0048] A second preprocessing module, used to preprocess the speech data based on a preset second processing strategy to obtain a corresponding target speech;

[0049] An extraction module, configured to perform feature extraction on the target light image to obtain a corresponding first image feature, perform feature extraction on the target infrared image to obtain a corresponding second image feature, and perform feature extraction on the target voice to obtain a corresponding voice feature;

[0050] A fusion module, used for performing feature fusion processing on the first image feature, the second image feature and the speech feature to obtain corresponding fusion features;

[0051] A calling module, used to call a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism;

[0052] The recognition module is used to perform recognition processing on the fused features based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0054] Collecting multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data;

[0055] Preprocessing the light image and the infrared image respectively based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image;

[0056] Preprocessing the speech data based on a preset second processing strategy to obtain a corresponding target speech;

[0057] Extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features;

[0058] Performing feature fusion processing on the first image feature, the second image feature, and the speech feature to obtain corresponding fusion features;

[0059] Calling a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism;

[0060] The fused features are recognized and processed based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0061] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0062] Collecting multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data;

[0063] Preprocessing the light image and the infrared image respectively based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image;

[0064] Preprocessing the speech data based on a preset second processing strategy to obtain a corresponding target speech;

[0065] Extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features;

[0066] Performing feature fusion processing on the first image feature, the second image feature, and the speech feature to obtain corresponding fusion features;

[0067] Calling a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism;

[0068] The fused features are recognized and processed based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0069] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0070] The present application first collects multimodal data corresponding to the user; then pre-processes the light image and the infrared image based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image; and pre-processes the voice data based on a preset second processing strategy to obtain a corresponding target voice; then performs feature extraction on the target light image to obtain a corresponding first image feature, performs feature extraction on the target infrared image to obtain a corresponding second image feature, and performs feature extraction on the target voice to obtain a corresponding voice feature; subsequently, performs feature fusion processing on the first image feature, the second image feature and the voice feature to obtain a corresponding fusion feature; further calls a pre-trained micro-expression recognition model; finally, performs recognition processing on the fusion feature based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user. This application obtains fused features by preprocessing, feature extraction and feature fusion processing on the collected multimodal data, and subsequently uses a micro-expression recognition model built based on the introduction of a teacher-student network and a knowledge distillation mechanism to recognize and process the fused features output by the cross-modal converter. This can effectively improve the recognition accuracy of micro-expression recognition processing for users and improve the accuracy of the obtained micro-expression recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0073] Figure 2 A flowchart of an embodiment of a micro-expression recognition method based on artificial intelligence according to the present application;

[0074] Figure 3 It is a structural schematic diagram of an embodiment of a micro-expression recognition device based on artificial intelligence according to the present application;

[0075] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0077] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0078] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0079] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0080] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0081] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0082] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0083] It should be noted that the artificial intelligence-based micro-expression recognition method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based micro-expression recognition device is generally set in the server / terminal device.

[0084] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0085] Continue to refer Figure 2, showing a flow chart of an embodiment of an artificial intelligence-based micro-expression recognition method according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The artificial intelligence-based micro-expression recognition method provided in the embodiment of the present application can be applied to any scenario that requires micro-expression recognition, and the artificial intelligence-based micro-expression recognition method can be applied to products in these scenarios, for example, micro-expression recognition in the fields of intelligent customer service, financial transactions, judicial interrogations, clinical medicine, etc. The artificial intelligence-based micro-expression recognition method comprises the following steps:

[0086] Step S201, collecting multimodal data corresponding to the user, wherein the multimodal data includes light images, infrared images and voice data.

[0087] In this embodiment, the electronic device (e.g. Figure 1 The server / terminal device shown in the figure) can obtain multimodal data through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. Among them, the above-mentioned multimodal data includes light images, infrared images and voice data. In addition, the specific implementation process of the above-mentioned collection of multimodal data corresponding to the user will be further described in detail in the subsequent specific embodiments of this application, which will not be elaborated here.

[0088] Step S202 , preprocessing the light image and the infrared image respectively based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image.

[0089] In the present embodiment, the above-mentioned specific implementation process of preprocessing the light image and the infrared image respectively based on the preset first processing strategy to obtain the target light image corresponding to the light image and the target infrared image corresponding to the infrared image will be further described in detail in the subsequent specific embodiments of the present application and will not be elaborated in detail here.

[0090] Step S203: pre-process the speech data based on a preset second processing strategy to obtain a corresponding target speech.

[0091] In this embodiment, the above-mentioned specific implementation process of preprocessing the voice data based on the preset second processing strategy to obtain the corresponding target voice will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated in detail here.

[0092] Step S204, extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target voice to obtain corresponding voice features.

[0093] In this embodiment, the specific implementation process of extracting the features of the target light image to obtain the corresponding first image features, extracting the features of the target infrared image to obtain the corresponding second image features, and extracting the features of the target speech to obtain the corresponding speech features will be described in further detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0094] Step S205: performing feature fusion processing on the first image feature, the second image feature and the speech feature to obtain corresponding fusion features.

[0095] In this embodiment, the above-mentioned feature fusion processing of the first image feature, the second image feature and the voice feature to obtain the corresponding fusion feature is carried out. This application will provide further details on this in subsequent specific embodiments and will not be elaborated on here.

[0096] Step S206, calling a pre-trained micro-expression recognition model.

[0097] In this embodiment, the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model obtained by training a preset initial student network based on the knowledge distillation mechanism. The specific implementation process corresponding to the process of training the preset initial student network based on the knowledge distillation mechanism will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0098] Step S207: performing recognition processing on the fused features based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0099] In this embodiment, the fused features may be input into the micro-expression recognition model, the fused features may be recognized and processed by the micro-expression recognition model, and a micro-expression recognition result corresponding to the user may be generated.

[0100] The present application first collects multimodal data corresponding to the user; then pre-processes the light image and the infrared image based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image; and pre-processes the voice data based on a preset second processing strategy to obtain a corresponding target voice; then performs feature extraction on the target light image to obtain a corresponding first image feature, performs feature extraction on the target infrared image to obtain a corresponding second image feature, and performs feature extraction on the target voice to obtain a corresponding voice feature; subsequently, performs feature fusion processing on the first image feature, the second image feature and the voice feature to obtain a corresponding fusion feature; further calls a pre-trained micro-expression recognition model; finally, performs recognition processing on the fusion feature based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user. This application obtains fused features by preprocessing, feature extraction and feature fusion processing on the collected multimodal data, and subsequently uses a micro-expression recognition model built based on the introduction of a teacher-student network and a knowledge distillation mechanism to recognize and process the fused features output by the cross-modal converter. This can effectively improve the recognition accuracy of micro-expression recognition processing for users and improve the accuracy of the obtained micro-expression recognition results.

[0101] In some optional implementations of this embodiment, before step S206, the electronic device may further perform the following steps:

[0102] Get the pre-built initial teacher network and initial student network.

[0103] In this embodiment, the initial student network may specifically adopt an LSTM+Softmax classification structure. The initial teacher network may specifically adopt a more complex model structure with superior performance to learn the characteristics and classification methods of micro-expressions. For example, a classification model based on attention or graph neural network may be used as the initial teacher network.

[0104] Acquire pre-collected multimodal micro-expression data.

[0105] In this embodiment, the multimodal micro-expression data may include micro-expression data of three modes corresponding to light images, infrared images and voice data. When the multimodal micro-expression data is used to train the initial teacher model, the micro-expression data is pre-processed, feature extracted and fused to obtain the corresponding multimodal fusion features, and the multimodal fusion features are used for the subsequent model training of the initial teacher model.

[0106] The initial teacher model is trained based on the multimodal micro-expression data to obtain a trained teacher network.

[0107] In this embodiment, the initial teacher model is trained by taking the above-mentioned multimodal fusion features obtained after processing corresponding to the multimodal micro-expression data as input, so that the initial teacher model can learn the characteristics and classification methods of micro-expressions, thereby obtaining a trained teacher network. During the training process, data enhancement, regularization and other technologies can be used to improve the generalization ability of the obtained teacher network. The obtained teacher network can provide more reliable soft labels to guide the student model learning.

[0108] The trained teacher network is used as the teacher model, and the initial student network is used as the student of the teacher model for training by minimizing the KL divergence loss function to obtain a trained student model.

[0109] In this embodiment, in order to transfer the prior knowledge learned by the teacher model to the student model, knowledge distillation is performed. Specifically, the migration of knowledge such as soft labels and feature maps is achieved by minimizing the KL divergence loss function, thereby effectively enhancing the generalization ability of the student model. The above-mentioned process of training the initial student network as a student of the teacher model by minimizing the KL divergence loss function includes: (1) Soft label migration: The softmax output layer of the teacher model represents the probability distribution of the classification category, which provides richer supervised information as a soft label. The soft label output by the teacher model is used as supervision, and the student model minimizes the KL divergence loss to make its own predicted probability distribution close to the teacher model: Loss {soft} =KL(P T ||P S ), where P T Output soft labels for the teacher model, P s is the probability distribution learned by the student model. Minimizing the KL divergence is equivalent to maximizing the log likelihood, which can transfer soft label knowledge. (2) Deep feature transfer: Let the student model imitate the deep features of the middle layer of the teacher model and establish a mapping relationship between features. Suppose the l-th layer features in the teacher model and the student model are: The loss function for transferring feature knowledge is: (3) Joint training: Combine soft labels and deep features for migration. The joint training loss function is: Loss = Loss {soft} +w*Loss {feat} w is the weight coefficient for adjusting the degree of feature transfer loss. Through the above knowledge distillation training based on minimizing the KL divergence loss function, the student model can imitate the teacher model and obtain better generalization performance.

[0110] The trained student model is fine-tuned and optimized to obtain a corresponding target student model.

[0111] In this embodiment, the performance of the student model in the micro-expression recognition task can be further improved by adjusting the parameters of the student model, adding additional feature extraction layers or classification layers, etc., thereby obtaining a target student model.

[0112] The target student model is used as the micro-expression recognition model.

[0113] The present application obtains a pre-constructed initial teacher network and an initial student network; and obtains pre-collected multimodal micro-expression data; then trains the initial teacher model based on the multimodal micro-expression data to obtain a trained teacher network; then uses the trained teacher network as the teacher model, and trains the initial student network as the student of the teacher model by minimizing the KL divergence loss function to obtain a trained student model; subsequently fine-tunes and optimizes the trained student model to obtain a corresponding target student model, and uses the target student model as the micro-expression recognition model. The present application introduces a teacher-student network and a knowledge distillation mechanism. After training the initial teacher model based on the multimodal micro-expression data to obtain a trained teacher network, the teacher network is used as the teacher model, and the initial student network is used as the student of the teacher model by minimizing the KL divergence loss function to obtain a trained student model, and subsequently fine-tunes and optimizes the trained student model to obtain the final micro-expression recognition model. This application transfers the prior knowledge learned by the teacher model to the student model for knowledge distillation, which effectively enhances the generalization ability of the student model, facilitates the subsequent use of the micro-expression recognition model to identify and process the fused features output by the cross-modal converter, and can effectively improve the recognition accuracy of micro-expression recognition processing for users, thereby improving the accuracy of the obtained micro-expression recognition results.

[0114] In some optional implementations, step S201 includes the following steps:

[0115] A light image corresponding to the face of the user is collected based on a preset camera.

[0116] In this embodiment, the camera can be a high-speed camera, such as a visible light camera corresponding to the smallest model of Photron FASTCAM. The high-speed camera is used to collect light video images corresponding to changes in facial expressions. Typical indicators of the high-speed camera are: resolution 1024x1024, and acquisition frame rate up to 2000fps. The user faces the camera and changes his expression, and the camera is fixed in position to ensure that the face is in the center of the picture. At the same time, three lighting lamps are set to illuminate the front of the subject's face.

[0117] A facial thermal distribution image corresponding to the user's face is collected based on a preset thermal imager, and the facial thermal distribution image is used as the infrared image.

[0118] In this embodiment, the thermal imager can be FLIR E95 24 degrees. The typical parameters of the thermal imager are: resolution 464x348, acquisition frame rate 50Hz, temperature measurement range -20℃~+150℃, temperature sensitivity <50mK. The user faces the thermal imager, roughly in the same position as the visible light camera. The thermal imager is fixed.

[0119] The voice data corresponding to the user is collected based on a preset recording device.

[0120] In this embodiment, the recording device may be a professional recording device, such as Neumann TLM 102, which is a large diaphragm condenser microphone with a frequency response of 20Hz-20kHz and a sensitivity of 7mV / Pa. The microphone is placed 30cm in front of the user's face to ensure the sound reception effect. The voice data of the user when the expression changes is collected and stored. Among them, the optical image, infrared image and voice data are collected synchronously by triggering, and the data storage format is MAT or AVI, etc.

[0121] The present application collects a light image corresponding to the user's face based on a preset camera; collects a facial thermal distribution image corresponding to the user's face based on a preset thermal imager, and uses the facial thermal distribution image as the infrared image; and collects voice data corresponding to the user based on a preset recording device. The present application can quickly and accurately collect multimodal data of the user corresponding to light images, infrared images, and voice data based on the use of cameras, thermal imagers, and recording devices, thereby improving the efficiency of obtaining multimodal data and ensuring the accuracy of multimodal data.

[0122] In some optional implementations of this embodiment, step S202 includes the following steps:

[0123] The optical image and the infrared image are respectively subjected to image cropping processing to obtain a corresponding first optical image and a first infrared image.

[0124] In this embodiment, the above-mentioned image cropping process can adopt a face detection method based on deep learning (such as MTCNN) to detect and obtain the facial area, and crop the light image and the infrared image into an image sequence containing only the face, thereby obtaining a first light image and a first infrared image.

[0125] Perform illumination normalization processing on the first light image and the first infrared image respectively to obtain a corresponding second light image and a second infrared image.

[0126] In this embodiment, the illumination normalization process may use an algorithm based on the Retinex theory to eliminate the influence of illumination, so that the illumination of the facial image is uniform.

[0127] The second light image and the second infrared image are scale-normalized to obtain a corresponding third light image and a third infrared image.

[0128] In this embodiment, the scale normalization process refers to scaling the cropped facial image to a uniform resolution. The resolution can be set according to actual usage requirements, for example, it can be set to 2048*2048.

[0129] The third light image and the third infrared image are respectively subjected to mean variance normalization processing to obtain a corresponding fourth light image and a fourth infrared image.

[0130] In this embodiment, the above-mentioned mean variance normalization processing refers to performing mean variance normalization on the three RGB channels of the image sequence respectively, even if the image data distribution is similar.

[0131] The fourth light image is used as the target light image, and the fourth infrared image is used as the target infrared image.

[0132] The present application performs image cropping, illumination normalization, scale normalization and mean variance normalization on the light image and the infrared image, thereby achieving rapid and accurate preprocessing of the light image and the infrared image, thereby improving the accuracy of the obtained target light image and target infrared image.

[0133] In some optional implementations, step S203 includes the following steps:

[0134] The voice data is subjected to silence elimination processing to obtain a corresponding first voice.

[0135] In this embodiment, the above-mentioned silence elimination processing refers to the processing of detecting and removing non-speech segments in a speech sequence and retaining only the speech content.

[0136] The first speech is subjected to denoising processing to obtain a corresponding second speech.

[0137] In this embodiment, the above-mentioned denoising process refers to the process of removing background noise in speech using a deep learning denoising algorithm (such as DeepXi).

[0138] The second speech is subjected to frame processing to obtain a corresponding third speech.

[0139] In this embodiment, the above-mentioned frame processing refers to the processing of dividing the speech signal into short-time frames with a length of 30ms and a frame shift of 10ms.

[0140] Perform spectrum extraction processing on the third speech to obtain a corresponding fourth speech.

[0141] In this embodiment, the spectrum extraction process refers to extracting Mel spectrum features for each speech frame as a representation of the speech sequence.

[0142] The fourth speech is used as the target speech.

[0143] The present application performs silence elimination, denoising, framing and spectrum extraction processing on the speech data, thereby achieving fast and accurate preprocessing of the speech data and improving the accuracy of the obtained target speech.

[0144] In some optional implementations, step S204 includes the following steps:

[0145] Based on a preset convolutional neural network, feature extraction is performed on the target light image to obtain the first image feature.

[0146] In this embodiment, the above-mentioned convolutional neural network can specifically be a pre-built convolutional neural network based on the attention mechanism. The network structure of the above-mentioned convolutional neural network is: input layer, convolution layer (including 3 groups of convolution blocks with residual structure), attention layer, and fully connected layer. Among them, the attention layer includes position attention mechanism and channel attention mechanism, which can learn the importance of facial areas, so as to focus on micro-expression related areas. The fully connected layer maps the features to 2048 dimensions as the final visible image descriptor, reflecting static visual information. In addition, the input of the convolutional neural network is a preprocessed 2048*2048 optical facial image.

[0147] Based on a preset convolutional stacked autoencoder, feature extraction is performed on the target infrared image to obtain the second image feature.

[0148] In this embodiment, the above-mentioned convolution stacked autoencoder contains multiple groups of convolutional layers to gradually obtain the deep feature expression of the infrared image. The corresponding decoder reconstructs the image through the deconvolution layer to achieve feature extraction. Among them, the input of the convolution stacked autoencoder is the preprocessed 2048*2048 infrared thermal facial image, and finally outputs 2048-dimensional features from the middle layer of the encoder to describe the facial thermal distribution and reflect the dynamic information of micro-expressions.

[0149] The target speech is subjected to feature extraction based on a preset speech feature learning network to obtain the speech feature.

[0150] In this embodiment, the speech feature learning network can be specifically a speech feature learning network based on an LSTM module. The input of the speech feature learning network is a Mel spectrum feature sequence of fixed length. The speech feature learning network includes two layers of LSTM modules, which can model the data time sequence dependency of speech. The Attention layer is added to the final time sequence output for feature focusing to obtain a speech descriptor. Finally, a 2048-dimensional feature is output through a fully connected layer to reflect the dynamic change information of speech.

[0151] The present application extracts features of the target light image based on a preset convolutional neural network to obtain the first image features; extracts features of the target infrared image based on a preset convolutional stacked autoencoder to obtain the second image features; and extracts features of the target speech based on a preset speech feature learning network to obtain the speech features. The present application extracts features of the target light image, the target infrared image, and the target speech respectively by using a convolutional neural network, a convolutional stacked autoencoder, and a speech feature learning network, thereby achieving fast and accurate feature extraction processing of the target light image, the target infrared image, and the target speech, thereby improving the data accuracy of the obtained first image features, second image features, and speech features.

[0152] In some optional implementations of this embodiment, step S205 includes the following steps:

[0153] Invokes a preset cross-modal converter.

[0154] In this embodiment, the cross-modal converter is a pre-built converter for realizing the equivalent conversion and feature alignment of modal representation. The cross-modal converter adopts an encoder-decoder structure, with multiple groups of multiple attention modules in the encoder and multiple cross-attention modules in the decoder. The output of the final converter forms a 2048-dimensional fusion feature. The cross-modal converter consists of an encoder and a decoder, and the overall form is an Encoder-Decoder structure. Specifically, the cross-modal converter includes: (1) Encoder: The encoder contains N = 6 identical layers, each of which contains two parts: a. Multi-head Self-Attention module: The input sequence is first mapped to the three vector representations of Query, Key, and Value, and then the correlation between words is calculated by scaling the dot product attention function, and the attention weight is obtained through the multi-head mechanism, and finally the weighted sum is performed with the Value value to output the enhanced feature representation. b. Fully connected network (Feed Forward Network): It contains two layers of linear mapping and RELU activation function, which is used to transform the feature representation. (2) Decoder The decoder also contains N = 6 identical layers, each of which contains three parts: a. Multi-head Self-Attention module: The structure is similar to that in the encoder. b. Multi-head Cross-Attention module: The Key and Value output by the encoder are subjected to dot product attention with the Query of the decoder to obtain features that model the global dependency between the encoder and the decoder. (3) Fully connected network (Feed Forward Network): The structure is similar to that in the encoder. Finally, the 2048-dimensional features of the three-modal information are fused and output from the decoder.

[0155] The first image feature, the second image feature, and the speech feature are input into the cross-modal converter.

[0156] In this embodiment, the first image feature, the second image feature and the speech feature are taken as input and input into a cross-modal converter.

[0157] The cross-modal converter performs feature conversion processing on the first image feature, the second image feature and the speech feature to obtain corresponding conversion features.

[0158] In this embodiment, feature conversion processing is performed on the first image feature, the second image feature and the speech feature through the cross-modal converter to achieve equivalent conversion and feature alignment of modal representation, and output corresponding conversion features.

[0159] The converted feature is used as the fused feature.

[0160] This application calls a preset cross-modal converter; then inputs the first image feature, the second image feature and the voice feature into the cross-modal converter; subsequently, the cross-modal converter performs feature conversion processing on the first image feature, the second image feature and the voice feature to obtain corresponding conversion features, and uses the conversion features as the fusion features. This application uses a cross-modal converter to perform feature conversion processing on the first image feature, the second image feature and the voice feature, which can realize the intelligent and accurate generation of fusion features that fuse three modal data, realize the equivalent mapping of modal representation, and enhance the association of different modal features. And subsequently, by using a micro-expression recognition model to recognize and process the fusion features output by the cross-modal converter, the recognition accuracy of the micro-expression recognition processing of the user can be effectively improved, and the accuracy of the obtained micro-expression recognition results can be improved.

[0161] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0162] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned micro-expression recognition results, the above-mentioned micro-expression recognition results can also be stored in a node of a blockchain.

[0163] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0164] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0165] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0167] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0168] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a micro-expression recognition device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0169] like Figure 3 As shown, the artificial intelligence-based micro-expression recognition device 300 described in this embodiment includes: a collection module 301, a first preprocessing module 302, a second preprocessing module 303, an extraction module 304, a fusion module 305, a calling module 306 and a recognition module 307. Among them:

[0170] The acquisition module 301 is used to acquire multimodal data corresponding to the user; wherein the multimodal data includes optical images, infrared images and voice data;

[0171] A first preprocessing module 302, configured to preprocess the light image and the infrared image based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image;

[0172] A second preprocessing module 303, configured to preprocess the speech data based on a preset second processing strategy to obtain a corresponding target speech;

[0173] An extraction module 304 is used to perform feature extraction on the target light image to obtain a corresponding first image feature, perform feature extraction on the target infrared image to obtain a corresponding second image feature, and perform feature extraction on the target voice to obtain a corresponding voice feature;

[0174] A fusion module 305 is used to perform feature fusion processing on the first image feature, the second image feature and the speech feature to obtain a corresponding fusion feature;

[0175] The calling module 306 is used to call a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism;

[0176] The recognition module 307 is used to perform recognition processing on the fused features based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

[0177] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0178] In some optional implementations of this embodiment, the micro-expression recognition device based on artificial intelligence further includes:

[0179] A first acquisition module is used to acquire a pre-built initial teacher network and an initial student network;

[0180] A second acquisition module is used to acquire pre-collected multimodal micro-expression data;

[0181] A first training module is used to train the initial teacher model based on the multimodal micro-expression data to obtain a trained teacher network;

[0182] A second training module is used to use the trained teacher network as the teacher model, and train the initial student network as a student of the teacher model by minimizing the KL divergence loss function to obtain a trained student model;

[0183] A processing module, used for fine-tuning and optimizing the trained student model to obtain a corresponding target student model;

[0184] A determination module is used to use the target student model as the micro-expression recognition model.

[0185] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0186] In some optional implementations of this embodiment, the acquisition module 301 includes:

[0187] A first acquisition submodule, configured to acquire a light image corresponding to the user's face based on a preset camera;

[0188] A second acquisition submodule is used to acquire a facial thermal distribution image corresponding to the user's face based on a preset thermal imager, and use the facial thermal distribution image as the infrared image;

[0189] The third collection submodule is used to collect voice data corresponding to the user based on a preset recording device.

[0190] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0191] In some optional implementations of this embodiment, the first preprocessing module 302 includes:

[0192] A first processing submodule, used to perform image cropping processing on the light image and the infrared image respectively to obtain a corresponding first light image and a first infrared image;

[0193] a second processing submodule, performing illumination normalization processing on the first light image and the first infrared image respectively to obtain a corresponding second light image and a second infrared image;

[0194] a third processing submodule, performing scale normalization processing on the second light image and the second infrared image respectively to obtain a corresponding third light image and a third infrared image;

[0195] a fourth processing submodule, performing mean variance normalization processing on the third light image and the third infrared image respectively to obtain a corresponding fourth light image and a fourth infrared image;

[0196] The first determining submodule uses the fourth light image as the target light image and uses the fourth infrared image as the target infrared image.

[0197] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation method, and will not be repeated here.

[0198] In some optional implementations of this embodiment, the second preprocessing module 303 includes:

[0199] A fifth processing submodule, configured to perform a mute removal process on the voice data to obtain a corresponding first voice;

[0200] A sixth processing submodule, configured to perform denoising on the first speech to obtain a corresponding second speech;

[0201] A seventh processing submodule, configured to perform frame processing on the second speech to obtain a corresponding third speech;

[0202] an eighth processing submodule, configured to perform spectrum extraction processing on the third speech to obtain a corresponding fourth speech;

[0203] The second determining submodule is used to use the fourth speech as the target speech.

[0204] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0205] In some optional implementations of this embodiment, the extraction module 304 includes:

[0206] A first extraction submodule, configured to extract features of the target light image based on a preset convolutional neural network to obtain the first image features;

[0207] A second extraction submodule, used for performing feature extraction on the target infrared image based on a preset convolutional stacked autoencoder to obtain the second image feature;

[0208] The third extraction submodule is used to extract features of the target speech based on a preset speech feature learning network to obtain the speech features.

[0209] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0210] In some optional implementations of this embodiment, the fusion module 305 includes:

[0211] A calling submodule is used to call a preset cross-modal converter;

[0212] An input submodule, configured to input the first image feature, the second image feature, and the speech feature into the cross-modal converter;

[0213] a conversion submodule, configured to perform feature conversion processing on the first image feature, the second image feature and the speech feature through the cross-modal converter to obtain corresponding conversion features;

[0214] The third determining submodule is configured to use the conversion feature as the fusion feature.

[0215] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the micro-expression recognition method based on artificial intelligence in the aforementioned implementation mode, and will not be repeated here.

[0216] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0217] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0218] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0219] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a micro-expression recognition method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0220] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the artificial intelligence-based micro-expression recognition method.

[0221] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0222] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based micro-expression recognition method.

[0223] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0224] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A micro-expression recognition method based on artificial intelligence, characterized in that: The steps include: Collecting multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data; Preprocessing the light image and the infrared image respectively based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image; Preprocessing the speech data based on a preset second processing strategy to obtain a corresponding target speech; Extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features; Performing feature fusion processing on the first image feature, the second image feature, and the speech feature to obtain corresponding fusion features; Calling a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism; The fused features are recognized and processed based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

2. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: Before the step of calling the pre-trained micro-expression recognition model, the method further includes: Get the pre-built initial teacher network and initial student network; Acquire pre-collected multimodal micro-expression data; Training the initial teacher model based on the multimodal micro-expression data to obtain a trained teacher network; The trained teacher network is used as the teacher model, and the initial student network is used as the student of the teacher model for training by minimizing the KL divergence loss function to obtain a trained student model; Fine-tune and optimize the trained student model to obtain a corresponding target student model; The target student model is used as the micro-expression recognition model.

3. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: The step of collecting multimodal data corresponding to the user specifically includes: Capturing a light image corresponding to the user's face based on a preset camera; collecting a facial thermal distribution image corresponding to the user's face based on a preset thermal imager, and using the facial thermal distribution image as the infrared image; The voice data corresponding to the user is collected based on a preset recording device.

4. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: The step of preprocessing the light image and the infrared image based on the preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image specifically includes: Performing image cropping processing on the light image and the infrared image respectively to obtain a corresponding first light image and a first infrared image; performing illumination normalization processing on the first light image and the first infrared image respectively to obtain a corresponding second light image and a second infrared image; performing scale normalization processing on the second light image and the second infrared image respectively to obtain a corresponding third light image and a third infrared image; performing mean-variance normalization processing on the third light image and the third infrared image respectively to obtain a corresponding fourth light image and fourth infrared image; The fourth light image is used as the target light image, and the fourth infrared image is used as the target infrared image.

5. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: The step of preprocessing the speech data based on the preset second processing strategy to obtain the corresponding target speech specifically includes: Performing a mute removal process on the voice data to obtain a corresponding first voice; Performing denoising on the first speech to obtain a corresponding second speech; Performing frame processing on the second speech to obtain a corresponding third speech; Performing spectrum extraction processing on the third speech to obtain a corresponding fourth speech; The fourth speech is used as the target speech.

6. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: The steps of extracting features from the target light image to obtain corresponding first image features, extracting features from the target infrared image to obtain corresponding second image features, and extracting features from the target speech to obtain corresponding speech features specifically include: Extracting features of the target light image based on a preset convolutional neural network to obtain the first image features; Extracting features of the target infrared image based on a preset convolutional stacked autoencoder to obtain the second image features; The target speech is subjected to feature extraction based on a preset speech feature learning network to obtain the speech feature.

7. The micro-expression recognition method based on artificial intelligence according to claim 1, characterized in that: The step of performing feature fusion processing on the first image feature, the second image feature and the voice feature to obtain corresponding fusion features specifically includes: Call the preset cross-modal converter; Inputting the first image feature, the second image feature, and the speech feature into the cross-modal converter; Performing feature conversion processing on the first image feature, the second image feature, and the speech feature through the cross-modal converter to obtain corresponding conversion features; The converted feature is used as the fused feature.

8. A micro-expression recognition device based on artificial intelligence, characterized in that: include: A collection module, used to collect multimodal data corresponding to the user; wherein the multimodal data includes light images, infrared images and voice data; A first preprocessing module, configured to preprocess the light image and the infrared image based on a preset first processing strategy to obtain a target light image corresponding to the light image and a target infrared image corresponding to the infrared image; A second preprocessing module, used to preprocess the speech data based on a preset second processing strategy to obtain a corresponding target speech; An extraction module, configured to perform feature extraction on the target light image to obtain a corresponding first image feature, perform feature extraction on the target infrared image to obtain a corresponding second image feature, and perform feature extraction on the target voice to obtain a corresponding voice feature; A fusion module, used for performing feature fusion processing on the first image feature, the second image feature and the speech feature to obtain corresponding fusion features; A calling module, used to call a pre-trained micro-expression recognition model; wherein the micro-expression recognition model is a model constructed based on a trained student model, wherein the trained student model is a teacher model trained by pre-collected multimodal micro-expression data, and is a model obtained by training a preset initial student network based on a knowledge distillation mechanism; The recognition module is used to perform recognition processing on the fused features based on the micro-expression recognition model to generate a micro-expression recognition result corresponding to the user.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the micro-expression recognition method based on artificial intelligence are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the micro-expression recognition method based on artificial intelligence as described in any one of claims 1 to 7 are implemented.

Images