CLIP-based micro-expression recognition method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of image processing and digital medical treatment, and particularly discloses a CLIP-based micro-expression recognition method and device, computer equipment and a storage medium. According to the method, feature recognition is performed on the to-be-recognized micro-expression image to obtain the image feature vector, similarity calculation is performed on the image feature vector and the text feature vector, and the micro-expression description text corresponding to the image feature vector is determined according to the similarity calculation result to serve as the recognition result of the to-be-recognized micro-expression. According to the method, the recognition result is obtained through image feature comparison, the text features are combined on the basis of the image features, the psychological activities contained in the micro-expressions to be recognized can be accurately recognized, and the accuracy of the micro-expression recognition result is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing and digital medical technology, and in particular to a CLIP-based micro-expression recognition method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer vision technology, image recognition technology has been gradually applied to the medical field. For example, by analyzing the patient's micro-expressions, the patient's pain condition can be determined to avoid the patient's unclear description of his or her pain condition and improve the accuracy of the doctor's diagnosis of the disease. At present, micro-expressions are identified by extracting features and assembling facial muscle information. However, to determine micro-expressions by extracting features and then assembling facial muscle information combinations, it is necessary to extract features and assemble multiple frames of micro-expression images. In addition, there are many types of micro-expressions, and micro-expressions cannot be accurately identified only by muscle information; secondly, there are many types of facial muscle combinations, and psychological activities are also a component of micro-expressions, which cannot be reflected in the dimensional space of muscles. This leads to low accuracy of micro-expression recognition results. Therefore, how to improve the accuracy of micro-expression recognition has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a CLI-based micro-expression recognition method, apparatus, computer equipment, and storage medium to improve the accuracy of micro-expression recognition.

[0004] In a first aspect, the present application provides a micro-expression recognition method based on CLIP, the method comprising:

[0005] Obtain a micro-expression image to be recognized;

[0006] Performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized;

[0007] Performing similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector;

[0008] Based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, a recognition result of the micro-expression to be recognized is obtained.

[0009] In a second aspect, the present application further provides a CLIP-based micro-expression recognition device, the device comprising:

[0010] A micro-expression image acquisition module to be identified, used to acquire the micro-expression image to be identified;

[0011] An image feature vector acquisition module, used for performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain the image feature vector of the micro-expression to be recognized;

[0012] A similarity result obtaining module is used to perform similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector;

[0013] The recognition result determination module is used for obtaining the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector.

[0014] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the CLIP-based micro-expression recognition method as described above when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the CLIP-based micro-expression recognition method as described above.

[0016] The present application discloses a micro-expression recognition method, device, computer equipment and storage medium based on CLIP, which obtains a micro-expression image to be recognized; performs feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized; performs similarity calculation between a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector; obtains a recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector. The present application obtains an image feature vector by performing feature recognition on the micro-expression image to be recognized, performs similarity calculation between the image feature vector and the text feature vector, and determines the micro-expression description text corresponding to the image feature vector as the recognition result of the micro-expression to be recognized according to the similarity calculation result. The present application not only obtains the recognition result by comparing the image features, but also combines the text features on the basis of the image features, and can accurately recognize the psychological activities contained in the micro-expression to be recognized, thereby improving the accuracy of the micro-expression recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic flow chart of a first embodiment of a micro-expression recognition method based on CLIP provided in an embodiment of the present application;

[0019] Figure 2 is a schematic flow chart of a second embodiment of a micro-expression recognition method based on CLIP provided in an embodiment of the present application;

[0020] Figure 3 A schematic block diagram of a micro-expression recognition device based on CLIP provided in an embodiment of the present application;

[0021] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] The embodiment of the present application provides a micro-expression recognition method, device, computer equipment and storage medium based on CLIP. The micro-expression recognition method based on CLIP can be applied to a server, by performing feature recognition on a micro-expression image to be recognized to obtain an image feature vector, performing similarity calculation on the image feature vector and the text feature vector, determining the text feature vector corresponding to the image feature vector according to the similarity calculation result, and then obtaining the micro-expression description text corresponding to the text feature vector as the recognition result of the micro-expression to be recognized, combining the text features on the basis of the image features, and being able to accurately recognize the psychological activities contained in the micro-expression to be recognized, thereby improving the accuracy of the micro-expression recognition result. The server can be an independent server or a server cluster.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] See also Figure 1 , Figure 1 This is a schematic flow chart of a micro-expression recognition method based on CLIP provided in an embodiment of the present application. The micro-expression recognition method based on CLIP can be applied to a server, by performing feature recognition on a micro-expression image to be recognized to obtain an image feature vector, performing similarity calculation on the image feature vector and the text feature vector, determining the text feature vector corresponding to the image feature vector according to the similarity calculation result, and then obtaining the micro-expression description text corresponding to the text feature vector as the recognition result of the micro-expression to be recognized, combining the text features on the basis of the image features, and being able to accurately recognize the psychological activities contained in the micro-expression to be recognized, thereby improving the accuracy of the micro-expression recognition result.

[0029] like Figure 1 As shown, the micro-expression recognition method based on CLI specifically includes steps S101 to S104.

[0030] S101, obtaining a micro-expression image to be recognized;

[0031] In one embodiment, the micro-expression image to be recognized is obtained based on the face image to be recognized. Specifically, face detection is performed on the face image to be recognized to obtain the area where the face is located, and the face image to be recognized is cropped according to the area where the face is located to obtain the micro-expression image to be recognized.

[0032] Furthermore, the method of obtaining the micro-expression image to be identified includes: obtaining a facial image to be identified, wherein the facial image to be identified includes at least one face; when the facial image to be identified is a single image, performing face detection on the facial image to be identified to determine an area where at least one face is located; and cropping the facial image to be identified based on the area where the face is located to obtain the micro-expression image to be identified.

[0033] In one embodiment, an image or video stream containing a face may be acquired through a camera or other image acquisition device. When the camera or other image acquisition device acquires a single image containing a face, the face image to be identified is a single image. When the camera or other image acquisition device acquires a video stream containing a face, the image in the video stream is extracted frame by frame to obtain continuous face images to be identified.

[0034] In one embodiment, a face detection algorithm is used to locate the face region in the image. Common face detection algorithms include Haar cascade classifiers, convolutional neural networks (CNNs) in deep learning, etc. The face detection algorithm will output the location information of the face, usually in the form of a rectangular frame to mark the location of the face, including the bounding box coordinates of the face. According to the face region determined by the face detection algorithm, the original image is cropped, only the part containing the face is retained, and the background and other interference elements in the image that are not related to micro-expression recognition are removed.

[0035] In one embodiment, the cropped image may be preprocessed to improve the accuracy of subsequent micro-expression recognition. The preprocessing steps may include grayscale, denoising, brightness adjustment, contrast enhancement, etc.

[0036] Furthermore, after acquiring the facial image to be identified, the method further includes: when the facial image to be identified is a continuous image, generating a time series based on the continuous image frames of the facial image to be identified; and processing the facial image to be identified according to the time series to obtain a continuous micro-expression image to be identified.

[0037] In one embodiment, when the face image to be recognized is a series of continuous image frames (for example, captured from a video stream), it is first necessary to generate a time sequence based on these image frames. A time sequence is a data structure representing the order of image frames, which can be a simple list of frame numbers or a complex structure containing timestamps.

[0038] In one embodiment, each frame of the image is processed according to the generated time sequence to obtain continuous micro-expression images to be identified. The processing steps include face recognition and cropping the image according to the area where the face is located to obtain continuous micro-expression images to be identified.

[0039] S102, performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized;

[0040] In one embodiment, the CLI P model, namely the Contrast i ve Language-Image Pre-training model, is a pre-training model for joint representation learning of images and texts, consisting of an image encoder and a text encoder, which are used to encode images and texts respectively, and learn the similarity between images and texts.

[0041] In one embodiment, the micro-expression image to be recognized is used as an input parameter of the CLIP model, and an image feature vector is obtained through an image encoder of the CLIP model.

[0042] S103, performing similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector;

[0043] Furthermore, the similarity calculation between the preset text feature vector and the image feature vector is performed to obtain the similarity calculation result between the image feature vector and the text feature vector, including: calculating the vector dot product between the image feature vector and the text feature vector; scaling the vector dot product based on a preset scaling factor to obtain a first parameter for similarity calculation; and obtaining the similarity calculation result between the image feature vector and the text feature vector based on a preset similarity calculation formula, the first parameter, the image feature vector and the text feature vector.

[0044] In one embodiment, the vector dot product of the image feature vector and the text feature vector is calculated. Assume that the image feature vector is I = (i1, i2, ..., i n ), the text feature vector is T = (t1, t2, ..., t m ), where n and m are the dimensions of the two vectors. The dot product of the vectors is calculated as:

[0045] K=∑(i1×t1)+∑(i2×t2)+...+∑(i n ×t m )

[0046] In one embodiment, the vector dot product is scaled based on a preset scaling factor to obtain a first parameter for similarity calculation. Assuming that the preset scaling factor is s, the scaled dot product is K_scaled=s*K.

[0047] In one embodiment, based on the first parameter, the image feature vector and the text feature vector, the similarity calculation result of the image feature vector and the text feature vector is obtained: assuming that the similarity calculation result is S, the similarity calculation formula is S=K_scaled / (||I||*||T||), where ||I|| and ||T|| are the moduli of the image feature vector and the text feature vector, respectively.

[0048] Among them, the determination of the scaling factor usually depends on multiple factors, including but not limited to the characteristics of the data, the requirements of the model, and the results of experiments and tuning. For example, analyzing the distribution characteristics of the feature vector, such as mean, variance, maximum value, minimum value, etc., if the range of element values ​​of the feature vector is large, it may cause the dot product result to be too large, thus affecting the calculation of similarity. At this time, a smaller scaling factor can be selected to reduce the numerical range of the dot product result. Or in the scaled dot product attention mechanism, the scaling factor is used to balance the size of the dot product result to avoid numerical problems caused by too high a dimension. At this time, the scaling factor is usually the inverse of the square root of the vector dimension.

[0049] In one embodiment, the vector dot product of the current image feature vector and each text feature vector is calculated one by one, and then each vector dot product is scaled according to the scaling factor to obtain the first parameter corresponding to each vector dot product. Finally, the similarity between the current image feature vector and each text feature vector is calculated according to the similarity calculation formula, the first parameter, the image feature vector and the text feature vector to obtain the similarity calculation result.

[0050] S104, obtaining a recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector.

[0051] Furthermore, the recognition result of the micro-expression to be recognized is obtained based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, including: when the micro-expression image to be recognized is a single image, taking the text feature vector with the highest similarity as the target feature vector; and obtaining the micro-expression description text corresponding to the target feature vector as the recognition result of the micro-expression to be recognized.

[0052] In one embodiment, when the micro-expression image to be recognized is a single image, the micro-expression description text corresponding to the similarity pair Amap text feature vector can be directly obtained as the recognition result of the micro-expression to be recognized.

[0053] In a specific embodiment, when a text feature vector with the highest similarity to an image feature vector is selected as a target feature vector, a micro-expression description text corresponding to the target feature vector is retrieved and obtained. The micro-expression description text specifically describes the features of the micro-expressions in the image, such as the type and intensity of the expression.

[0054] Furthermore, the method of obtaining the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector includes: when the micro-expression images to be recognized are continuous images, when the micro-expression images to be recognized are continuous images, based on the similarity calculation result, determining the micro-expression description text of each image in the continuous micro-expression images to be recognized; based on the time series, performing a comprehensive analysis on each of the micro-expression description texts to obtain the recognition result of the micro-expression to be recognized.

[0055] In one embodiment, the CLIP model is used to perform similarity calculation between the processed micro-expression image and a pre-set text feature vector to obtain a similarity calculation result. Based on the similarity calculation result, the text feature vector with the highest similarity to the image feature of each micro-expression image to be identified is used to determine the micro-expression description text corresponding to the text feature vector as the description text that best matches the micro-expression image to be identified.

[0056] In one embodiment, the determined micro-expression description text is further comprehensively analyzed using the time series information, which may include evaluating the change trend of the micro-expression description text over time, identifying the start and end points of the micro-expression, and the like.

[0057] In a specific embodiment, the micro-expression description texts are arranged in chronological order to form time series data. Statistical analysis is performed on the time series data, such as calculating the average length and occurrence frequency of the micro-expression description texts at each time point. Specifically, a time series analysis method (such as ARI MA model, seasonal decomposition, etc.) can be used to identify trends or periodic changes in the micro-expression description texts.

[0058] In one embodiment, the start and end points of micro-expressions are identified by analyzing the frequency and order of occurrence of keywords in the micro-expression description text. Specifically, natural language processing techniques (such as sentiment analysis, topic modeling, etc.) can be used to assist in identifying the start and end points of micro-expressions.

[0059] In one embodiment, cluster analysis is performed on the description text to find different micro-expression patterns or categories. Specifically, machine learning or deep learning methods can be used to classify the description text to identify different types of micro-expressions.

[0060] In another embodiment, association rules between different words in the description text can also be mined to discover potential micro-expression features or patterns. Specifically, association rule mining algorithms (such as April, FP-Growth, etc.) can be used to discover frequent item sets and association rules between words.

[0061] In another embodiment, sentiment analysis can also be performed on the obtained continuous micro-expression description text to understand the emotional color and emotional changes of the micro-expression. Specifically, sentiment analysis tools (such as sentiment dictionaries, sentiment classifiers, etc.) can be used to perform sentiment analysis on the description text.

[0062] The above embodiment provides a micro-expression recognition method, device, computer equipment and storage medium based on CLIP, which obtains a micro-expression image to be recognized; performs feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized; performs similarity calculation between a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector; obtains a recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector. The present application obtains an image feature vector by performing feature recognition on the micro-expression image to be recognized, performs similarity calculation between the image feature vector and the text feature vector, determines the text feature vector corresponding to the image feature vector according to the similarity calculation result, and then obtains the micro-expression description text corresponding to the text feature vector as the recognition result of the micro-expression to be recognized. The present application not only obtains the recognition result by comparing the image features, but also combines the text features on the basis of the image features, and can accurately recognize the psychological activities contained in the micro-expression to be recognized, thereby improving the accuracy of the micro-expression recognition result.

[0063] See also Figure 2 , Figure 2 This is a schematic flow chart of a micro-expression recognition method based on CLIP provided in an embodiment of the present application. The micro-expression recognition method based on CLIP can be applied to a server, and by performing semantic detection on the acquired natural language description text, the micro-expression description text related to the micro-expression is classified, and then the feature recognition of the micro-expression description text is performed to obtain a text feature vector, thereby improving the correlation between the text feature vector and the micro-expression and improving the accuracy of micro-expression recognition.

[0064] like Figure 2 As shown, the micro-expression recognition method based on CLI specifically includes steps S201 to S203.

[0065] S201, obtaining at least one natural language description text, and performing semantic detection on the natural language description texts one by one to obtain semantic detection results of the natural language description texts;

[0066] S202, classifying the natural language description text based on the semantic detection result to obtain a micro-expression description text;

[0067] S203: Perform text feature recognition on the micro-expression description text to obtain the text feature vector.

[0068] In one embodiment, the natural language description text includes but is not limited to comments on social media, forum posts, news reports, literary works, etc.

[0069] In one embodiment, semantic detection is performed on the acquired natural language description text to determine its semantic content. Semantic detection may include natural language processing technologies such as word segmentation, part-of-speech tagging, named entity recognition, and syntactic analysis. Through these technologies, semantic detection results of each natural language description text can be obtained, and these results can be used in subsequent classification and feature recognition steps.

[0070] In one embodiment, the natural language description text is classified according to the obtained semantic detection results. The classification can be based on predefined categories or customized according to actual needs. For example, the text can be classified based on emotion, theme, style, etc. For this application, the focus is on text related to micro-expressions, so the classification target is the micro-expression description text. Through classification, the text related to micro-expressions is screened out from a large amount of natural language description text, providing a basis for subsequent feature recognition.

[0071] In one embodiment, text feature recognition is performed on the selected micro-expression description text to convert the text into a feature vector. Text feature recognition may include keyword extraction, sentiment analysis, entity recognition, etc.

[0072] In the above embodiment, semantic detection is performed on the acquired natural language description text to classify the micro-expression description text related to the micro-expression, and then feature recognition is performed on the micro-expression description text to obtain a text feature vector, thereby improving the correlation between the text feature vector and the micro-expression and improving the accuracy of micro-expression recognition.

[0073] See also Figure 3 , Figure 3 The embodiment of the present application provides a schematic block diagram of a micro-expression recognition device based on CLIP, which is used to execute the aforementioned micro-expression recognition method based on CLIP. The micro-expression recognition device based on CLIP can be configured on a server.

[0074] like Figure 3 As shown, the micro-expression recognition device 300 based on CLIP includes:

[0075] The micro-expression image acquisition module 301 is used to acquire the micro-expression image to be recognized;

[0076] An image feature vector obtaining module 302 is used to perform feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized;

[0077] A similarity result obtaining module 303 is used to perform similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector;

[0078] In the recognition result determination module 304 , the user obtains the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector.

[0079] Furthermore, the CLI-based micro-expression recognition device 300 further includes:

[0080] A text semantic detection module, used to obtain at least one natural language description text, and perform semantic detection on the natural language description texts one by one to obtain semantic detection results of each natural language description text;

[0081] A description text acquisition module, used to classify the natural language description text based on the semantic detection result to obtain a micro-expression description text;

[0082] The feature vector acquisition module is used to perform text feature recognition on the micro-expression description text to obtain the text feature vector.

[0083] Furthermore, the to-be-recognized micro-expression image acquisition module 301 comprises:

[0084] A face image acquisition unit, used to acquire a face image to be identified, wherein the face image to be identified includes at least one face;

[0085] A face region determination unit, configured to perform face detection on the face image to be identified and determine a region where at least one face is located when the face image to be identified is a single image;

[0086] The micro-expression image obtaining unit is used to crop the face image to be identified based on the area where the face is located to obtain the micro-expression image to be identified.

[0087] Furthermore, the to-be-recognized micro-expression image acquisition module 301 further includes:

[0088] A time series obtaining unit, configured to generate a time series based on continuous image frames of the face images to be identified when the face images to be identified are continuous images;

[0089] The micro-expression image obtaining unit is used to process the face image to be identified according to the time sequence to obtain continuous micro-expression images to be identified.

[0090] Furthermore, the recognition result determination module 304 includes:

[0091] A description text determination unit, configured to determine the micro-expression description text of each image in the continuous micro-expression images to be identified based on the similarity calculation result when the micro-expression images to be identified are continuous images;

[0092] The recognition result obtaining unit is used to perform a comprehensive analysis on each of the micro-expression description texts based on the time series to obtain a recognition result of the micro-expression to be recognized.

[0093] Furthermore, the recognition result determination module 304 includes:

[0094] a target vector determination unit, configured to use the text feature vector with the highest similarity as a target feature vector when the micro-expression image to be identified is a single image;

[0095] The recognition result determination unit is used to obtain the micro-expression description text corresponding to the target feature vector as the recognition result of the micro-expression to be recognized.

[0096] Furthermore, the similarity result obtaining module 303 includes:

[0097] A vector dot product obtaining unit, used for calculating the vector dot product of the image feature vector and the text feature vector;

[0098] A first parameter obtaining unit, used to scale the vector dot product based on a preset scaling factor to obtain a first parameter for similarity calculation;

[0099] The similarity result obtaining unit is used to obtain the similarity calculation result of the image feature vector and the text feature vector based on a preset similarity calculation formula, the first parameter, the image feature vector and the text feature vector.

[0100] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0101] The above-mentioned device can be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0102] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0103] See also Figure 4 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0104] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any micro-expression recognition method based on CLIP.

[0105] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0106] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any micro-expression recognition method based on CLIP.

[0107] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0110] Obtain a micro-expression image to be recognized;

[0111] Performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized;

[0112] Performing similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector;

[0113] Based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, a recognition result of the micro-expression to be recognized is obtained.

[0114] In one embodiment, before the processor performs similarity calculation between the preset text feature vector and the image feature vector to obtain the similarity calculation result between the image feature vector and the text feature vector, it is further used to implement:

[0115] Acquire at least one natural language description text, and perform semantic detection on the natural language description texts one by one to obtain semantic detection results of each natural language description text;

[0116] Based on the semantic detection result, classify the natural language description text to obtain a micro-expression description text;

[0117] Perform text feature recognition on the micro-expression description text to obtain the text feature vector.

[0118] In one embodiment, when acquiring the micro-expression image to be identified, the processor is used to implement:

[0119] Acquire a face image to be identified, wherein the face image to be identified includes at least one face;

[0120] When the face image to be identified is a single image, performing face detection on the face image to be identified to determine a region where at least one face is located;

[0121] The face image to be identified is cropped based on the area where the face is located to obtain the micro-expression image to be identified.

[0122] In one embodiment, after obtaining the face image to be recognized, the processor is further configured to implement:

[0123] When the face images to be recognized are continuous images, generating a time series based on continuous image frames of the face images to be recognized;

[0124] The face image to be identified is processed according to the time sequence to obtain continuous micro-expression images to be identified.

[0125] In one embodiment, when the processor obtains the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, it is used to implement:

[0126] When the micro-expression images to be identified are continuous images, determining the micro-expression description text of each image in the continuous micro-expression images to be identified based on the similarity calculation result;

[0127] Based on the time series, a comprehensive analysis is performed on each of the micro-expression description texts to obtain a recognition result of the micro-expression to be recognized.

[0128] In one embodiment, when the processor obtains the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, it is used to implement:

[0129] When the micro-expression image to be identified is a single image, the text feature vector with the highest similarity is used as the target feature vector;

[0130] The micro-expression description text corresponding to the target feature vector is obtained as the recognition result of the micro-expression to be recognized.

[0131] In one embodiment, when the processor performs similarity calculation between a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector, the processor is used to implement:

[0132] Calculating the vector dot product of the image feature vector and the text feature vector;

[0133] Scaling the vector dot product based on a preset scaling factor to obtain a first parameter for similarity calculation;

[0134] Based on a preset similarity calculation formula, the first parameter, the image feature vector and the text feature vector, a similarity calculation result of the image feature vector and the text feature vector is obtained.

[0135] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the micro-expression recognition methods based on CLIP provided in the embodiments of the present application.

[0136] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Medi a Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc., equipped on the computer device.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A CLIP-based micro-expression recognition method, characterized in that: include: Obtain a micro-expression image to be recognized; Performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain an image feature vector of the micro-expression to be recognized; Performing similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector; Based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector, a recognition result of the micro-expression to be recognized is obtained.

2. The CLIP-based micro-expression recognition method according to claim 1, characterized in that: Before performing similarity calculation on the preset text feature vector and the image feature vector to obtain the similarity calculation result between the image feature vector and the text feature vector, the method further includes: Acquire at least one natural language description text, and perform semantic detection on the natural language description texts one by one to obtain semantic detection results of each natural language description text; Based on the semantic detection result, classify the natural language description text to obtain a micro-expression description text; Perform text feature recognition on the micro-expression description text to obtain the text feature vector.

3. The CLIP-based micro-expression recognition method according to claim 1, characterized in that: The step of obtaining the micro-expression image to be identified includes: Acquire a face image to be identified, wherein the face image to be identified includes at least one face; When the face image to be identified is a single image, performing face detection on the face image to be identified to determine a region where at least one face is located; The face image to be identified is cropped based on the area where the face is located to obtain the micro-expression image to be identified.

4. The CLIP-based micro-expression recognition method according to claim 3, characterized in that: After obtaining the face image to be recognized, the method further includes: When the face images to be recognized are continuous images, generating a time series based on continuous image frames of the face images to be recognized; The face image to be identified is processed according to the time sequence to obtain continuous micro-expression images to be identified.

5. The CLIP-based micro-expression recognition method according to claim 4, characterized in that: The obtaining the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector includes: When the micro-expression images to be identified are continuous images, determining the micro-expression description text of each image in the continuous micro-expression images to be identified based on the similarity calculation result; Based on the time series, a comprehensive analysis is performed on each of the micro-expression description texts to obtain a recognition result of the micro-expression to be recognized.

6. The CLIP-based micro-expression recognition method according to claim 3, characterized in that: The obtaining the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector includes: When the micro-expression image to be identified is a single image, the text feature vector with the highest similarity is used as the target feature vector; The micro-expression description text corresponding to the target feature vector is obtained as the recognition result of the micro-expression to be recognized.

7. The CLIP-based micro-expression recognition method according to any one of claims 1 to 6, characterized in that: The calculating the similarity between the preset text feature vector and the image feature vector to obtain the similarity calculation result between the image feature vector and the text feature vector includes: Calculating the vector dot product of the image feature vector and the text feature vector; Scaling the vector dot product based on a preset scaling factor to obtain a first parameter for similarity calculation; Based on a preset similarity calculation formula, the first parameter, the image feature vector and the text feature vector, a similarity calculation result of the image feature vector and the text feature vector is obtained.

8. A CLIP-based micro-expression recognition device, characterized in that: include: A micro-expression image acquisition module to be identified, used to acquire the micro-expression image to be identified; An image feature vector acquisition module, used for performing feature recognition on the micro-expression image to be recognized based on a preset CLIP model to obtain the image feature vector of the micro-expression to be recognized; A similarity result obtaining module is used to perform similarity calculation on a preset text feature vector and the image feature vector to obtain a similarity calculation result between the image feature vector and the text feature vector; The recognition result determination module is used for obtaining the recognition result of the micro-expression to be recognized based on the similarity calculation result and the micro-expression description text corresponding to the text feature vector.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the CLIP-based micro-expression recognition method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the CLIP-based micro-expression recognition method according to any one of claims 1 to 7.

Images