Face micro-expression video frame number processing method and device, equipment and storage medium

Abstract

The invention provides a face micro-expression video frame number processing method. The method comprises the following steps: acquiring at least two frames of face micro-expression images; extracting a first micro-expression feature from the face micro-expression image, wherein the first micro-expression feature is a key point feature related to a face posture and a facial expression action; generating a low-frame-rate face key point feature sequence according to the first micro-expression feature in each frame of face micro-expression image; and inputting the low-frame-rate face key point feature sequence into a time sequence super-division model to generate a high-frame-rate face key point feature sequence. According to the method, the continuous expression action frames can be generated in the time sequence, the frame rate of the whole video is improved, the problem of low micro expression frame rate can be solved, and high-frame-rate video input is provided for a subsequent micro expression detection algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device, equipment and storage medium for processing the frame number of a facial micro-expression video. Background Art

[0002] Micro-expression recognition algorithms mainly rely on computer vision and deep learning technology, aiming to identify the true emotional state of an individual by analyzing the tiny movements and expression changes of the face. These algorithms usually involve tracking facial feature points, feature extraction, and classifier training to achieve automatic recognition of micro-expressions. The micro-expression recognition method based on hierarchical network (HINN) is one of the micro-expression recognition algorithms. This method first extracts 68 feature points of each frame of the micro-expression video through the cascade regression tree algorithm, and then divides the micro-expression image into four sub-images (eyebrow, eye, nose and mouth images) with these feature points. Then, rich local features are extracted from the sub-images through a hierarchical convolutional network, and then these features are sent to a hierarchical bidirectional recurrent network to learn dynamic sequence features, and finally the obtained high-level features are classified1. There are also micro-expression recognition algorithms based on robust principal component analysis (RPCA) and bidirectional LSTM (RBLSTM). This method uses RPCA to extract sparse information of micro-expressions from the image frames of micro-expression videos, and then sends the extracted sparse information to a bidirectional long short-term memory neural network to obtain global sparse features and classify them. Among the micro-expression recognition algorithms, there are also micro-expression recognition algorithms based on deep fusion networks (DFNN). This method uses linear weighting to fuse the features of the HINN network and the RBLSTM network, and classifies micro-expressions by combining the extracted rich-sparse and local-global features. The experimental results of these algorithms on multiple spontaneous micro-expression databases (such as CASME I, CASME II, CAS(ME)2 and SAMM) show that these methods have good performance compared with existing algorithms. The research and application of micro-expression recognition algorithms have important significance and application prospects in the fields of sentiment analysis, security monitoring, etc.

[0003] Micro-expression algorithms play an important role in financial scenarios, medicine, and clinical applications. Micro-expressions usually last very short periods of time, generally between 1 / 25 and 1 / 5 seconds. In a 25fps video, the process from the beginning to the end of a micro-expression is only 1 to 5 frames. Too few frames will affect the accuracy of micro-expression detection.

[0004] The industry's micro-expression detection algorithms are all used in high frame rate scenarios (60-200fps). However, in actual business scenarios, it is difficult to meet the frame rate required for micro-expression detection.

[0005] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a method, device, equipment and storage medium for processing the frame number of facial micro-expression videos.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] The present invention provides a method for processing the number of frames of a facial micro-expression video, comprising:

[0009] Acquire at least two frames of facial micro-expression images;

[0010] Extracting a first micro-expression feature from the facial micro-expression image, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action;

[0011] Generate a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image;

[0012] The low frame rate facial key point feature sequence is input into the temporal super-resolution model to generate a high frame rate facial key point feature sequence.

[0013] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, after the step of "inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence", further comprises:

[0014] Extracting a second micro-expression feature from the facial micro-expression image, wherein the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement;

[0015] Generate facial micro-expression continuous image frames according to the second micro-expression feature and a high frame rate facial key point feature sequence.

[0016] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, the step of "generating facial micro-expression continuous image frames according to the second micro-expression feature and the high frame rate facial key point feature sequence" specifically includes:

[0017] Extracting facial key point coordinates of consecutive frames in a high frame rate facial key point feature sequence;

[0018] Inputting the coordinates of the facial key points of the continuous frames into the WarpMatrix network to generate an optical flow motion field at the feature level of the facial motion process;

[0019] Performing spatial transformation on the optical flow motion field and the second micro-expression feature to obtain a facial feature after spatial transformation;

[0020] Generate continuous image frames of facial micro-expressions based on the facial features after spatial transformation.

[0021] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, before the step of "inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence", further comprises:

[0022] Obtain facial feature sequences from high frame rate facial videos;

[0023] The facial feature sequence in the high frame rate facial video is used as a supervisory signal for the temporal super-resolution model.

[0024] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, before the step of "inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence", further comprises:

[0025] Obtain several facial micro-expression images from facial videos;

[0026] The acquired facial micro-expression images include multiple groups of facial micro-expression start frame images and facial micro-expression end frame images, forming a facial micro-expression training set.

[0027] Furthermore, the method for processing the number of frames of a facial micro-expression video, before the step of "inputting a facial key point feature sequence with a low frame rate into a temporal super-resolution model to generate a facial key point feature sequence with a high frame rate", after the step of "obtaining a plurality of facial micro-expression images including a plurality of groups of facial micro-expression start frame images and facial micro-expression end frame images to form a facial micro-expression training set", further comprises:

[0028] Extract the first micro-expression feature from the facial micro-expression training set;

[0029] Generate a low-frame-rate facial key point feature sequence training set based on the first micro-expression feature of each group of facial micro-expression start frame images to facial micro-expression end frame images;

[0030] The generated low frame rate facial key point feature sequence training set is input into the temporal super-resolution model for training, and the facial feature sequence in the high frame rate facial video is used as a supervisory signal.

[0031] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video further includes, after the step of "inputting the generated low-frame-rate facial key point feature sequence training set into the temporal super-resolution model for training, and using the facial feature sequence in the high-frame-rate facial video as a supervisory signal":

[0032] Comparing the facial key point feature sequence outputted from the temporal super-resolution model with the facial feature sequence of the supervisory signal;

[0033] Calculating the gap between the facial key point feature sequence and the facial feature sequence through a loss function, and generating a guidance signal according to the gap;

[0034] The guidance signal is input into the temporal super-resolution model.

[0035] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, the step of "extracting a first micro-expression feature from the facial micro-expression image, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action" specifically includes:

[0036] The first micro-expression feature K is extracted through the KpsNet model, where the first micro-expression feature K is a key point feature related to facial posture and facial expression movements.

[0037] In a preferred embodiment, the method for processing the number of frames of a facial micro-expression video, the step of "extracting a second micro-expression feature from the facial micro-expression image, wherein the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement" specifically includes:

[0038] The second micro-expression feature is extracted through the FaceFeat model, where the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement.

[0039] The step of “inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence” specifically includes:

[0040] The low frame rate facial key point feature sequence is input into the transformer model to generate a high frame rate facial key point feature sequence.

[0041] Preferably, the present invention also provides a facial micro-expression video frame number processing device, comprising:

[0042] A micro-expression acquisition module is used to acquire facial micro-expression image frames, wherein the acquired facial micro-expression image frames include facial micro-expression start frame images and facial micro-expression end frame images;

[0043] A facial micro-expression feature extraction module, used to extract a first micro-expression feature, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action;

[0044] A facial feature generation module, used to generate a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image;

[0045] The high frame rate output module is used to input the low frame rate facial key point feature sequence into the temporal super-resolution model to generate a high frame rate facial key point feature sequence.

[0046] Preferably, the present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned method for processing the frame number of facial micro-expression videos when executing the computer-readable instructions.

[0047] Preferably, the present invention also provides a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the steps of the method for processing the frame number of facial micro-expression videos as described above are implemented.

[0048] Compared with the prior art, the embodiments of the present invention mainly have the following beneficial effects:

[0049] The present invention provides a method for processing the number of frames of a facial micro-expression video, comprising: obtaining at least two frames of facial micro-expression images; extracting a first micro-expression feature from the facial micro-expression images, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action; generating a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of the facial micro-expression image; inputting the low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence. The present invention utilizes a temporal super-resolution algorithm, which can generate continuous expression action frames in a temporal sequence, improve the frame rate of the overall video, solve the problem of low micro-expression frame rate, and provide a high-frame-rate video input for a subsequent micro-expression detection algorithm.

[0050] In addition, unless otherwise defined, the meaning of all technical terms used herein is consistent with that commonly understood by those skilled in the art to which the present invention belongs. If there is any inconsistency, the present specification and its included definitions shall prevail. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 An exemplary system architecture diagram is shown in which the present invention can be applied;

[0053] Figure 2 A flowchart of an embodiment of a method for processing the number of frames of a facial micro-expression video according to the present invention is shown;

[0054] Figure 3 A schematic structural diagram of an embodiment of a device for processing the number of frames of a facial micro-expression video according to the present invention is shown;

[0055] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present invention is shown. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs; the terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0061] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0062] Server 105 can be a server that provides various services, such as a background server that provides support for pages displayed on terminal devices 101, 102, and 103. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0063] It should be noted that the method for processing the number of frames of a micro-facial expression video provided in the embodiment of the present invention is generally executed by a server, and accordingly, the device for processing the number of frames of a micro-facial expression video is generally arranged in the server.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for processing the number of frames of a video of micro-expressions of a human face according to the present invention. The present invention provides a method for processing the number of frames of a video of micro-expressions of a human face, comprising:

[0066] S210, acquiring at least two frames of facial micro-expression images;

[0067] S220, extracting a first micro-expression feature from the facial micro-expression image, where the first micro-expression feature is a key point feature representing a facial posture and a facial expression action;

[0068] S230, generating a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of the facial micro-expression image;

[0069] S240: Inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence.

[0070] Among them, the present invention utilizes a temporal super-resolution algorithm, which can generate continuous expression action frames in time sequence, improve the frame rate of the overall video, solve the problem of low frame rate of micro-expressions, and provide high-frame-rate video input for subsequent micro-expression detection algorithms. The facial micro-expression video frame number processing method proposed by the present invention can extract facial key point feature representations that are independent of face ID, and use key points as facial expression feature representations, which is more intuitive and highly interpretable; facial key points are specially provided to reduce the dimension of facial expression features to a certain extent, which is convenient for subsequent temporal super-resolution algorithms. At the same time, the facial temporal super-resolution algorithm based on intermediate features solves the problem of insufficient frame rate of original data in the micro-expression detection algorithm by using temporal super-resolution.

[0071] Preferably, the method for processing the number of frames of a facial micro-expression video provided by the present invention further comprises, after the step of “S240, inputting a facial key point feature sequence with a low frame rate into a temporal super-resolution model to generate a facial key point feature sequence with a high frame rate”:

[0072] S241, extracting a second micro-expression feature from the facial micro-expression image, where the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement;

[0073] S242: Generate facial micro-expression continuous image frames according to the second micro-expression feature and the high frame rate facial key point feature sequence.

[0074] Preferably, in the method for processing the number of frames of a facial micro-expression video provided by the present invention, the step of "S242, generating facial micro-expression continuous image frames according to the second micro-expression feature and the facial key point feature sequence with a high frame rate" specifically comprises:

[0075] S2421, extracting facial key point coordinates of consecutive frames in a facial key point feature sequence with a high frame rate;

[0076] S2422, inputting the coordinates of the facial key points of the continuous frames into the WarpMatrix network to generate an optical flow motion field at the feature level of the facial motion process;

[0077] S2423, performing spatial transformation through the optical flow motion field and the second micro-expression feature to obtain a facial feature after spatial transformation;

[0078] S2424, generating continuous image frames of facial micro-expressions according to the facial features after spatial transformation.

[0079] The overall structure of the present invention can be based on a GAN network, wherein the front part is mainly a face key point feature extraction part, and the back part is mainly a face image generation part.

[0080] Preferably, in the method for processing the number of frames of a facial micro-expression video provided by the present invention, the step of "S220, extracting a first micro-expression feature from the facial micro-expression image, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action" specifically comprises:

[0081] The first micro-expression feature K is extracted through the KpsNet model, where the first micro-expression feature K is a key point feature related to facial posture and facial expression movements.

[0082] Preferably, in the method for processing the number of frames of a facial micro-expression video provided by the present invention, the step of "S241, extracting a second micro-expression feature from the facial micro-expression image, wherein the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement" specifically comprises:

[0083] The second micro-expression feature is extracted through the FaceFeat model, where the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement.

[0084] Preferably, the step of "S240, inputting the low frame rate facial key point feature sequence into the temporal super-resolution model to generate a high frame rate facial key point feature sequence" specifically includes:

[0085] S250, inputting the low frame rate facial key point feature sequence into the transformer model to generate a high frame rate facial key point feature sequence.

[0086] The FaceFeat algorithm is an algorithm for face recognition, and its core steps include face detection and face recognition. Face detection is the process of locating a face, while face recognition is the process of matching the detected face with known face data to identify the face. The FaceFeat algorithm may use one or more of a variety of algorithms such as PCA, ICA, LDA, and EP to achieve efficient and accurate face recognition. In addition, the algorithm may also involve some self-supervised learning methods, such as using masked images to regress HOG features in feature space for network training to improve the performance and generalization ability of the algorithm.

[0087] The KpsNet algorithm is a neural network for key point detection and feature extraction, and is particularly suitable for point cloud registration tasks. The algorithm receives the features of the area where the human frame is located output by MS-RoIAlign, and uses operations such as convolution, downsampling, and upsampling to predict the heat map of human key points. The KpsNet algorithm has important application value in the field of point cloud data processing and analysis. It can efficiently extract key information and assist in subsequent point cloud registration work. The KpsNet algorithm can be used to detect and extract key points of faces.

[0088] FaceFeat represents a facial feature extraction algorithm model, which is used to extract facial features Fs that are independent of posture and facial expression. KpsNet represents a facial key point algorithm model, which extracts facial key point information. The key point information is used to represent facial posture and facial expression. Ks represents the facial key point coordinates of the source image (Source image), and Kd represents the facial key point coordinates of the final image (Driver image). Ks and Kd are used to construct the optical flow information of the facial motion process, and the WarpMatrix network is used to generate the optical flow motion field at the feature level. The generated optical flow motion field is used to transform Fs to obtain the facial features after spatial transformation. Finally, the generator (G) is used to generate an output image (prediction) based on the facial features of the source image and the final image action.

[0089] In the context of facial key point detection or face alignment, Ks and Kd represent the coordinates of facial key points in the "Source" image and the "Driver" image, respectively. These key points usually include the coordinates of the eyes, nose, mouth, eyebrows, chin, etc., which are used to describe the structure and features of the face. Specifically: Ks represents the coordinates of facial key points in the source image. The source image refers to the original image to be face aligned or transformed. Kd represents the coordinates of facial key points in the driver image. The driving image refers to the image that provides the target for face alignment or transformation, that is, we hope that the face in the source image can be transformed into the face style or posture in the driving image.

[0090] In the task of face transformation or face alignment, an algorithm is usually used to calculate the transformation relationship between the facial key points from the source image to the driving image, and then the face in the source image is adjusted according to this transformation relationship to make it more similar or aligned with the face in the driving image.

[0091] The method for processing the number of frames of facial micro-expression videos proposed in the present invention can extract facial key point feature representations that are independent of face ID, and use key points as facial expression feature representations, which is more intuitive and interpretable; facial key points can reduce the dimension of facial expression features to a certain extent, which is convenient for subsequent time series super-resolution algorithms. At the same time, the face time series super-resolution algorithm based on intermediate features solves the problem of insufficient original data frame rate in the micro-expression detection algorithm by using time series super-resolution.

[0092] The present invention relates to training of a model. Preferably, the method for processing the number of frames of a facial micro-expression video provided by the present invention further includes, before the step of "S240, inputting a facial key point feature sequence with a low frame rate into a temporal super-resolution model to generate a facial key point feature sequence with a high frame rate":

[0093] Obtain facial feature sequences from high frame rate facial videos;

[0094] The facial feature sequence in the high frame rate facial video is used as a supervisory signal for the temporal super-resolution model.

[0095] Preferably, the method for processing the number of frames of a facial micro-expression video provided by the present invention further includes, before the step of "S240, inputting a facial key point feature sequence with a low frame rate into a temporal super-resolution model to generate a facial key point feature sequence with a high frame rate":

[0096] Obtain several facial micro-expression images from facial videos;

[0097] The acquired facial micro-expression images include multiple groups of facial micro-expression start frame images and facial micro-expression end frame images, forming a facial micro-expression training set.

[0098] Furthermore, the method for processing the number of frames of facial micro-expression videos provided by the present invention, before the step of "inputting a facial key point feature sequence with a low frame rate into a temporal super-resolution model to generate a facial key point feature sequence with a high frame rate", after the step of "obtaining a plurality of facial micro-expression images including a plurality of groups of facial micro-expression start frame images and facial micro-expression end frame images to form a facial micro-expression training set", further comprises:

[0099] Extract the first micro-expression feature from the facial micro-expression training set;

[0100] Generate a low-frame-rate facial key point feature sequence training set based on the first micro-expression feature of each group of facial micro-expression start frame images to facial micro-expression end frame images;

[0101] The generated low frame rate facial key point feature sequence training set is input into the temporal super-resolution model for training, and the facial feature sequence in the high frame rate facial video is used as a supervisory signal.

[0102] Preferably, the method for processing the number of frames of a facial micro-expression video provided by the present invention further comprises, after the step of "inputting the generated low-frame-rate facial key point feature sequence training set into the temporal super-resolution model for training, and using the facial feature sequence in the high-frame-rate facial video as a supervisory signal":

[0103] Comparing the facial key point feature sequence outputted from the temporal super-resolution model with the facial feature sequence of the supervisory signal;

[0104] Calculating the gap between the facial key point feature sequence and the facial feature sequence through a loss function, and generating a guidance signal according to the gap;

[0105] The guidance signal is input into the temporal super-resolution model.

[0106] The temporal super-resolution model of the present invention is based on transformer, the input is a low frame rate facial key point feature sequence, and the output is a high frame rate facial key point feature sequence. The present invention extracts the facial feature sequence in the high frame rate facial video as the training data supervision signal, uses the low frame rate key point sequence as the algorithm model training input, and uses the original high frame rate sequence as the supervision signal of the algorithm model.

[0107] The present invention also provides a corresponding loss function:

[0108] L G =λ1L p +λ2L gan +λ3L fm +λ4L kps

[0109] Among them, λ1~λ4 correspond to the weights of each loss, and are adjusted accordingly according to the actual situation during the training process.

[0110] (1)L p Represents image perception loss, VGG19 can be used for pre-training, and VGG19 is used as the feature extractor. This paper uses three-level pyramid features for multi-level constraints.

[0111] (2)L gan Represents GAN Loss.

[0112] (3)L fm Represents the discriminator feature map loss, the specific formula is:

[0113]

[0114] Where d is the input driver image, and z represents the extracted intermediate feature points. f Represents the intermediate feature output of the Discriminator model, which can extract the intermediate features of the input image. m represents the number of extracted feature pixels (samples). (G(z)) represents the sample generated by the generator from random noise (z), and i=1 means that the starting number of samples is 1.

[0115] (4)L kps represents the key point loss, using the pre-trained key point model for supervision. The loss function formula is:

[0116]

[0117] Where N represents the number of key points; s i Represents the Euclidean distance between two key points.

[0118] Train the model so that the output value of the model is close to or reaches the true value.

[0119] The present invention provides a method for processing the number of frames of a facial micro-expression video, which specifically includes:

[0120] S210, acquiring at least two frames of facial micro-expression images;

[0121] S220, extracting a first micro-expression feature from the facial micro-expression image, where the first micro-expression feature is a key point feature representing a facial posture and a facial expression action;

[0122] S230, generating a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of the facial micro-expression image;

[0123] S240: Inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence.

[0124] S241, extracting a second micro-expression feature from the facial micro-expression image, where the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement;

[0125] S242: Generate facial micro-expression continuous image frames according to the second micro-expression feature and the high frame rate facial key point feature sequence.

[0126] Specifically, the step of “S240, inputting a low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence” specifically includes:

[0127] S250, inputting the low frame rate facial key point feature sequence into the transformer model to generate a high frame rate facial key point feature sequence.

[0128] The step of “S242, generating continuous facial micro-expression image frames according to the second micro-expression feature and the high frame rate facial key point feature sequence” specifically includes:

[0129] S2421, extracting facial key point coordinates of consecutive frames in a facial key point feature sequence with a high frame rate;

[0130] S2422, inputting the coordinates of the facial key points of the continuous frames into the WarpMatrix network to generate an optical flow motion field at the feature level of the facial motion process;

[0131] S2423, performing spatial transformation through the optical flow motion field and the second micro-expression feature to obtain a facial feature after spatial transformation;

[0132] S2424, generating continuous image frames of facial micro-expressions according to the facial features after spatial transformation.

[0133] Specifically, the method for processing the number of frames of a facial micro-expression video provided by the present invention, the step of "S220, extracting a first micro-expression feature from the facial micro-expression image, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action" specifically includes:

[0134] The first micro-expression feature K is extracted through the KpsNet model, where the first micro-expression feature K is a key point feature related to facial posture and facial expression movements.

[0135] Specifically, the step of “S241, extracting a second micro-expression feature from the facial micro-expression image, wherein the second micro-expression feature is a facial feature that is unrelated to facial posture and facial expression” specifically includes:

[0136] The second micro-expression feature is extracted through the FaceFeat model, where the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement.

[0137] Micro-expression algorithms play an important role in financial scenarios, medicine and clinical applications. Micro-expressions usually last very short periods of time, generally between 1 / 25 seconds and 1 / 5 seconds. In a 25fps video, the process from the beginning to the end of a micro-expression is only 1-5 frames. Too few frames will affect the accuracy of micro-expression detection. Micro-expression detection algorithms in the industry are all used in high frame rate scenarios (60-200fps), but it is difficult to meet the frame rate required for micro-expression detection in actual business scenarios. The present invention first extracts a low-frame rate facial key point feature sequence from a low-frame rate facial micro-expression action frame, and then inputs the low-frame rate facial key point feature sequence into the transformer temporal super-resolution model for processing, outputs a high-frame rate facial key point feature sequence, and finally generates a high-frame rate facial micro-expression action frame, and then generates a facial micro-expression continuous image frame, that is, generates a high-frame rate facial micro-expression video. The present invention can obtain facial micro-expression images from low-frame-rate facial micro-expression videos, and generate high-frame-rate facial micro-expression videos through the processing of the present invention, thereby meeting the frame rate required for micro-expression detection in actual business scenarios. The present invention also provides a training description of the temporal super-resolution model. In order to make the output results more in line with the requirements, general model training is often a necessary process.

[0138] The present invention can generate continuous expression action frames in time sequence through the time series super-resolution algorithm, improve the frame rate of the overall video, solve the problem of low frame rate of micro-expressions, and provide high frame rate video input for subsequent micro-expression detection algorithms. The facial micro-expression video frame number processing method proposed in the present invention can extract facial key point feature representations that are independent of face ID, and use key points as facial expression feature representations, which is more intuitive and highly interpretable; facial key points are specially provided to reduce the dimension of facial expression features to a certain extent, which is convenient for subsequent time series super-resolution algorithms. At the same time, the face time series super-resolution algorithm based on intermediate features solves the problem of insufficient frame rate of original data in the micro-expression detection algorithm by using time series super-resolution.

[0139] In the above embodiment, the present invention discloses a facial micro-expression video frame number processing device, which belongs to the field of micro-expression recognition. Figure 3 As shown, the facial micro-expression video frame number processing device of the present invention comprises:

[0140] The micro-expression acquisition module 310 acquires a facial micro-expression image frame, wherein the acquired facial micro-expression image frame includes a facial micro-expression start frame image and a facial micro-expression end frame image;

[0141] A facial micro-expression feature extraction module 320 is used to extract a first micro-expression feature, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action;

[0142] A facial feature generation module 330, configured to generate a low frame rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image;

[0143] The high frame rate output module 340 is used to input the low frame rate facial key point feature sequence into the temporal super-resolution model to generate a high frame rate facial key point feature sequence.

[0144] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a method for processing the number of frames of a micro-expression video of a face, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0148] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the method for processing the number of frames of a facial micro-expression video.

[0149] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0150] In the above embodiment, the present invention discloses a computer device, which belongs to the field of micro-expression processing technology. The present invention obtains at least two frames of facial micro-expression images; extracts a first micro-expression feature from the facial micro-expression images, wherein the first micro-expression feature is a key point feature representing facial posture and facial expression movements; generates a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image; inputs the low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence. The present invention utilizes a temporal super-resolution algorithm, which can generate continuous expression action frames in time sequence, improve the frame rate of the overall video, solve the problem of low micro-expression frame rate, and provide high-frame-rate video input for subsequent micro-expression detection algorithms.

[0151] The main goal of video super-resolution is to convert low-resolution videos into high-resolution videos, which is essentially a prediction problem involving spatial and temporal series. Through video super-resolution, we can reconstruct higher-resolution video content from low-resolution video sources. The traditional architecture of Transformer (including fully connected self-attention layers and word-level feedforward layers) is not ideal for video super-resolution problems, mainly due to the following two reasons:

[0152] First, the fully connected self-attention layer is unable to capture the local features of the data because it relies on the linear layer to generate the attention map;

[0153] Second, the word-level feed-forward layer lacks the ability to align features, which is crucial for video super-resolution tasks.

[0154] The Transformer model has been successfully adapted to the video super-resolution task. Specifically, to address the first problem, a spatiotemporal convolutional self-attention layer can be introduced to deeply mine local information in the video; for the second problem, a bidirectional optical flow feedforward layer can be designed to effectively mine the correlation between adjacent video frames and achieve accurate frame alignment.

[0155] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned facial micro-expression video frame number processing method.

[0156] In the above embodiment, the present invention discloses a storage medium, which belongs to the field of micro-expression processing technology. The present invention obtains at least two frames of facial micro-expression images; extracts a first micro-expression feature from the facial micro-expression images, wherein the first micro-expression feature is a key point feature representing facial posture and facial expression movements; generates a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image; inputs the low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence. The present invention utilizes a temporal super-resolution algorithm, which can generate continuous expression action frames in time sequence, improve the frame rate of the overall video, solve the problem of low micro-expression frame rate, and provide high-frame-rate video input for subsequent micro-expression detection algorithms.

[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods provided by the present invention in various embodiments of the present invention.

[0158] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0159] In summary, the present invention provides a method for processing the number of frames of a facial micro-expression video, comprising: obtaining at least two frames of facial micro-expression images; extracting a first micro-expression feature from the facial micro-expression images, wherein the first micro-expression feature is a key point feature related to facial posture and facial expression action; generating a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of the facial micro-expression image; inputting the low-frame-rate facial key point feature sequence into a temporal super-resolution model to generate a high-frame-rate facial key point feature sequence. The present invention utilizes a temporal super-resolution algorithm, which can generate continuous expression action frames in a temporal sequence, improve the frame rate of the overall video, solve the problem of low micro-expression frame rate, and provide a high-frame-rate video input for a subsequent micro-expression detection algorithm. The method for processing the number of frames of a facial micro-expression video proposed by the present invention can extract facial key point feature representations that are not related to face IDs, and use key points as facial expression feature representations, which is more intuitive and highly interpretable; facial key point features reduce the dimension of facial expression features to a certain extent, which is convenient for subsequent temporal super-resolution algorithms. At the same time, the face temporal super-resolution algorithm based on intermediate features uses temporal super-resolution to solve the problem of insufficient original data frame rate in the micro-expression detection algorithm.

[0160] Obviously, the embodiments described above are only some embodiments of the present invention, not all embodiments. The preferred embodiments of the present invention are shown in the drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention is described in detail with reference to the above embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the above specific embodiments, or to replace some of the technical features therein with equivalents. All equivalent structures made by using the contents of the present invention specification and drawings, directly or indirectly applied to other related technical fields, are also within the scope of patent protection of the present invention. Although the present invention has been described with reference to (one or more) exemplary embodiments, it will be understood by those skilled in the art that the present invention is not limited to the exact structure and components described herein, and that various modifications, changes and deformations can be understood from the previous description without departing from the spirit and scope of the present invention as defined in the attached claims. The present invention is not limited to the shown order of steps, because some steps can be performed in different orders and / or simultaneously with other steps. Therefore, the present invention is not limited to the disclosed (one or more) specific embodiments, but will include all embodiments falling within the scope of the attached claims.

[0161] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A method for processing the number of frames of a facial micro-expression video, characterized in that: include: Acquire at least two frames of facial micro-expression images; Extracting a first micro-expression feature from the facial micro-expression image, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action; Generate a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image; The low frame rate facial key point feature sequence is input into the temporal super-resolution model to generate a high frame rate facial key point feature sequence.

2. The method for processing the number of frames of a facial micro-expression video according to claim 1, characterized in that: After the step of inputting the low frame rate facial key point feature sequence into the temporal super-resolution model to generate a high frame rate facial key point feature sequence, the method further includes: Extracting a second micro-expression feature from the facial micro-expression image, wherein the second micro-expression feature is a facial feature that is unrelated to facial posture and expression movement; Generate facial micro-expression continuous image frames according to the second micro-expression feature and a high frame rate facial key point feature sequence.

3. The method for processing facial micro-expression video frames according to claim 2, characterized in that: The step of generating continuous facial micro-expression image frames according to the second micro-expression feature and the high frame rate facial key point feature sequence specifically comprises: Extracting facial key point coordinates of consecutive frames in a high frame rate facial key point feature sequence; Inputting the coordinates of the facial key points of the continuous frames into the WarpMatrix network to generate an optical flow motion field at the feature level of the facial motion process; Performing spatial transformation on the optical flow motion field and the second micro-expression feature to obtain a facial feature after spatial transformation; Generate continuous image frames of facial micro-expressions based on the facial features after spatial transformation.

4. The method for processing facial micro-expression video frames according to claim 1, characterized in that: Before the step of inputting the low frame rate facial key point feature sequence into the temporal super-resolution model to generate the high frame rate facial key point feature sequence, the method further includes: Obtain facial feature sequences from high frame rate facial videos; The facial feature sequence in the high frame rate facial video is used as a supervisory signal for the temporal super-resolution model.

5. The method for processing the number of frames of a facial micro-expression video according to claim 4, characterized in that: Before the step of inputting the low frame rate facial key point feature sequence into the temporal super-resolution model to generate the high frame rate facial key point feature sequence, the method further includes: Obtain several facial micro-expression images from facial videos; The acquired facial micro-expression images include multiple groups of facial micro-expression start frame images and facial micro-expression end frame images, forming a facial micro-expression training set.

6. The method for processing facial micro-expression video frames according to claim 5, characterized in that: Before the step of inputting the low frame rate facial key point feature sequence into the temporal super-resolution model to generate the high frame rate facial key point feature sequence, after the step of acquiring the plurality of facial micro-expression images including a plurality of groups of facial micro-expression start frame images and facial micro-expression end frame images to form a facial micro-expression training set, the method specifically includes: Extract the first micro-expression feature from the facial micro-expression training set; Generate a low-frame-rate facial key point feature sequence training set based on the first micro-expression feature of each group of facial micro-expression start frame images to facial micro-expression end frame images; The generated low frame rate facial key point feature sequence training set is input into the temporal super-resolution model for training, and the facial feature sequence in the high frame rate facial video is used as a supervisory signal.

7. The method for processing the number of frames of a facial micro-expression video according to claim 6, characterized in that: After the step of inputting the generated low frame rate facial key point feature sequence training set into the temporal super-resolution model for training, and using the facial feature sequence in the high frame rate facial video as a supervisory signal, the following step is further included: Comparing the facial key point feature sequence outputted from the temporal super-resolution model with the facial feature sequence of the supervisory signal; Calculating the gap between the facial key point feature sequence and the facial feature sequence through a loss function, and generating a guidance signal according to the gap; The guidance signal is input into the temporal super-resolution model.

8. A facial micro-expression video frame number processing device, characterized in that: include: A micro-expression acquisition module is used to acquire facial micro-expression image frames, wherein the acquired facial micro-expression image frames include facial micro-expression start frame images and facial micro-expression end frame images; A facial micro-expression feature extraction module, used to extract a first micro-expression feature, wherein the first micro-expression feature is a key point feature representing a facial posture and a facial expression action; A facial feature generation module, used to generate a low-frame-rate facial key point feature sequence according to the first micro-expression feature in each frame of facial micro-expression image; The high frame rate output module is used to input the low frame rate facial key point feature sequence into the temporal super-resolution model to generate a high frame rate facial key point feature sequence.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for processing the frame number of a facial micro-expression video as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method for processing the frame number of facial micro-expression videos according to any one of claims 1 to 7 are implemented.

Images