Macro expression image generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a macro expression image generation method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a sample micro-expression image and a sample macro-expression image; training a preset expression difference prediction model through an image style encoder, the sample micro-expression image and the sample macro-expression image to obtain a target expression difference prediction model; training a preset expression image generation model through an image style encoder, the sample micro-expression image, the sample macro-expression image and a target expression difference prediction model to obtain a target expression image generation model; and obtaining a target micro-expression image, and performing image generation through the target micro-expression image, the target expression difference prediction model and the target expression image generation model to obtain a target macro-expression image. According to the embodiment of the invention, the micro-expression image can be accurately converted into the macro-expression image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a macro expression image generation method and device, electronic equipment and storage medium. Background Art

[0002] Micro-expressions are short-term changes in facial expressions, usually lasting from tens of milliseconds to one second. This type of expression reflects the real emotions of the target object in an instant, and is often difficult to detect and recognize. Macro-expressions are continuous changes in facial expressions, usually lasting more than one second. This type of expression shows a more obvious emotional state, but may not completely and accurately reflect the target object's inner true emotions. Therefore, when judging a person's inner changes based on expressions, it is necessary to rely on capturing the real emotions of micro-expressions. For example, in medical psychological assessments, the micro-expressions of the target patient are captured and converted into macro-expressions, so as to accurately judge the psychological state of the target patient. In the existing technology for converting micro-expressions into macro-expressions, it is necessary to rely on manually set conversion coefficients. In actual tests, the image will be distorted due to inappropriate conversion coefficients, that is, it cannot be accurately converted into macro-expressions based on micro-expressions. Therefore, how to accurately convert micro-expression images into macro-expression images has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a macro expression image generation method and device, electronic device and storage medium, aiming to accurately convert micro expression images into macro expression images.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a macro expression image, the method comprising:

[0005] Obtain a sample micro-expression image and a sample macro-expression image of a sample object;

[0006] The expression difference loss value is calculated by using a preset image style encoder, the sample micro-expression image, the sample macro-expression image and a preset expression difference prediction model;

[0007] Training the preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model;

[0008] The preset expression image generation model is trained by using the image style encoder, the sample micro-expression image, the sample macro-expression image and the target expression difference prediction model to obtain a target expression image generation model;

[0009] A target micro-expression image of a target object is acquired, and image generation is performed through the target micro-expression image, the image style encoder, the target expression difference prediction model and the target expression image generation model to obtain a target macro-expression image.

[0010] In some embodiments, the calculating of the difference loss value by using a preset image style encoder, the sample micro-expression image, the sample macro-expression image, and a preset expression difference prediction model to obtain the expression difference loss value includes:

[0011] Performing expression difference prediction on the sample micro-expression image by using the preset expression difference prediction model to obtain predicted expression difference features;

[0012] Calculating the expression difference between the sample micro-expression image and the sample macro-expression image by using the image style encoder to obtain sample expression difference features;

[0013] An expression difference loss value is calculated according to the predicted expression difference feature and the sample expression difference feature to obtain the expression difference loss value.

[0014] In some embodiments, the preset expression difference prediction model includes an image semantic extractor and a difference amplifier, and the expression difference prediction of the sample micro-expression image is performed by the preset expression difference prediction model to obtain the predicted expression difference feature, including:

[0015] Performing image semantic extraction on the sample micro-expression image by the image semantic extractor to obtain the semantics of the sample micro-expression image;

[0016] The difference amplifier is used to perform difference amplification on the semantics of the sample micro-expression image to obtain the predicted expression difference feature.

[0017] In some embodiments, calculating the expression difference between the sample micro-expression image and the sample macro-expression image by the image style encoder to obtain the sample expression difference feature includes:

[0018] Performing style encoding on the sample micro-expression image by using the image style encoder to obtain a sample micro-expression code;

[0019] Style encoding is performed on the sample macro-expression image by the image style encoder to obtain a sample macro-expression code;

[0020] The difference between the sample micro-expression code and the sample macro-expression code is calculated to obtain the sample expression difference feature.

[0021] In some embodiments, the training of a preset expression image generation model by using the image style encoder, the sample micro-expression image, the sample macro-expression image, and the target expression difference prediction model to obtain a target expression image generation model includes:

[0022] Performing style encoding on the sample micro-expression image by using the image style encoder to obtain initial micro-expression encoding features;

[0023] Performing difference prediction on the sample micro-expression image by using the target expression difference prediction model to obtain initial expression difference features;

[0024] Generate an image by using the initial micro-expression coding features, the initial expression difference features and the preset expression image generation model to obtain an initial macro-expression image;

[0025] The preset expression image generation model is optimized in parameters according to the sample macro expression image and the initial macro expression image to obtain the target expression image generation model.

[0026] In some embodiments, the step of optimizing parameters of the preset expression image generation model according to the sample macro expression image and the initial macro expression image to obtain the target expression image generation model includes:

[0027] Extracting texture features from the sample macro expression image to obtain sample macro expression texture features;

[0028] Extracting texture features from the initial macro expression image to obtain initial macro expression texture features;

[0029] Calculate the image texture loss value according to the sample macro expression texture feature and the initial macro expression texture feature to obtain the expression texture loss value;

[0030] The preset expression image generation model is parameter optimized according to the expression texture loss value to obtain the target expression image generation model.

[0031] In some embodiments, obtaining a sample micro-expression image and a sample macro-expression image of a sample object includes:

[0032] Acquire expression sequence frames of the sample object;

[0033] Performing optical flow face alignment according to the expression sequence frames to obtain optical flow face sequence frames;

[0034] Expression images are extracted according to the optical flow face sequence frames to obtain the sample micro-expression images and the sample macro-expression images.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a macro expression image generating device, the device comprising:

[0036] An image acquisition module is used to acquire a sample micro-expression image and a sample macro-expression image of a sample object;

[0037] A loss calculation module, used to calculate a difference loss value through a preset image style encoder, the sample micro-expression image, the sample macro-expression image and a preset expression difference prediction model to obtain an expression difference loss value;

[0038] A difference training module, used for training the preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model;

[0039] Generate a training module, which is used to train a preset expression image generation model through the image style encoder, the sample micro-expression image, the sample macro-expression image and the target expression difference prediction model to obtain a target expression image generation model;

[0040] The image generation module is used to obtain a target micro-expression image of a target object, and to generate an image through the target micro-expression image, the image style encoder, the target expression difference prediction model and the target expression image generation model to obtain a target macro-expression image.

[0041] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The present application proposes a macro-expression image generation method and device, electronic device and storage medium, which obtain sample micro-expression images and sample macro-expression images of a sample object, and then train a preset expression difference prediction model through a style encoder, sample micro-expression images, and sample macro-expressions to obtain a target expression difference prediction model, so that the target expression difference prediction model can predict the difference between micro-expressions and macro-expressions based on micro-expressions; further, a preset expression image generation model is trained through an image style encoder, sample micro-expression images, sample macro-expression images, and a target expression difference prediction model to obtain a target expression image generation model, so that the target expression difference prediction model can generate images based on expression differences and micro-expression images to obtain macro-expressions; finally, a target micro-expression image of the target expression is obtained, and image generation is performed through the target micro-expression image, the target expression difference prediction model, and the target expression image generation model to obtain a target macro-expression image, thereby realizing accurate conversion of micro-expression images into macro-expression images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a macro expression image generation method provided in an embodiment of the present application;

[0045] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0046] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0047] Figure 4 yes Figure 3 Flow chart of step S301 in FIG.

[0048] Figure 5 yes Figure 3 Flow chart of step S302 in FIG.

[0049] Figure 6 yes Figure 5 Flow chart of step S104 in FIG.

[0050] Figure 7 yes Figure 6 Flow chart of step S604 in FIG.

[0051] Figure 8 is a structural schematic diagram of a macro expression image generating device provided in an embodiment of the present application;

[0052] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, some nouns involved in this application are analyzed:

[0057] Style-based Generative Adversarial Network (StyleGAN): It is a generative model, a variant of the Generative Adversarial Network (GAN), which is specifically designed to generate high-quality, detailed images. As part of the field of computer science and artificial intelligence, the core of StyleGAN is to generate realistic images by simulating and extending human visual creativity. By introducing the concept of style transfer, this technology allows the control of the style hierarchy of the generated image, thereby achieving gradual refinement from coarse to detailed, greatly improving the image quality and diversity of changes. The application areas of StyleGAN include but are not limited to digital art creation, game content generation, virtual reality, etc.

[0058] Contrastive Language-Image Pretraining Network (CLIP): It simulates and expands the machine's visual and language processing capabilities by understanding the image content and related text descriptions. This technology uses contrastive learning methods to pre-train with a large amount of image and text data, enabling the machine to recognize and understand the complex relationship between images and their language descriptions. CLIP's applications include image classification, text-to-image generation, and multimodal interaction. In this way, CLIP can not only simulate human visual and language cognitive processes, but also use these capabilities to improve decision-making and analysis.

[0059] Microexpression: Microexpression is an extremely brief and tiny movement in human facial expression, usually lasting no more than 1 / 25 to 1 / 5 second. Microexpression belongs to the field of psychology and behavioral science, reflecting the true emotions that people inadvertently reveal when they try their best to control their facial expressions. Microexpression research is an important branch of artificial intelligence, involving the simulation and analysis of subtle facial expressions of humans to understand underlying emotions and intentions. Research in this field includes facial recognition, emotion analysis, fraud prevention, psychological assessment and other aspects. By using high-speed photography technology and computer vision algorithms, researchers are able to capture and analyze these brief expressions that are usually difficult to observe with the naked eye. Microexpression analysis also involves the development and application of technologies such as data mining, machine learning, and artificial intelligence, as well as in-depth research in psychology and behavioral science.

[0060] Macroexpression: Macroexpression is a relatively obvious and long-lasting movement in human facial expressions, which can intuitively reflect the emotional state of an individual. Macroexpression is mainly used to analyze and understand basic human emotions such as happiness, sadness, anger, surprise, etc. Macroexpression research is an important branch of artificial intelligence, involving the simulation and analysis of significant human facial expressions to interpret emotions and intentions. Research in this field includes facial recognition, emotion analysis, human-computer interaction, etc. Through high-precision image processing technology and pattern recognition algorithms, researchers can automatically identify and classify facial expressions, which are further applied to video surveillance, psychological assessment, customer service, education and training, and other fields. The study of macroexpression not only improves the machine's ability to recognize human expressions, but also promotes the naturalness and intelligence of interactive systems. In addition, macroexpression analysis also involves the development and application of multiple technologies such as data mining, machine learning, pattern recognition, computational neuroscience, and psychology.

[0061] Micro-expressions are short-term changes in facial expressions, usually lasting from tens of milliseconds to one second. This type of expression reflects the real emotions of the target object in an instant, and is often difficult to detect and recognize. Macro-expressions are continuous changes in facial expressions, usually lasting more than one second. This type of expression shows a more obvious emotional state, but may not completely and accurately reflect the target object's inner true emotions. Therefore, when judging a person's inner changes based on expressions, it is necessary to rely on capturing the real emotions of micro-expressions. For example, in medical psychological assessments, the micro-expressions of the target patient are captured and converted into macro-expressions, so as to accurately judge the psychological state of the target patient. In the existing technology for converting micro-expressions into macro-expressions, it is necessary to rely on manually set conversion coefficients. In actual tests, the image will be distorted due to inappropriate conversion coefficients, that is, it cannot be accurately converted into macro-expressions based on micro-expressions. Therefore, how to accurately convert micro-expression images into macro-expression images has become an urgent problem to be solved.

[0062] Based on this, the embodiments of the present application provide a macro expression image generation method and device, an electronic device and a storage medium, which aim to accurately convert micro expression images into macro expression images.

[0063] The embodiments of the present application provide a macro expression image generation method and device, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the macro expression image generation method in the embodiments of the present application is described.

[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0066] The macro expression image generation method provided in the embodiment of the present application relates to the field of image processing technology. The macro expression image generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the macro expression image generation method, etc., but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0069] Figure 1 is an optional flow chart of the macro expression image generation method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S105.

[0070] Step S101, obtaining a sample micro-expression image and a sample macro-expression image of a sample object;

[0071] Step S102, calculating the difference loss value through a preset image style encoder, a sample micro-expression image, a sample macro-expression image and a preset expression difference prediction model to obtain an expression difference loss value;

[0072] Step S103, training a preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model;

[0073] Step S104, training a preset expression image generation model through an image style encoder, sample micro-expression images, sample macro-expression images, and a target expression difference prediction model to obtain a target expression image generation model;

[0074] Step S105, obtaining a target micro-expression image of the target object, and performing image generation through the target micro-expression image, an image style encoder, a target expression difference prediction model and a target expression image generation model to obtain a target macro-expression image.

[0075] In steps S101 to S105 shown in the embodiment of the present application, a sample micro-expression image and a sample macro-expression image of a sample object are obtained, and then a preset expression difference prediction model is trained through a style encoder, sample micro-expression images, and sample macro-expressions to obtain a target expression difference prediction model, so that the target expression difference prediction model can predict the difference between micro-expressions and macro-expressions based on micro-expressions; further, a preset expression image generation model is trained through an image style encoder, sample micro-expression images, sample macro-expression images, and a target expression difference prediction model to obtain a target expression image generation model, so that the target expression difference prediction model can generate images based on expression differences and micro-expression images to obtain macro-expressions; finally, a target micro-expression image of the target expression is obtained, and image generation is performed through the target micro-expression image, the target expression difference prediction model, and the target expression image generation model to obtain a target macro-expression image, thereby realizing accurate conversion of micro-expression images into macro-expression images.

[0076] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0077] Step S201, obtaining expression sequence frames of a sample object;

[0078] Step S202, performing optical flow face alignment according to the expression sequence frames to obtain optical flow face sequence frames;

[0079] Step S203, extracting expression images according to the optical flow face sequence frames to obtain sample micro-expression images and sample macro-expression images.

[0080] Steps S201 to S203 shown in the embodiment of the present application obtain expression sequence frames of the sample object, and then perform optical flow face alignment based on the expression sequence frames to obtain optical flow face sequence frames, thereby ensuring that subtle changes in expression will not be missed or misinterpreted due to slight movements or rotations of the head, thereby improving the accuracy and reliability of expression capture, and then performing expression image extraction based on the optical flow face sequence frames to obtain sample micro-expression images and sample macro-expression images, thereby providing a data basis for subsequent conversion from micro-expressions to macro-expressions.

[0081] In step S201 of some embodiments, expression sequence frames refer to a series of continuous image frames that capture the transition process from one expression to another. The entire sequence records in detail every slight change in the facial expression of the sample object. Methods for obtaining expression sequence frames include but are not limited to video capture and photo shooting.

[0082] In step S202 of some embodiments, optical flow face alignment is a method of spatially aligning consecutive face image frames using optical flow technology to detect and correct the position change of the face in consecutive frames caused by face motion or camera movement, thereby ensuring that facial features such as eyes, nose and mouth remain in a relatively fixed position in the sequence, and reducing the movement introduced by head movement, so that subsequent expression analysis is more accurate. Optical flow face sequence frames refer to a series of face images obtained after optical flow alignment, and the position and direction of the face in all frames are consistent.

[0083] In step S203 of some embodiments, micro-expressions are extremely short and tiny movements in human facial expressions, usually lasting no more than 1 / 25 to 1 / 5 second. Macro-expressions are relatively obvious and long-lasting movements in human facial expressions, usually lasting more than 1 second, and can intuitively reflect the emotional state of the individual. Expression image extraction is to track facial key points of each frame in the optical flow face sequence frame, and then construct a feature vector of facial expression based on the facial key points, and then classify based on the pre-trained support vector machine to classify the micro-expression sequence and macro-expression sequence in the optical flow face sequence frame, and then determine the micro-expression image based on the starting frame and ending frame of the micro-expression sequence, and determine the macro-expression image based on the starting frame and ending frame of the macro-expression sequence.

[0084] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S303:

[0085] Step S301, predicting the expression difference of the sample micro-expression image by using a preset expression difference prediction model to obtain predicted expression difference features;

[0086] Step S302, calculating the expression difference of the sample micro-expression image and the sample macro-expression image through an image style encoder to obtain a sample expression difference feature;

[0087] Step S303, calculating the expression difference loss value according to the predicted expression difference feature and the sample expression difference feature to obtain the expression difference loss value.

[0088] In steps S301 to S303 shown in the embodiment of the present application, expression differences are predicted for sample micro-expression images through a preset expression difference prediction model to obtain predicted expression difference features, and then expression differences are calculated for sample micro-expression images and sample macro-expression images through an image style encoder to obtain sample expression difference features, and finally expression difference loss values ​​are calculated based on the predicted expression difference features and the sample expression difference features to obtain expression difference loss values, thereby determining the loss between the predicted value and the true value of the preset expression difference prediction model, providing a data basis for subsequent training of the preset expression difference prediction model based on the loss value, so that the preset expression difference prediction model can improve the ability to perceive the difference between micro-expressions and macro-expressions at the image style level.

[0089] See also Figure 4 In some embodiments, the preset expression difference prediction model includes an image semantic extractor and a difference amplifier, and step S301 may include but is not limited to steps S401 to S402:

[0090] Step S401, extracting image semantics from the sample micro-expression image by an image semantic extractor to obtain the semantics of the sample micro-expression image;

[0091] Step S402 , performing difference amplification on the semantics of the sample micro-expression image through a difference amplifier to obtain predicted expression difference features.

[0092] In steps S401 to S402 shown in the embodiment of the present application, image semantics are extracted from the sample micro-expression image by an image semantic extractor to obtain the sample micro-expression image semantics, and then the sample micro-expression image semantics are difference amplified by a difference amplifier to obtain predicted expression difference features, thereby realizing the difference between the micro-expression image prediction and the macro-expression image.

[0093] In step S401 of some embodiments, image semantic extraction is used to identify and extract meaningful information from the sample micro-expression image, conduct in-depth analysis of the visual content of the sample micro-expression image, interpret the various elements in the image and their semantic relationships, and obtain the semantics of the sample micro-expression image. In one embodiment, the image semantic extractor is composed of a contrastive language-image pretraining network (CLIP). The sample micro-expression image is input into the image semantic extractor, i.e., CLIP, for image semantic extraction to obtain the semantics of the sample micro-expression image.

[0094] In step S402 of some embodiments, difference amplification refers to amplifying potential expression differences in the semantics of sample micro-expression images, and the potential differences in the semantics of sample micro-expressions are difference-amplified by a difference amplifier, thereby amplifying the characteristics of micro-expressions. In one embodiment, the difference amplifier is composed of a convolutional neural network, and the micro-expression image semantics are input into the convolutional neural network for difference amplification to obtain predicted expression difference characteristics.

[0095] See also Figure 5 In some embodiments, step S302 includes but is not limited to steps S501 to S503:

[0096] Step S501, style encoding the sample micro-expression image through an image style encoder to obtain a sample micro-expression code;

[0097] Step S502, style encoding the sample macro expression image through an image style encoder to obtain a sample macro expression code;

[0098] Step S503, calculating the difference between the sample micro-expression code and the sample macro-expression code to obtain the sample expression difference feature.

[0099] In steps S501 to S503 shown in the embodiment of the present application, the sample micro-expression image is style-encoded by an image style encoder to obtain a sample micro-expression code, and then the sample macro-expression image is style-encoded by an image style encoder to obtain a sample macro-expression code, and finally the difference between the sample micro-expression code and the sample macro-expression code is calculated to obtain a sample expression difference feature, thereby obtaining the style difference between the sample micro-expression image and the sample macro-expression image, which provides a data basis for subsequent training of a preset expression difference prediction model based on style differences, so that the preset expression difference prediction model can perceive the style difference between micro-expressions and macro-expressions, thereby improving the style difference prediction ability of the preset expression difference prediction model.

[0100] In step S501 of some embodiments, style coding is to extract visual style features from sample micro-expression images using an image style encoder. These features capture the overall visual effect and texture information of the image, and are relatively independent of the specific content of the image, such as the shape or position of the object. That is, style coding can capture the overall style and texture in the sample micro-expression image. In one embodiment, the image style encoder is composed of a pre-trained encoder in a style-based generative adversarial network (Style-basedGenerative Adversarial Network, StyleGAN).

[0101] The StyleGAN network includes an encoder, a generator, and a discriminator. By inputting a real image into the encoder, an encoded output is obtained, and then the encoded output is input into the generator to obtain a generated image. Then, the discriminator is used to determine whether the generated image is a real image. During the training process, the network parameters of the discriminator are first frozen, and then the encoder and generator are trained. After reaching the preset iteration conditions, the network parameters of the encoder and generator are frozen, and then the discriminator is trained. The encoder, generator, and discriminator are trained back and forth to improve the encoding ability of the encoder.

[0102] In one embodiment, the encoder is composed of StyleGAN-Encoder.

[0103] In step S502 of some embodiments, the sample macro expression image is input into an image style encoder for style encoding to obtain a sample macro expression code.

[0104] In step S503 of some embodiments, the difference between the sample micro-expression code and the sample macro-expression code is calculated to obtain the sample expression difference feature, and the difference calculation is shown in formula (1):

[0105] sC-sB (1),

[0106] Among them, sC is the sample macro expression code, sB is the sample micro expression code, and sC-sB is the sample expression difference feature.

[0107] In step S303 of some embodiments, expression difference loss is calculated based on predicted expression difference features and sample expression difference features. In one embodiment, expression difference loss is calculated as a mean square error loss value calculation, and mean square error is calculated based on predicted expression difference features and sample expression difference features to obtain expression difference loss value.

[0108] In step S103 of some embodiments, the parameters of the preset expression difference prediction model are optimized based on the expression difference loss value, and the model is back-propagated by the stochastic gradient descent method to obtain the target expression difference prediction model.

[0109] See also Figure 6 In some embodiments, step S104 includes but is not limited to steps S601 to S604:

[0110] Step S601, style encoding the sample micro-expression image through an image style encoder to obtain initial micro-expression coding features;

[0111] Step S602, performing difference prediction on the sample micro-expression image by using the target expression difference prediction model to obtain initial expression difference features;

[0112] Step S603, generating an image by using the initial micro-expression coding features, the initial expression difference features and a preset expression image generation model to obtain an initial macro-expression image;

[0113] Step S604, optimizing parameters of a preset expression image generation model according to the sample macro expression image and the initial macro expression image to obtain a target expression image generation model.

[0114] In steps S601 to S604 shown in the embodiment of the present application, style encoding is performed on the sample micro-expression image through an image style encoder to obtain initial micro-expression encoding features, and then difference prediction is performed on the sample micro-expression image through a target expression difference prediction model to obtain initial expression difference features, and then image generation is performed through the initial micro-expression encoding features, initial expression difference features and a preset expression image generation model to obtain an initial macro-expression image, and finally parameters of the preset expression image generation model are optimized according to the sample macro-expression image and the initial macro-expression image, so as to obtain a target expression image generation model that can generate macro-expression images based on expression difference features and micro-expression encoding features, thereby realizing accurate conversion of micro-expression images into macro-expression images.

[0115] In step S601 of some embodiments, the sample micro-expression image is input into an image style encoder for style encoding, thereby capturing the overall style and texture in the sample micro-expression image and obtaining initial micro-expression encoding features.

[0116] In step S602 of some embodiments, the sample micro-expression image is input into a target expression difference prediction model for difference prediction to obtain initial expression difference features, thereby predicting the difference between the micro-expression and the macro-expression.

[0117] In step S603 of some embodiments, the initial micro-expression coding features and the initial expression difference features are input into a preset expression image generation model for image generation, so as to generate a macro-expression image based on the micro-expressions and the predicted expression difference features. In one embodiment, the preset expression image generation model is a generator in a StyleGan network, and the initial micro-expression coding features and the initial expression difference features are input into the generator in the StyleGan network to obtain an initial macro-expression image.

[0118] See also Figure 7 In some embodiments, step S604 may include but is not limited to steps S701 to S704:

[0119] Step S701, extracting texture features according to the sample macro expression image to obtain sample macro expression texture features;

[0120] Step S702, extracting texture features according to the initial macro expression image to obtain initial macro expression texture features;

[0121] Step S703, calculating the image texture loss value according to the sample macro expression texture features and the initial macro expression texture features to obtain the expression texture loss value;

[0122] Step S704, optimizing the parameters of the preset expression image generation model according to the expression texture loss value to obtain a target expression image generation model.

[0123] Steps S701 to S703 shown in the embodiment of the present application perform texture feature extraction based on the sample macro expression image to obtain the sample macro expression texture features, then perform texture feature extraction based on the initial macro expression image to obtain the initial macro expression texture features, then calculate the image texture loss value based on the sample macro expression texture features and the initial macro expression texture features to obtain the expression texture loss value, and finally optimize the parameters of the preset expression image generation model based on the expression texture loss value, thereby improving the image generation accuracy of the preset expression image generation model, and obtaining the target expression image generation model that can generate accurate macro expressions based on the differences between micro-expressions and predicted expressions.

[0124] In step S701 of some embodiments, texture feature extraction is to extract information about the surface structure of the object from the image, including but not limited to repeated patterns, roughness or regularity within the image. In one embodiment, grayscale co-occurrence matrix calculation is performed based on the sample macro-expression image to obtain the sample macro-expression texture features.

[0125] In step S702 of some embodiments, texture features are extracted based on the initial macro expression image. In one embodiment, gray level co-occurrence matrix is ​​calculated based on the sample macro expression image to obtain the initial macro expression texture features.

[0126] In step S703 of some embodiments, an image texture loss value is calculated based on the sample macro expression texture features and the initial macro expression texture features to obtain an expression texture loss value. The image texture loss value may be calculated by a mean square error or an absolute difference loss, and this application does not impose any specific restrictions.

[0127] In step S704 of some embodiments, the parameters of the preset expression image generation model are optimized according to the expression texture loss value, so that the preset expression image generation model has good image generation perception ability, and a target expression image generation model is obtained that can accurately predict and generate macro expression images based on micro-expressions and expression differences.

[0128] In step S105 of some embodiments, the target micro-expression image is input into an image style encoder to obtain a target micro-expression style code, and the target micro-expression image is simultaneously input into a target expression difference prediction model for expression difference prediction to obtain a target expression difference feature, and then the target micro-expression style code and the target expression difference feature are input into a target expression image generation model for image generation to obtain a target macro-expression image.

[0129] See also Figure 8 The embodiment of the present application also provides a macro expression image generation device, which can implement the above macro expression image generation method, and the device includes:

[0130] An image acquisition module 801 is used to acquire a sample micro-expression image and a sample macro-expression image of a sample object;

[0131] The loss calculation module 802 is used to calculate the difference loss value through a preset image style encoder, a sample micro-expression image, a sample macro-expression image and a preset expression difference prediction model to obtain an expression difference loss value;

[0132] A difference training module 803 is used to train a preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model;

[0133] A generation training module 804 is used to train a preset expression image generation model through an image style encoder, sample micro-expression images, sample macro-expression images and a target expression difference prediction model to obtain a target expression image generation model;

[0134] The image generation module 805 is used to obtain a target micro-expression image of the target object, and generate an image through the target micro-expression image, an image style encoder, a target expression difference prediction model and a target expression image generation model to obtain a target macro-expression image.

[0135] The specific implementation of the macro expression image generating device is substantially the same as the specific implementation of the above-mentioned macro expression image generating method, and will not be described in detail herein.

[0136] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned macro expression image generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0137] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0138] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0139] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the macro expression image generation method of the embodiment of this application;

[0140] Input / output interface 903, used to implement information input and output;

[0141] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0142] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0143] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0144] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned macro expression image generation method is implemented.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The macro expression image generation method, macro expression image generation device, electronic device and storage medium provided in the embodiments of the present application obtain sample micro expression images and sample macro expression images of the sample object, and then train a preset expression difference prediction model through a style encoder, sample micro expression images, and sample macro expressions to obtain a target expression difference prediction model, so that the target expression difference prediction model can predict the difference between micro expression and macro expression based on micro expression; further, a preset expression image generation model is trained through an image style encoder, sample micro expression images, sample macro expression images, and a target expression difference prediction model to obtain a target expression image generation model, so that the target expression difference prediction model can generate images based on expression differences and micro expression images to obtain macro expressions; finally, a target micro expression image of the target expression is obtained, and image generation is performed through the target micro expression image, the target expression difference prediction model, and the target expression image generation model to obtain a target macro expression, thereby realizing accurate conversion of the micro expression image into a macro expression image.

[0147] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0148] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0151] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0152] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0157] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a macro expression image, characterized in that: The method comprises: Obtain a sample micro-expression image and a sample macro-expression image of a sample object; The expression difference loss value is calculated by using a preset image style encoder, the sample micro-expression image, the sample macro-expression image and a preset expression difference prediction model; Training the preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model; The preset expression image generation model is trained by using the image style encoder, the sample micro-expression image, the sample macro-expression image and the target expression difference prediction model to obtain a target expression image generation model; A target micro-expression image of a target object is acquired, and image generation is performed through the target micro-expression image, the image style encoder, the target expression difference prediction model and the target expression image generation model to obtain a target macro-expression image.

2. The method according to claim 1, characterized in that: The calculation of the difference loss value by using a preset image style encoder, the sample micro-expression image, the sample macro-expression image and a preset expression difference prediction model to obtain the expression difference loss value includes: Performing expression difference prediction on the sample micro-expression image by using the preset expression difference prediction model to obtain predicted expression difference features; Calculating the expression difference between the sample micro-expression image and the sample macro-expression image by using the image style encoder to obtain sample expression difference features; An expression difference loss value is calculated according to the predicted expression difference feature and the sample expression difference feature to obtain the expression difference loss value.

3. The method according to claim 2, characterized in that The preset expression difference prediction model includes an image semantic extractor and a difference amplifier. The expression difference prediction of the sample micro-expression image is performed by the preset expression difference prediction model to obtain the predicted expression difference feature, including: Performing image semantic extraction on the sample micro-expression image by the image semantic extractor to obtain the semantics of the sample micro-expression image; The difference amplifier is used to perform difference amplification on the semantics of the sample micro-expression image to obtain the predicted expression difference feature.

4. The method according to claim 2, characterized in that: The calculating the expression difference between the sample micro-expression image and the sample macro-expression image by the image style encoder to obtain the sample expression difference feature includes: Performing style encoding on the sample micro-expression image by using the image style encoder to obtain a sample micro-expression code; Style encoding is performed on the sample macro-expression image by the image style encoder to obtain a sample macro-expression code; The difference between the sample micro-expression code and the sample macro-expression code is calculated to obtain the sample expression difference feature.

5. The method according to any one of claims 1 to 4, characterized in that: The method of training a preset expression image generation model by using the image style encoder, the sample micro-expression image, the sample macro-expression image and the target expression difference prediction model to obtain a target expression image generation model includes: Performing style encoding on the sample micro-expression image by using the image style encoder to obtain initial micro-expression encoding features; Performing difference prediction on the sample micro-expression image by using the target expression difference prediction model to obtain initial expression difference features; Generate an image by using the initial micro-expression coding features, the initial expression difference features and the preset expression image generation model to obtain an initial macro-expression image; The preset expression image generation model is optimized in parameters according to the sample macro expression image and the initial macro expression image to obtain the target expression image generation model.

6. The method according to claim 5, characterized in that The step of optimizing the parameters of the preset expression image generation model according to the sample macro expression image and the initial macro expression image to obtain the target expression image generation model includes: Extracting texture features from the sample macro expression image to obtain sample macro expression texture features; Extracting texture features from the initial macro expression image to obtain initial macro expression texture features; Calculate the image texture loss value according to the sample macro expression texture feature and the initial macro expression texture feature to obtain the expression texture loss value; The preset expression image generation model is parameter optimized according to the expression texture loss value to obtain the target expression image generation model.

7. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a sample micro-expression image and a sample macro-expression image of a sample object includes: Acquire expression sequence frames of the sample object; Performing optical flow face alignment according to the expression sequence frames to obtain optical flow face sequence frames; Expression images are extracted according to the optical flow face sequence frames to obtain the sample micro-expression images and the sample macro-expression images.

8. A macro expression image generating device, characterized in that: The device comprises: An image acquisition module is used to acquire a sample micro-expression image and a sample macro-expression image of a sample object; A loss calculation module, used to calculate a difference loss value through a preset image style encoder, the sample micro-expression image, the sample macro-expression image and a preset expression difference prediction model to obtain an expression difference loss value; A difference training module, used for training the preset expression difference prediction model based on the expression difference loss value to obtain a target expression difference prediction model; Generate a training module, which is used to train a preset expression image generation model through the image style encoder, the sample micro-expression image, the sample macro-expression image and the target expression difference prediction model to obtain a target expression image generation model; The image generation module is used to obtain a target micro-expression image of a target object, and to generate an image through the target micro-expression image, the image style encoder, the target expression difference prediction model and the target expression image generation model to obtain a target macro-expression image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the macro expression image generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the macro expression image generating method according to any one of claims 1 to 7 is implemented.

Images