Facial expression transfer method, device and computer equipment

Abstract

The present application discloses a method for migrating facial expressions, which includes: obtaining a first facial image and a second facial image; performing a three-dimensional transformation on the first facial image and the second facial image respectively to obtain a corresponding third facial image and a fourth facial image; extracting the expression data of the fourth facial image through a preset self-supervised expression reconstruction model and fusing it to the third facial image to obtain a new third facial image; performing a two-dimensional transformation on the new third facial image to obtain a new first facial image, and the expression of the new first facial image is consistent with the expression of the second facial image. The present application also provides a facial expression migration device, a computer device, and a computer-readable storage medium. The present application extracts the expression data of a three-dimensional facial image through a self-supervised expression reconstruction model and then migrates it to another three-dimensional facial image, thereby realizing accurate and rapid facial expression migration.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method, apparatus, computer device and computer-readable storage medium for transferring facial expressions. Background Art

[0002] Facial expression transfer technology refers to mapping the expression of an input face to a target face through a certain mapping relationship. This technology not only allows users to control the facial expression in the target image or video by inputting a face, but also provides data enhancement services for face recognition tasks.

[0003] However, due to the diversity of facial features, facial data with the same expression and the same degree cannot be collected. Therefore, the process of facial expression reconstruction is very time-consuming and the expression cannot be accurately transferred. Summary of the invention

[0004] In view of this, the present application proposes a facial expression migration method, apparatus, computer device and computer-readable storage medium, which can solve the above-mentioned problems of long time consumption and low accuracy in facial expression migration.

[0005] First, to achieve the above purpose, the present application provides a method for transferring facial expressions, the method comprising:

[0006] Acquire a first facial image and a second facial image; perform a three-dimensional transformation on the first facial image and the second facial image respectively to obtain a corresponding third facial image and a fourth facial image; extract expression data of the fourth facial image through a preset self-supervised expression reconstruction model and replace it with the third facial image to obtain a new third facial image; perform a two-dimensional transformation on the new third facial image to obtain a new first facial image, wherein the expression of the new first facial image is consistent with the expression of the second facial image.

[0007] In one example, performing three-dimensional transformation on the first facial image and the second facial image respectively to obtain corresponding third facial image and fourth facial image includes: obtaining the third facial image and the fourth facial image respectively from the first facial image and the second facial image by learning to regress three-dimensional facial shape and expression from images without three-dimensional supervision.

[0008] In one example, the self-supervised expression reconstruction model is obtained by pre-training an initial self-supervised expression reconstruction model using a fifth face image and a sixth face image as training data, wherein the fifth face image and the sixth face image are both three-dimensional images.

[0009] In one example, the method of using the fifth facial image and the sixth facial image as training data to train the initial self-supervised expression reconstruction model includes: inputting the fifth facial image and the sixth facial image into the initial self-supervised expression reconstruction model for training, and outputting a seventh facial image and an eighth facial image, wherein the seventh facial image includes the face of the fifth facial image and the expression is consistent with the expression of the sixth facial image, and the eighth facial image includes the face of the sixth facial image and the expression is consistent with the expression of the fifth facial image; inputting the seventh facial image and the eighth facial image into the initial self-supervised expression reconstruction model for training, and outputting a ninth facial image and a tenth facial image, wherein the ninth facial image includes the fifth facial image The first facial image includes a face of the fifth facial image and the expression is consistent with the expression of the fifth facial image, and the tenth facial image includes the face of the sixth facial image and the expression is consistent with the expression of the sixth facial image; determine whether the similarity between the expression of the ninth facial image and the expression of the fifth face exceeds a preset first threshold and / or whether the similarity between the expression of the tenth facial image and the expression of the sixth face exceeds a preset second threshold; if the similarity between the expression of the ninth facial image and the expression of the fifth face exceeds the first threshold and / or the similarity between the expression of the tenth facial image and the expression of the sixth face exceeds the second threshold, output the trained initial self-supervised expression reconstruction model as the self-supervised expression reconstruction model; otherwise, repeat the above training steps.

[0010] In one example, the expression data of the fourth facial image is extracted through a preset self-supervised expression reconstruction model and merged with the third facial image to obtain a new third facial image, including: encoding the third and fourth facial images according to the encoder in the self-supervised expression reconstruction model to obtain the encoded third and fourth facial expression features; extracting features of the difference between the expression data of the fourth face and the expressionless data of the fourth face according to the feature extraction network in the self-supervised expression reconstruction model to obtain features of the difference; fusing and decoding the features of the fourth face difference and the expression features of the third face through the decoder in the self-supervised expression reconstruction model to obtain a new third facial expression image.

[0011] In one example, the encoder and / or the decoder is a deep learning subnetwork in the self-supervised expression reconstruction model.

[0012] In one example, the formula of the deep learning convolutional neural network in the self-supervised expression reconstruction model includes: Among them, A function represents the graphical model; l is the layer of the deep learning convolutional neural network, v is all the landmarks in the face image, u is the common landmarks in the face image, a is the key point in the face image, and M is the dynamic weighting matrix.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a facial expression migration device, the device comprising:

[0014] An acquisition module is used to acquire a first facial image and a second facial image; a first conversion module is used to perform three-dimensional conversion on the first facial image and the second facial image respectively to obtain corresponding third facial images and fourth facial images; a reconstruction module is used to extract expression data of the fourth facial image through a preset self-supervised expression reconstruction model and merge it into the third facial image to obtain a new third facial image; a second conversion module is used to perform two-dimensional conversion on the new third facial image to obtain a new first facial image, and the expression of the new first facial image is consistent with the expression of the second facial image.

[0015] Furthermore, the present application also proposes a computer device, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of the facial expression migration method as described above are implemented.

[0016] Furthermore, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the facial expression migration method as described above.

[0017] Compared with the prior art, the facial expression migration method, device, computer equipment and computer-readable storage medium proposed in this application can obtain a first facial image and a second facial image; perform three-dimensional transformation on the first facial image and the second facial image respectively to obtain a corresponding third facial image and a fourth facial image; extract the expression data of the fourth facial image through a preset self-supervised expression reconstruction model and merge it into the third facial image to obtain a new third facial image; perform a two-dimensional transformation on the new third facial image to obtain a new first facial image, and the expression of the new first facial image is consistent with the expression of the second facial image. The expression data of a three-dimensional facial image is extracted through a self-supervised expression reconstruction model and then migrated to another three-dimensional facial image, thereby realizing accurate and rapid facial expression migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an application environment of an embodiment of the present application;

[0019] Figure 2 It is a flowchart of a specific embodiment of the facial expression transfer method of the present application;

[0020] Figure 3 is a flow chart of the training process of the self-supervised expression reconstruction model in an exemplary example of the present application;

[0021] Figure 4 This is an effect diagram of the facial landmark points corresponding to the deep learning convolutional neural network constructed in an exemplary example of the present application;

[0022] Figure 5 This is a flow chart of performing expression migration on a face image in video data in an exemplary example of the present application;

[0023] Figure 6 is a schematic diagram of a program module of an embodiment of the facial expression transfer device of the present applicant;

[0024] Figure 7 It is a schematic diagram of an optional hardware architecture of the computer device of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0026] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] At present, there are several main technologies for realizing facial expression transfer: affine parameter estimation of each face grid in the facial image of the expression source, and directly assign it to the background 3D (3-Dimensions) target face template, and perform affine transformation on other local grids based on the affine parameters to realize deformation operation, and finally achieve the purpose of spatial transformation of the overall face structured information; such algorithms need to perform parameter estimation and reconstruction operations on a grid-by-grid basis, which is very time-consuming. Alternatively, the decoupled model is used to reduce the dimension of the expression information of the source 3D face and the target face, and the expression information of the two is directly exchanged, and then the template face is reversely reconstructed using the new expression information; the dimensionality reduction process of such algorithms has limited representation ability of expression features and cannot achieve the transfer of rich expressions. Alternatively, a method based on deep convolutional neural networks is used to learn various facial expressions; however, this method requires different faces to have exactly the same degree of expression, but such data cannot be obtained in real scenes, which limits the reconstruction effect of expression transfer to a certain extent.

[0028] Figure 1 Schematic diagram of the application environment of an embodiment of the present application. Figure 1 As shown, the computer device 1 can obtain two face images to be used for expression migration from the network or other terminals connected to the computer device 1, or directly obtain two face images to be used for expression migration from its own storage, one of which is used as the first face image to be used for expression migration, and the other face image is the face expression source data, i.e., the second face image. Of course, the computer device 1 can also obtain a face image in video data as the first face image, and then use another face image in video data as the face expression source data.

[0029] In this embodiment, the computer device 1 includes a self-supervised expression reconstruction model. After the computer device 1 obtains the first face image and the second face image, the first face image and the second face image are respectively subjected to a three-dimensional transformation to obtain a corresponding third face image and a fourth face image; then, the expression data of the fourth face image is extracted through a preset self-supervised expression reconstruction model and replaced with the third face image to obtain a new third face image; finally, the new third face image is subjected to a two-dimensional transformation to obtain a new first face image, and the expression of the new first face image is consistent with the expression of the second face image, that is, a new first face image with the expression of the second face image. In this embodiment, the computer device 1 can be used as a terminal device such as a face expression server, a mobile terminal, a PC, etc. In other embodiments, the computer device 1 can also be used as an independent functional module, and then attached to a face expression server, a mobile terminal, or a PC to realize the function of face expression migration, which is not limited here.

[0030] Embodiment 1

[0031] Figure 2 FIG. 1 is a flow chart of an embodiment of the facial expression transfer method of the present invention. It is understood that the flow chart in the embodiment of the present method is not used to limit the order of executing the steps. Figure 1 The application environment shown is taken as an example, and an exemplary description is given with a computer device as the execution subject.

[0032] like Figure 2 As shown, the facial expression transfer method may include steps S200 to S206, wherein:

[0033] Step S200: Acquire a first face image and a second face image.

[0034] Step S202: Perform three-dimensional transformation on the first face image and the second face image respectively to obtain a corresponding third face image and a fourth face image.

[0035] Specifically, the computer device 1 can provide a user interaction interface, and then receive a facial expression migration request for a target facial image, i.e., a first facial image, triggered by a user on the user interaction interface, and receive a facial image corresponding to the facial expression source data specified by the user, i.e., a second facial image. For example, a user can input two video data, one of which is used as the facial expression migration to be performed, and the other is used as the facial expression source data. Then, the computer device 1 will read the facial image video frames frame by frame from the video data for the facial expression migration to be performed as the first facial image, and at the same time, read the facial image video frames frame by frame from the video data corresponding to the facial expression source data as the second facial image.

[0036] Next, the computer device 1 performs three-dimensional conversion on the first face image and the second face image respectively to obtain corresponding third face image and fourth face image, including: obtaining the third face image and the fourth face image respectively from the first face image and the second face image by learning to regress the three-dimensional face shape and expression from the image without three-dimensional supervision. Among them, the technology of learning to regress the three-dimensional face shape and expression from the image without three-dimensional supervision can effectively convert the two-dimensional face image into a three-dimensional face image and maintain rich facial expressions.

[0037] Step S204, extracting expression data of the fourth facial image through a preset self-supervised expression reconstruction model and fusing it into the third facial image to obtain a new third facial image.

[0038] In this embodiment, the self-supervised expression reconstruction model is obtained by the computer device 1 using the fifth face image and the sixth face image as training data to train the initial self-supervised expression reconstruction model, wherein the fifth face image and the sixth face image are both three-dimensional images. For example, the computer device 1 pre-collects a preset number of levels, such as tens of millions of three-dimensional images, as training data to train the preset initial self-supervised expression reconstruction model, thereby obtaining the self-supervised expression reconstruction model.

[0039] The computer device 1 uses the fifth face image and the sixth face image as training data to train the initial self-supervised expression reconstruction model, including: inputting the fifth face image and the sixth face image into the initial self-supervised expression reconstruction model for training, and outputting a seventh face image and an eighth face image, wherein the seventh face image includes the face of the fifth face image and the expression is consistent with the expression of the sixth face image, and the eighth face image includes the face of the sixth face image and the expression is consistent with the expression of the fifth face image; inputting the seventh face image and the eighth face image into the initial self-supervised expression reconstruction model for training, and outputting a ninth face image and a tenth face image, wherein the ninth face image includes the fifth face image The face image includes a face and an expression that is consistent with the expression of the fifth face image, and the tenth face image includes a face and an expression that is consistent with the expression of the sixth face image; it is determined whether the similarity between the expression of the ninth face image and the expression of the fifth face exceeds a preset first threshold and / or whether the similarity between the expression of the tenth face image and the expression of the sixth face exceeds a preset second threshold; if the similarity between the expression of the ninth face image and the expression of the fifth face exceeds the first threshold and / or the similarity between the expression of the tenth face image and the expression of the sixth face exceeds the second threshold, the trained initial self-supervised expression reconstruction model is output as the self-supervised expression reconstruction model; otherwise, the above training steps are repeated.

[0040] In an exemplary example, the self-supervised expression reconstruction model is a model in which multiple network models are cascaded to a deep learning convolutional neural network to jointly realize the facial expression reconstruction function, including: a feature extraction network, an encoding network and a decoding network.

[0041] The computer device 1 extracts the expression data of the fourth face image through a preset self-supervised expression reconstruction model and fuses it to the third face image to obtain a new third face image, including: encoding the third and fourth face images according to the encoder in the self-supervised expression reconstruction model to obtain the encoded third and fourth face expression features; extracting features of the difference between the expression data of the fourth face and the expressionless data of the fourth face according to the feature extraction network to obtain the features of the difference; fusing and decoding the features of the fourth face difference and the third face expression features through the decoder in the self-supervised expression reconstruction model to obtain a new third face expression image. Wherein, the encoder and / or the decoder are deep learning subnetworks in the self-supervised expression reconstruction model.

[0042] See also Figure 3 As shown, Figure 3 It is a flow chart of the training process of the self-supervised expression reconstruction model in an exemplary example of the present application. In this embodiment, the computer device 1 cascades two initial self-supervised expression reconstruction models, that is, the first initial self-supervised expression reconstruction model is cascaded with the second initial self-supervised expression reconstruction model, and shares weights. Then, the computer device 1 inputs the fifth face image Xt and the sixth face image Ys into the first initial self-supervised expression reconstruction model, extracts the feature s in the difference between the expressionless sixth face image Y0 and the sixth face image in the sixth face image Ys through the feature extraction network T(·), and then inputs s into the deep learning convolutional neural network through the encoding network E(·) for training, and finally reconstructs the input image data through the decoding network D(·), thereby outputting the seventh face image Xs1 with the same expression as the sixth face image Ys, and the eighth face image Yt1 with the same expression as the fifth face image Xt.

[0043] Next, the computer device 1 inputs the seventh facial image Xs1 and the eighth facial image Ys1 into the second initial self-supervised expression reconstruction model, extracts feature t from the difference between the expressionless fifth facial image X0 and the fifth facial image in the fifth facial image Xt through the feature extraction network T(·) therein, and then inputs t into the deep learning convolutional neural network for training through the encoding network E(·), and finally reconstructs the expression of the input image data through the decoding network D(·), thereby outputting the tenth facial image Ys2 with the same expression as the sixth facial image Ys, and the ninth facial image Xt2 with the same expression as the fifth facial image Xt.

[0044] Finally, determine whether the similarity between Xt and Xt2 exceeds a preset first threshold; if it exceeds the first threshold, end the training and output the trained initial self-supervised expression reconstruction model as the final self-supervised expression reconstruction model; otherwise, repeat the above training process.

[0045] Among them, the formula of the deep learning convolutional neural network in the self-supervised expression reconstruction model includes: Among them, A function represents the graph model; l is the layer of the deep learning convolutional neural network, v is all the landmarks in the face image, u is the common landmarks in the face image, a is the key point in the face image, and M is the dynamic weight matrix. Figure 4 As shown, Figure 4 This is an effect diagram of the facial landmark points corresponding to the deep learning convolutional neural network constructed in an exemplary example of this application. Among them, Represents the relationship function between all the identification points on the face image corresponding to the lth layer of the deep learning convolutional neural network; It represents the relationship function between points corresponding to the lth layer of the deep learning convolutional neural network describing the common landmarks on the face image; Represents all the identification points and key points on the face image corresponding to the lth layer of the deep learning convolutional neural network (i.e., Figure 4 The relationship function between the key points in the circle in the upper part of the figure and other common identification points); M represents the weighted matrix, that is, the distance matrix between the key points and all other identification points, Manifested as Figure 4 In the lower part of the figure, the key point and each other identification point have not only the relationship between points but also the distance relationship, which can well show the position linkage relationship between the key point and other face identification points when the position of another face identification point changes. Therefore, the final calculated It can well reflect the positional and motion relationships of all landmarks on the face image, so it can better describe facial expressions.

[0046] Step S206, performing a two-dimensional transformation on the new third facial image to obtain a new first facial image, wherein the expression of the new first facial image is consistent with the expression of the second facial image.

[0047] Finally, the computer device 1 performs a two-dimensional conversion on the new third facial image after expression reconstruction via the self-supervised expression reconstruction model, for example, directly mapping the three-dimensional facial image data into two-dimensional facial image data, thereby obtaining a new first facial image, that is, a first facial image having an expression consistent with the second facial image.

[0048] See also Figure 5 As shown, Figure 5 This is a flow chart of expression transfer for a face image in video data in an exemplary example of the present application. In this embodiment, the computer device 1 first receives a 2D face video input by a user, and then processes the 2D face video frame by frame, and sequentially performs: spatial mapping of the face image from 2D to 3D, and then inputs it into a self-supervised expression reconstruction model for 3D face data to obtain a 3D face image after expression reconstruction; then spatial mapping of the face image from 3D to 2D to obtain a 2D face image after expression transfer, and finally splicing it into a 2D video, thereby outputting an expression transfer video.

[0049] As can be seen from the above, the facial expression migration method proposed in this embodiment can obtain a first facial image and a second facial image; perform three-dimensional transformation on the first facial image and the second facial image respectively to obtain corresponding third facial images and fourth facial images; extract the expression data of the fourth facial image through a preset self-supervised expression reconstruction model and replace it with the third facial image to obtain a new third facial image; perform a two-dimensional transformation on the new third facial image to obtain a new first facial image, and the expression of the new first facial image is consistent with the expression of the second facial image. The expression data of a three-dimensional facial image is extracted through a self-supervised expression reconstruction model and then migrated to another three-dimensional facial image, thereby realizing accurate and rapid facial expression migration.

[0050] Embodiment 2

[0051] Figure 6 A block diagram of a facial expression migration device according to Embodiment 2 of the present application is schematically shown, and the facial expression migration device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0052] like Figure 6 As shown, the facial expression transfer device 400 may include an acquisition module 410, a first conversion module 420, a reconstruction module 430 and a second conversion module 440, wherein:

[0053] The acquisition module 410 is used to acquire a first face image and a second face image.

[0054] The first conversion module 420 is used to perform three-dimensional conversion on the first face image and the second face image respectively to obtain a corresponding third face image and a fourth face image.

[0055] The reconstruction module 430 is used to extract the expression data of the fourth facial image through a preset self-supervised expression reconstruction model and merge it into the third facial image to obtain a new third facial image.

[0056] The second conversion module 440 is used to perform a two-dimensional conversion on the new third facial image to obtain a new first facial image, wherein the expression of the new first facial image is consistent with the expression of the second facial image.

[0057] In an exemplary example, the first conversion module 420 is also used to obtain the third facial image and the fourth facial image respectively from the first facial image and the second facial image by learning to regress the three-dimensional facial shape and expression from the image without three-dimensional supervision.

[0058] In an exemplary example, the self-supervised expression reconstruction model is obtained by pre-training the initial self-supervised expression reconstruction model using the fifth face image and the sixth face image as training data, wherein the fifth face image and the sixth face image are both three-dimensional images. The formula of the deep learning convolutional neural network in the self-supervised expression reconstruction model includes: Among them, A function represents the graphical model; l is the layer of the deep learning convolutional neural network, v is all the landmarks in the face image, u is the common landmarks in the face image, a is the key point in the face image, and M is the dynamic weighting matrix.

[0059] In an exemplary example, the reconstruction module 430 is further used to: input the fifth facial image and the sixth facial image into an initial self-supervised expression reconstruction model for training, and output a seventh facial image and an eighth facial image, wherein the seventh facial image includes the face of the fifth facial image and the expression is consistent with the expression of the sixth facial image, and the eighth facial image includes the face of the sixth facial image and the expression is consistent with the expression of the fifth facial image; input the seventh facial image and the eighth facial image into an initial self-supervised expression reconstruction model for training, and output a ninth facial image and a tenth facial image, wherein the ninth facial image includes the face of the fifth facial image and the expression is consistent with the expression of the fifth facial image. The expression of the facial image is consistent with that of the facial image, the tenth facial image includes the face of the sixth facial image and the expression is consistent with the expression of the sixth facial image; it is determined whether the similarity between the expression of the ninth facial image and the expression of the fifth face exceeds a preset first threshold and / or whether the similarity between the expression of the tenth facial image and the expression of the sixth face exceeds a preset second threshold; if the similarity between the expression of the ninth facial image and the expression of the fifth face exceeds the first threshold and / or the similarity between the expression of the tenth facial image and the expression of the sixth face exceeds the second threshold, the trained initial self-supervised expression reconstruction model is output as the self-supervised expression reconstruction model; otherwise, the above training steps are repeated.

[0060] In an illustrative example, the reconstruction module 430 is also used to: encode the third and fourth facial images according to the encoder in the self-supervised expression reconstruction model to obtain encoded third and fourth facial expression features; extract features of the difference between the expression data of the fourth face and the expressionless data of the fourth face according to the feature extraction network in the self-supervised expression reconstruction model to obtain features of the difference; fuse and decode the features of the fourth face difference and the third facial expression features through the decoder in the self-supervised expression reconstruction model to obtain a new third facial expression image; wherein the encoder and / or the decoder is a deep learning subnetwork in the self-supervised expression reconstruction model.

[0061] Embodiment 3

[0062] Figure 7 The schematic diagram of the hardware architecture of a computer device corresponding to a computer device 1 suitable for implementing a facial expression migration method according to the third embodiment of the present application is shown schematically. In this embodiment, the computer device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, blade server, tower server or cabinet server (including an independent server, or a server cluster composed of multiple servers) with a gateway function. Figure 7As shown, the computer device at least includes but is not limited to: a memory 510, a processor 520, and a network interface 530 that can communicate with each other through a system bus. Among them:

[0063] The memory 510 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 510 can be an internal storage module of the computer device, such as a hard disk or memory of the computer device 1. In other embodiments, the memory 510 can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device 1, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 510 can also include both the internal storage module of the computer device 1 and its external storage device. In this embodiment, the memory 510 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the facial expression migration method, etc. In addition, the memory 510 can also be used to temporarily store various types of data that have been output or are to be output.

[0064] In some embodiments, the processor 520 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 520 is generally used to control the overall operation of the computer device, such as performing control and processing related to data interaction or communication with the computer device 1. In this embodiment, the processor 520 is used to run the program code stored in the memory 510 or process data.

[0065] The network interface 530 may include a wireless network interface or a wired network interface, and the network interface 530 is generally used to establish a communication link between the computer device and other devices. For example, the network interface 530 is used to connect the computer device to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 1 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0066] It should be pointed out that Figure 7 The computer device 1 is shown only with components 510 - 530 , but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0067] In this embodiment, the program code of the facial expression migration method stored in the memory 510 can also be divided into one or more program modules and executed by one or more processors (processor 520 in this embodiment) to complete the embodiment of the present application.

[0068] Embodiment 4

[0069] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0070] Acquire a first facial image and a second facial image; perform three-dimensional transformation on the first facial image and the second facial image respectively to obtain corresponding third facial image and fourth facial image; extract expression data of the fourth facial image through a preset self-supervised expression reconstruction model and fuse it into the third facial image to obtain a new third facial image; perform two-dimensional transformation on the new third facial image to obtain a new first facial image, wherein the expression of the new first facial image is consistent with the expression of the second facial image.

[0071] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code of the facial expression migration method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0072] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0073] The above are only preferred embodiments of the embodiments of the present application, and are not intended to limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the embodiments of the present application.

Claims

1. A facial expression transfer method, characterized in that: The method comprises: Acquire a first face image and a second face image; Performing three-dimensional transformation on the first face image and the second face image respectively to obtain a corresponding third face image and a fourth face image; Extracting expression data of the fourth facial image through a preset self-supervised expression reconstruction model and fusing it into the third facial image to obtain a new third facial image; Performing a two-dimensional transformation on the new third facial image to obtain a new first facial image, wherein an expression of the new first facial image is consistent with an expression of the second facial image; The self-supervised expression reconstruction model is obtained by pre-training an initial self-supervised expression reconstruction model using a fifth face image and a sixth face image as training data, wherein both the fifth face image and the sixth face image are three-dimensional images; The step of using the fifth face image and the sixth face image as training data to train the initial self-supervised expression reconstruction model includes: Inputting the fifth face image and the sixth face image into an initial self-supervised expression reconstruction model for training, and outputting a seventh face image and an eighth face image, wherein the seventh face image includes the face of the fifth face image and the expression is consistent with the expression of the sixth face image, and the eighth face image includes the face of the sixth face image and the expression is consistent with the expression of the fifth face image; Inputting the seventh face image and the eighth face image into an initial self-supervised expression reconstruction model for training, and outputting a ninth face image and a tenth face image, wherein the ninth face image includes the face of the fifth face image and the expression is consistent with the expression of the fifth face image, and the tenth face image includes the face of the sixth face image and the expression is consistent with the expression of the sixth face image; Determining whether the similarity between the expression of the ninth face image and the expression of the fifth face exceeds a preset first threshold and / or whether the similarity between the expression of the tenth face image and the expression of the sixth face exceeds a preset second threshold; If the similarity between the expression of the ninth facial image and the expression of the fifth face exceeds the first threshold and / or the similarity between the expression of the tenth facial image and the expression of the sixth face exceeds the second threshold, the trained initial self-supervised expression reconstruction model is output as the self-supervised expression reconstruction model; otherwise, the above training steps are repeated.

2. The facial expression transfer method according to claim 1, wherein: The performing three-dimensional conversion on the first face image and the second face image respectively to obtain a corresponding third face image and a fourth face image comprises: The third face image and the fourth face image are obtained from the first face image and the second face image respectively by learning a method of regressing the three-dimensional face shape and expression from the image without three-dimensional supervision.

3. The facial expression transfer method according to any one of claims 1 to 2, characterized in that: The method of extracting expression data of the fourth face image through a preset self-supervised expression reconstruction model and fusing the expression data to the third face image to obtain a new third face image includes: encoding the third and fourth facial images according to the encoder in the self-supervised facial expression reconstruction model to obtain encoded third and fourth facial expression features; Extracting features of the difference between the expression data of the fourth face and the expressionless data of the fourth face according to the feature extraction network in the self-supervised expression reconstruction model to obtain features of the difference; The features of the fourth face difference and the third face expression features are fused and decoded through the decoder in the self-supervised expression reconstruction model to obtain a new third face expression image.

4. The facial expression transfer method as claimed in claim 3, characterized in that: The encoder and / or the decoder is a deep learning subnetwork in the self-supervised expression reconstruction model.

5. The facial expression transfer method according to any one of claims 1 to 4, characterized in that: The formula of the deep learning convolutional neural network in the self-supervised expression reconstruction model includes: ； Among them, A function represents the graphical model; l is the layer of the deep learning convolutional neural network, v is all the landmarks in the face image, u is the common landmarks in the face image, a is the key point in the face image, and M is the dynamic weighting matrix.

6. A facial expression transfer device, characterized in that: The device comprises: An acquisition module, used to acquire a first face image and a second face image; A first conversion module, configured to perform three-dimensional conversion on the first face image and the second face image respectively to obtain a corresponding third face image and a fourth face image; A reconstruction module, used for extracting expression data of the fourth face image through a preset self-supervised expression reconstruction model and fusing it into the third face image to obtain a new third face image, wherein the self-supervised expression reconstruction model is obtained by pre-training an initial self-supervised expression reconstruction model using a fifth face image and a sixth face image as training data, wherein the fifth face image and the sixth face image are both three-dimensional images; wherein the training of the initial self-supervised expression reconstruction model using the fifth face image and the sixth face image as training data comprises: inputting the fifth face image and the sixth face image into the initial self-supervised expression reconstruction model for training, and outputting a seventh face image and an eighth face image, wherein the seventh face image includes the face of the fifth face image and the expression is consistent with the expression of the sixth face image, and the eighth face image includes the face of the sixth face image and the expression is consistent with the expression of the fifth face image; The seventh face image and the eighth face image are input into the initial self-supervised expression reconstruction model for training, and a ninth face image and a tenth face image are output, wherein the ninth face image includes the face of the fifth face image and the expression is consistent with the expression of the fifth face image, and the tenth face image includes the face of the sixth face image and the expression is consistent with the expression of the sixth face image; it is determined whether the similarity between the expression of the ninth face image and the expression of the fifth face exceeds a preset first threshold and / or whether the similarity between the expression of the tenth face image and the expression of the sixth face exceeds a preset second threshold; if the similarity between the expression of the ninth face image and the expression of the fifth face exceeds the first threshold and / or the similarity between the expression of the tenth face image and the expression of the sixth face exceeds the second threshold, the trained initial self-supervised expression reconstruction model is output as the self-supervised expression reconstruction model; otherwise, the above training steps are repeated; The second conversion module is used to perform a two-dimensional conversion on the new third facial image to obtain a new first facial image, wherein the expression of the new first facial image is consistent with the expression of the second facial image.

7. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the steps of the facial expression migration method as described in any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the facial expression transfer method according to any one of claims 1 to 5.

9. A computer program product comprising computer instructions, characterized in that: When the computer instruction is executed by the processor, the steps of the facial expression transfer method as described in any one of claims 1-5 are implemented.

Images