Face image fusion method and device, and storage medium

Abstract

The present application relates to image fusion and medical field, disclose a kind of face image fusion method, comprising: using face image fusion model to the feature of source face image is fused to target face image, obtain initial face image;Using the fully connected layer of model to initial face image, target face image expression recognition obtains first group of vectors, using the classification function of model to initial face image, target face image expression recognition obtains second group of vectors;According to first group of vectors and second vector respectively calculate the first and second cosine similarity of initial face image and target face image;According to loss value formula and first and second cosine similarity, calculate loss value, according to loss value, call face image fusion model to initial face image is modified, obtain face fusion image.The present application also provides device and medium.The present application can improve the expression natural degree of facial expression of image obtained after face image fusion, improve the propaganda effect of medical knowledge video.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and more particularly to a method, apparatus and storage medium for face image fusion. Background Technology

[0002] Face image fusion refers to transferring facial features from a source face to a target face while maintaining the attributes of the source face (such as expression, pose, lighting, etc.) unchanged on the target face.

[0003] For example, in the medical field, it is necessary to promote some important medical knowledge to the public, such as how individuals can prevent AIDS. This requires producing relevant videos to disseminate this information and finding suitable individuals to film them. In this case, facial image fusion technology can be used to create promotional videos featuring specific individuals promoting medical knowledge. Using images of specific faces to promote medical knowledge can increase public interest and thus achieve a more effective dissemination of medical information.

[0004] With the development of machine learning technology, there are currently two main types of face image fusion methods: source-oriented methods that operate on the source face at the image level and target-oriented methods that operate on the target face at the feature level.

[0005] Source-oriented approaches transfer attributes from the target face to the source face, then blend the source face into the target image. Early methods used 3D models to transfer pose and lighting, but required human intervention. Other approaches utilize 3D face datasets to transfer representations and poses, then use Poisson mixing to merge the source face into the target image. However, due to the limited expressive power of 3D face datasets, methods that reconstruct from 3D models often fail to accurately reproduce these representations.

[0006] Target-oriented approaches utilize neural networks to extract features from target images, then modify these features to reconstruct the output face-swapping image. This requires training a generator capable of swapping faces with a specific identity; once trained, it can swap faces between two specific identities, but lacks generalization capabilities. Methods like this combine latent representations from source face regions and target non-face regions to produce results, but fail to preserve the target's facial expressions.

[0007] Therefore, the fused facial images obtained by the two current methods of facial image fusion often exhibit unnatural facial expressions. Summary of the Invention

[0008] In view of the above, it is necessary to provide a method for face image fusion, which aims to solve the technical problem that the facial expressions of the faces obtained by face image fusion in the existing technology are unnatural. In the medical field, it also aims to address the technical problem of how to use face image fusion technology to create videos that can increase public interest in medical information.

[0009] This invention provides a method for face image fusion, comprising:

[0010] The facial features of the source face image are fused into the target face image using a preset face image fusion model to obtain the initial face image;

[0011] The initial face image and the target face image are subjected to expression recognition using the fully connected layer of the expression recognition model to obtain the first set of vectors;

[0012] The initial face image and the target face image are subjected to expression recognition using the classification function of the expression recognition model to obtain a second set of vectors;

[0013] The first cosine similarity between the initial face image and the target face image is calculated based on the first set of vectors, and the second cosine similarity between the initial face image and the target face image is calculated based on the second set of vectors.

[0014] The expression supervision loss values ​​of the initial face image and the target face image are calculated according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity. The face image fusion model is then called to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

[0015] Optionally, a preset face image fusion model is used to fuse the facial features of the source face image into the target face image to obtain an initial face image, including:

[0016] The encoder of the face image fusion model is used to extract the facial features of the target face image, the identity extraction module of the face image fusion model is used to extract the identity information of the source face image, and the identity injection module is used to embed the identity information of the source face image into the facial features of the target face image.

[0017] The initial face image is generated by performing bilinear interpolation and convolution operations on the facial features of the target face image, which contains the identity information of the source face image.

[0018] Optionally, the expression recognition model is used to perform expression recognition on the initial face image and the target face image to obtain a first set of vectors, including:

[0019] The fully connected layer of the VGG-16 network of the expression recognition model is used to perform expression recognition on the initial face image and the target face image;

[0020] The vectors of the first preset dimension of the identified initial face image and the target face image are respectively used as the first vector and the second vector, and the first vector and the second vector are used as the first set of vectors.

[0021] Optionally, the initial face image and the target face image are subjected to expression recognition using the classification function of the expression recognition model to obtain a second set of vectors, including:

[0022] The expression recognition model's classification function is used to perform expression recognition on the initial face image and the target face image;

[0023] The vectors of the second preset dimension of the identified initial face image and the target face image are respectively used as the third vector and the fourth vector, and the third vector and the fourth vector are used as the second set of vectors.

[0024] Optionally, calculating the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and calculating the second cosine similarity between the initial face image and the target face image based on the second set of vectors, includes:

[0025] The first cosine similarity is obtained by calculating the cosine similarity between the first vector and the second vector, and the second cosine similarity is obtained by calculating the cosine similarity between the third vector and the fourth vector.

[0026] Optionally, the expression supervision loss value of the initial face image and the target face image is calculated according to the preset loss value calculation formula and the first cosine similarity, including:

[0027] The first expression supervision loss value, Loss1, is calculated using the preset first loss value calculation formula:

[0028] Loss1 = 1 – Fcos,

[0029] Where Loss1 represents the high-dimensional latent feature loss of the expressions of the initial face image and the target face image, and Fcos represents the first cosine similarity.

[0030] Optionally, the expression supervision loss value of the initial face image and the target face image is calculated according to a preset loss value calculation formula and the second cosine similarity, including:

[0031] The second expression supervision loss value, Loss2, is calculated using a pre-defined formula:

[0032] Loss2 = 1 – Scos,

[0033] Where Loss1 represents the expression classification loss between the initial face image and the target face image, and Scos represents the second cosine similarity.

[0034] To address the above problems, the present invention also provides a face image fusion device, the device comprising:

[0035] The generation module is used to fuse the facial features of the source face image into the target face image using a preset face image fusion model to obtain an initial face image;

[0036] The recognition module is used to perform expression recognition on the initial face image and the target face image using the fully connected layer of the expression recognition model to obtain a first set of vectors;

[0037] The recognition module is further configured to perform expression recognition on the initial face image and the target face image using the classification function of the expression recognition model to obtain a second set of vectors;

[0038] The calculation module is used to calculate the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and to calculate the second cosine similarity between the initial face image and the target face image based on the second set of vectors.

[0039] The correction module is used to calculate the expression supervision loss value of the initial face image and the target face image according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity, and call the face image fusion model to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

[0040] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0041] At least one processor; and,

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a face image fusion program that can be executed by the at least one processor, the face image fusion program being executed by the at least one processor to enable the at least one processor to perform the aforementioned face image fusion method.

[0044] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing a face image fusion program, which can be executed by one or more processors to implement the aforementioned face image fusion method.

[0045] Compared to existing technologies, face fusion is performed directly using a pre-set face image fusion model.

[0046] The present invention utilizes a face image fusion model to fuse the features of a source face image into a target face image to obtain an initial face image. Then, the fully connected layer of the model is used to perform expression recognition on the initial face image and the target face image to obtain a first set of vectors. Finally, the model's classification function is used to perform expression recognition on the initial face image and the target face image to obtain a second set of vectors.

[0047] The first cosine similarity and the second cosine similarity of the initial face image and the target face image are calculated based on the first set of vectors and the second set of vectors, respectively. The loss value is calculated according to the loss value formula and the first and second cosine similarities. The face image fusion model is then called to correct the initial face image based on the loss value to obtain the face fusion image. This can improve the accuracy of the preset face image fusion model, thereby improving the naturalness of the facial expressions in the face image fusion. In the medical field, the more natural the facial expressions of the people in the promotional videos produced using face image fusion technology, the more the public's attention can be focused on the medical knowledge presented in the video, thus achieving the purpose of promoting medical-related knowledge. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a face image fusion method according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of a face image fusion device according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device that implements a method for face image fusion according to an embodiment of the present invention;

[0051] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0053] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0054] Reference Figure 1 The diagram shown is a flowchart illustrating a face image fusion method according to an embodiment of the present invention. This method is executed by an electronic device.

[0055] In this embodiment, a method for face image fusion includes steps S1-S5.

[0056] S1. Using a preset face image fusion model, the facial features of the source face image are fused into the target face image to obtain the initial face image.

[0057] In this embodiment, the face image fusion model uses the Sim-swap model, which includes an encoder, an identity extraction module, an identity injection module, a decoder, and a loss module.

[0058] The source face image is an image using the VGG-Face2 face dataset. The source face image and the target face image are images selected for face swapping. For example, if user A wants to swap their face onto user B's face, then user A's face image is the source face image, and user B's face image is the target face image.

[0059] In the medical field, such as in the field of medical knowledge promotion, User A is a celebrity who has already communicated with the hospital in advance and is willing to appear on camera. A celebrity is someone with a certain degree of fame. User B is a non-public figure who has recorded a video in advance to promote medical knowledge.

[0060] For example, in the field of medical consultation, in online consultations on a hospital platform, user A is a doctor on the hospital platform who regularly consults with patient C, and user B is another doctor on the hospital platform. When user A does not have time to consult with patient C, and patient C is afraid of other doctors, all the steps of this application can be executed.

[0061] In one embodiment, step S1 involves extracting facial features from the target face image using the Sim-swap model encoder of the face image fusion model, then extracting the identity information of the source face image using the identity extraction module of the face image fusion model, embedding the identity information of the source face image into the facial features of the target face image using the identity injection module of the face image fusion model, and then performing bilinear interpolation and convolution operations on the facial features of the target face image with the embedded identity information of the source face image to generate the initial face image.

[0062] The encoder of the Sim-swap model is implemented based on the VGG-13 convolutional network.

[0063] The identity extraction module is implemented using a deep convolutional neural network based on additive angular margin loss. It extracts the identity information of the source face image and then embeds the extracted identity information of the source face image into the face features of the target face image based on residual blocks.

[0064] The method of embedding the identity information extracted from the source face image into the facial features of the target face image based on residual blocks includes the following steps: performing a first convolution on the facial features of the target face image; performing a first adaptive instance regularization (AdaIN) on the facial features of the convolved target face image and the identity information of the source face image; performing a second convolution on the facial features of the target face image output after adaptive instance regularization; and then performing a second adaptive instance regularization on the facial features of the target face image after the second convolution and the identity information of the source face image to obtain the facial features of the target face image embedded with the identity information of the source face image.

[0065] Adaptive instance regularization is used to align the mean and standard deviation of the facial features in the target face image with the source face identity vector, thereby ensuring that the facial features in the target face image are consistent with those in the source face image.

[0066] Then, the Sim-swap model's decoder is used to enlarge the facial features of the target face image, which contains the source face identity information, into an image with higher resolution.

[0067] In one embodiment, the decoder is implemented based on upsampling and convolution. Upsampling is achieved using bilinear interpolation, which interpolates the feature map to produce a larger feature map based on the upsampling rate. The upsampling rate is not limited here. The upsampling of the feature maps is independent of each other, so the number of channels in the convolution does not change. The convolution operation is performed on the feature map, and then the loss module of the Sim-swap model is used to train the loss on the feature map after the convolution operation to obtain the initial face image.

[0068] In the field of medical promotion, the face of user A (a celebrity) has been superimposed onto the face of user B (a non-public figure). The resulting initial face image is an image of user B's face replaced with user A's face. This allows us to replace all of user B's faces in videos that user B has pre-recorded with a camera to promote medical knowledge with user A's face. This provides the first step in preparing for the production of videos featuring celebrities promoting medical knowledge.

[0069] In the field of medical consultation, online video consultations on hospital platforms can provide the first step in making the face of user A more accurate when consulting with patient C.

[0070] In some embodiments, other face image fusion models can be used, and the Sim-swap model is not limited to this one.

[0071] In other embodiments, the source face image may also use the CelebA face dataset.

[0072] In other embodiments, upsampling can also be implemented using deconvolution operations, which increase the size of the input value by transposing the convolution.

[0073] S2. Perform expression recognition on the initial face image and the target face image using the fully connected layer of the expression recognition model to obtain the first set of vectors.

[0074] In this embodiment, an expression recognition model is used to perform expression recognition on the initial face image and the target face image, respectively. The expression recognition model is implemented based on the VGG16 network and the softmax classification function. The VGG16 network consists of 6 block structures, each with the same number of convolutional channels. The entire VGG16 network uses 3x3 kernels for all convolutional layers and 2x2 max pooling layers for all pooling layers. Each block structure is delimited by a pooling layer. Specifically, the VGG16 network consists of convolution-convolution-pooling-convolution-convolution-pooling-convolution-convolution-convolution-pooling-convolution-convolution-convolution-pooling-convolution-convolution-pooling-convolution-convolution-pooling-fully connected-fully connected-fully connected layers. The number of convolutional channels in the first five block structures are 64, 128, 512, 512, and 512, respectively, and the number of channels in the last three fully connected layers are 4096, 4096, and 1000, respectively.

[0075] The VGG16 network was chosen as part of the facial expression recognition model because VGG16, like other convolutional neural networks, requires fewer parameters while having the same receptive field, allowing computers to perform facial expression recognition more quickly.

[0076] The third fully connected layer of the VGG-16 network is used to perform expression recognition on the initial face image and the target face image. The 1000-dimensional vectors of the recognized initial face image and the target face image are used as the first vector and the second vector, respectively, and the first vector and the second vector are used as the first set of vectors. The reason for choosing the third fully connected layer of the VGG16 network is that the first and second fully connected layers both have 4096 neurons, while the third fully connected layer only has 1000 neurons. Therefore, the parameters of the first and second fully connected layers are much more than those of the third fully connected layer, so choosing the third fully connected layer can relatively improve the computing speed of the computer.

[0077] In the medical field and in the field of publicity, step 2 can obtain the features of the initial face image (the image of user B's face has been replaced with the image of user A's face) and the features of user B's face image. Extracting the features of the above two can provide data support for the following calculation of cosine similarity, ultimately making the celebrity expressions in the generated videos of celebrities promoting medical knowledge more natural.

[0078] In the field of online consultations, it can also make the expressions of user A who is consulting with patient C more accurate, making patient C feel more comfortable when consulting online on the hospital platform.

[0079] In some embodiments, the facial expression recognition model can be implemented based on the VGG19 network and the softmax function. The VGG19 network is basically the same as the VGG16 network, but the VGG19 network has one more convolutional layer in the third, fourth, and fifth blocks, respectively, with the same number of convolutional channels as the VGG16 network in each block.

[0080] S3. Perform expression recognition on the initial face image and the target face image using the classification function of the expression recognition model to obtain a second set of vectors.

[0081] In this embodiment, an expression recognition model is used to identify the initial face image and the target face image respectively. After feature extraction through the VGG16 network of the model, each image is classified into seven expressions using a softmax classification function: 0. Anger; 1. Disgust; 2. Fear; 3. Happy; 4. Normal; 5. Sad; 6. Surprised. The initial face image and the target face image are each classified using the softmax function, outputting seven predicted expression values. The seven-dimensional vectors of the identified initial face image and the target face image are used as the third and fourth vectors, respectively, and these three vectors are used as the second set of vectors. To better reflect the difference between the model and the actual data, cross-entropy is used as the loss function.

[0082] In the medical field, the facial expressions of User B when recording a medical promotional video with a camera may differ from those captured by User A with a camera. Step 3 can obtain the facial expression features of the initial face image and User B's face image. This allows us to identify where the generated face image of User B replaced with User A's face is relatively unnatural compared to User B's face image, providing data support for calculating the loss function and correcting the entire model.

[0083] In the field of online consultation, this step can obtain the facial features of the initial face image and the face image of user B. This allows for more accurate facial expressions when user B, who is using user A's facial expressions, is consulting with patient C online, thus reducing the patient's anxiety.

[0084] S4. Calculate the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and calculate the second cosine similarity between the initial face image and the target face image based on the second set of vectors.

[0085] In this embodiment, the cosine similarity between the first vector and the second vector is calculated using a preset cosine similarity calculation formula to obtain the first cosine similarity, and the cosine similarity between the third vector and the fourth vector is calculated to obtain the second cosine similarity.

[0086] In the medical field, calculating these two cosine similarities can provide direct computational data for the loss function below. Then, by correcting the initial face image, the facial image generated by replacing user B's face with user A's face can be more similar to the facial expression of user B's face image. This is beneficial for generating more realistic facial images in videos of celebrities promoting medical knowledge, thus enabling medical knowledge to be disseminated more widely.

[0087] In the field of online consultation, by calculating the two cosine similarities mentioned above, in the scenario of online consultation on a hospital platform, the facial expressions of the person whose face is replaced with that of user A when consulting with patient C are more accurate, and patient C feels more at ease.

[0088] S5. Calculate the expression supervision loss value of the initial face image and the target face image according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity. Call the face image fusion model to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

[0089] In this embodiment, the first expression supervision loss value Loss1 is calculated using a preset first loss value calculation formula, where Loss1 = 1 – Fcos, Loss1 is the high-dimensional latent feature loss of the expression between the initial face image and the target face image, and Fcos is the first cosine similarity.

[0090] The second expression supervision loss value Loss2 is calculated using the preset second loss value calculation formula. Loss2 = 1 – Scos, where Loss1 is the expression classification loss between the initial face image and the target face image, and Scos is the second cosine similarity.

[0091] Based on the two expression supervision loss values ​​above, the face image fusion model is called to correct the initial face image and obtain a face fusion image. The expression of this face image will be more natural. In the medical field, the more natural the expression of the person in the promotional video is, the more the public's attention can be focused on the medical knowledge presented in the video, thereby achieving the purpose of promoting medical-related knowledge.

[0092] In the medical field, the facial expressions of celebrities in videos promoting medical knowledge generated by a pre-set facial image fusion model often appear unnatural. The steps described above can make the facial expressions of celebrities in medical promotional videos more natural, which is beneficial for the dissemination of medical knowledge.

[0093] In the field of online consultations, specifically within hospital platforms, the above five steps result in a more accurate facial expression when the face of user B, who is being used to consult with patient C, is replaced with that of user A. This significantly reduces patient C's anxiety during the online consultation, making them feel more comfortable and at ease.

[0094] like Figure 2 The diagram shown is a schematic diagram of a face image fusion device according to an embodiment of the present invention.

[0095] The face image fusion device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the face image fusion device 100 may include a generation module 110, a recognition module 120, a calculation module 130, and a correction module 140. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0096] In this embodiment, the functions of each module / unit are as follows:

[0097] The generation module 110 is used to fuse the facial features of the source face image into the target face image using a preset face image fusion model to obtain an initial face image;

[0098] The recognition module 120 is used to perform expression recognition on the initial face image and the target face image using the fully connected layer of the expression recognition model to obtain a first set of vectors;

[0099] The recognition module 120 is further configured to perform expression recognition on the initial face image and the target face image using the classification function of the expression recognition model to obtain a second set of vectors;

[0100] The calculation module 130 is used to calculate the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and to calculate the second cosine similarity between the initial face image and the target face image based on the second set of vectors.

[0101] The correction module 140 is used to calculate the expression supervision loss value of the initial face image and the target face image according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity, and call the face image fusion model to correct the initial face image according to the expression supervision loss value to obtain a face fusion image.

[0102] In one embodiment, a preset face image fusion model is used to fuse facial features of a source face image into a target face image to obtain an initial face image, including:

[0103] The encoder of the face image fusion model is used to extract the facial features of the target face image, the identity extraction module of the face image fusion model is used to extract the identity information of the source face image, and the identity injection module is used to embed the identity information of the source face image into the facial features of the target face image.

[0104] The initial face image is generated by performing bilinear interpolation and convolution operations on the facial features of the target face image, which contains the identity information of the source face image.

[0105] In one embodiment, the fully connected layer of an expression recognition model is used to perform expression recognition on the initial face image and the target face image to obtain a first set of vectors, including:

[0106] The fully connected layer of the VGG-16 network of the expression recognition model is used to perform expression recognition on the initial face image and the target face image;

[0107] The vectors of the first preset dimension of the identified initial face image and the target face image are respectively used as the first vector and the second vector, and the first vector and the second vector are used as the first set of vectors.

[0108] In one embodiment, the expression recognition function of the expression recognition model is used to recognize the expressions of the initial face image and the target face image to obtain a second set of vectors, including:

[0109] The expression recognition model's classification function is used to perform expression recognition on the initial face image and the target face image;

[0110] The vectors of the second preset dimension of the identified initial face image and the target face image are respectively used as the third vector and the fourth vector, and the third vector and the fourth vector are used as the second set of vectors.

[0111] In one embodiment, calculating a first cosine similarity between the initial face image and the target face image based on the first set of vectors, and calculating a second cosine similarity between the initial face image and the target face image based on the second set of vectors, includes:

[0112] The first cosine similarity is obtained by calculating the cosine similarity between the first vector and the second vector, and the second cosine similarity is obtained by calculating the cosine similarity between the third vector and the fourth vector.

[0113] In one embodiment, calculating the expression supervision loss value of the initial face image and the target face image according to the preset loss value calculation formula and the first cosine similarity includes:

[0114] The first expression supervision loss value, Loss1, is calculated using the preset first loss value calculation formula:

[0115] Loss1 = 1 – Fcos,

[0116] Where Loss1 represents the high-dimensional latent feature loss of the expressions of the initial face image and the target face image, and Fcos represents the first cosine similarity.

[0117] In one embodiment, calculating the expression supervision loss value of the initial face image and the target face image according to a preset loss value calculation formula and the second cosine similarity includes:

[0118] The second expression supervision loss value, Loss2, is calculated using a pre-defined formula:

[0119] Loss2 = 1 – Scos,

[0120] Where Loss1 represents the expression classification loss between the initial face image and the target face image, and Scos represents the second cosine similarity.

[0121] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a method for face image fusion according to an embodiment of the present invention.

[0122] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected via a system bus. The memory 11 stores a face image fusion program 10, which can be executed by the processor 12. Figure 3 Only the electronic device 1, which includes components 11-13 and the face image fusion program 10, is shown. Those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0123] The memory 11 includes RAM and at least one type of readable storage medium. The RAM provides a cache for the operation of the electronic device 1; the readable storage medium can be a non-volatile storage medium such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the readable storage medium can be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium can also be an external storage device of the electronic device 1, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. In this embodiment, the readable storage medium of the memory 11 is typically used to store the operating system and various application software installed on the electronic device 1, such as storing the code of the face image fusion program 10 in one embodiment of the present invention. Furthermore, the memory 11 can also be used to temporarily store various types of data that have been output or will be output.

[0124] In some embodiments, processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 12 is typically used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run program code stored in the memory 11 or process data, such as running a face image fusion program 10.

[0125] The network interface 13 may include a wireless network interface or a wired network interface, which is used to establish a communication connection between the electronic device 1 and the terminal (not shown in the figure).

[0126] Optionally, the electronic device 1 may further include a user interface, which may include a display, an input unit such as a keyboard, and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0127] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0128] The face image fusion program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 12, can achieve the following:

[0129] The facial features of the source face image are fused into the target face image using a preset face image fusion model to obtain the initial face image;

[0130] The initial face image and the target face image are subjected to expression recognition using the fully connected layer of the expression recognition model to obtain the first set of vectors;

[0131] The initial face image and the target face image are subjected to expression recognition using the classification function of the expression recognition model to obtain a second set of vectors;

[0132] The first cosine similarity between the initial face image and the target face image is calculated based on the first set of vectors, and the second cosine similarity between the initial face image and the target face image is calculated based on the second set of vectors.

[0133] The expression supervision loss values ​​of the initial face image and the target face image are calculated according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity. The face image fusion model is then called to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

[0134] Specifically, the processor 12's implementation method for the aforementioned face image fusion program 10 can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0135] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-combustible. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0136] The computer-readable storage medium stores a face image fusion program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the various embodiments of the face image fusion method described above, and will not be repeated here.

[0137] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0138] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0140] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0141] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0142] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device using software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for face image fusion, characterized in that, The method includes: The method involves fusing facial features from a source face image to a target face image using a pre-defined face image fusion model to obtain an initial face image. This includes: extracting facial features from the target face image using the encoder of the face image fusion model; extracting identity information from the source face image using the identity extraction module of the face image fusion model; embedding the identity information from the source face image into the facial features of the target face image using an identity injection module; and performing bilinear interpolation and convolution operations on the facial features of the target face image embedded with the identity information from the source face image to generate the initial face image. The initial face image and the target face image are subjected to expression recognition using the fully connected layer of the expression recognition model to obtain the first set of vectors; The expression recognition model is used to perform expression recognition on the initial face image and the target face image to obtain a second set of vectors. This includes: performing expression recognition on the initial face image and the target face image using the classification function of the expression recognition model; using the recognized vectors of the second preset dimension of the initial face image and the target face image as the third vector and the fourth vector, respectively, and using the third vector and the fourth vector as the second set of vectors. The calculation of the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and the calculation of the second cosine similarity between the initial face image and the target face image based on the second set of vectors, includes: calculating the cosine similarity between the first set of vectors and the second set of vectors to obtain the first cosine similarity, and calculating the cosine similarity between the third vector and the fourth vector to obtain the second cosine similarity; The expression supervision loss values ​​of the initial face image and the target face image are calculated according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity. The face image fusion is then called to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

2. The face image fusion method as described in claim 1, characterized in that, The expression recognition model utilizes a fully connected layer to perform expression recognition on the initial face image and the target face image, obtaining a first set of vectors, including: The fully connected layer of the VGG-16 network of the expression recognition model is used to perform expression recognition on the initial face image and the target face image; The vectors of the first preset dimension of the identified initial face image and the target face image are respectively used as the first vector and the second vector, and the first vector and the second vector are used as the first set of vectors.

3. The face image fusion method as described in claim 1, characterized in that, The expression supervision loss value of the initial face image and the target face image is calculated according to the preset loss value calculation formula and the first cosine similarity, including: The first expression supervision loss value, Loss1, is calculated using the preset first loss value calculation formula: Loss1 = 1 – Fcos, Where Loss1 represents the high-dimensional latent feature loss of the expressions of the initial face image and the target face image, and Fcos represents the first cosine similarity.

4. The face image fusion method as described in claim 1, characterized in that, The expression supervision loss values ​​of the initial face image and the target face image are calculated according to the preset loss value calculation formula and the second cosine similarity, including: The second expression supervision loss value, Loss2, is calculated using a pre-defined formula: Loss2 = 1 –Scos, Where Loss2 represents the expression classification loss between the initial face image and the target face image, and Scos represents the second cosine similarity.

5. A face image fusion apparatus for implementing the face image fusion method as described in any one of claims 1 to 4, characterized in that, The device includes: The generation module is used to fuse the facial features of the source face image into the target face image using a preset face image fusion model to obtain an initial face image; The recognition module is used to perform expression recognition on the initial face image and the target face image using the fully connected layer of the expression recognition model to obtain a first set of vectors; The recognition module is further configured to perform expression recognition on the initial face image and the target face image using the classification function of the expression recognition model to obtain a second set of vectors; The calculation module is used to calculate the first cosine similarity between the initial face image and the target face image based on the first set of vectors, and to calculate the second cosine similarity between the initial face image and the target face image based on the second set of vectors. The correction module is used to calculate the expression supervision loss value of the initial face image and the target face image according to the preset loss value calculation formula and the first cosine similarity and the second cosine similarity, and call the face image fusion model to correct the initial face image according to the expression supervision loss value to obtain the face fusion image.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a face image fusion program that can be executed by the at least one processor, the face image fusion program being executed by the at least one processor to enable the at least one processor to perform the face image fusion method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a face image fusion program, which can be executed by one or more processors to implement the face image fusion method as described in any one of claims 1 to 4.

Images