Digital face image replacement method, device, equipment and storage medium

Abstract

The present application relates to artificial intelligence technology, and discloses a digital face image replacement method, comprising: performing face coding processing on a face video to be replaced to obtain a face representation to be replaced; injecting an identity feature vector of a new face extracted from a new face image into the face representation to be replaced to obtain a face representation after identity replacement; performing feature merging processing on the face representation after identity replacement and a multi-channel speech representation map obtained from the face video to be replaced to obtain a merged feature map; and performing decoding processing on the merged feature map. The present application also relates to blockchain technology, and the face video to be replaced is stored in a blockchain. The present application can solve the problems in the prior art, such as that the digital face generation technology mainly focuses on speech-driven digital human speaker face generation tasks, and lacks a feasible scheme for quickly generating the image of a face and changing the boundary.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for digital face image replacement. Background Technology

[0002] Voice-driven face generation is a widely discussed research topic in artificial intelligence and digital media. It's a key technology involved in the practical application of virtual digital humans and other technologies, possessing significant practical and economic value. For example, in face-to-face customer service, human customer service representatives often provide services according to fixed scripts. This process can be replaced by a voice-driven, realistic digital face generation system using artificial intelligence. Such a system can create an extremely realistic conversational image for customers, not only improving service quality but also significantly freeing up human resources.

[0003] Existing voice-driven digital face generation technologies are relatively limited in functionality. While they can generate highly synchronized voice and lip-synced target face videos given any voice and target face video, to present a satisfactory digital image to clients, this process requires recording a live-action template video of the digital avatar (i.e., the target face video). Recording video data to obtain visually appealing facial dynamics consumes significant manpower, time, and effort. If a change in appearance is needed, it exacerbates time and communication costs, failing to effectively address changing requirements.

[0004] The process of generating a 2D or 3D digital face from an input voice signal is complex and cross-modal. Current digital face generation technology mainly focuses on voice-driven digital human speaking face generation tasks, that is, generating a target face video with highly synchronized speech and lip movements given any voice and any target face video. If you want to change the face image, you need to re-record a real-life template video of the person, which consumes a lot of manpower, time and energy. At present, there is a lack of feasible solutions for quickly changing the image of the generated face. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for digital face image replacement. Its main purpose is to solve the problems in the existing technology, which mainly focuses on voice-driven digital human speaking face generation tasks and lacks a feasible solution for generating face images quickly and with boundary changes.

[0006] Firstly, to achieve the above objectives, the present invention provides a digital face image replacement method, the method comprising:

[0007] The video of the face to be replaced is subjected to face encoding processing to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and the identity feature vector of the face to be replaced corresponding to the face composition information;

[0008] The identity feature vector of the new face extracted from the pre-acquired new face image is injected into the face representation to be replaced to obtain the face representation after identity replacement;

[0009] The facial representation after identity replacement is combined with the multi-channel speech representation map obtained from the video of the face to be replaced to obtain a merged feature map.

[0010] The merged feature map is decoded to generate a replacement face video with new facial identity features while retaining the image background information in the video of the face to be replaced.

[0011] Secondly, in order to solve the above problems, the present invention also provides a digital face image changing device, the device comprising:

[0012] The face encoding module is used to perform face encoding processing on the video of the face to be replaced to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and the identity feature vector of the face to be replaced corresponding to the face composition information;

[0013] The identity feature injection module is used to inject the identity feature vector of the new face extracted from the pre-acquired new face image into the face representation to be replaced, so as to obtain the face representation after identity replacement.

[0014] The feature map stitching module performs feature merging processing on the face representation after identity replacement and the multi-channel speech representation map obtained from the video of the face to be replaced, to obtain the merged feature map;

[0015] The decoding module is used to decode the merged feature map to generate a replacement face video with the identity features of the new face and while maintaining the image background information in the video of the face to be replaced.

[0016] Thirdly, in order to solve the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0017] At least one processor; and,

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the steps of the digital face image replacement method as described above.

[0020] Fourthly, in order to solve the above problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the digital face image replacement method as described above.

[0021] The digital face image replacement method, apparatus, device, and storage medium proposed in this invention preprocess the video of the face to be replaced, aligning the face and voice in the video in a temporal sequence to obtain the target face video. Then, by injecting the identity feature vector extracted from the new face image into the face representation to be replaced extracted from the target face video, the identity features in the face representation to be replaced are replaced with the identity features of the new face image. Next, the identity-replaced face representation is merged with the multi-channel speech representation map obtained from the target face video to obtain a merged feature map. After decoding, the image retains the identity features of the face to be replaced from the target face video. This invention replaces the background information of digital faces in video images, thus filling the current gap in the lack of a feasible solution for quickly and conveniently changing digital facial images. The end-to-end face replacement method of this invention simplifies the complex technical process pipeline in engineering practice and improves the operational efficiency of deployment on servers. While maintaining the accuracy of mouth shape generation for digital faces, it endows them with the ability to customize facial images, better responding to ever-changing business needs and improving the reliability of the system in practical applications. The convenient image replacement can reduce the expensive manual time cost currently required to replace 2D digital human images, helping customers obtain satisfactory service with lower costs and higher efficiency. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a digital face replacement method according to an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a digital face image changing device according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the internal structure of an electronic device that implements a digital face image replacement method according to an embodiment of the present invention.

[0025] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0028] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0029] This invention provides a method for digital face replacement. (See reference...) Figure 1 The diagram shown is a flowchart illustrating a digital face replacement method according to an embodiment of the present invention. This method can be executed by a device, which can be implemented in software and / or hardware.

[0030] In this embodiment, the digital face image replacement method includes:

[0031] Step S110: Perform face encoding processing on the video of the face to be replaced to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and the identity feature vector of the face to be replaced corresponding to the face composition information.

[0032] Specifically, a face encoder can be used to process the face in the video of the face to be replaced, resulting in a representation of the face to be replaced. This representation includes facial composition information, the identity feature vector of the face to be replaced corresponding to the facial composition information, and image background information. The facial composition information includes the facial features, such as the facial features themselves. The identity feature vector of the face to be replaced refers to the detailed facial features corresponding to the facial composition, such as wrinkles and lip lines. By changing the identity feature vector of the face to be replaced, the facial image can be altered.

[0033] As an optional embodiment of the present invention, the video of the face to be replaced is stored in a blockchain. Before performing face encoding processing on the video of the face to be replaced to obtain the representation of the face to be replaced, the method further includes:

[0034] For each frame of the video with the face to be replaced, crop out an image containing the face, centered on the face.

[0035] The images containing faces are arranged according to the cropping time of each frame to obtain the sequence of faces to be replaced, and the speech waveform data is extracted from the speech signal in the video of the face to be replaced and converted into Mel spectrum.

[0036] Align the sequence of face images to be replaced with the Mel spectrum in time.

[0037] Specifically, the face to be replaced in this invention refers to the original digital face video, such as a 2D or 3D face video. The recorded face video and its accompanying audio signal are preprocessed to align the face image in each frame of the face video with the Mel spectrum in the audio waveform data of the audio signal in terms of timing. That is, the face in the face video is aligned with the audio in the audio signal in terms of timing, achieving synchronization between the lip movements and audio in the face video, ensuring that the facial image (lip movements and facial expressions) and sound in the original digital face video are consistent. The audio signal can be the audio recorded during the face video's recording or a dubbing added later based on the face video. Videos are composed of individual frames. To facilitate the alignment of different frames with audio, each frame of the video containing the face to be replaced needs to be cropped. Since the face occupies a very small percentage of the overall image, cropping is necessary to ensure the successful injection of the new face's feature vector into the face representation. To make the face clearer, the cropped image containing the face can be adjusted to a preset resolution, and then each frame containing the face can be scaled to the preset resolution, such as 96x96. During the cropping process, the image containing the face can be cropped using the two symmetrical points at the outermost edge of the recognized face image's horizontal axis and the symmetrical point at the top of its vertical axis. The image containing the face can be rectangular, circular, or diamond-shaped; no particular limitation is made here. The images containing faces are sorted according to the cropping time to obtain the sequence of faces to be replaced. For the speech signal in the video of the face to be replaced, speech waveform data needs to be extracted, then converted into Mel spectrograms. The face image sequence is then aligned temporally with the Mel spectrogram. For example, if the video of the face to be replaced is 25 frames per second, with each frame corresponding to a 40ms time window, the Mel spectrogram of a 200ms time window centered on a specific frame is considered to be aligned with that frame. The video presentation is a fast playback of a sequence of images and the corresponding synchronized playback of sound. In previous related studies, approximately 200ms of speech information is sufficient to accurately predict lip movements at intermediate moments, so a window length of 00ms is often used. Of course, other suitable window lengths can be used instead, and no particular limitation is made here.

[0038] As an optional embodiment of the present invention, the video of the face to be replaced is subjected to face encoding processing to obtain the representation of the face to be replaced, including:

[0039] Obtain a preset number of consecutive frame images and an equal number of reference frame images from the frame images of the video of the face to be replaced, and obtain a set of consecutive frame images and a set of reference frame images respectively;

[0040] The images in the continuous frame image set are masked according to a preset area to obtain the masked continuous frame image set.

[0041] The masked continuous frame image set and the reference frame image set are input into the face encoder for face encoding processing to obtain a face feature map set, and the face feature map set is used as the face representation to be replaced; wherein, the face feature map set includes the preset number of face feature maps.

[0042] Specifically, since a video is composed of individual frames, not every frame clearly shows a face. Therefore, it's necessary to extract a predetermined number of frames from the video to be replaced. This means obtaining a predetermined number of consecutive frames from the video. To increase selectivity and accuracy when encoding the facial representation in the frames, an equal number of reference frames are also obtained. The images in the consecutive frames are then masked according to a predetermined region; for example, the area below the mouth can be selected as the masking region. This results in a masked set of consecutive frames. The masked set of consecutive frames and the reference frame set are then input into the face encoder. Face encoding is performed to obtain a face feature map set, which is then used as the representation of the face to be replaced. The face feature map set includes a predetermined number of face feature maps. For example, five consecutive frame images and five reference frame images are obtained from the video of the face to be replaced. The lower part of the face below the mouth in the five consecutive frame images is masked, resulting in an image set including five masked consecutive frame images. The face encoder receives the five reference frame image set and the five masked consecutive frame image set, processes them, and outputs a face feature map set consisting of face feature maps of a predetermined dimension. For example, five face feature maps with a dimension of 512*12*12 are output, thus obtaining the face feature map set, which is the representation of the face to be replaced. The frame images in the reference frame image set are arbitrarily extracted, while the consecutive frame images are continuously extracted.

[0043] Step S120: Inject the identity feature vector of the new face extracted from the pre-acquired new face image into the face representation to be replaced, and obtain the face representation after identity replacement.

[0044] Specifically, the identity features in the video of the face to be replaced are replaced with the identity features in the new face image. Therefore, it is necessary to extract the identity feature vector of the new face from the new face image, and then inject the extracted identity feature vector of the new face into the representation of the face to be replaced, so as to obtain the face representation after identity replacement. The identity feature vector of the new face in the new face image can be extracted by the face recognition model in the existing technology.

[0045] As an optional embodiment of the present invention, the identity feature vector of the new face extracted from the pre-acquired new face image is injected into the face representation to be replaced, and the face representation after identity replacement includes:

[0046] Extract the identity feature information of the new face from the new face image; wherein, the identity feature information of the new face includes face composition information and the identity feature vector of the new face corresponding to the face composition information;

[0047] Based on the same facial composition information in the new face image and the face representation to be replaced, the identity feature vector of the new face is used to replace the identity feature vector of the face representation to be replaced, thus obtaining the face representation after identity replacement.

[0048] Specifically, the identity feature information of the new face is extracted from the new face image using existing face recognition technology. The identity feature information of the new face includes face composition information and the identity feature vector of the new face corresponding to the face composition information. Then, based on the same face composition information in the new face image and the face representation to be replaced, the identity feature vector of the new face extracted from the new face image is used to replace the identity feature vector of the face to be replaced in the face representation to be replaced, thereby obtaining the face representation after identity replacement.

[0049] As an optional embodiment of the present invention, based on the same facial composition information in the new face image and the face representation to be replaced, the identity feature vector of the new face is used to replace the identity feature vector of the face to be replaced in the face representation to be replaced, resulting in a face representation with the replaced identity, including:

[0050] The identity feature vector of the new face and each face feature map in the face representation to be replaced are used as input pairs to obtain the input pair set; wherein, the face representation to be replaced includes a preset number of face feature maps; each face feature map corresponds to the identity feature vector of the face to be replaced; the dimension of the identity feature vector of the face to be replaced is equal to the dimension of the identity feature vector of the new face;

[0051] The input pairs in the input pair set are successively input into the pre-defined identity injection model, wherein the identity injection model includes a fully connected layer and an image style transfer layer;

[0052] The fully connected layer predicts a computation channel with the same dimension as the identity feature vector of the new face in the input pair, and calculates the mean and variance of the identity feature vector of the new face and the mean and variance of the identity feature vector of the face to be replaced in each computation channel.

[0053] Based on the mean and variance of the identity feature vectors of the new face and the identity feature vectors of the face to be replaced, respectively, obtained from each computation channel of the fully connected layer, style transfer calculations are performed on each computation channel through the image style transfer layer, and the style transfer calculation results of all computation channels are statistically analyzed.

[0054] Based on the style transfer calculation results of all computation channels, the face feature maps after identity replacement of the corresponding output pairs are obtained.

[0055] Collect all facial feature maps after identity replacement to obtain a set of facial feature maps after identity replacement, and use the set of facial feature maps after identity replacement as the facial representation after identity replacement.

[0056] Specifically, the identity feature vector of the new face is paired one by one with each face feature map in the face representation to be replaced (i.e., the face feature map set), thus obtaining an input pair set. For example, if the face feature map set includes 5 face feature maps, then the input pair set includes 5 input pairs, each consisting of the identity feature vector of the new face and one of the 5 face feature maps. Then, the input pairs in the input pair set are sequentially input into a preset identity injection model. The identity injection model performs identity injection processing on each input pair, resulting in 5 face feature maps with replaced identities. The identity injection model includes fully connected layers and image style transfer layers. The fully connected layers predict computational channels with the same dimension as the identity feature vector of the new face in the input pair, and calculate the mean and variance of the identity feature vector of the new face and the identity feature vector of the face to be replaced within each computational channel. For example, if the dimension of the input face feature map is 512*12*12 and the dimension of the identity feature vector of the new face is 512, then the fully connected layer... The layer predicts 512 computation channels based on the input identity feature vector y, with each channel corresponding to the first dimension of the face feature map (512 dimensions corresponding to 512 channels). Then, within each computation channel, the mean and variance of the identity feature vector of the new face and the identity feature vector of the face to be replaced are calculated. Next, based on the mean and variance of the identity feature vectors of the new face and the face to be replaced, obtained from the fully connected layer for each computation channel, style transfer is performed on each computation channel through an image style transfer layer. The style transfer calculation results for all computation channels are then statistically analyzed. Based on the style transfer calculation results for all computation channels, the face feature map after identity replacement for the corresponding output pair is obtained. All face feature maps after identity replacement are statistically analyzed to obtain a set of face feature maps after identity replacement, which is used as the face representation after identity replacement. The formula for style transfer calculation is as follows:

[0057]

[0058] Among them, AdaIN(f tgt v src ) represents the output value of the image style transfer layer, f tgt and v src μ represents the facial feature map and the identity features of the new face, respectively. src and σ src Let μ(f) represent the mean and variance obtained by predicting the input from the fully connected layer, respectively. tgt ) and σ(f tgtThe values ​​() represent the mean and variance calculated for each channel. During this process, the identity information of the original video frame in the face feature map is gradually replaced by the identity information of the source face, making the identity of the generated frame image output by the decoder closely resemble the source face. Since identity features are strongly correlated with facial texture, the introduced identity information helps the model better preserve facial details.

[0059] Step S130: Perform feature merging processing on the face representation after identity replacement and the multi-channel speech representation map obtained from the face video to be replaced to obtain the merged feature map.

[0060] Specifically, a multi-channel speech representation map is obtained from the video of the face to be replaced. Since the video of the face to be replaced has been preprocessed and the sound corresponds to the video frame, it is only necessary to stitch the face representation after identity replacement with the multi-channel speech representation map to obtain the feature map to be decoded. After decoding, the video of the face after identity replacement can be obtained.

[0061] As an optional embodiment of the present invention, the method for obtaining a multi-channel speech representation map includes:

[0062] The speech feature vector set is obtained by extracting speech feature vectors for each frame of the Mel spectrum in the video of the face to be replaced using a convolutional neural network.

[0063] The speech feature vectors in the speech feature vector set are sequentially subjected to batch normalization and piecewise linear function processing to obtain a multi-channel speech representation map.

[0064] Specifically, a speech feature extraction model can be pre-trained, and then the Mel spectrum obtained from the video of the face to be replaced can be used as input to this model to directly output a speech feature map of a preset dimension, for example, a speech feature map with a dimension of 256*12*12. The speech feature extraction model can include a speech encoder composed of multiple residual convolutional blocks. This encoder extracts a 512-dimensional speech feature vector from each frame of Mel spectrum, and then processes it through several layers of residual convolutional blocks to output the speech feature map. The residual convolutional blocks consist of convolutional neural network layers, batch normalization layers, and piecewise linear function layers.

[0065] Step S140: Decode the merged feature map to generate a replacement face video with the identity features of the new face and retain the image background information in the video of the face to be replaced.

[0066] Specifically, since the merged feature map is composed of a face representation with a new identity and a multi-channel speech representation map, and both the face representation and the multi-channel speech representation map are encoded feature maps, decoding is required. After decoding, a replacement face video with the identity features of the new face and retaining the image background information of the video of the face to be replaced can be generated. For example, a speech feature map with a dimension of 256*12*12 and a face feature map with a dimension of 512*12*12 are concatenated along the first dimension to obtain a concatenated feature map with a dimension of 768*12*12. The concatenated feature map is then fed into a preset decoder, and finally, the generated replacement face video is output. The preset decoder can be composed of multiple residual convolutional blocks.

[0067] As an optional embodiment of the present invention, after decoding the merged feature map to generate a replacement face video with new face identity features while maintaining the image background information in the target face video, the method further includes:

[0068] Take any face image from the replacement face video and use it as the generated frame image;

[0069] Extract the identity feature vector from the generated frame image to obtain the identity feature vector of the generated frame;

[0070] Calculate the similarity value between the identity feature vector of the generated frame and the new face identity feature vector extracted from the new face image, and use it as the face identity judgment value;

[0071] When the facial identity judgment value is greater than the preset facial identity threshold, a message indicating that the replacement of the face video was successful is generated.

[0072] Specifically, to ensure that the identity features in the generated replacement face video are the same as those of the source face, an arbitrary face image is obtained from the replacement face video and used as the generated frame image. Then, the identity feature vector is extracted from the generated frame image to obtain the identity feature vector of the generated frame. Next, the similarity value between the identity feature vector of the generated frame and the face identity feature vector extracted from the new face image is calculated as the face identity judgment value. Finally, when the face identity judgment value is greater than the preset face identity threshold, the information that the replacement face video was successfully generated is generated, thereby ensuring that the identity features in the generated replacement face video tend to be similar to the source face, achieving the purpose of replacing the digital face image.

[0073] To further evaluate the lip movements, voice, and visual effects of the generated replacement face video, a pre-trained lip movement synchronization discriminator and a visual discriminator can be used to judge the synchronization of the lip movements and voice and the visual realism of the replacement face video, respectively.

[0074] like Figure 2The diagram shown is a functional block diagram of a digital face image changing device according to an embodiment of the present invention.

[0075] The digital face image changing device 200 of this invention can be installed in an electronic device. Depending on the functions implemented, the digital face image changing device may include a face encoding module 210, an identity feature injection module 220, a feature map stitching module 230, and a decoding module 240. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0076] In this embodiment, the functions of each module / unit are as follows:

[0077] The face encoding module 210 is used to perform face encoding processing on the video of the face to be replaced to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and the identity feature vector of the face to be replaced corresponding to the face composition information.

[0078] Specifically, a face encoder can be used to process the face in the video of the face to be replaced, resulting in a representation of the face to be replaced. This representation includes facial composition information, the identity feature vector of the face to be replaced corresponding to the facial composition information, and image background information. The facial composition information includes the facial features, such as the facial features themselves. The identity feature vector of the face to be replaced refers to the detailed facial features corresponding to the facial composition, such as wrinkles and lip lines. By changing the identity feature vector of the face to be replaced, the facial image can be altered.

[0079] As an optional embodiment of the present invention, the digital face image changing device 200 further includes a preprocessing module and an audio alignment module (not shown in the figure).

[0080] The preprocessing module is used to arrange the images containing faces according to the cropping time of each frame to obtain the sequence of faces to be replaced, and to extract speech waveform data from the speech signal in the video of faces to be replaced and convert it into Mel spectrum.

[0081] The audio alignment module is used to align the sequence of face images to be replaced with the Mel spectrum in time.

[0082] Specifically, the face to be replaced in this invention refers to the original digital face video, such as a 2D or 3D face video. A preprocessing module preprocesses the recorded face video and its audio signal. An audio alignment module then aligns each frame of the face image in the face video with the Mel spectrum in the audio waveform data of the audio signal in terms of timing. That is, the face in the face video is aligned with the audio in the audio signal in terms of timing, achieving synchronization between the lip movements and the audio in the face video, ensuring that the facial image (lip movements and facial expressions) and sound in the original digital face video are consistent. The audio signal can be the audio recorded during the face video's recording or a dubbing added later based on the face video. Videos are composed of individual frames. To facilitate the alignment of different frames with audio, each frame of the video containing the face to be replaced needs to be cropped. Since the face occupies a very small percentage of the overall image, cropping is necessary to ensure the successful injection of the new face's feature vector into the face representation. To make the face clearer, the cropped image containing the face can be adjusted to a preset resolution, and then each frame containing the face can be scaled to the preset resolution, such as 96x96. During the cropping process, the image containing the face can be cropped using the two symmetrical points at the outermost edge of the recognized face image's horizontal axis and the symmetrical point at the top of its vertical axis. The image containing the face can be rectangular, circular, or diamond-shaped; no particular limitation is made here. The images containing faces are sorted according to the cropping time to obtain the sequence of faces to be replaced. For the speech signal in the video of the face to be replaced, speech waveform data needs to be extracted, then converted into Mel spectrograms. The face image sequence is then aligned temporally with the Mel spectrogram. For example, if the video of the face to be replaced is 25 frames per second, with each frame corresponding to a 40ms time window, the Mel spectrogram of a 200ms time window centered on a specific frame is considered to be aligned with that frame. The video presentation is a fast playback of a sequence of images and the corresponding synchronized playback of sound. In previous related studies, approximately 200ms of speech information is sufficient to accurately predict lip movements at intermediate moments, so a window length of 00ms is often used. Of course, other suitable window lengths can be used instead, and no particular limitation is made here.

[0083] As an optional embodiment of the present invention, the face encoding module 210 further includes: an image set acquisition unit, a occlusion processing unit, and an encoding unit (not shown in the figure).

[0084] The image set acquisition unit is used to acquire a preset number of consecutive frame images and an equal number of reference frame images from the frame images of the video of the face to be replaced, so as to obtain a continuous frame image set and a reference frame image set respectively.

[0085] The masking processing unit is used to mask the images in the continuous frame image set according to a preset area to obtain the masked continuous frame image set.

[0086] The encoding unit is used to input the masked continuous frame image set and the reference frame image set into the face encoder for face encoding processing to obtain a face feature map set, and use the face feature map set as the face representation to be replaced; wherein, the face feature map set includes the preset number of face feature maps.

[0087] Specifically, since a video is composed of individual frames, not every frame can clearly represent a face. Therefore, the image set acquisition unit needs to extract a preset number of frame images from the video to be replaced. To increase selectivity and accuracy when encoding the face representation in the frame images, an equal number of reference frame images are also acquired. These reference frame images are then masked according to a preset area by the masking unit. For example, the area below the mouth of the face in the consecutive frame image set can be selected as the masking area, resulting in a masked consecutive frame image set. Finally, the encoding unit combines the masked consecutive frame image set with the reference frames. The image set is input into the face encoder for face encoding processing to obtain a face feature map set, which serves as the representation of the face to be replaced. The face feature map set includes a preset number of face feature maps. For example, five consecutive frame images and five reference frame images are obtained from the video of the face to be replaced. The lower part of the face below the mouth in the five consecutive frame images is masked, resulting in an image set including five masked consecutive frame images. The face encoder receives the five reference frame image set and the five masked consecutive frame image set, processes them, and outputs a face feature map set composed of face feature maps of a preset dimension. For example, five face feature maps with a dimension of 512*12*12 are output, thus obtaining the face feature map set, which is the representation of the face to be replaced. The frame images in the reference frame image set are arbitrarily extracted, while the consecutive frame images are continuously extracted.

[0088] The identity feature injection module 220 is used to inject the identity feature vector of the new face extracted from the pre-acquired new face image into the face representation to be replaced, so as to obtain the face representation after identity replacement.

[0089] Specifically, the identity features in the video of the face to be replaced are replaced with the identity features in the new face image. Therefore, it is necessary to extract the identity feature vector of the new face from the new face image, and then inject the extracted identity feature vector of the new face into the representation of the face to be replaced, so as to obtain the face representation after identity replacement. The identity feature vector of the new face in the new face image can be extracted by the face recognition model in the existing technology.

[0090] As an optional embodiment of the present invention, the identity feature injection module 220 further includes: a feature extraction unit and a feature injection unit (not shown in the figure).

[0091] The feature extraction unit is used to extract the identity feature information of the new face from the new face image; wherein, the identity feature information of the new face includes face composition information and the identity feature vector of the new face corresponding to the face composition information;

[0092] The feature injection unit is used to replace the identity feature vector of the face to be replaced in the face representation of the face to be replaced with the identity feature vector of the new face based on the same face composition information in the new face image and the face representation of the face to be replaced, so as to obtain the face representation after identity replacement.

[0093] Specifically, the feature extraction unit uses existing face recognition technology to extract the identity feature information of the new face from the new face image. The identity feature information of the new face includes face composition information and the identity feature vector of the new face corresponding to the face composition information. Then, the feature injection unit replaces the identity feature vector of the new face extracted from the new face image with the identity feature vector of the face to be replaced in the face representation based on the same face composition information in the new face image and the face representation to be replaced, thereby obtaining the face representation after identity replacement.

[0094] As an optional embodiment of the present invention, the feature injection unit further includes: an input pair matching subunit, an input subunit, a calculation subunit, a style transfer subunit, an output subunit, and a statistical subunit (not shown in the figure).

[0095] The input pair matching subunit is used to take the identity feature vector of the new face and each face feature map in the face representation to be replaced as input pairs to obtain the input pair set; wherein, the face representation to be replaced includes a preset number of face feature maps; the face feature map corresponds to the identity feature vector of the face to be replaced; the dimension of the identity feature vector of the face to be replaced is equal to the dimension of the identity feature vector of the new face;

[0096] The input subunit is used to inject a preset identity into the model by successively inputting the input pairs from the input pair set. The identity injection model includes a fully connected layer and an image style transfer layer.

[0097] The computational subunit is used to predict computational channels with the same dimension as the identity feature vector of the new face in the input pair through the fully connected layer, and to calculate the mean and variance of the identity feature vector of the new face and the mean and variance of the identity feature vector of the face to be replaced in each computational channel.

[0098] The style transfer subunit is used to perform style transfer calculations on each calculation channel through the image style transfer layer based on the mean and variance of the identity feature vector of the new face and the identity feature vector of the face to be replaced, which are the corresponding dimensions of the output of each calculation channel obtained by the fully connected layer, and to statistically analyze the style transfer calculation results of all calculation channels.

[0099] The output subunit is used to obtain the face feature map after identity replacement for the corresponding output pair based on the style transfer calculation results of all calculation channels.

[0100] The statistical subunit is used to count all facial feature maps after identity replacement, obtain a set of facial feature maps after identity replacement, and use the set of facial feature maps after identity replacement as the facial representation after identity replacement.

[0101] Specifically, the input pair matching subunit pairs the identity feature vector of the new face with each face feature map in the face representation to be replaced (i.e., the face feature map set) to obtain the input pair set. For example, if the face feature map set includes 5 face feature maps, then the input pair set includes 5 input pairs. These 5 input pairs are composed of the identity feature vector of the new face and the 5 face feature maps, respectively. Then, the input subunit sequentially inputs the input pairs in the input pair set into the preset identity injection model. The identity injection model performs identity injection processing on each input pair to obtain 5 face feature maps with replaced identities. The identity injection model includes a fully connected layer and an image style transfer layer. The fully connected layer predicts computational channels with the same dimension as the identity feature vector of the new face in the input pair. Within each computational channel, computational subunits calculate the mean and variance of the identity feature vector of the new face and the mean and variance of the identity feature vector of the face to be replaced. For example, if the dimension of the input face feature map is 512*12*12 and the dimension of the identity feature vector of the new face is 512, then the fully connected layer... The Layer predicts 512 computation channels based on the input identity feature vector y, with each channel corresponding to the first dimension of the face feature map (512 dimensions corresponding to 512 channels). Then, within each computation channel, the mean and variance of the identity feature vector of the new face and the identity feature vector of the face to be replaced are calculated. Next, based on the mean and variance of the identity feature vectors of the new face and the face to be replaced, obtained from the fully connected layer for each computation channel, style transfer calculations are performed on each computation channel using an image style transfer layer through a style transfer subunit. The style transfer calculation results for all computation channels are then statistically analyzed. Based on the style transfer calculation results for all computation channels, the face feature map after identity replacement for the corresponding output pair is obtained. All face feature maps after identity replacement are statistically analyzed to obtain a set of face feature maps after identity replacement, which is used as the face representation after identity replacement. The formula for style transfer calculation is as follows:

[0102]

[0103] Among them, AdaIN(f tgt v src ) represents the output value of the image style transfer layer, f tgt and v src μ represents the facial feature map and the identity features of the new face, respectively. src and σ src Let μ(f) represent the mean and variance obtained by predicting the input from the fully connected layer, respectively. tgt ) and σ(f tgtThe values ​​() represent the mean and variance calculated for each channel. During this process, the identity information of the original video frame in the face feature map is gradually replaced by the identity information of the source face, making the identity of the generated frame image output by the decoder closely resemble the source face. Since identity features are strongly correlated with facial texture, the introduced identity information helps the model better preserve facial details.

[0104] The feature map stitching module 230 is used to perform feature merging processing on the face representation after identity replacement and the multi-channel speech representation map obtained from the video of the face to be replaced, so as to obtain the merged feature map.

[0105] Specifically, a multi-channel speech representation map is obtained from the target face video. Since the sound corresponds to the image frame after the face video to be replaced has been preprocessed, it is only necessary to stitch the face representation after identity replacement with the multi-channel speech representation map to obtain the feature map to be decoded. After decoding, the face video after identity replacement can be obtained.

[0106] As an optional embodiment of the present invention, the method for obtaining a multi-channel speech representation map includes:

[0107] The speech feature vector set is obtained by extracting speech feature vectors from each frame of the Mel spectrum in the face-swap video using a convolutional neural network.

[0108] The speech feature vectors in the speech feature vector set are sequentially subjected to batch normalization and piecewise linear function processing to obtain a multi-channel speech representation map.

[0109] Specifically, a speech feature extraction model can be pre-trained, and then the Mel spectrum obtained from the video of the face to be replaced can be used as input to this model to directly output a speech feature map of a preset dimension, for example, a speech feature map with a dimension of 256*12*12. The speech feature extraction model can include a speech encoder composed of multiple residual convolutional blocks. This encoder extracts a 512-dimensional speech feature vector from each frame of Mel spectrum, and then processes it through several layers of residual convolutional blocks to output the speech feature map. The residual convolutional blocks consist of convolutional neural network layers, batch normalization layers, and piecewise linear function layers.

[0110] The decoding module 240 is used to decode the merged feature map to generate a replacement face video with the identity features of the new face and retain the image background information in the video of the face to be replaced.

[0111] Specifically, since the merged feature map is composed of a face representation with a new identity and a multi-channel speech representation map, and both the face representation and the multi-channel speech representation map are encoded feature maps, decoding is required. After decoding, a replacement face video with the identity features of the new face and retaining the image background information of the video of the face to be replaced can be generated. For example, a speech feature map with a dimension of 256*12*12 and a face feature map with a dimension of 512*12*12 are concatenated along the first dimension to obtain a concatenated feature map with a dimension of 768*12*12. The concatenated feature map is then fed into a preset decoder, and finally, the generated replacement face video is output. The preset decoder can be composed of multiple residual convolutional blocks.

[0112] As an optional embodiment of the present invention, the digital face image changing device 200 further includes: a frame image acquisition module, an identity feature vector extraction module, a similarity calculation module, and an information generation module (not shown in the figure).

[0113] The frame image acquisition module is used to acquire any face image from the replacement face video as the generated frame image;

[0114] The identity feature vector extraction module is used to extract the identity feature vector from the generated frame image to obtain the identity feature vector of the generated frame.

[0115] The similarity calculation module is used to calculate the similarity value between the identity feature vector of the generated frame and the identity feature vector of the new face extracted from the new face image, and use it as the face identity judgment value.

[0116] The information generation module is used to generate a message indicating that the face replacement video has been successfully completed when the face identity judgment value is greater than the preset face identity threshold.

[0117] Specifically, to ensure that the identity features in the generated replacement face video are the same as those of the source face, the following steps are taken: First, a frame image acquisition module arbitrarily acquires a face image from the replacement face video as the generated frame image. Then, an identity feature vector extraction module extracts the identity feature vector from the generated frame image to obtain the identity feature vector of the generated frame. Next, a similarity calculation module calculates the similarity value between the identity feature vector of the generated frame and the identity feature vector of the new face extracted from the new face image, using this value as the face identity judgment value. Finally, an information generation module generates a message indicating successful replacement of the face video when the face identity judgment value is greater than a preset face identity threshold. This ensures that the identity features in the generated replacement face video closely resemble the source face, achieving the goal of replacing the digital face image.

[0118] To further evaluate the lip movements, voice, and visual effects of the generated replacement face video, a pre-trained lip movement synchronization discriminator and a visual discriminator can be used to judge the synchronization of the lip movements and voice and the visual realism of the replacement face video, respectively.

[0119] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a digital face image replacement method according to an embodiment of the present invention.

[0120] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a digital face image changing program 12.

[0121] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a digital face image changing program, but also to temporarily store data that has been output or will be output.

[0122] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (such as digital face image changing programs) stored in the memory 11, and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0123] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0124] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0125] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0126] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0127] Optionally, the electronic device 1 may further include a client interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual client interface.

[0128] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0129] The digital face image changing program 12 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0130] The video of the face to be replaced is subjected to face encoding processing to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and identity feature vector of the face to be replaced corresponding to the face composition information;

[0131] The identity feature vector of the new face extracted from the pre-acquired new face image is injected into the face representation to be replaced, thus obtaining the face representation after identity replacement;

[0132] The facial representation after identity replacement is combined with the multi-channel speech representation map obtained from the video of the face to be replaced to obtain the merged feature map.

[0133] The merged feature map is decoded to generate a replacement face video with the new face's identity features while retaining the image background information of the video of the face to be replaced.

[0134] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here. It should be emphasized that, to further ensure the privacy and security of the aforementioned face video to be replaced, the face video to be replaced can also be stored in a blockchain node.

[0135] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0136] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0137] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0139] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0140] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0141] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0142] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for replacing digital facial images, applied to electronic devices, characterized in that, The method includes: For each frame of the video of the face to be replaced, an image containing the face is cropped with the face as the center; the images containing the face are arranged according to the cropping time of each frame to obtain the image sequence of the face to be replaced; and speech waveform data is extracted from the speech signal in the video of the face to be replaced and converted into Mel spectrum. Align the sequence of faces to be replaced with the Mel spectrum in time. The video of the face to be replaced is subjected to face encoding processing to obtain the face representation to be replaced; wherein, the face representation to be replaced includes image background information, face composition information and the identity feature vector of the face to be replaced corresponding to the face composition information; The identity feature vector of the new face extracted from the pre-acquired new face image is injected into the face representation to be replaced to obtain the face representation after identity replacement; The facial representation after identity replacement is combined with the multi-channel speech representation map obtained from the video of the face to be replaced to obtain a merged feature map. The merged feature map is decoded to generate a replacement face video with new facial identity features while retaining the image background information in the video of the face to be replaced.

2. The digital face image replacement method according to claim 1, characterized in that, The identity feature vector of the new face extracted from the pre-acquired new face image is injected into the face representation to be replaced, resulting in the face representation after identity replacement, including: Extract the identity feature information of the new face from the new face image; wherein, the identity feature information of the new face includes face composition information and the identity feature vector of the new face corresponding to the face composition information; Based on the same facial composition information in the new face image and the face representation to be replaced, the identity feature vector of the new face is used to replace the identity feature vector of the face representation to be replaced, thus obtaining the face representation after identity replacement.

3. The digital face image replacement method according to claim 2, characterized in that, The step of replacing the identity feature vector of the face to be replaced in the face representation based on the same facial composition information in the new face image and the face representation to be replaced, to obtain the face representation after identity replacement, includes: The identity feature vector of the new face and each face feature map in the face representation to be replaced are used as input pairs to obtain an input pair set; wherein, the face representation to be replaced includes a preset number of face feature maps; each face feature map corresponds to the identity feature vector of the face to be replaced; the dimension of the identity feature vector of the face to be replaced is equal to the dimension of the identity feature vector of the new face; The input pairs in the input pair set are successively input into the preset identity injection model, wherein the identity injection model includes a fully connected layer and an image style transfer layer; The fully connected layer predicts a computation channel with the same dimension as the identity feature vector of the new face in the input pair, and calculates the mean and variance of the identity feature vector of the new face and the mean and variance of the identity feature vector of the face to be replaced in each computation channel. Based on the mean and variance of the identity feature vector of the new face and the identity feature vector of the face to be replaced, respectively, obtained by each calculation channel of the fully connected layer, style transfer calculation is performed on each calculation channel through the image style transfer layer, and the style transfer calculation results of all calculation channels are statistically analyzed. Based on the style transfer calculation results of all the aforementioned calculation channels, the face feature maps of the corresponding output pairs after identity replacement are obtained. Collect all facial feature maps after identity replacement to obtain a set of facial feature maps after identity replacement, and use the set of facial feature maps after identity replacement as the facial representation after identity replacement.

4. The digital face image replacement method according to claim 1, characterized in that, The video of the face to be replaced is stored in the blockchain.

5. The digital face image replacement method according to claim 1, characterized in that, The process of performing face encoding on the video of the face to be replaced to obtain the representation of the face to be replaced includes: From the frame images of the video of the face to be replaced, obtain a preset number of consecutive frame images and an equal number of reference frame images to obtain a set of consecutive frame images and a set of reference frame images, respectively. The images in the continuous frame image set are masked according to a preset area to obtain a masked continuous frame image set. The masked continuous frame image set and the reference frame image set are input into the face encoder for face encoding processing to obtain a face feature map set, and the face feature map set is used as the face representation to be replaced; wherein, the face feature map set includes the preset number of face feature maps.

6. The digital face image replacement method according to claim 1, characterized in that, The method for obtaining the multi-channel speech representation map includes: The speech feature vector set is obtained by extracting speech feature vectors from each frame of the Mel spectrum in the video of the face to be replaced using a convolutional neural network. The speech feature vectors in the speech feature vector set are sequentially subjected to batch normalization and piecewise linear function processing to obtain a multi-channel speech representation map.

7. The digital face image replacement method according to claim 1, characterized in that, After decoding the merged feature map to generate a replacement face video with new facial identity features while retaining the image background information in the video of the face to be replaced, the process further includes: Take any face image from the replacement face video and use it as the generated frame image; Extract the identity feature vector from the generated frame image to obtain the identity feature vector of the generated frame; Calculate the similarity value between the identity feature vector of the generated frame and the identity feature vector of the new face, and use it as the face identity judgment value; When the facial identity judgment value is greater than the preset facial identity threshold, a message indicating that the replacement of the face video was successful is generated.

8. A digital face image changing device, characterized in that, The device includes: The preprocessing module is used to crop out an image containing the face centered on the face for each frame of the video to be replaced; arrange the images containing the face according to the cropping time of each frame to obtain a sequence of images of the face to be replaced; and extract speech waveform data from the speech signal in the video to be replaced and convert it into Mel spectrum. An audio alignment module is used to align the sequence of face images to be replaced with the Mel spectrum in time. A face encoding module is used to perform face encoding processing on the video of the face to be replaced to obtain a face representation of the face to be replaced; wherein, the face representation of the face to be replaced includes image background information, face composition information, and an identity feature vector of the face to be replaced corresponding to the face composition information. The identity feature injection module is used to inject the identity feature vector of the new face extracted from the pre-acquired new face image into the face representation to be replaced, so as to obtain the face representation after identity replacement. The feature map stitching module performs feature merging processing on the face representation after identity replacement and the multi-channel speech representation map obtained from the video of the face to be replaced, to obtain the merged feature map; The decoding module is used to decode the merged feature map to generate a replacement face video with the identity features of the new face and while maintaining the image background information in the video of the face to be replaced.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the digital face image replacement method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the digital face image replacement method as described in any one of claims 1 to 7.

Images