An image processing method, an electronic device, and a storage medium

Abstract

This application provides an image processing method, an electronic device, and a storage medium. The method includes: acquiring a sample video corresponding to a first object, and acquiring a sample image corresponding to a second object, wherein the sample video includes a facial region of the first object, and the sample image includes a facial region of the second object; selecting target image frames from the sample video that include the face of the first object; replacing the face of the first object in the target image frame with the face of the second object in the sample image to obtain a replaced image frame; determining a relative displacement feature map between image frames in the sample video; and replacing the face of the first object in the image frames of the sample video with the face of the second object based on the replaced image frame and the relative displacement feature map between image frames in the sample video.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, an electronic device, and a storage medium. Background Technology

[0002] Deepfake, also known as deep fake technology, is a portmanteau of deep learning and fake. It specifically refers to the application of human image synthesis technology based on artificial intelligence. Deepfake technology can be used to overlay existing images or videos onto target images or videos.

[0003] In some scenarios, such as live streaming and comedy, it may be necessary to replace one face in an image with another. Currently, the mainstream approach for this is to use deepfake technology to replace one face in one video with another. However, deepfake technology requires extracting facial features from the video (such as facial expressions and lip movements), encoding these features, and then decoding them into the face-swapped image. These feature extraction, encoding, and decoding operations are usually time-consuming and only suitable for non-real-time applications. However, in real-time applications, such as live streaming with anonymous guests requiring real-time face blurring, or comedy scenarios where one actor's face needs to be replaced with another's in real-time, current image processing methods are insufficient to meet the demands of real-time applications. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, electronic device, and storage medium to improve the problem that replacing faces in videos is difficult to meet the needs of real-time application scenarios.

[0005] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring a sample video corresponding to a first object, and acquiring a sample image corresponding to a second object, wherein the sample video includes a facial region of the first object, and the sample image includes a facial region of the second object; filtering out target image frames including the face of the first object from the sample video; replacing the face of the first object in the target image frame with the face of the second object in the sample image to obtain a replacement image frame; determining a relative displacement feature map between image frames in the sample video; and replacing the face of the first object in the image frames of the sample video with the face of the second object based on the replacement image frame and the relative displacement feature map between image frames in the sample video. In the above implementation process, the target image frame in the sample video where the first object's face appears is first replaced, and then the other image frames in the sample video are replaced according to the relative displacement feature map between the target image frame and the target image frame. This results in a face-swapping video with real-time face replacement, achieving the effect of replacing the face in the sample video in real time. This avoids time-consuming operations such as feature encoding and feature decoding, allowing the face-swapping video to effectively meet the needs of real-time application scenarios. It achieves a three-dimensional face-swapping effect without the need for face modeling and training.

[0006] In one optional implementation of the first aspect, replacing the face of a first object in a target image frame with the face of a second object in a sample image to obtain a replacement image frame includes: determining a sample texture map corresponding to the sample image, and determining a face mask texture map based on the sample image; filtering non-face regions in the sample texture map using the face mask texture map to obtain the face region of the second object; and replacing the face region of the first object in the target image frame with the face region of the second object to obtain a replacement image frame.

[0007] In the above implementation process, face replacement is performed based on sample texture map and mask texture map to obtain the replaced image frame, thereby more accurately finding the face location area. Thus, when replacing each pixel, only the face area is replaced, effectively improving the accuracy of replacing the face area in the video.

[0008] In one optional implementation of the first aspect, determining the sample texture map corresponding to the sample image includes: performing face detection on the sample image to obtain an initial face region of the second object; performing target processing on the initial face region to obtain a face region to be processed of the second object, wherein the target processing includes cropping; and predicting the spatial coordinate values ​​corresponding to each pixel of the sample face image based on the horizontal and vertical coordinate values ​​of each pixel in the face region to be processed to obtain the sample texture map corresponding to the face region to be processed, wherein each pixel in the sample texture map includes: a spatial coordinate value, a horizontal coordinate value, and a vertical coordinate value.

[0009] In the above implementation process, by performing face detection and cropping on the sample image, more background areas unrelated to the face are effectively removed. Furthermore, by replacing the sample face image with the spatial coordinate values ​​corresponding to each pixel, more effective face information can be added, and the accuracy of replacing the face area can be improved.

[0010] In one optional implementation of the first aspect, determining the face mask texture map based on the sample image includes: for each pixel in the sample image, using a mask neural network model to predict the mask value corresponding to each pixel to obtain a face mask map corresponding to the sample image, wherein each pixel in the face mask map includes: an abscissa value and a ordinate value; predicting the spatial coordinate value corresponding to each pixel in the face mask map based on the abscissa and ordinate values ​​of each pixel in the face mask map, performing texture extraction, and obtaining a face mask texture map, wherein each pixel in the face mask texture map includes: a spatial coordinate value, an abscissa value, and a ordinate value.

[0011] In the above implementation process, the mask value corresponding to each pixel is predicted by using a mask neural network model to obtain the face mask map corresponding to the sample image, and the spatial coordinate dimension information of the face mask map is predicted, which effectively improves the accuracy of replacing the face region in the video based on the spatial coordinate dimension information.

[0012] In one optional implementation of the first aspect, replacing the face region of the first object in the target image frame with the face region of the second object to obtain a replaced image frame includes: performing face detection on the target image frame to obtain the face region of the first object; mapping the face region of the second object to the face region of the first object in the target image frame to obtain a mapped image frame, wherein the mapped image frame is an image frame in which the face region of the second object is aligned with the face region of the first object; determining the mapped image frame as the foreground image and determining the target image frame as the background image; and fusing each pixel in the foreground image with the corresponding pixel in the background image.

[0013] In one optional implementation of the first aspect, fusing each pixel in the foreground image with the corresponding pixel in the background image includes: for the face region of the second object in the foreground image, calculating multiple distances between each pixel in the face region of the second object and all edge pixels of the face region of the second object; determining the weight of each pixel in the face region of the second object based on the shortest distance among the multiple distances and a preset distance, obtaining a weight matrix corresponding to the face region of the second object, wherein the preset distance is obtained by multiplying the longest distance among the multiple distances by a preset ratio; multiplying the weight matrix corresponding to the face region of the second object with the matrix formed by all pixels of the face region of the second object value by value to obtain a smoothed foreground image; and fusing each pixel in the smoothed foreground image with the corresponding pixel in the background image point by point.

[0014] In the above implementation process, by performing a smooth gradient fusion on the second face in the face image frame, the problem of obvious style difference between the replaced face and the background is avoided, effectively ensuring the smoothness of the edges and the consistency of style after face swapping.

[0015] In one optional implementation of the first aspect, determining the relative displacement feature map between image frames in the sample video includes: for each image frame in the sample video, calculating the relative displacement between each pixel in the image frame and a movingly similar pixel in the target image frame, obtaining the relative displacement corresponding to each pixel in the image frame; for each pixel in the image frame, obtaining the displacement weight of the pixel; and determining the product of the pixel's displacement weight and its corresponding relative displacement to obtain the relative displacement feature map between the image frame and the target image frame. In the above implementation, by performing relative displacement calculations between the image frame after face replacement and the relative displacement feature map corresponding to each image frame, time-consuming operations such as feature encoding and feature decoding are avoided, effectively achieving real-time and stable conversion from un-face-swapped images to face-swapped videos.

[0016] In one optional implementation of the first aspect, based on the relative displacement feature map between the replacement image frame and the image frames in the sample video, replacing the face of the first object in the image frames of the sample video with the face of the second object includes: for each image frame in the sample video, determining the pixel to be replaced that has a similar motion to each pixel in the replacement image frame; determining the displacement product corresponding to the pixel to be replaced in the replacement image frame based on the relative displacement feature map; and moving the pixel to be replaced according to the sum of the displacement product and the pixel value of each pixel in the replacement image frame to obtain multiple moved image frames, wherein the face of the first object in each of the moved image frames has been replaced with the face of the second object.

[0017] In one alternative implementation of the first aspect, after obtaining multiple moving image frames, the method further includes: for blank areas appearing in the moving image frames, using a generative adversarial network to predict the pixel value of each pixel in the blank area based on the pixel values ​​of existing pixels in the moving image frames.

[0018] Secondly, embodiments of this application also provide an image processing method, including: acquiring a sample video corresponding to a first object, and acquiring a sample image corresponding to a second object, wherein the sample video includes a facial region of the first object, and the sample image includes a facial region of the second object; filtering out target image frames including the face of the first object from the sample video; replacing the face of the first object in the target image frame with the face of the second object in the sample image to obtain a replaced image frame; after each replacement, determining a relative displacement feature map between the currently replaced image frame and its adjacent image frames, and based on the replaced image frame and the relative displacement feature map, replacing the face of the first object in the adjacent image frames with the face of the second object, thereby realizing the replacement of the face of the first object in the image frames of the sample video with the face of the second object.

[0019] Thirdly, embodiments of this application also provide an image processing apparatus, comprising: a video image acquisition device, configured to acquire a sample video corresponding to a first object, and to acquire a sample image corresponding to a second object, wherein the sample video includes a facial region of the first object, and the sample image includes a facial region of the second object; a target image filtering module, configured to filter out target image frames including the face of the first object from the sample video; a target image replacement module, configured to replace the face of the first object in the target image frame with the face of the second object in the sample image, thereby obtaining a replaced image frame; a displacement feature determination module, configured to determine a relative displacement feature map between image frames in the sample video; and a face image replacement module, configured to replace the face of the first object in the image frames of the sample video with the face of the second object based on the replaced image frame and the relative displacement feature map between image frames in the sample video.

[0020] Fourthly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, wherein when the machine-readable instructions are executed by the processor, they perform the method described in the first aspect above or any optional implementation of the first aspect, or, when the machine-readable instructions are executed by the processor, they perform the method described in the second aspect above.

[0021] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the method described in the first aspect or any optional implementation thereof above, or, when machine-readable instructions are executed by a processor, executes the method described in the second aspect above.

[0022] Sixthly, embodiments of this application also provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method described in the first aspect or any optional implementation thereof above; or, when the machine-readable instructions are executed by a processor, they perform the method described in the second aspect above. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The illustration shown is a schematic flowchart of the image processing method provided in an embodiment of this application;

[0025] Figure 2 The illustration shows a schematic diagram of the image frame-by-frame replacement process provided in an embodiment of this application;

[0026] Figure 3 The diagram shown is a structural schematic of the image processing apparatus provided in an embodiment of this application;

[0027] Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security and prevention, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial recognition payment, facial recognition unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beauty filters, cosmetics, medical aesthetics, and intelligent temperature measurement.

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0030] Before introducing the image processing method provided in the embodiments of this application, let's first introduce some concepts involved in the embodiments of this application:

[0031] Face detection is a computer technique used to locate and measure the size of faces in any digital image. It detects facial features, such as the location and size of faces, while ignoring other elements like buildings, trees, and bodies. A face detection model is a neural network model that performs face detection on an image; it can determine whether a face exists in the image.

[0032] OpenCV is a cross-platform computer vision library; it can be used to develop real-time image processing, computer vision, and pattern recognition programs.

[0033] Generative Adversarial Networks (GANs), also known as generative adversarial networks, are a learning paradigm in machine learning that learns by having two neural networks compete against each other. A GAN consists of a generator and a discriminator. The generator randomly samples data from the latent space as input data, and the generator's output should mimic real samples in the training set as closely as possible.

[0034] It should be noted that the image processing method provided in this application embodiment can be executed by an electronic device. Here, an electronic device refers to a device terminal or server with the function of executing computer programs. Device terminals include, for example, smartphones, personal computers, tablets, personal digital assistants, or mobile internet devices. A server refers to a device that provides computing services through a network. Servers include, for example, x86 servers and non-x86 servers. Non-x86 servers include, for example, mainframes, minicomputers, and UNIX servers.

[0035] The following describes the applicable scenarios for this image processing method, which include, but are not limited to: Since this image processing method replaces the first face image frame in a sample video where a face appears, and then calculates based on the relative displacement feature map between other image frames in the sample video and the first face image frame, it can achieve the effect of replacing faces in the video in real time; therefore, this image processing method can be used to make software in the form of software development kits (SDKs) and applications (APPs), and the software can be installed and deployed on terminal devices. These software and terminal devices are suitable for scenarios such as entertainment videos, cultural relic photography, and live advertising.

[0036] Please see Figure 1The illustrated diagram shows a flowchart of the image processing method provided in this application embodiment. The main idea of ​​this image processing method is to first replace the target image frame containing the face of a first object in the sample video, and then replace it based on the relative displacement feature map between other image frames in the sample video and the target image frame, thereby obtaining a face-swapping video with real-time face replacement. This achieves the effect of replacing faces in the sample video in real time, thus avoiding time-consuming operations such as feature encoding and feature decoding, and allowing the face-swapping video to effectively meet the needs of real-time application scenarios. Specifically, the above image processing method may include:

[0037] Step S110: Obtain a sample video corresponding to the first object, and obtain a sample image corresponding to the second object, wherein the sample video includes the facial region of the first object, and the sample image includes the facial region of the second object.

[0038] The sample video refers to the video of the face to be replaced (i.e., the face of the first subject). The sample video can be represented by the formula V. A ={I Ai |i∈[1,N]};where N is the total number of image frames in the sample video, A represents the first object, i represents the i-th image frame in the sample video, I Ai This represents the i-th image frame in the sample video that contains the first object. In some scenarios, the face of the first object is always present, such as in a live broadcast where the presenter faces the camera the entire time. In this case, all image frames in the sample video will include the face of the first object. However, in practice, it is also possible that only some image frames include the face of the first object, while others do not.

[0039] A sample image refers to an image that includes the face to be replaced (i.e., the face of the second object). This sample image can be represented by the formula I. B Where B represents the face of the second object, and I... B This represents a sample image that includes a second face.

[0040] Step S120: Filter out target image frames that include the face of the first object from the sample video.

[0041] Step S130: Replace the face of the first object in the target image frame with the face of the second object in the sample image to obtain the replaced image frame.

[0042] Step S140: Determine the relative displacement feature map between image frames in the sample video.

[0043] Step S150: Based on the relative displacement feature map between the replacement image frame and the image frames in the sample video, replace the face of the first object in the image frame of the sample video with the face of the second object.

[0044] It should be noted that when only some image frames in the above sample video include the face of the first object, the image frames that all contain the face of the first object can be filtered out first, and then only the image frames containing the face of the first object are processed (including steps S140 and S150 above). That is, other image frames that do not contain the face of the first object are ignored, and only the relative displacement feature map between the image frames containing the face of the first object and other image frames is determined, and the face replacement operation is performed based on the relative displacement feature map.

[0045] In the above implementation process, the target image frame in the sample video where the first object's face appears is first replaced, and then the other image frames in the sample video are replaced according to the relative displacement feature map between the target image frame and the target image frame. This results in a face-swapping video with real-time face replacement, achieving the effect of replacing the face in the sample video in real time. This avoids time-consuming operations such as feature encoding and feature decoding, allowing the face-swapping video to effectively meet the needs of real-time application scenarios. It achieves a three-dimensional face-swapping effect without the need for face modeling and training.

[0046] As an optional implementation of step S120 above, this implementation includes, for example, performing face detection on each image frame in the sample video in chronological order. Specifically, a face detection model can be used to perform face detection on each image frame in the sample video to obtain face detection results. Based on the face detection results, it is determined whether the image frame contains a first face. If so, the image frame is determined as a face image frame containing the first face; otherwise, face detection continues on the next image frame. The face detection models that can be used include: Multi-task Cascaded Convolutional Networks (MTCNN) and Region Convolutional Neural Networks (RCNN).

[0047] It is understandable that the above-mentioned selection of face image frames, including the first face, from the sample video as target image frames can be represented as I. A1 A1 can represent the image frame where a face first appears in the sample video, or it can be any image frame where a face appears any number of times. The above process can be specifically expressed by the formula: in, D represents the face of the first object in the target image frame of the sample video. AI represents the action of performing face detection on the first face. A1 This represents the target image frame. Then, the PRNet network model is used to extract texture from the face of the first object, obtaining the first texture map corresponding to the target image frame. For example: in, Represents the first texture map. This represents the coordinate inverse mapping function of the texture map. This represents the face of the first object in the target image frame.

[0048] As an optional implementation of step S130, the specific face replacement process may include:

[0049] Step S131: Determine the sample texture map corresponding to the sample image.

[0050] Step S132: Determine the face mask texture map based on the sample image.

[0051] Step S133: Use the face mask texture map to filter the non-face areas in the sample texture map to obtain the face area of ​​the second object.

[0052] Step S134: Replace the face region of the first object in the target image frame with the face region of the second object to obtain the replaced image frame.

[0053] As an optional implementation of determining the sample texture map corresponding to the sample image in step S131, the above texture extraction process may include:

[0054] Step S131a: Perform face detection on the sample image to obtain the initial face region of the second object.

[0055] Step S131b: Perform target processing on the initial face region to obtain the face region to be processed of the second object, wherein the target processing includes cropping processing.

[0056] Understandably, besides cropping, the aforementioned target processing may also involve expanding or scaling the initial face region to ensure the image size matches the network's input size, in order to fuse the face-swapping results and feed them into the network. Therefore, the target processing described above can also include expansion and / or scaling.

[0057] Step S131c: Based on the horizontal and vertical coordinate values ​​of each pixel in the face region to be processed, predict the spatial coordinate values ​​corresponding to each pixel in the face region to be processed to obtain a sample texture map corresponding to the face region to be processed. Each pixel in the sample texture map includes: spatial coordinate value, horizontal coordinate value, and vertical coordinate value.

[0058] An example of implementing steps S131a to S131c above is as follows: using a face detection model to perform face detection and cropping on the sample image to obtain a sample face image, specifically for example: in, I represents a sample face image. B D represents a sample image including a second face. B This indicates that face detection is performed on the second face to obtain the face bounding box and class probability. If the class probability of the face is greater than a preset threshold, it is cropped according to the face bounding box to obtain the sample face image. Specifically, the face detection model in this embodiment can use the MTCNN model or the RCNN model for face detection. Here, MTCNN is used as an example, and can be expressed by the formula: (x0,y0,x1,y1)=MTCNN(I B ); where I B The text describes a sample image containing the second face, where (x0, y0, x1, y1) represents the detected face bounding box, and MTCNN indicates that MTCNN is used for face detection. Then, based on the x and y coordinates of each pixel in the sample face image, the text predicts the spatial coordinates of each pixel to obtain a sample texture map. Each pixel in the sample texture map includes spatial coordinates, x-coordinates, and y-coordinates.

[0059] Optionally, considering the need to fuse the face-swapping results and feed them into the network as input, we can expand, crop, and scale the face detection data. Specifically, this can be represented as: I f =resize(crop(margin(I B ,x0,y0,x1,y1,k)));where, I B This represents a sample image including the second face. (x0, y0, x1, y1) represents the detected face bounding box, where (x0, y0) and (x1, y1) represent the coordinates of the top-left and bottom-right points of the face bounding box detected by MTCNN, respectively. k represents the expansion ratio, margin represents the expansion operation, corp represents the cropping operation, and resize represents the scaling operation. Then, the PRNet network model is used to extract textures from the sample face image to obtain the corresponding sample texture map. For example: in, Represents the sample texture map. This represents the coordinate inverse mapping function of the texture map. This represents a sample human face image.

[0060] As an optional implementation of step S132, the process of obtaining a face mask texture map based on the sample image includes:

[0061] Step S132a: For each pixel in the sample image, use a mask neural network model to predict the mask value corresponding to each pixel to obtain the face mask image corresponding to the sample image. Each pixel in the face mask image includes: horizontal coordinate value and vertical coordinate value.

[0062] Step S132b: Based on the horizontal and vertical coordinate values ​​of each pixel in the face mask image, predict the spatial coordinate values ​​corresponding to each pixel in the face mask image to obtain the face mask texture image. Each pixel in the face mask texture image includes: spatial coordinate value, horizontal coordinate value, and vertical coordinate value.

[0063] An example of implementing steps S132a to S132b above is as follows: For each pixel in the sample image, a mask neural network model is used to predict the mask value corresponding to each pixel to obtain the face mask image corresponding to the sample image. Specifically, for example: in, This represents a face mask image. This represents a sample face image, and mask represents the masking operation of the mask neural network model. The mask neural network model here can be a deep neural network (DNN) or similar segmentation model. For example, if the input of the segmentation model is 256*256*3, then the output of the segmentation model is 256*256*c, where c is the dimension vector of each point.

[0064] Then, based on the horizontal and vertical coordinates of each pixel in the face mask image, the spatial coordinates corresponding to each pixel in the face mask image are predicted. This is equivalent to extracting texture from the face mask image to obtain the mask texture map corresponding to the sample image. For example: in, This represents the image after texture extraction from the face mask image. This represents the coordinate inverse mapping function of the texture map. This represents a face mask image. Represents a mask texture map, p f ={(w,h)} represents the coordinates of the preset region, where w represents the width of the preset region and h represents the height of the preset region.

[0065] As an optional implementation of step S134, the process of replacing the location region in the target image frame with the face region of the second object may include:

[0066] Step S134a: Perform face detection on the target image frame to obtain the face region of the first object.

[0067] Step S134b: Map the face region of the second object to the face region of the first object in the target image frame to obtain a mapped image frame. The mapped image frame is an image frame in which the face region of the second object is aligned with the face region of the first object.

[0068] Step S134c: Determine the mapped image frame as the foreground image and the target image frame as the background image.

[0069] Step S134d: Merge each pixel in the foreground image with the corresponding pixel in the background image.

[0070] The implementation of steps S134a to S134d above is as follows: Face detection is performed on the target image frame to obtain the position region of the face of the first object in the target image frame. The face region of the second object is then mapped to the face region of the first object in the target image frame, so that the face region of the second object is aligned with the face region of the first object. Here, alignment refers to the fact that since the size of the video image frame containing the first object is different from the size of the sample image containing the second object, during the replacement process, the face region of the second object must first be aligned with the face region of the first object before replacement can occur. There are many ways to achieve this alignment, such as the above method of reverse mapping the face region of the second object to the face region of the first object in the target image frame, which completes the alignment operation. After the face region of the second object is aligned with the position region in the target image frame, the face region of the second object is determined as the foreground image, and the target image frame is determined as the background image; specifically, for example: in, This represents the texture image after it has been filtered by the masked texture image. Represents a mask texture map. This indicates element-wise multiplication. Represents the sample texture map. This represents a face image that is mapped. This represents the coordinate inverse mapping function of the texture map. This represents the effect of the mask. This represents the inverse mapping operation for face detection. The coordinate mapping function of the texture map. This represents the process of mapping texture coordinates onto the texture image after filtering it using a masked texture image, resulting in a mapped face image. This represents the foreground image. Indicates background image, I A1This represents the target image frame in the sample video. Further, each pixel in the foreground image is fused with the corresponding pixel in the background image, for example: Among them, I′ A1→B This represents the pixels in the image frame after the face has been replaced. This represents each pixel in the foreground image. This represents the corresponding pixel in the background image.

[0071] As an optional implementation of step S134b, after aligning the facial region of the second object with the position region in the target image frame, a smooth gradient blending can be performed on the facial region of the second object. The smooth gradient blending process may include:

[0072] Step S134e: For the face region of the second object in the foreground image, calculate multiple distances between each pixel in the face region of the second object and all edge pixels of the face region of the second object.

[0073] Step S134f: Determine the weight of each pixel in the face region of the second object based on the shortest distance among multiple distances and the preset distance, and obtain the weight matrix corresponding to the face region of the second object. The preset distance is obtained by multiplying the longest distance among multiple distances by a preset ratio.

[0074] Step S134g: Multiply the weight matrix corresponding to the face region of the second object with the matrix composed of all pixels of the face region of the second object one by one to obtain the smoothed foreground image.

[0075] Step S134h: Add and merge each pixel in the smoothed foreground image with the corresponding pixel in the background image point by point.

[0076] For example, the implementation of steps S134e to S134h above is as follows: All edge pixels (i.e., edge regions) of the face region of the second object are detected using an edge detection algorithm from the OpenCV library, assuming it is represented by an n*2 array. Then, the shortest distance between each pixel in the face region of the second object and all edge pixels of that face region is calculated, and this distance is taken as d. The smoothed foreground image can then be represented as follows: Where d represents the shortest distance from each pixel in the face region of the second object to all edge pixels of that face region. th This represents a preset distance (i.e., a gradient distance threshold). This preset distance is obtained by first acquiring multiple distances from each pixel in the face region of the second object to all edge pixels of that face region, and then multiplying the longest of these distances by a preset ratio. Then, this is used... Replace the above steps By combining Poisson fusion in OpenCV, the styles of the replaced face and background are merged to obtain a face-swapped video that removes the edge face-swapping traces.

[0077] As an optional implementation of step S140, the specific calculation process of the relative displacement feature map may include:

[0078] Step S141: For each image frame in the sample video, calculate the relative displacement between each pixel in the image frame and the moving similar pixel in the target image frame, and obtain the relative displacement corresponding to each pixel in the image frame.

[0079] An example implementation of step S141 above is as follows: Assume the target image frame is the first image frame in the sample video, and the pixel p1 in the first image frame corresponds to pixel p in the i-th image frame. i Then, pixel p1 is relative to point p i The relative displacement can be expressed as: Where p1 represents a pixel in the first image frame (assuming it is the target image frame), p i This represents the pixel point corresponding to the i-th image frame. This represents the pixel p1 in the first image frame relative to the corresponding pixel p in the i-th image frame. i The relative displacement.

[0080] Step S142: For each pixel in the image frame, obtain the displacement weight of the pixel.

[0081] Step S143: Determine the product of the pixel displacement weight and the corresponding relative displacement of the pixel to obtain the relative displacement feature map between the image frame and the target image frame.

[0082] For example, in implementing steps S142 to S143 above: if the motion trend of point q near pixel p is similar to that of pixel p, then Taylor expansion can be used, and the motion trend here can be calculated using an optical flow algorithm to obtain the motion trend of point q near pixel p. Because the motion trend of point q near pixel p is similar to that of pixel p, it can be predicted that: in, p1 represents the relative displacement of pixel q from the i-th video frame to the 1-th video frame, and p1 represents the pixel in the 1-th video frame. i Let f(q) represent the pixel in the i-th video frame. Here, discarding the higher-order o terms is the first-order approximation of the relative displacement. The remaining f(q) represents the displacement weight of the pixel, so it can be called a first-order motion model.

[0083] As an optional implementation of step S150, the process of replacing image frames in the sample video based on the relative displacement feature map may include:

[0084] Step S150a: For each image frame in the sample video, determine the pixel to be replaced that has a similar motion to each pixel in the replacement image frame.

[0085] Step S150b: Based on the relative displacement feature map, determine the displacement product corresponding to the pixel to be replaced in the replacement image frame.

[0086] It is understandable that the relative displacement feature map mentioned above is composed of the displacement product of each pixel. The displacement product here refers to the product of the displacement weight corresponding to the pixel to be replaced and the relative displacement. The displacement weight here represents the weight of the pixel's movement in each frame, that is, the degree of movement of the same pixel in different image frames. Since the degree of movement of the same pixel is different in different image frames, weights are needed to represent the degree of movement of the pixel.

[0087] Step S150c: Move the pixel to be replaced corresponding to the pixel according to the product of displacement and the sum of the pixel values ​​of each pixel in the replacement image frame to obtain multiple moving image frames, wherein the face of the first object in the moving image frames has been replaced with the face of the second object.

[0088] For example, the implementation of steps S150a and S150c described above is as follows: For pixels that appear simultaneously in both the replacement image frame and the image frame to be replaced, i.e., have a corresponding relationship, the pixel replacement result can be directly calculated. Specifically, using an optical flow algorithm, the pixel to be replaced that has a similar movement to the pixel in the replacement image frame is found in each image frame. Assuming the replacement image frame is the first image frame in the sample video, and the pixel p1 in the first image frame corresponds to pixel p in the i-th image frame. i Then, pixel p1 is relative to point p i The relative displacement can be expressed as: Where p1 represents the pixel in the first replacement image frame, p i This represents the pixel point corresponding to the i-th image frame. This represents the pixel p1 in the first replaced image frame relative to the corresponding pixel p in the i-th image frame. iThe relative displacement is determined. For each image frame in the sample video, the relative displacement of the pixel to be replaced in the relative displacement feature map is determined, and the pixel to be replaced is moved according to this relative displacement to obtain multiple moved image frames; that is, the displacement coordinates of the corresponding pixels in the relative displacement feature map corresponding to each image frame in the sample video are added to calculate the displacement of the replacement image frame, resulting in multiple image frames with calculated displacements. Therefore, it is possible to use I Ai→B =I A1→B +Δ 1→i The relative displacement feature maps corresponding to each image frame in the sample video are calculated by replacing the image frame, thereby obtaining multiple motion image frames after displacement calculation; where Δ 1→i This represents the displacement product of a pixel in the relative displacement feature map from the i-th image frame to the 1-th image frame. In other words, this relative displacement feature map includes the product of the displacement weight and the relative displacement of each pixel in each image frame of the sample video (excluding the target image frame). A1→B I represents the pixels in the replacement image frame after the replacement. Ai→B This represents the pixels of a moving image frame after multiple displacement calculations.

[0089] As an optional implementation of step S150, for pixels that do not appear simultaneously in the replacement image frame and the image frame to be replaced, i.e., there is no corresponding relationship, GAN can be used to supplement them, and the supplemented pixels are used as the replacement result. Specifically, after moving the pixel to be replaced, the process further includes:

[0090] Step S150c: For blank areas appearing in the moving image frame, use a generative adversarial network to predict the pixel value of each pixel in the blank area based on the pixel values ​​of existing pixels in the moving image frame, and obtain multiple replacement image frames.

[0091] An example implementation of step S150c is as follows: For the blank areas that appear in each of the multiple moving image frames after movement, a generative adversarial network (GAN) is used to predict the pixel value of each pixel in the blank area based on the pixel values ​​of existing pixels in the moving image frame. The generative adversarial networks that can be used include: GAN, styleGAN, styleGAN2, WGAN (Wasserstein GAN), WGAN-GP (Wasserstein GAN-gradient penalty), and Auxiliary Classifier Generative Adversarial Network (AC-GAN), etc.

[0092] Optionally, after obtaining the multiple replaced image frames, the multiple replaced image frames can also be encapsulated into a face-swapping video. The process of encapsulating into a face-swapping video may include:

[0093] Step S150d: Encapsulate the obtained replacement image frames into a face-swapping video.

[0094] For example, the implementation of step S150d above is as follows: the relative displacement feature map of each pixel in the sample video from the nth frame to the 1st frame (i.e. the replacement image frame) is calculated in the above manner, and multiple replacement image frames are determined according to the replacement image frame and the relative displacement feature map corresponding to each image frame. Finally, the multiple image frames obtained are packaged into a face-swapping video according to the video format.

[0095] It is understandable that the above replacement is based on the relative displacement feature map between the target image frame and each image frame. In practice, after each replacement of the adjacent previous image frame, the image frame can be used as the target image frame, and then the replacement can be performed based on the relative displacement feature map between the target image frame and each image frame, thereby achieving the effect of replacing image frames one by one.

[0096] Please see Figure 2 The illustration shows a flowchart of image frame-by-frame replacement provided in an embodiment of this application; the image processing method for image frame-by-frame replacement here may include:

[0097] Step S210: Obtain a sample video corresponding to the first object, and obtain a sample image corresponding to the second object, wherein the sample video includes the facial region of the first object, and the sample image includes the facial region of the second object.

[0098] Step S220: Filter out target image frames that include the face of the first object from the sample video.

[0099] Step S230: Replace the face of the first object in the target image frame with the face of the second object in the sample image to obtain the replaced image frame.

[0100] Step S240: After each replacement, determine the relative displacement feature map between the replacement image frame obtained by the current replacement and its adjacent image frames, and based on the replacement image frame and the relative displacement feature map, replace the face of the first object in the adjacent image frame with the face of the second object, thereby realizing the replacement of the face of the first object in the image frame of the sample video with the face of the second object.

[0101] To clearly describe the process of steps S220 to S250 above, the replacement process from the first target image frame to the third target image frame is used as an example for detailed explanation. Specifically, for example: a first target image frame including the face of the first object is selected from the sample video. The face of the first object in the first target image frame is replaced with the face of the second object in the sample image to obtain the replaced first replacement image frame. A first relative displacement feature map between the first target image frame and its adjacent second target image frame is determined, and based on the first replacement image frame and the first relative displacement feature map, the face of the first object in the second target image frame is replaced with the face of the second object to obtain the second replacement image frame. A second relative displacement feature map between the second target image frame and its adjacent third target image frame is determined, and based on the second replacement image frame and the second relative displacement feature map, the face of the first object in the third target image frame is replaced with the face of the second object to obtain the third replacement image frame.

[0102] It is understood that there are two ways to determine the second relative displacement feature map mentioned above. The first way is to directly calculate the relative displacement feature map between the second target image frame and the third target image frame according to the method in step S140 above. For the specific calculation method, please refer to step S140, which will not be repeated here. The second way is to calculate the relative displacement feature map between the first target image frame and all image frames in advance to obtain multiple relative displacement feature maps. Then, the relative displacement feature maps that need to be calculated are subtracted from each other. For example, assuming that the relative displacement feature map between the first target image frame and the second target image frame (denoted as A) has been calculated, and the relative displacement feature map between the first target image frame and the third target image frame (denoted as B) has been calculated, then B can be directly subtracted from A to obtain the second relative displacement feature map mentioned above. Here, the first target image frame and the third target image frame are different image frames.

[0103] Please see Figure 3 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this application; this application provides an image processing apparatus 300, including:

[0104] The video image acquisition device 310 is used to acquire a sample video corresponding to a first object and to acquire a sample image corresponding to a second object, wherein the sample video includes the facial region of the first object and the sample image includes the facial region of the second object.

[0105] The target image filtering module 320 is used to filter target image frames, including the face of the first object, from the sample video.

[0106] The target image replacement module 330 is used to replace the face of the first object in the target image frame with the face of the second object in the sample image to obtain the replaced image frame.

[0107] The displacement feature determination module 340 is used to determine the relative displacement feature map between image frames in the sample video.

[0108] The face image replacement module 350 is used to replace the face of a first object in an image frame of a sample video with the face of a second object based on the relative displacement feature map between the replacement image frame and the image frames in the sample video.

[0109] Optionally, in this embodiment of the application, the target image replacement module includes:

[0110] The image texture determination module is used to determine the sample texture map corresponding to the sample image.

[0111] The mask texture determination module is used to determine the face mask texture map based on the sample image.

[0112] The mask texture filtering module is used to filter non-face areas in the sample texture map using a face mask texture map to obtain the face area of ​​the second object.

[0113] The replacement image acquisition module is used to replace the face region of the first object in the target image frame with the face region of the second object to obtain the replacement image frame.

[0114] Optionally, in this embodiment of the application, the image texture extraction module includes:

[0115] The face detection and cropping module is used to perform face detection on the sample image to obtain the initial face region of the second object.

[0116] The face target processing module is used to perform target processing on the initial face region to obtain the face region to be processed of the second object. The target processing includes cropping.

[0117] The sample texture acquisition module is used to predict the spatial coordinates of each pixel in the face region to be processed based on the horizontal and vertical coordinates of each pixel, and obtain the sample texture map corresponding to the face region to be processed. Each pixel in the sample texture map includes: spatial coordinates, horizontal coordinates, and vertical coordinates.

[0118] Optionally, in this embodiment of the application, the mask texture extraction module includes:

[0119] The pixel mask prediction module is used to predict the mask value corresponding to each pixel in the sample image using a mask neural network model, thereby obtaining the face mask image corresponding to the sample image. Each pixel in the face mask image includes: horizontal coordinate value and vertical coordinate value.

[0120] The mask texture acquisition module is used to predict the spatial coordinates of each pixel in the face mask image based on the horizontal and vertical coordinates of each pixel in the face mask image, and obtain the face mask texture image. Each pixel in the face mask texture image includes: spatial coordinates, horizontal coordinates, and vertical coordinates.

[0121] Optionally, in embodiments of this application, the image acquisition module is replaced, including:

[0122] The image face detection module is used to perform face detection on the target image frame to obtain the face region of the first object;

[0123] The foreground / background determination module is used to map the facial region of the second object to the facial region of the first object in the target image frame to obtain a mapped image frame. The mapped image frame is an image frame in which the facial region of the second object is aligned with the facial region of the first object.

[0124] The foreground / background determination module is used to determine the mapped image frame as the foreground image and the target image frame as the background image.

[0125] The foreground / background blending module is used to blend each pixel in the foreground image with the corresponding pixel in the background image.

[0126] Optionally, in this embodiment of the application, the foreground / background determination module includes:

[0127] The edge distance calculation module is used to calculate multiple distances between each pixel in the face region of the second object and all edge pixels of the face region of the second object, for the face region of the second object in the foreground image.

[0128] The weight matrix acquisition module is used to determine the weight of each pixel in the face region of the second object based on the shortest distance among multiple distances and a preset distance, and to obtain the weight matrix corresponding to the face region of the second object. The preset distance is obtained by multiplying the longest distance among multiple distances by a preset ratio.

[0129] The region matrix multiplication module is used to multiply the weight matrix corresponding to the face region of the second object with the matrix composed of all the pixels of the face region of the second object value by value to obtain a smoothed foreground image.

[0130] The point-by-point addition and fusion module is used to add and fuse each pixel in the smoothed foreground image with the corresponding pixel in the background image point by point.

[0131] Optionally, in this embodiment of the application, the displacement feature determination module includes:

[0132] The relative displacement calculation module is used to calculate the relative displacement between each pixel in the sample video and the moving similar pixel in the target image frame for each image frame, thereby obtaining the relative displacement corresponding to each pixel in the image frame.

[0133] The weighted displacement multiplication module is used to obtain the displacement weight of each pixel in the image frame.

[0134] The displacement feature map acquisition module is used to determine the product of the displacement weight of a pixel and the relative displacement of the corresponding pixel, and to obtain the relative displacement feature map between the image frame and the target image frame.

[0135] Optionally, in this embodiment of the application, the face image replacement module includes:

[0136] The motion similarity determination module is used to determine, for each image frame in the sample video, the pixels to be replaced that are motion-similar to those in the replacement image frame.

[0137] The pixel movement module is used to determine the displacement product corresponding to the pixel to be replaced in the replacement image frame based on the relative displacement feature map.

[0138] The moving image frame acquisition module is used to move the pixel to be replaced corresponding to the pixel according to the product of displacement and the sum of the pixel values ​​of each pixel in the replacement image frame, and obtain multiple moving image frames, wherein the face of the first object in the moving image frame has been replaced with the face of the second object.

[0139] Optionally, in this embodiment of the application, the face image replacement module further includes:

[0140] The blank area supplementation module is used to predict the pixel value of each pixel in the blank area based on the pixel values ​​of existing pixels in the moving image frame, for blank areas that appear in the moving image frame.

[0141] The specific implementation process of the functions corresponding to each module in the image processing device provided in this application embodiment can be referred to the above method embodiment. The image processing device corresponds to the above image processing method embodiment and can execute the various steps involved in the above image processing method embodiment. The specific functions of the image processing device can be referred to the description above, and will not be repeated here.

[0142] This application also provides a computer program product, including computer program instructions. When the computer program instructions are read and executed by a processor, they perform the methods described above in steps S110 to S150, or the methods described above in steps S210 to S240. It should be understood that this program product corresponds to the above-described image processing method embodiments and is capable of performing the various steps involved in the above method embodiments. The specific functions of this program product can be found in the description above; detailed descriptions are omitted here to avoid repetition. The program product includes at least one software functional module that can be stored in a memory or embedded in the operating system (OS) of the program product in the form of software or firmware.

[0143] Please see Figure 4 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 400 provided in this application includes a processor 410 and a memory 420. The memory 420 stores machine-readable instructions executable by the processor 410. When the machine-readable instructions are executed by the processor 410, the method described in steps S110 to S150 above is performed; or, when the machine-readable instructions are executed by the processor 410, the method described in steps S210 to S240 above is performed.

[0144] This application embodiment also provides a computer-readable storage medium 430, on which a computer program is stored. When the computer program is run by the processor 410, the method described in steps S110 to S150 above is executed. Alternatively, when the machine-readable instructions are executed by the processor 410, the method described in steps S210 to S240 above is executed.

[0145] The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0146] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0147] In addition, the functional modules of each embodiment in the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0148] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0149] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Obtain a sample video corresponding to a first object, and obtain a sample image corresponding to a second object, wherein the sample video includes the facial region of the first object, and the sample image includes the facial region of the second object; Target image frames including the face of the first object are selected from the sample video; The face of the first object in the target image frame is replaced with the face of the second object in the sample image to obtain the replaced image frame; Determine the relative displacement feature map between image frames in the sample video; Based on the relative displacement feature map between the replacement image frame and the image frames in the sample video, the face of the first object in the image frame of the sample video is replaced with the face of the second object; The relative displacement feature map is composed of the displacement product of each pixel, where the displacement product is the product of the displacement weight corresponding to the pixel to be replaced and the relative displacement. The displacement weight represents the degree of movement of the same pixel in different image frames. Determining the relative displacement feature map between image frames in the sample video includes: for each image frame in the sample video, calculating the relative displacement between each pixel in the image frame and a moving pixel in the target image frame to obtain the relative displacement corresponding to each pixel in the image frame; for each pixel in the image frame, obtaining the displacement weight of the pixel; and determining the product of the displacement weight of the pixel and the relative displacement corresponding to the pixel to obtain the relative displacement feature map between the image frame and the target image frame.

2. The method according to claim 1, characterized in that, The step of replacing the face of the first object in the target image frame with the face of the second object in the sample image to obtain the replaced image frame includes: Determine the sample texture map corresponding to the sample image, and determine the face mask texture map based on the sample image; The face mask texture map is used to filter the non-face areas in the sample texture map to obtain the face area of ​​the second object; The replacement image frame is obtained by replacing the face region of the first object in the target image frame with the face region of the second object.

3. The method according to claim 2, characterized in that, Determining the sample texture map corresponding to the sample image includes: Face detection is performed on the sample image to obtain the initial face region of the second object; The initial facial region is subjected to target processing to obtain the facial region to be processed of the second object, wherein the target processing includes cropping processing; Based on the horizontal and vertical coordinate values ​​of each pixel in the face region to be processed, the spatial coordinate values ​​corresponding to each pixel in the face region to be processed are predicted to obtain a sample texture map corresponding to the face region to be processed. Each pixel in the sample texture map includes: spatial coordinate value, horizontal coordinate value, and vertical coordinate value.

4. The method according to claim 2, characterized in that, Determining the facial mask texture map based on the sample image includes: For each pixel in the sample image, a mask neural network model is used to predict the mask value corresponding to each pixel to obtain a face mask image corresponding to the sample image. Each pixel in the face mask image includes: horizontal coordinate value and vertical coordinate value. Based on the horizontal and vertical coordinate values ​​of each pixel in the face mask image, the spatial coordinate values ​​corresponding to each pixel in the face mask image are predicted to obtain a face mask texture image. Each pixel in the face mask texture image includes: spatial coordinate value, horizontal coordinate value, and vertical coordinate value.

5. The method according to claim 2, characterized in that, The step of replacing the face region of the first object in the target image frame with the face region of the second object to obtain the replaced image frame includes: Face detection is performed on the target image frame to obtain the facial region of the first object; The facial region of the second object is mapped to the facial region of the first object in the target image frame to obtain a mapped image frame, wherein the mapped image frame is an image frame in which the facial region of the second object is aligned with the facial region of the first object; The mapped image frame is determined as the foreground image, and the target image frame is determined as the background image; Each pixel in the foreground image is merged with the corresponding pixel in the background image.

6. The method according to claim 5, characterized in that, The step of fusing each pixel in the foreground image with the corresponding pixel in the background image includes: For the face region of the second object in the foreground image, calculate multiple distances between each pixel in the face region of the second object and all edge pixels of the face region of the second object; The weight of each pixel in the facial region of the second object is determined based on the shortest distance among the plurality of distances and a preset distance, thereby obtaining a weight matrix corresponding to the facial region of the second object. The preset distance is obtained by multiplying the longest distance among the plurality of distances by a preset ratio. The weight matrix corresponding to the face region of the second object is multiplied one by one with the matrix composed of all the pixels of the face region of the second object to obtain the smoothed foreground image. Each pixel in the smoothed foreground image is added to the corresponding pixel in the background image and then blended point by point.

7. The method according to any one of claims 1-6, characterized in that, The step of replacing the face of the first object in the image frame of the sample video with the face of the second object based on the relative displacement feature map between the replacement image frame and the image frames in the sample video includes: For each image frame in the sample video, identify the pixels in the image frame whose motion is similar to that of each pixel in the replacement image frame; Based on the relative displacement feature map, determine the displacement product corresponding to the pixel to be replaced in the replacement image frame; The pixel to be replaced is moved according to the sum of the displacement product and the pixel value of each pixel in the replacement image frame, to obtain multiple moving image frames, wherein the face of the first object in the moving image frames has been replaced with the face of the second object.

8. The method according to claim 7, characterized in that, After obtaining multiple moving image frames, the process further includes: For blank areas appearing in the moving image frame, a generative adversarial network is used to predict the pixel value of each pixel in the blank area based on the pixel values ​​of the existing pixels in the moving image frame.

9. An image processing method, characterized in that, include: Obtain a sample video corresponding to a first object, and obtain a sample image corresponding to a second object, wherein the sample video includes the facial region of the first object, and the sample image includes the facial region of the second object; Target image frames including the face of the first object are selected from the sample video; The face of the first object in the target image frame is replaced with the face of the second object in the sample image to obtain the replaced image frame; After each replacement, the relative displacement feature map between the replaced image frame and its adjacent image frames is determined. Based on the replaced image frame and the relative displacement feature map, the face of the first object in the adjacent image frame is replaced with the face of the second object, thereby replacing the face of the first object in the image frame of the sample video with the face of the second object. The relative displacement feature map is composed of the displacement product of each pixel. The displacement product is the product of the displacement weight corresponding to the pixel to be replaced and the relative displacement. The displacement weight represents the degree of movement of the same pixel in different image frames. Determining the relative displacement feature map between the currently replaced image frame and its adjacent image frames includes: for each image frame, calculating the relative displacement between each pixel in the image frame and the moving similar pixel in the target image frame to obtain the relative displacement corresponding to each pixel in the image frame; for each pixel in the image frame, obtaining the displacement weight of the pixel; and determining the product of the displacement weight of the pixel and the relative displacement corresponding to the pixel to obtain the relative displacement feature map between the image frame and the target image frame.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as claimed in any one of claims 1 to 8; or, the machine-readable instructions, when executed by the processor, perform the method as claimed in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 8; or, when executed by a processor, the computer program performs the method as described in claim 9.

12. A computer program product, characterized in that, It includes computer program instructions that, when read and executed by a processor, perform the method as described in any one of claims 1-8; or, when read and executed by a processor, perform the method as described in claim 9.

Images