Video generation method and device, computer equipment, medium and program product

Abstract

The embodiment of the invention discloses a video generation method and device, computer equipment, a medium and a program product, and the method comprises the steps: obtaining a face corresponding relation and an original video, obtaining face movement tracks corresponding to all historical faces included in previous i-1 original video frames, and according to a plurality of face movement tracks, obtaining a plurality of original video frames; predicting a prediction area corresponding to each historical face in the i-th original video frame, detecting the i-th original video frame to obtain a detection area of the target face in the i-th original video frame, determining a target area of the target face in the i-th original video frame according to the detection area and the plurality of prediction areas, and fusing the reference face to the target area of the ith original video frame based on the indication of the face corresponding relation, and generating a target video. Therefore, the face changing accuracy is improved, and the face changing stability is guaranteed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video generation method, apparatus, computer equipment, medium, and program product. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, video face-swapping technology has been widely used in film special effects, games, advertising, live streaming, and other fields. Taking live streaming as an example, during the live streaming process, an original video is generated, and the original face included in the original video is replaced with a reference face to obtain the target video.

[0003] In related technologies, generative adversarial networks are generally used to extract the identity features of a reference face, and the identity features of the reference face are used to replace the identity features of the original face, while retaining the attribute features of the original face (such as facial pose and facial expression). Each original video frame is then replaced to obtain the target video with a successful face swap.

[0004] However, the above methods only achieve good face-swapping results in some simple scenarios, and the stability of face-swapping results in complex scenarios cannot be guaranteed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a video generation method and related apparatus to improve the stability of face-swapping effects.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a video generation method, the method comprising:

[0008] Obtain facial correspondence and original video. The facial correspondence is used to characterize the reference face for replacing the target face. The target face is the original face waiting to be replaced among multiple original faces included in the original video. The original video includes n original video frames, where n is a positive integer.

[0009] Obtain the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among multiple original faces, where i is a positive integer less than or equal to n.

[0010] Based on the multiple facial motion trajectories, predict the prediction region corresponding to each of the historical faces in the i-th original video frame;

[0011] The i-th original video frame is detected to obtain the detection region of the target face in the i-th original video frame;

[0012] Based on the detection area and the multiple prediction areas, the target area of ​​the target face in the i-th original video frame is determined;

[0013] Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate the target video.

[0014] On the other hand, embodiments of this application provide a video generation apparatus, the apparatus comprising: an acquisition unit, a prediction unit, a detection unit, a determination unit, and a generation unit;

[0015] The acquisition unit is used to acquire facial correspondence and original video. The facial correspondence is used to characterize the reference face for replacing the target face. The target face is the original face waiting to be replaced among multiple original faces included in the original video. The original video includes n original video frames, where n is a positive integer.

[0016] The acquisition unit is further configured to acquire the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among multiple original faces, where i is a positive integer less than or equal to n.

[0017] The prediction unit is used to predict the prediction region corresponding to each of the historical faces in the i-th original video frame based on the multiple facial motion trajectories.

[0018] The detection unit is used to detect the i-th original video frame to obtain the detection area of ​​the target face in the i-th original video frame;

[0019] The determining unit is configured to determine the target region of the target face in the i-th original video frame based on the detection region and the multiple predicted regions;

[0020] The generation unit is used to fuse the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence, thereby generating the target video.

[0021] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0022] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0023] The processor is configured to execute the methods described above according to instructions in the computer program.

[0024] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0025] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described above.

[0026] As can be seen from the above technical solution, the correspondence between the original video and the faces is obtained. This correspondence describes the relationship between the reference face and the target face, i.e., which reference face is used to replace which target face. Since not all original faces included in the original video will necessarily appear in the first i-1 original video frames, the original faces appearing in the first i-1 original video frames are called historical faces. The facial motion trajectory of each historical face is obtained, i.e., the change in its position across multiple original video frames. Due to the continuity between original video frames with short time intervals, the position change of the original face across multiple short-interval original video frames is continuous and smooth. Therefore, based on the facial motion trajectory of each historical face, the prediction region corresponding to each historical face in the i-th original video frame can be predicted, i.e., the possible position of each original face in the i-th original video frame. Thus, based on the continuity between multiple original video frames, stable detection of each original face is achieved to avoid facial jitter and make the face-swapping effect more natural. Detection is performed on the i-th original video frame to obtain the detection region of the target face in the i-th original video frame, i.e., the position of the target face in the original video frame. Based on multiple predicted regions and detection regions, the target region of the target face in the i-th original video frame is determined. This allows for the identification of a historical face that is similar to the target face from multiple historical faces, thereby avoiding false detections or missed detections, improving detection accuracy, and thus improving the stability of face swapping. The correspondence between the reference face and the target face indicated by the facial correspondence is established, and the reference face is fused to the corresponding target region. This allows the reference face to replace the corresponding target face to obtain the target video. This improves the accuracy of face swapping and ensures its stability. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application;

[0032] Figure 5 A flowchart illustrating the video generation method provided in this application embodiment;

[0033] Figure 6 A schematic diagram of a facial movement trajectory provided in an embodiment of this application;

[0034] Figure 7 A flowchart illustrating the video generation method provided in this application embodiment;

[0035] Figure 8 A flowchart illustrating the video generation method provided in this application embodiment;

[0036] Figure 9 A schematic diagram illustrating a video face-swapping method provided in an embodiment of this application;

[0037] Figure 10 A schematic diagram illustrating the correspondence between target resolution and enhancement intensity provided in an embodiment of this application;

[0038] Figure 11 A schematic diagram of a video generation method provided in an embodiment of this application;

[0039] Figure 12 A schematic diagram illustrating facial motion trajectory detection provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application;

[0041] Figure 14 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0042] Figure 15 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application will now be described with reference to the accompanying drawings.

[0044] Video face swapping can be performed by, given an original video and a reference image, replacing the original faces in each frame of the original video with reference faces included in the reference image to obtain a target video. This results in the faces in each frame of the target video retaining the identity features of the reference face while also possessing the facial posture, facial expression, and other attributes of the original face. Additionally, the face-swapped image must appear realistic and natural.

[0045] In related technologies, although generative adversarial networks are used to improve the similarity of video face swapping, they only include two steps: face detection and face replacement. This method has good face swapping effect in some simple scenes with a single person, but it cannot handle some complex face swapping scenes, such as face swapping with multiple people or multiple tasks appearing one after another. Since it cannot be solved by simply restricting the input, the stability of its face swapping effect in complex scenes cannot be guaranteed.

[0046] Based on this, embodiments of this application provide a video generation method and related apparatus. For an original video including multiple original faces, based on the fact that the positional changes of the face between multiple original video frames with short intervals are continuous and smooth, it is possible to determine an original face that is more similar to the target face from multiple original faces by using the facial motion trajectory of each original face, thereby obtaining the target area that the reference face should replace. The original face in the target area is replaced by the reference face to obtain the target video, thereby improving the accuracy of face swapping and ensuring the stability of face swapping.

[0047] The video generation method provided in this application can be applied to computer devices with video generation capabilities, such as terminal devices and servers.

[0048] Specifically, terminal devices can be desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Smart in-vehicle devices can be in-vehicle navigation terminals and in-vehicle computers, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc., but are not limited to these.

[0049] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be connected directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this.

[0050] To facilitate understanding of the video generation method provided in this application embodiment, the following example uses a server as the execution subject of the video generation method to illustrate its application scenarios.

[0051] See Figure 1 This figure is a schematic diagram illustrating an application scenario of a video generation method provided in an embodiment of this application. For example... Figure 1 As shown, this application scenario includes a terminal device 110 and a server 120, which can communicate with each other via a communication network. The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0052] Terminal device 110 is equipped with a face-swapping service for videos. This client allows users to upload or specify reference faces to clearly identify which reference face replaces which target face. It can even directly upload or specify face correspondences. Taking face correspondences as an example, after obtaining the face correspondences and the original video, terminal device 110 sends them to server 120.

[0053] Server 120 is the server corresponding to the client, used to provide face-swapping services for videos. After obtaining the facial correspondence and the original video, server 120 needs to replace each target face in the original video (including n original video frames) with the corresponding reference face based on the indication of the facial correspondence, i.e., which reference face replaces which target face. The replacement process in the i-th original video frame is illustrated below.

[0054] The original video consists of n original video frames, each containing zero or one or more original faces, resulting in m original faces per frame, where m is a positive integer greater than 1. However, not all m original faces in the original video will necessarily appear in the first i-1 original video frames. Therefore, the k original faces that appear in the first i-1 original video frames are called historical faces, where k is a positive integer less than or equal to m.

[0055] Server 120 acquires the facial motion trajectories of each historical face, that is, the changes in the position of each historical face in multiple original video frames, such as... Figure 1 As shown, the facial motion trajectories of the two historical faces are based on the detection boxes. The changes of the detection boxes in multiple video frames are the facial motion trajectories of the corresponding historical faces.

[0056] Because the original video frames with short time intervals are continuous, the positional changes of the original face between multiple short-interval original video frames are continuous and smooth. Therefore, server 120 can predict the prediction region corresponding to each historical face in the i-th original video frame based on the facial motion trajectory of each historical face, that is, the possible location of each original face in the i-th original video frame, such as... Figure 1 As shown, the prediction area is represented by the detection box indicated by the solid line, while the detection box indicated by the dashed line is the detection box corresponding to the facial motion trajectory. Thus, server 120 achieves stable detection of each original face based on the continuity between multiple original video frames, avoiding facial jitter and resulting in a more natural face-swapping effect.

[0057] Server 120 obtains the detection region of the target face in the i-th original video frame, i.e., the position of the target face in the original video frame. Based on multiple predicted regions and detection regions, it determines the target region of the target face in the i-th original video frame, such as... Figure 1 As shown, among the two predicted regions obtained based on the two facial motion trajectories, the detection region overlaps more with the first predicted region. Therefore, the face identified by the first predicted region is most similar to the target face. Furthermore, the detection region can be adjusted based on this predicted region to obtain the target region of the target face in the i-th original video frame, thereby avoiding false detections or missed detections, improving the accuracy of detection, and enhancing the stability of face swapping.

[0058] Based on the indication of facial correspondence, server 120 fuses the reference face into the target region, thereby replacing the target face with the reference face, realizing face swapping for the i-th original video frame, and then performing face swapping for each original video frame in the original video to obtain the target video with the completed face swap.

[0059] Therefore, for an original video containing multiple original faces, based on the fact that the positional changes of the face between multiple short-interval original video frames are continuous and smooth, it is possible to determine the original face that is more similar to the target face from multiple original faces by using the facial motion trajectory of each original face, thereby obtaining the target area that the reference face should replace, and using the reference face to replace the original face in the target area to obtain the target video, thereby improving the accuracy of face swapping and ensuring the stability of face swapping.

[0060] The video generation method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal device may also have similar functions to the server to execute the video generation method provided in this application embodiment, or the terminal device and the server may jointly execute the video generation method provided in this application embodiment. This embodiment does not limit this.

[0061] The video generation method provided in this application can be applied to various scenarios, including but not limited to film and television production, social interaction, security verification, customer service, education and training, and live streaming. Three scenarios are given below as examples.

[0062] Scene 1: Film and television production scene.

[0063] For example, in film and television dramas, when an actor is unable to participate in filming due to unforeseen circumstances or when the character's appearance needs to be adjusted during post-production, the video generation method provided in this application can be used to replace the face of another actor onto the corresponding character, achieving rapid character transformation. Figure 2 As shown, for the i-th original video frame in the original video of a film or television drama, after determining the replacement relationship between the target face and the reference face, the target face in the i-th original video is replaced by the reference face using the video generation method provided in this application embodiment to obtain the i-th target video frame in the target video. Each original video frame in the original video is then replaced to obtain the target video.

[0064] For example, in science fiction, fantasy and other types of film and television works, the video generation method provided in the embodiments of this application can be used to create special effects, such as close-ups of the faces of monsters, aliens and other characters, as well as effects such as character deformation and aging.

[0065] Scenario 2: Social Interaction Scenario.

[0066] For example, on short video platforms, the video generation method provided in this application can be used to replace one's own face with other virtual faces to create various types of funny, imitation, and prank videos, increasing the video's entertainment value and enhancing interactivity. Figure 3As shown, for the i-th original video frame in the short video, after determining the replacement relationship between the target face and the reference face, the video generation method provided in this application is used to replace the target face in the i-th original video with the reference face to obtain the i-th target video frame in the target video. Each original video frame in the original video is then replaced to obtain the target video.

[0067] For example, the video generation method provided in this application can replace the face of a person in the video with a virtual face, thereby creating virtual idols or characters and injecting new vitality into the entertainment industry. These virtual idols can participate in concerts, live broadcasts, advertising endorsements, and other activities to interact with fans.

[0068] Scenario 3: Customer service scenario.

[0069] For example, during a video call with a customer, a reference face can be specified according to the customer's request. The video generation method provided in this application then replaces the customer service representative's face with the reference face, thereby providing the customer with a better visual experience, increasing its fun and practicality, and improving the customer service experience. Figure 4 As shown, during a video call with a customer, if the user wants to communicate with a customer service representative who has a reference face, then for the i-th original video frame in the original video, after determining the replacement relationship between the target face and the reference face, the target face in the i-th original video is replaced by the reference face using the video generation method provided in this application embodiment, resulting in the i-th target video frame in the target video. Each original video frame in the original video is then replaced to obtain the target video of the customer service representative with the reference face.

[0070] It should be noted that the above application scenarios are merely examples, and the video generation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0071] The following describes in detail a video generation method provided in this application through method embodiments.

[0072] See Figure 5 This figure is a schematic flowchart of the video generation method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of the video generation method as an example. Figure 5 As shown, the video generation method includes S501-S506.

[0073] S501: Obtain facial correspondence and original video.

[0074] The facial correspondence is used to characterize the reference face for replacing the target face, that is, which reference face is used to replace which target face. The target face is the original face to be replaced among multiple original faces included in the original video, the original face is the face included in the original video, and the reference face is the face used to replace the target face. For example, reference face A is used to replace original face B or original face C in the original video. Or, reference face A replaces both original face B and original face C in the original video. It should be noted that the face can be a human face, or it can be the face of an animal, robot, virtual human, etc., and this application does not make specific limitations in this regard.

[0075] This application does not specifically limit the number of reference faces and target faces. For example, there may be one reference face and one target face. Alternatively, there may be multiple reference faces and multiple target faces. In the embodiments corresponding to S501-S506, only one target face is used as an example for explanation. This application does not specifically limit the method of obtaining face correspondence; three methods are described below as examples.

[0076] Method 1 involves the user uploading an image including a reference face and specifying which original face in the original video the reference face should replace, thus obtaining the facial correspondence. This method clearly and quickly identifies the replacement relationship between the reference and target faces.

[0077] If the user is unaware that the original video contains multiple original faces, the original video can be pre-detected to obtain multiple original faces for the user to select, thus obtaining the target face, and subsequently the replacement relationship between the target face and the reference face. If the user knows that the original video contains multiple original faces, such as in a live streaming scenario where the original video typically contains only one original face, a reference face can be directly specified, and the original face can be replaced based on the reference face, thus obtaining the face correspondence relationship.

[0078] Method Two: Users select the desired reference face from a pool of pre-prepared reference faces and specify which original face in the original video it should replace. This allows users to directly choose from multiple pre-prepared reference faces without needing to accurately reference a face, thus improving the user experience.

[0079] Method 3: Users can directly specify or upload images of p reference faces, where p is a positive integer. For example, an image may contain p reference faces, or p images may exist, each containing one reference face. The original video contains q original faces, where q is a positive integer, and p is less than or equal to q. After obtaining the p reference faces, if the user specifies the original face to be replaced for each reference face (i.e., each reference face has its own original face to be replaced), the server obtains the original faces that correspond one-to-one with each reference face. The original face to be replaced is the target face, thus obtaining the face correspondence system. This face correspondence system describes which reference face replaces which target face.

[0080] If the user does not specify the replacement relationship between the reference face and the original face, that is, no original face corresponding one-to-one with the reference face is obtained, then the number of times each of the q original faces appears in the original video can be determined. Then, the p original faces with the most occurrences are determined as the target faces (e.g., based on the number of occurrences from largest to smallest, the original faces corresponding to the top p occurrences are determined as the target faces), and a one-to-one correspondence is established between the p target faces and the p reference faces, thus obtaining the face correspondence relationship.

[0081] Therefore, without specifying the target face in the original face or the replacement relationship between the reference faces and the target faces, one or more original faces that appear most frequently in the original video can be identified as target faces. A one-to-one correspondence between each target face and each reference face can then be established, resulting in a facial correspondence. This correspondence allows for face swapping in the video. This eliminates the need for users to specify the replacement relationship between each reference face and each target face individually, reducing user workload, shortening the time to generate the target video, and improving the user experience.

[0082] It is understood that in the specific embodiments of this application, data related to the user's face (such as original face, reference face, etc.) is involved. When the above embodiments of this application are applied to specific products or technologies, the data collection requires the consent and authorization of the data owner (such as user, institution or enterprise). The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0083] S502: Obtain the facial motion trajectory corresponding to each historical face in the first i-1 original video frames.

[0084] The original video consists of n original video frames, and there is a temporal relationship between the n original video frames. n is a positive integer. Taking the i-th original video frame among the n original video frames as an example, the face-swapping process in the i-th original video frame is explained. i is a positive integer less than or equal to n.

[0085] The first i-1 original video frames refer to the original video frames whose temporal sequence precedes the i-th original video frame. Although the original video includes multiple original faces, not all original faces may appear in the first i-1 original video frames. For example, if the original video includes m original faces (m is a positive integer greater than 1), fewer than m original faces may appear in the first i-1 original video frames, such as k original faces (k is a positive integer less than or equal to m). The remaining un-appeared original faces (i.e., mk original faces) appear after the i-th original video frame. Therefore, to distinguish them, the original faces appearing in the first i-1 original video frames are called historical faces.

[0086] Obtain the facial motion trajectory corresponding to each historical face. Different facial motion trajectories correspond to different historical faces. That is, each facial motion trajectory corresponds to a historical face, and multiple facial motion trajectories can overlap.

[0087] The facial motion trajectory is used to describe the positional changes of the corresponding historical face across multiple original video frames. This application does not specifically limit the relationship between the facial motion trajectory and the original video frames. The facial motion trajectory can be obtained based on multiple original video frames that include the corresponding historical face, either continuously or with small intermittent time intervals (i.e., can be considered continuous). These multiple original video frames can be all or part of the first i-1 original video frames that include the historical face. For example, if the historical face appears in all of the first i-1 original video frames, the facial motion trajectory can be obtained based on the first i-1 original video frames (i.e., based on all of them), or it can be obtained based on the first i-1, first i-2, and first i-3 original video frames. This application does not specifically limit this.

[0088] The embodiments of this application do not specifically limit the method for determining facial motion trajectories; those skilled in the art can set it according to actual needs. For example, the original faces in the original video can be tracked and detected using the Simple Online and Realtime Tracking (SORT) algorithm to obtain the facial motion trajectories of each original face. For example, each historical face is marked with a detection box, and the center of the detection box is used as the trajectory point. Based on the detection boxes of multiple historical faces, the facial motion trajectory of that historical face is obtained, and thus the facial motion trajectories of each historical face are obtained.

[0089] See Figure 6 This figure is a schematic diagram of a facial movement trajectory provided in an embodiment of this application. Figure 6 In this process, the facial motion trajectory of the historical face is obtained based on the (i-1), (i-2), (i-3), and (i-4)th original video frames. Based on the detection boxes identifying the historical face in the (i-4)th original video frame, the center of the detection box is used as the first trajectory point. Similarly, based on the detection boxes identifying the historical face in the (i-3)th original video frame, the center of the detection box is used as the second trajectory point, and so on, obtaining the trajectory points corresponding to the four original video frames. These four trajectory points constitute the facial motion trajectory of the historical face. It is understood that, for ease of subsequent calculation, the detection boxes can also be directly used as trajectory points to obtain the facial motion trajectory; this application does not specifically limit this approach.

[0090] The embodiments of this application do not specifically limit the method of obtaining the detection boxes. For example, a You Only Look Once Face Detection Model (YOLOFACE) can be used to detect the original video frames and obtain the detection boxes corresponding to each original face.

[0091] S503: Based on multiple facial motion trajectories, predict the prediction region corresponding to each historical face in the i-th original video frame.

[0092] Because the original video frames with short time intervals are continuous, the positional changes of the original face between multiple original video frames with short intervals are continuous and smooth. Therefore, based on the facial motion trajectory of each historical face, the prediction region corresponding to each historical face in the i-th original video frame can be predicted, that is, the possible position of each historical face in the i-th original video frame.

[0093] This means that while a historical face may appear in the first i-1 original video frames, it may not necessarily appear in the ith original video frame. Based on this, the predicted region of each historical face in the ith original video frame can be predicted based on the facial motion trajectory of each historical face and the time interval between the last original video frame associated with that trajectory and the ith original video frame. Specifically:

[0094] For a target facial motion trajectory among multiple facial motion trajectories, if the time interval between the last original video frame associated with the target facial motion trajectory and the i-th original video frame is less than or equal to the first interval threshold, then based on the target facial motion trajectory, the predicted region corresponding to the historical face in the i-th original video frame is predicted. Multiple facial motion trajectories are used as target facial motion trajectories respectively to obtain the predicted regions corresponding to each historical face in the i-th original video frame.

[0095] Among them, the original video frames associated with the facial motion trajectory are multiple original video frames used to calculate the facial motion trajectory, and the last original video frame associated with the facial motion trajectory is the last original video frame in the temporal sequence among the multiple original video frames used to calculate the facial motion trajectory.

[0096] For example, if facial motion trajectory 1 is obtained based on the (i-4)th original video frame and the (i-1)th original video frame, and the time interval between the (i-1)th original video frame and the ith original video frame is small, then the predicted region of the historical face in the ith original video frame can be predicted based on facial motion trajectory 1. As another example, if facial motion trajectory 2 is obtained based on the (i-20)th original video frame and the (i-10)th original video frame, since the time interval between the (i-10)th original video frame and the ith original video frame is large, it is not necessary to predict the predicted region of the historical face in the ith original video frame based on facial motion trajectory 2; that is, the predicted region of the historical face corresponding to facial motion trajectory 2 in the ith original video frame is 0.

[0097] If the time interval between the last original video frame associated with the target facial motion trajectory and the i-th original video frame is greater than the first interval threshold, it means that the target facial motion trajectory is no longer helpful for predicting the motion trajectory of the historical face, and prediction can no longer be made based on the target facial motion trajectory. The prediction area of ​​the historical face corresponding to the target facial motion trajectory in the i-th original video frame is 0.

[0098] Therefore, based on the time interval between the last original video frame associated with the facial motion trajectory and the i-th original video frame, it is determined whether to make a prediction based on the facial motion trajectory, thereby ensuring that more of the predicted regions are useful, reducing the computational load of prediction, increasing the speed of video face swapping, and improving the user experience.

[0099] S504: Detect the i-th original video frame to obtain the detection region of the target face in the i-th original video frame.

[0100] For example, each original face included in the i-th original video frame is detected separately to obtain the detection region of each original face in the i-th original video frame, that is, the position of the original face in the i-th original video frame. If the target face is in the i-th original video frame, then it is one of the original faces included in the i-th original video frame, and subsequent replacement operations can be performed; if the target face is not in the i-th original video frame, then the subsequent steps are not performed, and i = i + 1 is set, which is equivalent to performing replacement for the next original video frame after the i-th original video frame, and S502-S504 is executed again until the target face is detected, and S505 and S506 are executed.

[0101] As one possible implementation, the detection region can be identified using a detection bounding box. This application does not specifically limit the method of detecting the original face in the i-th original video frame. For example, YOLOFACE can be used to detect each original face included in the i-th original video frame. If the original video only includes original faces, then the detected original faces are the target faces. If the original video includes multiple original faces, then one or zero of the detected original faces are the target faces; this application does not specifically limit this.

[0102] S505: Based on the detected region and multiple predicted regions, determine the target region of the target face in the i-th original video frame.

[0103] Although multiple facial motion trajectories are obtained based on the aforementioned steps, each facial motion trajectory is used to describe the positional changes of the same historical face, it may not be known which facial motion trajectory belongs to the historical face, and therefore it is unknown which facial motion trajectory the target face corresponds to, or which prediction region the target face corresponds to.

[0104] The predicted regions identify the positions of each original face in the i-th original video frame. Therefore, the detected region can be compared with multiple predicted regions to determine which predicted region is more similar to the detected region. Taking the bounding box as an example, if the bounding box identifying the predicted region has a high degree of overlap with the bounding box identifying the detected region, it indicates that the predicted region and the detected region are likely to represent the same face. Thus, a predicted region that is more similar to the detected region can be obtained from multiple predicted regions.

[0105] Furthermore, the accuracy of the detection area for identifying the target face may be low due to factors such as detection precision. Sudden facial movements or camera movements can also cause abrupt changes in facial trajectory, leading to lower accuracy in the predicted area for identifying historical faces. Therefore, the detection area and the predicted area can be mutually corrected. If the positions of the detection area and the predicted area are similar, the midpoint between them can be used as the target area. Conversely, if the positions of the detection area and the predicted area differ significantly, detection can be performed again, or the area with higher accuracy can be used as the target area. This application does not impose specific limitations on these methods.

[0106] The target region of the target face in the i-th original video frame is used to indicate the most likely location of the target face in the i-th original video frame.

[0107] S506: Based on the indication of facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate the target video.

[0108] Facial correspondence indicates which reference face replaces which target face. The target region in the i-th original video frame is the most likely location of the target face in the i-th original video. Based on the facial correspondence, the reference face corresponding to the target face at that target location is determined. Then, based on the indication of the facial correspondence, the reference face is fused to the target location to obtain the i-th target video frame. In the i-th target video frame, the reference face replaces the target face, thus achieving the effect of replacing the target face based on the reference face in the i-th original video frame. Subsequently, each original video frame can be used as the i-th original video frame to obtain the target video frame corresponding to each original video frame. Thus, the target video is obtained based on multiple target video frames, thereby achieving the face-swapping effect on the original video.

[0109] As can be seen from the above technical solution, the correspondence between the original video and the faces is obtained. This correspondence describes the relationship between the reference face and the target face, i.e., which reference face is used to replace which target face. Since not all original faces included in the original video will necessarily appear in the first i-1 original video frames, the original faces appearing in the first i-1 original video frames are called historical faces. The facial motion trajectory of each historical face is obtained, i.e., the change in its position across multiple original video frames. Due to the continuity between original video frames with short time intervals, the position change of the original face across multiple short-interval original video frames is continuous and smooth. Therefore, based on the facial motion trajectory of each historical face, the prediction region corresponding to each historical face in the i-th original video frame can be predicted, i.e., the possible position of each original face in the i-th original video frame. Thus, based on the continuity between multiple original video frames, stable detection of each original face is achieved to avoid facial jitter and make the face-swapping effect more natural. Detection is performed on the i-th original video frame to obtain the detection region of the target face in the i-th original video frame, i.e., the position of the target face in the original video frame. Based on multiple predicted regions and detection regions, the target region of the target face in the i-th original video frame is determined. This allows for the identification of a historical face that is similar to the target face from multiple historical faces, thereby avoiding false detections or missed detections, improving detection accuracy, and thus improving the stability of face swapping. The correspondence between the reference face and the target face indicated by the facial correspondence is established, and the reference face is fused to the corresponding target region. This allows the reference face to replace the corresponding target face to obtain the target video. This improves the accuracy of face swapping and ensures its stability.

[0110] As one possible implementation, the following example illustrates the concept of having multiple reference faces and multiple target faces.

[0111] See Figure 7 The figure is a flowchart illustrating the video generation method provided in an embodiment of this application.

[0112] S701: Obtain facial correspondence and original video.

[0113] In this embodiment, the facial correspondence is used to describe which reference face is used to replace which target face, that is, which reference face among multiple reference faces is used to replace which target face among multiple target faces.

[0114] S702: Obtain the facial motion trajectory corresponding to each historical face in the first i-1 original video frames.

[0115] See S502 for details.

[0116] S703: Based on multiple facial motion trajectories, predict the prediction region corresponding to each historical face in the i-th original video frame.

[0117] See S503 for details.

[0118] S704: Detect the i-th original video frame to obtain the detection regions corresponding to each target face in the i-th original video frame.

[0119] For example, by performing detection on the i-th original video frame based on YOLOFACE, the detection regions corresponding to each historical face in the i-th original video frame can be obtained.

[0120] Similarly, if the target face is in the i-th original video frame, then it is one of the original faces included in the i-th original video frame. If the target face is not in the i-th original video frame, then let i = i + 1, which is equivalent to replacing the target face in the next original video frame after the i-th original video frame. Then execute S702-S704 again until the target face is detected, and execute S705-S709.

[0121] The following steps, based on S705-S707, determine which predicted region corresponds to which detection region, i.e., which target face corresponds to which reference face.

[0122] S705: Determine the overlap between each detection area and each prediction area.

[0123] The detection region corresponds to a portion of the original video frame, and the prediction region also corresponds to a portion of the original video frame. The degree of overlap between the detection region and the prediction region is called the overlap degree. The overlap degree can be obtained by calculating between A detection regions and B prediction regions.

[0124] The embodiments of this application do not specifically limit the calculation method of the overlap degree. For example, the overlap degree can be based on the number of pixels included in the detection region and the number of pixels included in the prediction region being the same. Another example is based on the intersection-union ratio between the detection boxes of the detection region and the detection boxes of the prediction region.

[0125] S706: Determine the optimal solution for multiple objectives based on multiple overlap degrees.

[0126] A multi-objective optimal solution is the combination that maximizes the overlap between each detection region and each prediction region. In a multi-objective programming problem, a multi-objective optimal solution is the solution that simultaneously optimizes multiple objective functions. Each objective function is equivalent to maximizing the overlap between a detection region and a prediction region. By finding a multi-objective optimal solution, we can avoid getting trapped in local optima and achieve global optima, that is, ensuring that multiple detection regions match the prediction regions with the highest overlap, or in other words, matching each detection region with the most suitable prediction region.

[0127] In multi-objective programming problems, since there are multiple objective functions and these objective functions may conflict, it is difficult to find a solution that can optimize all objective functions at the same time. Usually, we seek to find a "satisfactory" solution, namely the Pareto optimal solution, which is a solution in a given multi-objective function in which no one objective function can be further improved without harming at least one other objective function.

[0128] The embodiments in this application do not specifically limit the method for determining the optimal solution for multiple objectives. For example, hierarchical solution methods, goal programming models, constraint models, Hungarian algorithms, etc.

[0129] S707: Based on the optimal solution for multiple targets, determine the target regions corresponding to each target face in the i-th original video frame.

[0130] The multi-target optimal solution indicates the most suitable prediction region matched with each detection region. Based on the detection regions and their most suitable prediction regions, adjustments are made to obtain multiple target regions, i.e., the target regions corresponding to each target face in the i-th original video frame. The specific adjustment method is described in S505 and will not be repeated here.

[0131] S708: Based on the facial correspondence, determine the reference face corresponding to each target face, and obtain the mapping relationship between the reference face and the target region.

[0132] There is a correspondence between the target face and the target region, and a correspondence between the reference face and the target face (i.e., a face correspondence). Therefore, based on the target face, the correspondence between the target region and the reference face (i.e., a mapping relationship) can be obtained. That is, the target region and the reference face with the mapping relationship correspond to the same target face.

[0133] S709: Based on the mapping relationship, each reference face indicated by the facial correspondence is fused into the corresponding target region in the i-th original video frame to generate the target video.

[0134] The mapping relationship describes the correspondence between the reference face and the target region in the i-th original video frame. This allows each reference face to be fused into the target region with which it has a mapping relationship, thus achieving the effect of replacing the target face in the target region with the reference face, obtaining the i-th target video frame, and ultimately the target video. For the specific fusion method, see S506, which will not be elaborated here.

[0135] Therefore, when facial correspondence indicates the replacement relationship between multiple reference faces and multiple target faces, i.e., in the scenario of multi-person face swapping for an original video, multiple target faces each have a detection region, that is, there are multiple detection regions. When multiple detection regions are matched with multiple prediction regions, it may not be guaranteed that each detection region will match the prediction region with the maximum overlap. However, by finding the optimal solution for multiple targets, the most suitable prediction region can be matched for each detection region, thereby avoiding getting trapped in local optima and improving the accuracy of subsequent replacement.

[0136] One possible approach is to first detect the facial motion trajectories of each original video frame, obtaining the corresponding facial motion trajectories for each original face. Based on these facial correspondences, a reference face (i.e., a mapping relationship) can then be determined for each facial motion trajectory. This mapping relationship can then be used to achieve the video face-swapping effect.

[0137] It's important to note that after obtaining the mapping relationship, the process of replacing the target face with the reference face is time-independent. That is, the replacement processes in each original video frame are independent of each other, allowing for parallel processing. This means that the process of replacing the target face with the reference face can be implemented in parallel for each original video frame. For example, each original video frame can use a separate process for this process. Similarly, the 1st to 10th original video frames can each use a single process, and so on. Therefore, parallel processing can improve the processing speed of video face swapping and enhance the user experience.

[0138] This application provides a specific implementation of S505, namely, a specific implementation of determining the target region of the target face in the i-th original video frame based on the detection region and multiple prediction regions, as detailed in S5051-S5054.

[0139] S5051: Determine the overlap between the detection area and multiple prediction areas.

[0140] See S705 for details. The following example illustrates how to determine the overlap between one or more detection areas.

[0141] S5052: If multiple overlaps are less than the overlap threshold, the target face is the original face that appears in the i-th original video frame, and the facial motion trajectory corresponding to the target face is created.

[0142] If multiple overlaps are all less than the overlap threshold, it means that the facial motion trajectory based on each historical face cannot predict the target face. That is, the target face is not similar to the historical faces that appeared in the first i-1 original video frames. Therefore, compared with the historical faces included in the first i-1 original video frames, the target face is the original face that appears in the i-th original video frame.

[0143] At this point, a facial motion trajectory can be created based on the newly emerging original face, that is, the facial motion trajectory corresponding to the target face, and the detection area of ​​the target face in the i-th original video frame is the starting point of the facial motion trajectory, so that the predicted position of the target face in the next original video frame can be predicted based on the facial motion trajectory.

[0144] The embodiments of this application do not specifically limit the overlap threshold. Those skilled in the art can set it according to actual needs, such as 70%.

[0145] S5053: If at least one of the multiple overlap degrees is greater than or equal to the overlap degree threshold, then the prediction region corresponding to the overlap degree that is greater than or equal to the overlap degree threshold and meets the overlap condition is determined as the prediction region corresponding to the detection region.

[0146] If at least one of the multiple overlap degrees is greater than or equal to the overlap degree threshold, that is, the target face is more similar to at least one historical face that appears in the previous i-1 original video frames, it means that the target face is not a newly appearing original face in the i-th original video frame compared to the historical faces included in the previous i-1 original video frames.

[0147] When the target face is not a newly appearing original face in the i-th original video, an overlap degree that meets the overlap condition can be selected from one or more overlap degrees that are greater than or equal to the overlap degree threshold. In this case, the target face included in the predicted region corresponding to the overlap degree is most similar to the historical faces included in the detection region. Therefore, the predicted region corresponding to the overlap degree can be determined as the predicted region corresponding to the detection region, that is, a correspondence between the detection region and the predicted region is established. This enables the determination of the face most similar to the target face from multiple historical faces, thereby improving the accuracy of subsequent video face swapping.

[0148] The embodiments of this application do not specifically limit the overlap conditions. For example, the overlap is sorted from largest to smallest, and the overlap with the highest overlap is the overlap that meets the overlap conditions.

[0149] S5054: Adjust the detection area according to the predicted area corresponding to the detection area to obtain the target area of ​​the target face in the i-th original video frame, and update the facial motion trajectory corresponding to the target face according to the target area.

[0150] The predicted region is the position of a historical face in the i-th original video frame, predicted from the historical faces in the previous i-1 original video frames. Since the motion trajectory of a face in short intervals (e.g., consecutive original video frames) is continuous, the predicted region has high reliability. The detected region is the position of the detected target face in the i-th original video frame and also has high reliability. Therefore, the two can be used interchangeably. The corresponding detected region can be adjusted based on the predicted region, such as using the average of the predicted and detected regions as the target region for the target face in the i-th original video frame, thereby improving the accuracy of the target region.

[0151] Meanwhile, after obtaining the target location, the target location can be used as the next trajectory point of the facial motion trajectory corresponding to the predicted area to update the facial motion trajectory, thereby obtaining the updated facial motion trajectory for subsequent use.

[0152] Furthermore, if the detection area and the prediction area are both successfully matched, that is, if multiple reference faces and the target face are all successfully matched, but there are prediction areas among the multiple prediction areas that are not successfully matched with the detection area, that is, there are historical faces among the multiple historical faces that are not corresponding to the reference faces, and the time interval between the last original video frame associated with the unmatched prediction area and the i-th original video frame is greater than the first interval threshold, then the facial motion trajectory corresponding to the unmatched prediction area will no longer be predicted, so as to reduce the computational load of prediction.

[0153] Therefore, by using the overlap threshold to determine whether the target face is a newly appearing original face in the i-th original video frame, a new facial motion trajectory is created for the newly appearing original face, and the old facial motion trajectory is updated for the historical faces that have already appeared, so as to ensure the correctness of the facial motion trajectory, improve the prediction accuracy of subsequent original video frames, improve the accuracy of video face swapping, and improve the user experience.

[0154] As mentioned above, the predicted position of the face in the i-th original video frame will only be predicted based on the face motion trajectory if the time interval between the last original video frame associated with the face motion trajectory and the i-th original video frame is less than or equal to the first interval threshold. In other words, the original video includes multiple face motion trajectories, but the same original face may correspond to multiple face motion trajectories. For example, if the original face A appears after the 1st to the 10th original video frames and reappears in the 15th to the 20th original video frames, it will be identified as two face motion trajectories.

[0155] Based on this, multiple facial motion trajectories can be merged so that the same facial motion trajectory corresponds to the same original face. Subsequently, the target face corresponding to a facial motion trajectory can be replaced based on a reference face. That is, the target region in each original video frame is determined based on the original video frame associated with the facial motion trajectory, and the target face in the target region is replaced based on the reference face to achieve the video face-swapping effect.

[0156] In addition, after obtaining a facial motion trajectory, it is possible to determine whether the facial motion trajectory is repeated with an already determined facial motion trajectory, so as to subsequently analyze whether the original face described by the facial motion trajectory is the main face in the original video, etc.

[0157] This application does not specifically limit the method of merging facial movement trajectories. Two methods are described below as examples.

[0158] Method 1: Merge facial movement trajectories with similar facial features and non-overlapping start and end times.

[0159] A1: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames.

[0160] The initial facial motion trajectory is the facial motion trajectory that has not yet been merged. It can be obtained directly from the facial motion trajectory obtained from the first i-1 original video frames. For the specific acquisition method, please refer to the aforementioned S502.

[0161] A2: Obtain the start and end times of the associated video frames of the target's initial facial motion trajectory, as well as the facial features of the target's historical face in each associated video frame.

[0162] The target's initial facial motion trajectory is one of multiple initial facial motion trajectories. The associated video frame is one of multiple original video frames associated with this target's initial facial motion trajectory; it is the original video frame from which the target's initial facial motion trajectory was calculated. The start and end times of this associated video frame are based on the first and last original video frames among these multiple original video frames. For example, the start time of the first original video frame and the end time of the last original video frame are the start and end times. Alternatively, the start time of the first original video frame and the start time of the last original video frame are also considered the start and end times; this application does not specifically limit this.

[0163] The target historical face is the historical face corresponding to the initial facial motion trajectory of the target, and facial features are used to describe the features of the face.

[0164] A3: Based on the facial features of the target's historical face in each associated video frame, determine the average facial features corresponding to the target's initial facial motion trajectory.

[0165] The facial features of the target historical face may have subtle differences in each original video frame, just as the facial features of a person's frontal face and side face may have subtle differences. Therefore, it is necessary to determine the average facial features of the target historical face relative to the target's initial facial motion trajectory based on the facial features of the target historical face in each associated video frame. The average facial features are obtained based on the average of multiple facial features, so as to more accurately describe the characteristics of the target historical face in the target's initial facial motion trajectory.

[0166] A4: Take multiple initial facial motion trajectories as target initial facial motion trajectories and obtain the average facial features of each initial facial motion trajectory.

[0167] Multiple initial facial motion trajectories are used as target initial facial motion trajectories, and A2 and A3 are executed to obtain the average facial features of each initial facial motion trajectory.

[0168] A5: If the similarity of the average facial features of two initial facial motion trajectories is greater than the similarity threshold, and the start and end times of the associated video frames of the two initial facial motion trajectories do not overlap, then the two initial facial motion trajectories will be merged.

[0169] If the similarity of the average facial features of two initial facial motion trajectories is greater than a similarity threshold, then the historical faces corresponding to these two initial facial motion trajectories are relatively similar. Given that the two historical faces are relatively similar, if the start and end times of the associated video frames of the initial facial motion trajectories corresponding to the two historical faces do not overlap, it indicates a high probability that the two historical faces belong to the same historical face. If the start and end times of the associated video frames of the initial facial motion trajectories corresponding to the two historical faces overlap, then the two historical faces are generally not the same historical face, because the same historical face generally does not appear twice in the same original video frame.

[0170] Based on this, if the similarity of the average facial features of two initial facial motion trajectories is greater than the similarity threshold, and the start and end times of the associated video frames of the two initial facial motion trajectories do not overlap, then the two initial facial motion trajectories will be merged.

[0171] Taking the fourth and fifth initial facial motion trajectories among multiple initial facial motion trajectories as examples, the fourth and fifth initial facial motion trajectories are different facial motion trajectories. If the similarity between the average facial features of the fourth initial facial motion trajectory and the average facial features of the fifth initial facial motion trajectory is greater than the similarity threshold, and the start and end times of the associated video frames of the fourth initial facial motion trajectory and the associated video frames of the fifth initial facial motion trajectory do not coincide, then the fourth and fifth initial facial motion trajectories are merged to obtain a single facial motion trajectory.

[0172] The embodiments of this application do not specifically limit the similarity threshold. Those skilled in the art can set it according to actual needs, such as 90%.

[0173] If, after obtaining an initial facial motion trajectory, it is determined whether the initial facial motion trajectory is repeated with the already determined initial facial motion trajectory through A1-A5, the repeated initial facial motion trajectories are merged to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

[0174] If multiple initial facial motion trajectories are obtained after recognizing all original video frames included in the original video, it can be determined whether there is any overlap between these initial facial motion trajectories using methods A1-A5. The overlapping initial facial motion trajectories are then merged to obtain the facial motion trajectory corresponding to each original face in the original video. Specifically, A1 should refer to obtaining multiple initial facial motion trajectories based on the multiple original video frames included in the original video. At this point, the initial facial motion trajectories are unmerged facial motion trajectories and can be directly obtained from the multiple original video frames included in the original video.

[0175] Therefore, if the original video uses camera movement techniques such as cuts and splits, resulting in continuous storylines being distributed across multiple original video frames with large time intervals, although the first interval threshold is used to divide them into multiple initial facial motion trajectories, these initial facial motion trajectories can be merged through merging. That is, initial facial motion trajectories with the same average facial features can be merged, and initial facial motions describing the same original face can be merged to obtain facial motion trajectories. This can maintain the aforementioned continuity, expand application scenarios, and improve the accuracy of subsequent target area determination.

[0176] Method 2: Merge facial movement trajectories that are similar in location and occur at short intervals.

[0177] B1: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames.

[0178] See A1 above for details.

[0179] B2: If the time interval between the first associated video frame and the second associated video frame is less than the second interval threshold, and the difference between the position of the historical face in the first associated video frame and the position of the historical face in the second associated video frame is less than the position threshold, then the first initial facial motion trajectory and the second initial facial motion trajectory are merged.

[0180] Wherein, the first associated video frame is the last original video frame among a plurality of original video frames associated with the first initial facial motion trajectory, and the second associated video frame is the first original video frame among a plurality of original video frames associated with the second initial facial motion trajectory. The first initial facial motion trajectory is one of a plurality of initial facial motion trajectories, and the second initial facial motion trajectory is one of a plurality of initial facial motion trajectories; the first initial facial motion trajectory and the second initial facial motion trajectory are different initial facial motion trajectories.

[0181] This application does not specifically limit the first interval threshold and the second interval threshold; those skilled in the art can set them according to actual needs. For example, the first interval threshold is the duration corresponding to one original video frame, and the second interval threshold is the duration corresponding to five original video frames. Furthermore, the first interval threshold and the second interval threshold can be the same or different.

[0182] If the time interval between the first associated video frame and the second associated video frame is less than the second interval threshold, it indicates that the time interval between the first associated video frame and the second associated video frame is small. This may be because they were separated by camera movement techniques such as cuts and shot splits, but they may depict a continuous plot. However, since the first i-1 original video frames include multiple historical faces, each historical face will have one or more initial facial motion trajectories.

[0183] To ensure that the two initial facial motion trajectories describe a coherent scene of the same original face, the position of the historical face identified by the two initial facial motion trajectories can be determined. Specifically, the position of the historical face identified by the first initial facial motion trajectory in the first associated video frame and the position of the historical face identified by the second initial facial motion trajectory in the second associated video frame are used. If the difference between the two positions is less than the position threshold, it indicates that the historical face described by the two initial facial motion trajectories is the same original face. Then, the two initial facial motion trajectories are merged to obtain the merged facial motion trajectory, and then the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames are obtained.

[0184] Furthermore, to further reduce the computational load caused by the merging operation, all initial facial motion trajectories can be obtained first based on the original video, and multiple initial facial motion trajectories obtained based on the original video can be obtained. Then, the multiple initial facial motion trajectories obtained based on the original video can be merged at the same time, i.e., B2. After merging, the facial motion trajectories corresponding to each historical face included in the original video can be obtained, thereby avoiding multiple merging of two initial facial motion trajectories, reducing the computational load, improving the speed of video face swapping, and improving the user experience.

[0185] Therefore, if the original video uses camera movement techniques such as cuts and split shots, resulting in a continuous storyline distributed across multiple discontinuous original video frames, multiple initial facial motion trajectories can be merged through merging. That is, facial motion trajectories with similar positions and short time intervals can be merged. This allows two initial facial motion trajectories describing a coherent storyline of the same original face to be restored, thus obtaining the facial motion trajectory. This can maintain the aforementioned continuity, expand application scenarios, and improve the accuracy of subsequent target area determination.

[0186] As one possible approach, since the original video frames often show fleeting glimpses of the original face, such as passersby in street interview videos, subsequent replacement or calculation of this original face could result in a large computational load and poor stability of the video face swapping.

[0187] Based on this, unimportant original faces can be deleted, see C1-C2 for details.

[0188] C1: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames.

[0189] See A1 above for details.

[0190] C2: If the number of original video frames associated with the third initial facial motion trajectory is less than the video frame threshold, then delete the third initial facial motion trajectory and the original face corresponding to the third initial facial motion trajectory to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

[0191] The third initial facial motion trajectory is one of multiple initial facial motion trajectories. If the number of original video frames associated with the third initial facial motion trajectory is less than the video frame threshold, it indicates that the historical face corresponding to the third initial facial motion trajectory appeared for a short time in the first i-1 original video frames. It may be a fleeting original face or an unimportant face in the original video. Based on this, the third initial facial motion trajectory and the original face corresponding to the third initial facial motion trajectory can be used to treat each initial facial motion trajectory as the third initial facial motion trajectory. Executing C2, we can then obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

[0192] The embodiments of this application do not specifically limit the video frame threshold, and those skilled in the art can set it according to actual needs.

[0193] One possible implementation is to obtain multiple initial facial motion trajectories based on all original video frames included in the original video. If the number of original video frames associated with a third initial facial motion trajectory is less than a video frame threshold, then the third initial facial motion trajectory and the original face corresponding to the third initial facial motion trajectory are deleted, resulting in facial motion trajectories corresponding to each original face included in the original video. This allows for convenient and quick deletion of all original video frames.

[0194] Therefore, by determining whether the number of original video frames associated with the initial facial motion trajectory is less than the video frame threshold, it is possible to determine whether the original face corresponding to the initial facial motion trajectory is an unimportant face. This allows for the deletion of unimportant original faces from the original video, thereby reducing the number of original faces in subsequent processing, reducing computational load, increasing video face-swapping speed, and improving user experience.

[0195] As one possible implementation, multiple original video frames included in the original video can be identified to obtain r initial facial motion trajectories, where r is a positive integer. If the number of original video frames associated with the initial facial motion trajectory is less than the video frame threshold, the initial facial motion trajectory is deleted (see C1-C2), resulting in s initial facial motion trajectories, where s is a positive integer less than or equal to r.

[0196] If the time interval between the first associated video frame and the second associated video frame is less than the second interval threshold, and the difference between the position of the historical face in the first associated video frame and the position of the historical face in the second associated video frame is less than the position threshold, then the first initial facial motion trajectory and the second initial facial motion trajectory are merged for the first time (see B1-B2), thereby obtaining t initial facial motion trajectories based on s initial facial motion trajectories, where t is a positive integer less than s.

[0197] If the similarity of the average facial features of two initial facial motion trajectories among the t initial facial motion trajectories is greater than the similarity threshold, and the start and end times of the associated video frames of the two initial facial motion trajectories do not overlap, then the two initial facial motion trajectories are merged a second time (see A1-A5) to obtain the facial motion trajectory.

[0198] Therefore, by uniformly processing the initial facial motion trajectories included in the original video, through one deletion operation and two merging operations, not only can useless original faces be effectively reduced, improving the stability of video face swapping and reducing the computational load of subsequent steps, but also unqualified original faces can be filtered out, further reducing the computational load of subsequent steps.

[0199] As one possible implementation, this application provides a specific implementation of S506, namely, based on the indication of facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate the target video. See D1-D5 for details.

[0200] D1: Based on the facial correspondence indication, obtain the reference face and the target region of the i-th original video frame.

[0201] The facial correspondence is used to indicate which reference face is used to replace which target face. In this embodiment, we will continue to use one reference face and one target face as an example. The target region in the i-th original video frame identifies the position of the target face in the i-th original video frame.

[0202] D2: Align the target face in the target region of the i-th original video frame using affine transformation to obtain the adjusted face of the i-th original video frame.

[0203] Affine transformation, also known as affine mapping, is a geometric process where a linear transformation followed by a translation is applied to a vector space to transform it into another vector space. Specifically, affine transformations preserve the flatness and parallelism of an image; that is, after an affine transformation, straight lines remain straight lines, and parallel lines remain parallel lines. By performing operations such as translation, rotation, scaling, and shearing on an image through affine transformations, the position, orientation, and size of the image can be changed, while maintaining its basic shape and features.

[0204] This allows for the alignment of the target face within the target region of the i-th original video frame using an affine transformation. This aligns the facial key points of the adjusted face in the i-th original video frame with preset positions. In other words, the affine transformation aligns the facial key points (such as eyes, nose, and mouth) of the target face to designated positions, thus achieving spatial facial alignment. For example, the target region of the i-th original video frame can be cropped, and the cropped image can include the target face. An affine transformation can then be performed on the cropped image to obtain the aligned image.

[0205] Therefore, affine transformation can align facial images under different poses and lighting conditions to a standard position and size, thereby eliminating the influence of factors such as angle and lighting on facial feature extraction and improving the accuracy of face replacement. Furthermore, original video frames may be affected by factors such as camera, shooting angle, and lighting conditions, resulting in variations in the size, position, and angle of each original video frame. Affine transformation can transform these original video frames to a unified standard, facilitating subsequent standardized processing and analysis. In addition, in some cases, the original face may be affected by noise, blur, or other factors, leading to a deterioration in the quality of the original video frames. Affine transformation can be used to rotate and scale the original video frames to optimize their quality, making them more suitable for subsequent operations.

[0206] D3: Extract features from the reference face to obtain the identity features of the reference face.

[0207] Facial features include identity features and attribute features. Identity features describe who the face belongs to, while attribute features describe the facial characteristics, such as facial posture and facial expressions.

[0208] The embodiments of this application do not specifically limit the feature extraction method, such as feature extraction through a pre-trained network.

[0209] D4: Based on the identity features of the reference face and the adjusted face of the i-th original video frame, the target video frame is obtained by fusing the data through a generative adversarial network.

[0210] For example, the identity features of the reference face and the aforementioned aligned image (the image obtained through affine transformation) are input into a generative adversarial network. The generative adversarial network then fuses the identity features into the target region in the aligned image to obtain the target video frame for the i-th original video frame.

[0211] Generative Adversarial Networks (GANs) are deep learning models that consist of two neural networks: a generator and a discriminator. The generator aims to produce fake data that closely resembles real data, while the discriminator aims to accurately determine whether the input data is real or generated by the generator. During training, these two networks compete against each other, continuously optimizing their performance until a dynamic equilibrium is reached. Once training is complete, the generator can be used for face replacement to improve the accuracy of target video frames.

[0212] As one possible implementation, for example, non-facial regions of target video frames generated by generative adversarial networks are prone to artifacts. Based on this, facial features in the target region can be cropped, allowing only the cropped area to be replaced. The replaced area is then fused with the non-facial regions of the target region to obtain the target video frame. This reduces the impact on non-facial regions during the replacement process, improving the face-swapping effect.

[0213] For example, by adding a facial mask to the pixels included in the target region, a masked region and a non-masked region can be obtained. The masked region includes the initial pixels of the target region, and the non-masked region includes the initial pixels of the region other than the masked region. The initial pixels are pixels from the original video frame library. For example, by identifying the facial region of the target face through a masking network and retaining the initial pixels of the non-facial regions, the non-masked region is obtained, and then the masked region is obtained.

[0214] Furthermore, based on the identity features of the reference face and the adjusted face of the i-th original video frame, a generative adversarial network is used to generate multiple fused pixels. The image corresponding to the multiple fused pixels includes both the identity features of the reference face and the attribute features of the adjusted face. The initial pixels in the masking region are replaced with the multiple fused pixels, and the initial pixels and fused pixels in the boundary region are smoothed to obtain the target video frame for the i-th original video frame. Alternatively, the fused pixels and the non-masked region are fused, and the initial pixels and fused pixels in the boundary region are smoothed to obtain the target video frame for the i-th original video frame.

[0215] The intersection of the masked and unmasked regions forms the boundary line between the two regions. The boundary region is obtained based on this boundary line; for example, three pixels can be obtained from the left and three pixels from the right side of the boundary line, resulting in a boundary region composed of multiple obtained pixels. Smoothing can be achieved by sampling a weighted weight from a Gaussian function, thus enabling a smooth transition between the masked and unmasked regions.

[0216] In addition, the presence of bangs or other facial features may cause irregularities in the facial contours. Therefore, compared to the aforementioned cropping methods, pixel-level precision facial masking can more accurately cover the facial area, thereby reducing the impact on non-facial areas during the replacement process and improving the video face-swapping effect.

[0217] Therefore, by using pixel-level precision facial masking, masked and unmasked regions can be obtained. For masked regions, fused pixels output by a generative adversarial network are used. For unmasked regions, the initial pixels of the original video frame are used. For the boundary between masked and unmasked regions, the initial pixels and fused pixels are smoothed to make them blend better, thereby achieving a smooth transition and obtaining a target video frame with high accuracy, avoiding artifacts in non-facial areas.

[0218] D5: Generate the target video based on the target video frame of the i-th original video frame.

[0219] For example, each original video frame included in the original video is taken as the i-th original video frame, and the target video frame corresponding to each original video frame is obtained, thereby obtaining the target video for achieving video face swapping.

[0220] Therefore, by combining affine transformation and generative adversarial networks (GANs), target faces in the original video frames can be replaced. Specifically, the target face is adjusted through affine transformation to obtain an adjusted face, thereby improving the accuracy, standard, and quality of subsequent face replacement. The fusion with GANs further enhances the accuracy of face replacement, specifically increasing the similarity between the successfully replaced face in the target video frame and the reference face, thus improving the user experience.

[0221] As one possible approach to improve the processing speed of video face swapping, a lightweight generative adversarial network can be used, and the images can be processed at a lower resolution, which will be explained in detail below.

[0222] See Figure 8 The figure is a flowchart illustrating the video generation method provided in an embodiment of this application.

[0223] S801: Obtain a reference image including the reference face.

[0224] S802: Detect the reference image to obtain a sub-image that includes the reference face.

[0225] If the obtained reference image is 1024×1024, the reference face only occupies a part of the reference image. Therefore, the reference image is detected to obtain the part of the reference face that it occupies, resulting in a sub-image. This sub-image includes the reference face and has a resolution of 128×128.

[0226] The embodiments of this application do not specifically limit the preset resolution threshold. Those skilled in the art can set it according to actual needs, such as setting the preset resolution threshold to 128, etc.

[0227] S803: Obtain facial correspondence and original video.

[0228] For example, a target face can be specified, thereby obtaining a facial correspondence based on a reference face and the target face.

[0229] S804: Obtain the facial motion trajectory corresponding to each historical face in the first i-1 original video frames.

[0230] S805: Based on multiple facial motion trajectories, predict the prediction region corresponding to each historical face in the i-th original video frame.

[0231] S806: Detect the i-th original video frame to obtain the detection region of the target face in the i-th original video frame.

[0232] Understandably, the resolution of the detection area can be equal to the resolution of the sub-image to facilitate subsequent replacement.

[0233] S807: Based on the detected region and multiple predicted regions, determine the target region of the target face in the i-th original video frame.

[0234] S803-S807 can be found in the aforementioned S501-S505.

[0235] S808: Based on the indication of facial correspondence, obtain a sub-image including the reference face and the target region of the i-th original video frame.

[0236] Since facial correspondence can indicate which reference face replaces which target face, the following example of replacing a target face with a reference face is used to obtain a sub-image including the reference face and the target region including the target face.

[0237] S809: Extract features from the sub-image including the reference face to obtain the identity features of the reference face.

[0238] For details, please refer to S5063, which will not be repeated here.

[0239] S810: Based on the identity features of the reference face and the target region of the i-th original video, a generative adversarial network is used to fuse them to obtain a fused sub-image.

[0240] See S5064 for details. The difference is that in S5064, the output of the generative adversarial network is the target video frame for the i-th original video frame, while in S810, the output of the generative adversarial network is the fused sub-image.

[0241] Furthermore, in S810, the number of parameters in the generative adversarial network is less than the parameter threshold, meaning a lightweight generative adversarial network is used to fuse the reference face and the target face. The fused sub-image includes not only the identity features of the reference face but also the attribute features of the target face. The resolution of the fused sub-image is equal to the resolution of the reference face, which facilitates the subsequent acquisition of the target video frame, making the process convenient and fast.

[0242] It should be noted that although the Generative Adversarial Network (GAN) used has a low parameter count, which can improve the speed of video face-swapping, the detection of the original face based on facial motion trajectories is highly stable, resulting in high robustness of the entire video face-swapping process. Therefore, the fusion accuracy requirements of the GAN are not high, thus improving the speed of video face-swapping while maintaining accuracy.

[0243] S811: Based on the fused sub-image, obtain the target video frame of the i-th original video frame.

[0244] See Figure 9 This figure is a schematic diagram of a video face-swapping method provided in an embodiment of this application. Figure 9 In this process, the i-th original video frame can be cropped to obtain the target region, including the target face, and the remaining image excluding the target region. It's understandable that the image is enlarged for illustrative purposes. Figure 9 The target region in the image has a resolution of 128×128, while the resolution of the i-th original video frame is 1024×1024. The fused sub-image integrates the identity features of the reference face into the target region; the resolution of the fused sub-image is the same as that of the target region, i.e., both can be 120×128. Finally, the fused sub-image is fused with the remaining images to obtain the target video frame for the i-th original video frame.

[0245] S812: Generate the target video based on the target video frame of the i-th original video frame.

[0246] For example, each original video frame included in the original video is taken as the i-th original video frame, thereby obtaining the target video frame for each original video frame, and thus obtaining the target video.

[0247] Therefore, by using low-resolution images to perform video face-swapping, the speed of video face-swapping can be improved. For example, performing face-swapping at a resolution of 128×128 can achieve a speed of 120 FPS (frames per second). Moreover, combining the advantages of generative adversarial networks can improve facial similarity. Furthermore, using adversarial generative networks with fewer parameters can further improve the speed of video face-swapping and enhance the user experience.

[0248] As one possible approach, since the skin color of the reference face and the skin color of the target face may be different, the skin color of the face after face swapping will be different from that of the target face, resulting in poor fusion and a poor quality target video, such as overly obvious fusion marks.

[0249] Based on this, the embodiments of this application further process the skin color to improve the blending effect, as detailed in S8111-S8113.

[0250] S8111: Obtain the mean hue and mean luminance of the i-th original video frame.

[0251] As one possible implementation, the i-th original video frame can be transformed into a color (Hue, Saturation, Value, HSV) space to calculate the average hue and the average luminance of the i-th original video frame.

[0252] S8112: Adjust the hue of the fused sub-image to the average hue of the i-th original video frame, and adjust the brightness of the fused sub-image to the average brightness of the i-th original video frame to obtain the adjusted sub-image.

[0253] As one possible implementation, a global linear transformation is performed on the fused sub-image, so that the mean hue of the resulting adjusted sub-image is consistent with the mean hue of the i-th original video frame, and the mean luminance of the adjusted sub-image is consistent with the mean luminance of the i-th original video frame.

[0254] Therefore, by adjusting the mean hue and mean luminance of the fused sub-image, the resulting adjusted sub-image is made consistent with the mean hue and mean luminance of the i-th original video frame, so that the skin color in the subsequent target video frame obtained based on the adjusted sub-image remains consistent.

[0255] S8113: Based on the adjusted sub-image, obtain the target video frame of the i-th original video frame.

[0256] Therefore, based on the mean hue and mean luminance of the i-th original video frame, the mean hue and mean luminance of the fused sub-image are adjusted, ensuring consistent skin color across the faces in the resulting target video frame and improving the face-swapping effect. Furthermore, adjusting the mean hue and mean luminance is convenient and quick, increasing the speed of face-swapping.

[0257] One possible approach is to enhance the facial region to improve facial clarity and even the accuracy of feature extraction (if applicable), thereby improving the effect of video face-swapping. This will be explained in detail below; please refer to E1-E5 for specific details.

[0258] E1: Obtain the target resolution of the original video.

[0259] The target resolution is the resolution of the original video. Since the resolutions of all the original video frames are usually equal, the target resolution of the original video can be determined based on the resolution of any one of the original video frames, thus reducing computational complexity by determining the resolution only once.

[0260] As one possible approach, if the original video contains multiple original video frames with different resolutions, the resolution of each original video frame can be determined during the replacement process, thereby performing face swapping on each original video frame based on its resolution, thus improving the accuracy of video face swapping.

[0261] E2: Determine the enhancement intensity of the reference face based on the target resolution.

[0262] As mentioned above, the facial resolution in the target video frame differs from that in the original video frame, resulting in issues such as excessive sharpness or excessive blurriness, leading to poor face-swapping results. For example, enhancing the resolution of a low-resolution reference face may cause local inconsistencies in the target video frame due to excessive sharpness.

[0263] Based on this, the embodiments of this application determine the enhancement intensity of the reference face based on the target resolution, that is, adjust the enhancement intensity of the reference face based on the resolution of the original video frame, so that the enhancement intensity is adapted to the clarity of the original video, thereby avoiding the problem of poor video face-swapping effect caused by local inconsistency.

[0264] As one possible approach, the correspondence between the target resolution and the enhancement intensity can be established in advance. The following explanation uses a piecewise function obtained from an experiment as an example. In this experiment, the resolution of the original video frame was 512×512.

[0265] If the target resolution is less than or equal to the first resolution threshold, the enhancement intensity is the first intensity value;

[0266] If the target resolution is greater than the first resolution threshold and less than or equal to the second resolution threshold, the enhancement intensity is the second intensity value. The first resolution threshold is less than the second resolution threshold. As the target resolution increases, the second intensity value increases.

[0267] If the target resolution is greater than the second resolution threshold, the enhancement intensity is the third intensity threshold, which is greater than the first intensity threshold.

[0268] See Figure 10 This figure is a schematic diagram illustrating the correspondence between target resolution and enhancement intensity provided in an embodiment of this application. Figure 10 In this context, the correspondence between target resolution and enhancement intensity is a piecewise function, which can be expressed as follows:

[0269]

[0270] Where y is the enhancement intensity, x is the target resolution, the first resolution threshold is 64, and the second resolution threshold is 384.

[0271] E3: Based on the indication of facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame.

[0272] E4: Based on the enhancement intensity of the reference face, enhance the undetermined video frame for the i-th original video frame to obtain the target video frame for the i-th original video frame.

[0273] By applying an enhancement intensity suitable for the original video, the faces included in the target video frame are enhanced, thereby reducing the problem of local inconsistencies in the faces in the target video frame.

[0274] E5: Generate the target video based on the target video frame for the i-th original video frame.

[0275] For example, each original video frame included in the original video is taken as the i-th original video frame, and the target video frame corresponding to each original video frame is obtained, thus obtaining the target video.

[0276] Therefore, compared to using the same enhancement intensity to enhance the clarity of different reference faces, which leads to local inconsistencies in the target video frame, in this embodiment, the enhancement intensity suitable for the target video frame is determined based on the target resolution of the original video. That is, the enhancement intensity is adaptively adjusted based on the original video, so that the enhanced reference face based on the enhancement intensity is more suitable for the target video frame, and the subsequent fusion is more natural, thus improving the effect of video face swapping.

[0277] As one possible approach, in some scenarios, the eyebrows in the target video frame may exhibit significant distortion, resulting in poor video face-swapping effects.

[0278] Research has revealed that due to factors such as hats and bangs obscuring the face, the reference face may not reveal its complete features, especially the eyebrows, which are frequently obscured in practical applications. This lack of distinctive eyebrow features leads to significant distortion of the eyebrows in the target video frame. In other words, the eyebrow distortion is caused by the occlusion of the eyebrows by the reference face. Based on this, the target video frame can be adjusted according to the proportion of eyebrow occlusion to make the generated target video more natural. This will be explained in detail below; please refer to F1-F4 for specific instructions.

[0279] F1: Based on the indication of facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame.

[0280] F2: Determine the size of the eyebrow area based on the pending video frames.

[0281] For example, a first facial semantic segmentation network can be used to identify a given video frame and determine whether each pixel is a pixel used to identify eyebrows, thereby obtaining the eyebrow region composed of pixels used to identify eyebrows.

[0282] The size of the eyebrow region can be determined based on the size of the image or the number of pixels it includes; this application does not impose any specific limitations on this.

[0283] F3: If the size of the eyebrow region is smaller than the eyebrow threshold, then replace the eyebrow of the pending video frame with the eyebrow of the target face to obtain the target video frame for the i-th original video frame.

[0284] If the size of the eyebrow area is smaller than the eyebrow threshold, it means that the eyebrow may be obscured, and the facial identity features may lack the identity features of the eyebrow, resulting in obvious distortion of the eyebrow in the target video frame.

[0285] Based on this, if the size of the eyebrow region is smaller than the eyebrow threshold, the eyebrow in the undetermined video frame is replaced with the eyebrow of the target face. This is equivalent to retaining the original eyebrow in the i-th original video frame to maximize the preservation of the eyebrow without distortion, thereby obtaining the target video for the i-th original video frame.

[0286] This application does not specifically limit the method of obtaining the eyebrows of the target face. For example, the i-th original video frame can be identified by the aforementioned first facial semantic segmentation network to determine whether each pixel is a pixel used to identify eyebrows, thereby obtaining the eyebrow region in the i-th original video frame, that is, the region where the eyebrows of the target face are located.

[0287] This application does not specifically limit the eyebrow threshold. The eyebrow threshold can be set based on the size of the eyebrow region, so as to determine whether the eyebrows in the image to be determined are occluded. For example, the eyebrow threshold can be obtained based on the ratio of the number of pixels in the eyebrow to the number of pixels in the image to be determined. Alternatively, the number of pixels included in the eyebrow can be directly determined as the eyebrow threshold.

[0288] F4: Generate the target video based on the target video frame of the i-th original video frame.

[0289] For example, each original video frame included in the original video is taken as the i-th original video frame, thereby obtaining the target video frame corresponding to each original video frame, and thus obtaining the target video.

[0290] Therefore, by determining whether the eyebrows in the image to be determined are occluded through the eyebrow region, it is decided whether to use the eyebrows of the target face for replacement. If the probability of the eyebrows being occluded is high (i.e., the size of the eyebrow region is smaller than the eyebrow threshold), then the eyebrows of the target face are used to replace the eyebrows in the image to be determined; if the probability of the eyebrows being occluded is low, then the eyebrows of the reference face are used, thereby further avoiding the situation where occluded eyebrows cannot provide sufficient identity features, leading to eyebrow distortion, and thus improving the effect of video face swapping.

[0291] As one possible approach, if the reference face has a lot of bangs, the bangs might be mistakenly identified as eyebrows, leading to distortion at the eyebrow area in the target video frame (referred to as eyebrow distortion). Therefore, the size of not only the eyebrow region but also the bangs region can be considered; see [link to details].

[0292] G1: Based on the indication of facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame.

[0293] The pending video frames include the identity features of the reference face and the attribute features of the target face.

[0294] G2: Determine the size of the eyebrow area based on the pending video frames, and determine the size of the bangs area based on the pending video frames.

[0295] The method for determining the size of the eyebrow area based on the image to be determined can be found in F2, and will not be repeated here.

[0296] For example, a second facial semantic segmentation network can be used to identify a given video frame, determining whether each pixel is a pixel used to identify the bangs, thereby obtaining the bangs region composed of the pixels used to identify the bangs. The second facial semantic segmentation network and the first facial semantic segmentation network are different networks.

[0297] The size of the bangs area can be determined based on the size of the image or the number of pixels it includes; this application does not impose any specific limitations on this.

[0298] G3: If the size of the eyebrow region is smaller than the eyebrow threshold and the size of the bangs region is larger than the bangs threshold, then replace the eyebrow of the undetermined video frame with the eyebrow of the target face to obtain the target video frame for the i-th original video frame.

[0299] If the size of the eyebrow region is smaller than the eyebrow threshold, it indicates that the eyebrows may be obscured, meaning the reference facial features may lack eyebrow identification features, resulting in significant distortion of the eyebrows in the target video frame. If the size of the bangs region is larger than the bangs threshold, it indicates that the reference face has a lot of bangs, which may obscure the eyebrows or be more likely to be identified as eyebrows. Meeting both conditions further increases the likelihood of eyebrow obscuration or misidentification.

[0300] Based on this, if the size of the eyebrow region is smaller than the eyebrow threshold and the size of the bangs region is larger than the bangs threshold, then the eyebrows in the undetermined video frame are replaced with the eyebrows of the target face. This is equivalent to retaining the original eyebrows in the i-th original video frame to maximize the preservation of the eyebrows without distortion, thereby obtaining the target video for the i-th original video frame.

[0301] This application does not specifically limit the method of obtaining the bangs of the target face. For example, the i-th original video frame can be identified by the aforementioned second facial semantic segmentation network to determine whether each pixel is a pixel used to identify the bangs, thereby obtaining the bangs region in the i-th original video frame, that is, the region where the bangs of the target face are located.

[0302] This application does not specifically limit the eyebrow threshold (or bangs threshold). The eyebrow threshold (or bangs threshold) can be set based on the size of the eyebrow region (or bangs region) to determine whether the eyebrows in the image to be determined are occluded. For example, the eyebrow threshold (or bangs threshold) can be obtained based on the ratio of the number of pixels in the eyebrow (or bangs) to the number of pixels in the image to be determined. Alternatively, the number of pixels included in the eyebrow (or bangs) can be directly determined as the eyebrow threshold (or bangs threshold).

[0303] G4: Generate the target video based on the target video frame of the i-th original video frame.

[0304] For example, each original video frame included in the original video is taken as the i-th original video frame, thereby obtaining the target video frame corresponding to each original video frame, and thus obtaining the target video.

[0305] Therefore, by using the eyebrow region and the bangs region to determine whether the eyebrows in the image to be determined are occluded, it is decided whether to use the eyebrows of the target face for replacement. If the probability of the eyebrows being occluded is high (i.e., the size of the eyebrow region is smaller than the eyebrow threshold, and the size of the bangs region is larger than the bangs threshold), then the eyebrows of the target face are used to replace the eyebrows in the image to be determined; if the probability of the eyebrows being occluded is low, then the eyebrows of the reference face are used, thereby further avoiding the situation where occluded eyebrows cannot provide sufficient identity features, leading to eyebrow distortion, and thus improving the effect of video face swapping.

[0306] To facilitate a further understanding of the technical solutions provided in the embodiments of this application, the following description takes the execution subject of the video generation method provided in the embodiments of this application as a server as an example, and provides an overall exemplary introduction to the video generation method.

[0307] Among related technologies, face-swapping for videos still has certain limitations in terms of real-time performance and stability. For example, in application scenarios such as partial facial occlusion, multi-person face-swapping, and the sequential appearance of multiple faces, it is difficult to achieve stable and consistent face-swapping, and users may perceive flickering. For high-resolution face-swapping, as facial resolution increases and the number of parameters in the neural network increases, the computational load is large, making it difficult to guarantee real-time performance.

[0308] Based on this, embodiments of this application provide a video generation method that can guarantee real-time performance and stability. This video generation method includes three stages: a face detection stage, a face-swapping stage, and a face optimization stage. It can be understood that this video face-swapping is based on a given reference face, editing the target face in the original video so that the identity features of the edited face are similar to the reference face, but the facial expressions and movements remain consistent with the target face.

[0309] See Figure 11This figure is a schematic diagram of a video generation method provided in an embodiment of this application. S1101-S1109 belong to the face detection stage, S1110 to the face swapping stage, and S1111-S1114 to the face optimization stage. These will be described separately below.

[0310] S1101: Obtain the reference image.

[0311] The reference image includes a reference face, and one or more reference faces can be obtained based on one or more reference images.

[0312] S1102: Facial detection.

[0313] The face detection in this step involves detecting each reference face in the reference image. Subsequently, the i-th original video frame in the original video will be used as an example to detect the reference image using face detection networks such as YOLOFACE, resulting in a detection box. Each detection box identifies the reference face in the reference image.

[0314] S1103: Facial enhancement.

[0315] The face enhancement step refers to enhancing the reference face, specifically the reference face within the detection bounding box in the reference image, to obtain the enhanced reference face. Face enhancement improves the clarity of the reference face and the accuracy of subsequent feature extraction, thus avoiding low similarity caused by a blurred reference face.

[0316] S1104: Feature extraction.

[0317] Feature extraction is performed on the enhanced reference face to obtain the identity features of the reference face, and these identity features are stored for later use.

[0318] It is understood that processes S1101-S1104 and S1105-S1108 can be executed in parallel or sequentially, and this application embodiment does not specifically limit this.

[0319] S1105: Obtain the original video.

[0320] The original video consists of multiple original video frames. To adapt to more complex application scenarios, a tracking detection method is added after face detection. The SORT algorithm is used as an example below.

[0321] S1106: Facial detection.

[0322] The face detection in this step is to detect faces in each original video frame. Taking the i-th original video frame as an example, face detection networks such as YOLOFACE are used to detect each original face in the i-th original video frame, resulting in multiple detection boxes. Each detection box is used to identify the detection region of an original face, and the number of original faces can be zero or one or more.

[0323] S1107: Follow-up detection.

[0324] For each detected original face, the original face is first enhanced using techniques such as GFPGAN (Generative Face ProgressiveGAN, an image inpainting and enhancement technique based on generative adversarial networks) to improve facial clarity and the accuracy of feature extraction.

[0325] Then, the SORT algorithm is used to track and detect the original faces in the original video, that is, to obtain the target regions corresponding to each target face in the i-th original video frame based on the facial motion trajectory. Specifically, Kalman filtering is used to propagate the original faces to the next original video frame to obtain the predicted regions of each original face in the next original video frame. Then, the intersection over union (IOU) ratio of each predicted region and each detected region is calculated, which is used as a measure of overlap. Finally, the Hungarian algorithm is used to determine the predicted regions corresponding to each target face in the i-th original video frame, thus obtaining the target regions corresponding to each target face in the i-th original video frame. See S702-S708 for details.

[0326] Furthermore, based on the facial motion trajectory of each historical face and the time interval between the last original video frame and the i-th original video frame among multiple original video frames associated with that facial motion trajectory, the prediction region of each historical face in the i-th original video frame can be predicted. Specifically, for the target facial motion trajectory among multiple facial motion trajectories, if the time interval between the last original video frame associated with the target facial motion trajectory and the i-th original video frame is less than or equal to a first interval threshold, then based on the target facial motion trajectory, the prediction region corresponding to the historical face in the i-th original video frame is predicted. By using multiple facial motion trajectories as the target facial motion trajectory, the prediction regions corresponding to each historical face in the i-th original video frame are obtained.

[0327] As one possible implementation, to better detect facial motion trajectories, a trajectory identifier can be set for each facial motion trajectory, and recognition can be performed one by one based on the trajectory identifier. Kalman filtering can then be used to update each facial motion trajectory, i.e., updating one trajectory at a time based on the trajectory identifier. The following section will combine... Figure 12 Please provide an explanation.

[0328] See Figure 12 The figure is a schematic diagram of facial motion trajectory detection provided in an embodiment of this application.

[0329] S1201: Obtain the detection regions of each original face in the i-th original video frame.

[0330] S1202: Obtain the facial motion trajectory of each original face in the first i-1 original video frames.

[0331] S1203: Based on the facial motion trajectories of each original face in the first i-1 original video frames, predict the prediction region of each original face in the i-th original video frame.

[0332] S1204: Calculate the cross-union ratio between each predicted region and each detected region.

[0333] S1205: Obtaining the optimal solution for multiple objectives based on the Hungarian matching algorithm.

[0334] S1206: If the multi-objective optimal solution indicates that there is no corresponding prediction region for the detection region, then create the facial motion trajectory corresponding to the detection region.

[0335] If the multi-objective optimal solution indicates that the detection area does not have a corresponding prediction area, that is, the overlap of multiple detection areas is less than the overlap threshold, then the original face corresponding to the detection area is the original face that appears in the i-th original video frame, and the facial motion trajectory corresponding to the original face is created.

[0336] S1207: If the multi-objective optimal solution indicates that the detection area has a corresponding prediction area, then update the facial motion trajectory corresponding to the detection area.

[0337] If the optimal solution for multiple targets indicates that the detection area has a corresponding prediction area, that is, at least one of the multiple overlap degrees of the detection area is greater than or equal to the overlap degree threshold, then the prediction area corresponding to the overlap degree that is greater than or equal to the overlap degree threshold and satisfies the overlap condition is determined as the prediction area corresponding to the detection area. Based on the prediction area corresponding to the detection area, the detection area is adjusted to obtain the target area of ​​the target face in the i-th original video frame, and the facial motion trajectory corresponding to the target face is updated based on the target area.

[0338] S1208: If the multi-objective optimal solution indicates that there is no corresponding detection region in the facial motion trajectory of each original face in the first i-1 original video frames, then prediction will no longer be performed for that facial motion trajectory.

[0339] For details related to S1201-S1208, please refer to S5051-S5054.

[0340] Furthermore, optimization can be performed on multiple facial motion trajectories obtained from the original video. For example, unimportant original faces can be removed from the original video using a C1-C2 method, reducing the computational load on the original faces. Finally, multiple facial motion trajectories are merged for the first time using a B1-B2 method, and then merged a second time using an A1-A5 method. Thus, through one deletion operation and two merging operations, not only can useless original faces be effectively reduced, improving the stability of video face swapping and reducing the computational load of subsequent steps, but also unqualified original faces can be filtered out, further reducing the computational load of subsequent steps.

[0341] S1108: Feature extraction.

[0342] Feature extraction is performed on the original faces corresponding to each facial movement trajectory to obtain facial key points, attribute features, etc. of each original face.

[0343] S1109: Facial matching.

[0344] The system can provide the user with images including the original face and reference faces, allowing the user to specify which reference face replaces which original face. This involves establishing a correspondence between the target face and reference faces based on facial mapping, specifying which reference face replaces the original face corresponding to which facial motion trajectory. Matching the facial features of the original and reference faces yields the facial motion trajectories for each reference face (these trajectories reflect the target region), enabling face swapping in video based on the trajectory identifiers of these motion trajectories.

[0345] Alternatively, instead of specifying the correspondence between reference faces and target faces, the user can determine the number of times each of the q original faces appears in the original video, and then identify the p original faces that appear the most frequently as target faces. A one-to-one correspondence is established between the p target faces and the p reference faces to obtain the face correspondence, and then the correspondence between the reference faces and the face motion trajectory is obtained. See Method 3 above for details.

[0346] S1110: Face replacement.

[0347] After obtaining the facial correspondence, the corresponding target face can be replaced using the reference face based on the indication of the facial correspondence. That is, based on the indication of the facial correspondence, the reference face is fused into the target region of the i-th original video frame to achieve replacement. For example, the i-th target video frame is obtained by replacing the corresponding target face with each reference face in the i-th original video frame using the D1-D4 method.

[0348] Furthermore, during the replacement process, pixel-level precision facial masking can be used, combined with smoothing processing, to avoid artifacts in non-facial areas.

[0349] In addition, face replacement can be performed using a low-resolution reference face, that is, face replacement is performed on a low-resolution image, and then the resolution of the replaced image is restored, thereby improving the speed of video face swapping. For relevant details, please refer to S808-S811.

[0350] S1111: Facial enhancement.

[0351] For details, refer to E1-E5 above. This enables adaptive adjustment of the resolution of each undetermined video frame based on the resolution of each original video frame, resulting in a target video frame with consistent resolution to avoid local inconsistencies.

[0352] S1112: Skin color correction.

[0353] See S8111-S8113 for details.

[0354] S1113: Eyebrow correction.

[0355] See F1-F4 or G1-G4 for details.

[0356] S1114: Generate the target video.

[0357] It should be noted that, except for facial motion trajectory detection which requires the temporal relationship of the original video, all other steps are independent of the individual original video frames. That is, not only can the various networks used be implemented in a lightweight manner, but each step that is independent of the temporal relationship of the original video frames can be processed in parallel. For example, after obtaining the correspondence between each original face and the reference face in the original video, each original video frame can be replaced separately based on multiple steps to obtain the target video frame corresponding to each original video frame, and thus obtain the target video.

[0358] It should be noted that although many additional steps have been added to ensure the quality and stability of video face swapping, the entire process can achieve high-quality, robust real-time video face swapping (20FPS) because most of the process is performed at low resolution and all support time-independent parallel processing.

[0359] In relation to the video generation method described above, this application also provides a corresponding video generation apparatus so that the above video generation method can be applied and implemented in practice.

[0360] See Figure 13 This figure is a schematic diagram of the structure of a video generation device provided in an embodiment of this application. Figure 13 As shown, the video generation device 1300 includes: an acquisition unit 1301, a prediction unit 1302, a detection unit 1303, a determination unit 1304, and a generation unit 1305;

[0361] The acquisition unit 1301 is used to acquire facial correspondence and original video. The facial correspondence is used to characterize a reference face for replacing the target face. The target face is the original face waiting to be replaced among a plurality of original faces included in the original video. The original video includes n original video frames, where n is a positive integer.

[0362] The acquisition unit 1301 is further configured to acquire the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in the multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among the multiple original faces, where i is a positive integer less than or equal to n.

[0363] The prediction unit 1302 is used to predict the prediction region corresponding to each of the historical faces in the i-th original video frame based on the multiple facial motion trajectories.

[0364] The detection unit 1303 is used to detect the i-th original video frame to obtain the detection area of ​​the target face in the i-th original video frame;

[0365] The determining unit 1304 is configured to determine the target region of the target face in the i-th original video frame based on the detection region and the plurality of predicted regions;

[0366] The generation unit 1305 is used to fuse the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence relationship, thereby generating a target video.

[0367] As can be seen from the above technical solution, the correspondence between the original video and the faces is obtained. This correspondence describes the relationship between the reference face and the target face, i.e., which reference face is used to replace which target face. Since not all original faces included in the original video will necessarily appear in the first i-1 original video frames, the original faces appearing in the first i-1 original video frames are called historical faces. The facial motion trajectory of each historical face is obtained, i.e., the change in its position across multiple original video frames. Due to the continuity between original video frames with short time intervals, the position change of the original face across multiple short-interval original video frames is continuous and smooth. Therefore, based on the facial motion trajectory of each historical face, the prediction region corresponding to each historical face in the i-th original video frame can be predicted, i.e., the possible position of each original face in the i-th original video frame. Thus, based on the continuity between multiple original video frames, stable detection of each original face is achieved to avoid facial jitter and make the face-swapping effect more natural. Detection is performed on the i-th original video frame to obtain the detection region of the target face in the i-th original video frame, i.e., the position of the target face in the original video frame. Based on multiple predicted regions and detection regions, the target region of the target face in the i-th original video frame is determined, thereby identifying a historical face that is similar to the target face from multiple historical faces, thus avoiding false detections or missed detections, improving detection accuracy, and improving the stability of face swapping. The correspondence between the reference face and the target face indicated by the facial correspondence is established, and the reference face is fused to the corresponding target region, thereby replacing the corresponding target face with the reference face to obtain the target video.

[0368] Therefore, for an original video containing multiple original faces, based on the fact that the positional changes of the face between multiple short-interval original video frames are continuous and smooth, it is possible to determine the original face that is more similar to the target face from multiple original faces by using the facial motion trajectory of each original face, thereby obtaining the target area that the reference face should replace, and using the reference face to replace the original face in the target area to obtain the target video, thereby improving the accuracy of face swapping and ensuring the stability of face swapping.

[0369] As one possible implementation, if there are multiple reference faces and multiple target faces, then the detection unit 1303 is specifically used for:

[0370] The i-th original video frame is detected to obtain the detection regions corresponding to each target face in the i-th original video frame;

[0371] The determining unit 1304 is specifically used for:

[0372] Determine the degree of overlap between each of the detection regions and each of the prediction regions;

[0373] Based on the multiple overlap degrees, a multi-objective optimal solution is determined, wherein the multi-objective optimal solution is the combination that maximizes the overlap degree between each detection region and each prediction region.

[0374] Based on the multi-objective optimal solution, the target regions corresponding to each of the target faces in the i-th original video frame are determined respectively;

[0375] The generation unit 1305 is specifically used for:

[0376] Based on the facial correspondence, a reference face corresponding to each of the target faces is determined, and the mapping relationship between the reference face and the target region is obtained.

[0377] Based on the mapping relationship, each of the reference faces indicated by the facial correspondence is fused into the corresponding target region in the i-th original video frame to generate the target video.

[0378] As one possible implementation, the determining unit 1304 is specifically used for:

[0379] Determine the degree of overlap between the detection region and the plurality of prediction regions;

[0380] If multiple overlap degrees are less than the overlap degree threshold, then the target face is the original face that appears newly in the i-th original video frame, and a facial motion trajectory corresponding to the target face is created.

[0381] If at least one of the multiple overlap degrees is greater than or equal to the overlap degree threshold, then the prediction region corresponding to the overlap degree that is greater than or equal to the overlap degree threshold and satisfies the overlap condition is determined as the prediction region corresponding to the detection region.

[0382] Based on the predicted region corresponding to the detection region, the detection region is adjusted to obtain the target region of the target face in the i-th original video frame, and the facial motion trajectory corresponding to the target face is updated based on the target region.

[0383] As one possible implementation, the prediction unit 1302 is specifically used for:

[0384] If the time interval between the last original video frame associated with the target facial motion trajectory and the i-th original video frame is less than or equal to a first interval threshold, then based on the target facial motion trajectory, the predicted region of the historical face corresponding to the target facial motion trajectory in the i-th original video frame is predicted, and the target facial motion trajectory is one of a plurality of facial motion trajectories;

[0385] By using the multiple facial motion trajectories as the target facial motion trajectory, the predicted regions corresponding to each historical face in the i-th original video frame are obtained.

[0386] As one possible implementation, the device further includes a trajectory optimization unit for:

[0387] Obtain multiple initial facial motion trajectories based on the first i-1 original video frames;

[0388] The start and end times of the associated video frames of the target's initial facial motion trajectory are obtained, as well as the facial features of the target's historical face in each of the associated video frames. The target's initial facial motion trajectory is one of a plurality of initial facial motion trajectories, the associated video frames are a plurality of original video frames associated with the target's initial facial motion trajectory, and the target's historical face is the historical face corresponding to the target's initial facial motion trajectory.

[0389] Based on the facial features of the target's historical face in each of the associated video frames, determine the average facial features corresponding to the target's initial facial motion trajectory;

[0390] The average facial features of each initial facial motion trajectory are obtained by taking the multiple initial facial motion trajectories as the target initial facial motion trajectory.

[0391] If the similarity of the average facial features of two initial facial motion trajectories among the multiple initial facial motion trajectories is greater than the similarity threshold, and the start and end times of the associated video frames of the two initial facial motion trajectories do not overlap, then the two initial facial motion trajectories are merged to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

[0392] As one possible implementation, the device further includes a trajectory optimization unit for:

[0393] Obtain multiple initial facial motion trajectories based on the first i-1 original video frames;

[0394] If the time interval between the first associated video frame and the second associated video frame is less than the second interval threshold, and the difference between the position of the historical face in the first associated video frame and the position of the historical face in the second associated video frame is less than the position threshold, then the first initial facial motion trajectory and the second initial facial motion trajectory are merged to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

[0395] Wherein, the first associated video frame is the last original video frame associated with the first initial facial motion trajectory, the second associated video frame is the first original video frame associated with the second initial facial motion trajectory, the first initial facial motion trajectory is one of a plurality of initial facial motion trajectories, and the second initial facial motion trajectory is one of a plurality of initial facial motion trajectories.

[0396] As one possible implementation, the acquisition unit 1301 is specifically used for:

[0397] Obtain multiple initial facial motion trajectories based on the first i-1 original video frames;

[0398] If the number of original video frames associated with the third initial facial motion trajectory is less than the video frame threshold, then the third initial facial motion trajectory and the original face corresponding to the third initial facial motion trajectory are deleted to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames. The third initial facial motion trajectory is one of the multiple initial facial motion trajectories.

[0399] As one possible implementation, the acquisition unit 1301 is further configured to:

[0400] Obtain p reference faces and q original faces, where p is a positive integer less than or equal to q, and q is a positive integer;

[0401] If an original face corresponding one-to-one with each of the reference faces is obtained, the face correspondence is obtained, and the original face is the target face;

[0402] If no original face is obtained that corresponds one-to-one with the reference face, then determine the number of times each of the q original faces appears in the original video, and establish a one-to-one correspondence between the p original faces that appear most frequently and the p reference faces to obtain the face correspondence relationship. Each of the p original faces is the target face.

[0403] As one possible implementation, the generation unit 1305 is specifically used for:

[0404] Based on the indication of the facial correspondence, the reference face and the target region of the i-th original video frame are obtained;

[0405] The target face in the target region of the i-th original video frame is aligned by affine transformation to obtain the adjusted face of the i-th original video frame. The facial key points of the adjusted face are aligned with preset positions.

[0406] Feature extraction is performed on the reference face to obtain the identity features of the reference face;

[0407] Based on the identity features of the reference face and the adjusted face of the i-th original video frame, a generative adversarial network is used to fuse them to obtain a target video frame for the i-th original video frame;

[0408] The target video is generated based on the target video frame of the i-th original video frame.

[0409] As one possible implementation, the device further includes a smoothing unit for:

[0410] A facial mask is added to the pixels included in the target area to obtain a masked area and a non-masked area;

[0411] The step of fusing the identity features of the reference face and the adjusted face of the i-th original video frame through a generative adversarial network to obtain a target video frame for the i-th original video frame includes:

[0412] Based on the identity features of the reference face and the adjusted face of the i-th original video frame, multiple fused pixels are generated through the generative adversarial network.

[0413] The initial pixels within the masked area are replaced by multiple fused pixels, and the initial pixels and fused pixels included in the boundary area are smoothed to obtain the target video frame for the i-th original video frame. The boundary area is the area determined based on the boundary line between the masked area and the unmasked area.

[0414] As one possible implementation, the acquisition unit 1301 is further configured to:

[0415] Obtain a reference image including the reference face;

[0416] The reference image is detected to obtain a sub-image including the reference face, wherein the resolution of the sub-image including the reference face is less than a preset resolution threshold;

[0417] The generation unit 1305 is specifically used for:

[0418] Based on the indication of the facial correspondence, obtain a sub-image including the reference face and the target region of the i-th original video frame;

[0419] Feature extraction is performed on the sub-image including the reference face to obtain the identity features of the reference face;

[0420] Based on the identity features of the reference face and the target region of the i-th original video, a generative adversarial network is used to fuse them to obtain a fused sub-image. The number of parameters of the generative adversarial network is less than the parameter threshold, and the resolution of the fused sub-image is the same as the resolution of the image including the reference face.

[0421] Based on the fused sub-image, the target video frame of the i-th original video frame is obtained;

[0422] The target video is generated based on the target video frame of the i-th original video frame.

[0423] As one possible implementation, the generation unit 1305 is specifically used for:

[0424] Obtain the average hue and average luminance of the i-th original video frame;

[0425] The hue of the fused sub-image is adjusted to the average hue of the i-th original video frame, and the brightness of the fused sub-image is adjusted to the average brightness of the i-th original video frame to obtain the adjusted sub-image.

[0426] Based on the adjusted sub-image, the target video frame of the i-th original video frame is obtained.

[0427] As one possible implementation, the acquisition unit 1301 is further configured to:

[0428] Obtain the target resolution of the original video;

[0429] The enhancement intensity of the reference face is determined based on the target resolution;

[0430] The generation unit 1305 is specifically used for:

[0431] Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame;

[0432] Based on the enhancement intensity of the reference face, the undetermined video frame for the i-th original video frame is enhanced to obtain the target video frame for the i-th original video frame.

[0433] The target video is generated based on the target video frame for the i-th original video frame.

[0434] As one possible implementation, the generation unit 1305 is specifically used for:

[0435] If the target resolution is less than or equal to the first resolution threshold, then the enhancement intensity is the first intensity value;

[0436] If the target resolution is greater than the first resolution threshold and less than or equal to the second resolution threshold, then the enhancement intensity is the second intensity value, the first resolution threshold is less than the second resolution threshold, and the second intensity value increases as the target resolution increases;

[0437] If the target resolution is greater than the second resolution threshold, then the enhancement intensity is a third intensity threshold, which is greater than the first intensity threshold.

[0438] As one possible implementation, the generation unit 1305 is specifically used for:

[0439] Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame;

[0440] The size of the eyebrow region is determined based on the undetermined video frames;

[0441] If the size of the eyebrow region is smaller than the eyebrow threshold, then the eyebrows of the undetermined video frame are replaced with the eyebrows of the target face to obtain the target video frame for the i-th original video frame.

[0442] The target video is generated based on the target video frame of the i-th original video frame.

[0443] As one possible implementation, the generation unit 1305 is specifically used for:

[0444] Based on the image to be determined, determine the size of the bangs area;

[0445] If the size of the eyebrow region is smaller than the eyebrow threshold and the size of the bangs region is larger than the bangs threshold, then the eyebrows of the undetermined image are replaced with the eyebrows of the target face to obtain the target video frame for the i-th original video frame.

[0446] This application also provides a computer device, which can be a server or a terminal device. The computer device provided in this application will be described below from a hardware implementation perspective. Figure 14 The diagram shown is a schematic of the server's structure. Figure 15 The diagram shown is a structural schematic of the terminal device.

[0447] See Figure 14 This figure is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1422, such as a central processing unit (CPU), memory 1432, and one or more application programs 1442 or data storage media 1430 (e.g., one or more mass storage devices). The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the server 1400.

[0448] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0449] The steps performed by the server in the above embodiments can be based on this Figure 14 The server structure shown.

[0450] The processor 1422 is used to perform the following steps:

[0451] Obtain facial correspondence and original video. The facial correspondence is used to characterize the reference face for replacing the target face. The target face is the original face waiting to be replaced among multiple original faces included in the original video. The original video includes n original video frames, where n is a positive integer.

[0452] Obtain the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among multiple original faces, where i is a positive integer less than or equal to n.

[0453] Based on the multiple facial motion trajectories, predict the prediction region corresponding to each of the historical faces in the i-th original video frame;

[0454] The i-th original video frame is detected to obtain the detection region of the target face in the i-th original video frame;

[0455] Based on the detection area and the multiple prediction areas, the target area of ​​the target face in the i-th original video frame is determined;

[0456] Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate the target video.

[0457] Optionally, the processor 1422 may also execute method steps of any specific implementation of the video generation method in the embodiments of this application.

[0458] See Figure 15 This figure is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The description will be based on a smartphone as an example. Figure 15 The diagram shown is a partial structural block diagram of the smartphone, which includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a Wi-Fi module 1570, a processor 1580, and a power supply 1590, among other components. Those skilled in the art will understand that... Figure 15 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0459] The following is combined Figure 15 A detailed introduction to the various components of a smartphone:

[0460] The RF circuit 1510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1580; in addition, it transmits uplink data to the base station.

[0461] The memory 1520 can be used to store software programs and modules, and the processor 1580 runs the software programs and modules stored in the memory 1520 to realize various functions and data processing of the smartphone.

[0462] Input unit 1530 can be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the smartphone. Specifically, input unit 1530 may include touch panel 1531 and other input devices 1532. Touch panel 1531, also known as a touch screen, can collect touch operations on or near the user and drive corresponding connected devices according to a pre-set program. In addition to touch panel 1531, input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0463] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0464] Smartphones may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be detailed here.

[0465] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, where speaker 1561 converts them into sound signals for output. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuit 1560, converted into audio data, and then processed by processor 1580 before being transmitted via RF circuit 1510 to, for example, another smartphone, or the audio data can be output to memory 1520 for further processing.

[0466] The processor 1580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1520, and by calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0467] The smartphone also includes a power supply 1590 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0468] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0469] In this embodiment of the application, the memory 1520 included in the smartphone can store computer programs and transmit the computer programs to the processor.

[0470] The processor 1580 included in the smartphone can execute the video generation method provided in the above embodiments according to the instructions in the computer program.

[0471] This application also provides a computer-readable storage medium for storing a computer program for executing the video generation method provided in the above embodiments.

[0472] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video generation method provided in the various optional implementations of the above aspects.

[0473] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0474] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0475] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0476] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0477] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video generation method, characterized in that, The method includes: Obtain facial correspondence and original video. The facial correspondence is used to characterize the reference face for replacing the target face. The target face is the original face waiting to be replaced among multiple original faces included in the original video. The original video includes n original video frames, where n is a positive integer. Obtain the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among multiple original faces, where i is a positive integer less than or equal to n. Based on the multiple facial motion trajectories, predict the prediction region corresponding to each of the historical faces in the i-th original video frame; The i-th original video frame is detected to obtain the detection region of the target face in the i-th original video frame; Based on the detection area and the multiple prediction areas, the target area of ​​the target face in the i-th original video frame is determined; Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate the target video.

2. The method according to claim 1, characterized in that, If there are multiple reference faces and multiple target faces, then the step of detecting the i-th original video frame to obtain the detection region of the target face in the i-th original video frame includes: The i-th original video frame is detected to obtain the detection regions corresponding to each target face in the i-th original video frame; Determining the target region of the target face in the i-th original video frame based on the detected region and multiple predicted regions includes: Determine the degree of overlap between each of the detection regions and each of the prediction regions; Based on the multiple overlap degrees, a multi-objective optimal solution is determined, wherein the multi-objective optimal solution is the combination that maximizes the overlap degree between each detection region and each prediction region. Based on the multi-objective optimal solution, the target regions corresponding to each of the target faces in the i-th original video frame are determined respectively; The step of fusing the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence to generate the target video includes: Based on the facial correspondence, a reference face corresponding to each of the target faces is determined, and the mapping relationship between the reference face and the target region is obtained. Based on the mapping relationship, each of the reference faces indicated by the facial correspondence is fused into the corresponding target region in the i-th original video frame to generate the target video.

3. The method according to claim 1, characterized in that, Determining the target region of the target face in the i-th original video frame based on the detected region and multiple predicted regions includes: Determine the degree of overlap between the detection region and the plurality of prediction regions; If multiple overlap degrees are less than the overlap degree threshold, then the target face is the original face that appears newly in the i-th original video frame, and a facial motion trajectory corresponding to the target face is created. If at least one of the multiple overlap degrees is greater than or equal to the overlap degree threshold, then the prediction region corresponding to the overlap degree that is greater than or equal to the overlap degree threshold and satisfies the overlap condition is determined as the prediction region corresponding to the detection region. Based on the predicted region corresponding to the detection region, the detection region is adjusted to obtain the target region of the target face in the i-th original video frame, and the facial motion trajectory corresponding to the target face is updated based on the target region.

4. The method according to claim 1, characterized in that, The step of predicting the prediction region corresponding to each of the historical faces in the i-th original video frame based on multiple facial motion trajectories includes: If the time interval between the last original video frame associated with the target facial motion trajectory and the i-th original video frame is less than or equal to a first interval threshold, then based on the target facial motion trajectory, the predicted region of the historical face corresponding to the target facial motion trajectory in the i-th original video frame is predicted, and the target facial motion trajectory is one of a plurality of facial motion trajectories; By using the multiple facial motion trajectories as the target facial motion trajectory, the predicted regions corresponding to each historical face in the i-th original video frame are obtained.

5. The method according to claim 4, characterized in that, The method further includes: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames; The start and end times of the associated video frames of the target's initial facial motion trajectory are obtained, as well as the facial features of the target's historical face in each of the associated video frames. The target's initial facial motion trajectory is one of a plurality of initial facial motion trajectories, the associated video frames are a plurality of original video frames associated with the target's initial facial motion trajectory, and the target's historical face is the historical face corresponding to the target's initial facial motion trajectory. Based on the facial features of the target's historical face in each of the associated video frames, determine the average facial features corresponding to the target's initial facial motion trajectory; The average facial features of each initial facial motion trajectory are obtained by taking the multiple initial facial motion trajectories as the target initial facial motion trajectory. If the similarity of the average facial features of two initial facial motion trajectories among the multiple initial facial motion trajectories is greater than the similarity threshold, and the start and end times of the associated video frames of the two initial facial motion trajectories do not overlap, then the two initial facial motion trajectories are merged to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames.

6. The method according to claim 4, characterized in that, The method further includes: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames; If the time interval between the first associated video frame and the second associated video frame is less than the second interval threshold, and the difference between the position of the historical face in the first associated video frame and the position of the historical face in the second associated video frame is less than the position threshold, then the first initial facial motion trajectory and the second initial facial motion trajectory are merged to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames. Wherein, the first associated video frame is the last original video frame associated with the first initial facial motion trajectory, the second associated video frame is the first original video frame associated with the second initial facial motion trajectory, the first initial facial motion trajectory is one of a plurality of initial facial motion trajectories, and the second initial facial motion trajectory is one of a plurality of initial facial motion trajectories.

7. The method according to claim 1, characterized in that, The acquisition of the facial motion trajectories corresponding to each historical face in the first i-1 original video frames includes: Obtain multiple initial facial motion trajectories based on the first i-1 original video frames; If the number of original video frames associated with the third initial facial motion trajectory is less than the video frame threshold, then the third initial facial motion trajectory and the original face corresponding to the third initial facial motion trajectory are deleted to obtain the facial motion trajectories corresponding to each historical face included in the first i-1 original video frames. The third initial facial motion trajectory is one of the multiple initial facial motion trajectories.

8. The method according to claim 1, characterized in that, The method further includes: Obtain p reference faces and q original faces, where p is a positive integer less than or equal to q, and q is a positive integer; If an original face corresponding one-to-one with each of the reference faces is obtained, the face correspondence is obtained, and the original face is the target face; If no original face is obtained that corresponds one-to-one with the reference face, then determine the number of times each of the q original faces appears in the original video, and establish a one-to-one correspondence between the p original faces that appear most frequently and the p reference faces to obtain the face correspondence relationship. Each of the p original faces is the target face.

9. The method according to claim 1, characterized in that, The step of fusing the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence to generate the target video includes: Based on the indication of the facial correspondence, the reference face and the target region of the i-th original video frame are obtained; The target face in the target region of the i-th original video frame is aligned by affine transformation to obtain the adjusted face of the i-th original video frame. The facial key points of the adjusted face are aligned with preset positions. Feature extraction is performed on the reference face to obtain the identity features of the reference face; Based on the identity features of the reference face and the adjusted face of the i-th original video frame, a generative adversarial network is used to fuse them to obtain a target video frame for the i-th original video frame; The target video is generated based on the target video frame of the i-th original video frame.

10. The method according to claim 9, characterized in that, The method further includes: A facial mask is added to the pixels included in the target area to obtain a masked area and a non-masked area; The step of fusing the identity features of the reference face and the adjusted face of the i-th original video frame through a generative adversarial network to obtain a target video frame for the i-th original video frame includes: Based on the identity features of the reference face and the adjusted face of the i-th original video frame, multiple fused pixels are generated through the generative adversarial network. The initial pixels within the masked area are replaced by multiple fused pixels, and the initial pixels and fused pixels included in the boundary area are smoothed to obtain the target video frame for the i-th original video frame. The boundary area is the area determined based on the boundary line between the masked area and the unmasked area.

11. The method according to claim 1, characterized in that, The method further includes: Obtain a reference image including the reference face; The reference image is detected to obtain a sub-image including the reference face, wherein the resolution of the sub-image including the reference face is less than a preset resolution threshold; The step of fusing the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence to generate the target video includes: Based on the indication of the facial correspondence, obtain a sub-image including the reference face and the target region of the i-th original video frame; Feature extraction is performed on the sub-image including the reference face to obtain the identity features of the reference face; Based on the identity features of the reference face and the target region of the i-th original video, a generative adversarial network is used to fuse them to obtain a fused sub-image. The number of parameters of the generative adversarial network is less than the parameter threshold, and the resolution of the fused sub-image is the same as the resolution of the image including the reference face. Based on the fused sub-image, the target video frame of the i-th original video frame is obtained; The target video is generated based on the target video frame of the i-th original video frame.

12. The method according to claim 11, characterized in that, The step of obtaining the target video frame of the i-th original video frame based on the fused sub-image includes: Obtain the average hue and average luminance of the i-th original video frame; The hue of the fused sub-image is adjusted to the average hue of the i-th original video frame, and the brightness of the fused sub-image is adjusted to the average brightness of the i-th original video frame to obtain the adjusted sub-image. Based on the adjusted sub-image, the target video frame of the i-th original video frame is obtained.

13. The method according to claim 1, characterized in that, The method further includes: Obtain the target resolution of the original video; The enhancement intensity of the reference face is determined based on the target resolution; The step of fusing the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence to generate the target video includes: Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame; Based on the enhancement intensity of the reference face, the undetermined video frame for the i-th original video frame is enhanced to obtain the target video frame for the i-th original video frame. The target video is generated based on the target video frame for the i-th original video frame.

14. The method according to claim 13, characterized in that, Determining the enhancement intensity of the reference face based on the target resolution includes: If the target resolution is less than or equal to the first resolution threshold, then the enhancement intensity is the first intensity value; If the target resolution is greater than the first resolution threshold and less than or equal to the second resolution threshold, then the enhancement intensity is the second intensity value, the first resolution threshold is less than the second resolution threshold, and the second intensity value increases as the target resolution increases; If the target resolution is greater than the second resolution threshold, then the enhancement intensity is a third intensity threshold, which is greater than the first intensity threshold.

15. The method according to claim 1, characterized in that, The step of fusing the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence to generate the target video includes: Based on the indication of the facial correspondence, the reference face is fused to the target region of the i-th original video frame to generate a pending video frame for the i-th original video frame; The size of the eyebrow region is determined based on the undetermined video frames; If the size of the eyebrow region is smaller than the eyebrow threshold, then the eyebrows of the undetermined video frame are replaced with the eyebrows of the target face to obtain the target video frame for the i-th original video frame. The target video is generated based on the target video frame of the i-th original video frame.

16. The method according to claim 15, characterized in that, The method further includes: Based on the image to be determined, determine the size of the bangs area; If the size of the eyebrow region is smaller than an eyebrow threshold, then the eyebrows of the undetermined image are replaced with the eyebrows of the target face to obtain the target video frame for the i-th original video frame, including: If the size of the eyebrow region is smaller than the eyebrow threshold and the size of the bangs region is larger than the bangs threshold, then the eyebrows of the undetermined image are replaced with the eyebrows of the target face to obtain the target video frame for the i-th original video frame.

17. A video generation apparatus, characterized in that, The device includes: an acquisition unit, a prediction unit, a detection unit, a determination unit, and a generation unit; The acquisition unit is used to acquire facial correspondence and original video. The facial correspondence is used to characterize the reference face for replacing the target face. The target face is the original face waiting to be replaced among multiple original faces included in the original video. The original video includes n original video frames, where n is a positive integer. The acquisition unit is further configured to acquire the facial motion trajectory corresponding to each historical face included in the first i-1 original video frames. The facial motion trajectory is used to describe the position change of the corresponding historical face in multiple original video frames. Different facial motion trajectories correspond to different historical faces. The historical face is the original face that appears in the first i-1 original video frames among multiple original faces, where i is a positive integer less than or equal to n. The prediction unit is used to predict the prediction region corresponding to each of the historical faces in the i-th original video frame based on the multiple facial motion trajectories. The detection unit is used to detect the i-th original video frame to obtain the detection area of ​​the target face in the i-th original video frame; The determining unit is configured to determine the target region of the target face in the i-th original video frame based on the detection region and the multiple predicted regions; The generation unit is used to fuse the reference face into the target region of the i-th original video frame based on the indication of the facial correspondence, thereby generating the target video.

18. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-16 according to the computer program.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-16.

20. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-16.

Images