Layer-based face replacement display method and face replacement display system

Abstract

The invention provides a layer-based face replacement display method and a layer-based face replacement display system. The method comprises the following steps: acquiring an original video; a virtual image layer is constructed, and the virtual image layer is used for displaying the replaced face image; under the condition that a face replacement demand is received, extracting an original face image in the original video, and replacing the extracted original face image with a target face image corresponding to the face replacement demand; the original video is displayed on the original image layer, the target face image is displayed on the virtual image layer, and the target face image of the virtual image layer covers the original face image of the original image layer. According to the scheme, the problem that the stability of the original video stream is poor due to the fact that the original video stream is processed by a face replacement scheme in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a layer-based face replacement display method, device, computer program product and face replacement display system. Background Art

[0002] In today's digital entertainment and live broadcasting, users want to be able to personalize and customize their viewing experience. Traditional face replacement solutions require full-frame processing in each frame of high-resolution video, including face detection, feature extraction, face replacement, and fusion. This method requires processing a large number of pixels, involves high computing power consumption, and requires modifying the video stream itself. Therefore, the face-changing software must intervene in the data transmission process of the video stream, which will cause poor stability of the original video stream. Summary of the invention

[0003] The main purpose of the present application is to provide a layer-based face replacement display method, device, computer program product and face replacement display system, so as to at least solve the problem that the face replacement scheme in the prior art processes the original video stream, resulting in poor stability of the original video stream.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a layer-based face replacement display method is provided, including: obtaining an original video; constructing a virtual layer, wherein the virtual layer is used to display a replaced face image; upon receiving a face replacement requirement, extracting the original face image in the original video, and replacing the extracted original face image with a target face image corresponding to the face replacement requirement; displaying the original video in the original layer, and displaying the target face image in the virtual layer, wherein the target face image of the virtual layer covers the original face image of the original layer.

[0005] Optionally, extracting the original face image in the original video includes: constructing a face recognition model, wherein the face recognition model is trained by a target recognition algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical original videos acquired within a historical time period, and historical original face images corresponding to the historical original videos, wherein the target recognition algorithm is one or more of MobileNet, SSD and YOLO; inputting the original video into the face recognition model to obtain the original face image corresponding to the original video.

[0006] Optionally, replacing the extracted original facial image with a target facial image corresponding to the face replacement requirement includes: using a face replacement tool to replace the extracted original facial image with the target facial image corresponding to the face replacement requirement, wherein the face replacement tool is DeepFaceLab or rope.

[0007] Optionally, after constructing the virtual layer, the method further includes: when the face replacement demand is not received, adjusting the transparency of the virtual layer to a first transparency, wherein when the transparency of the virtual layer is the first transparency, the virtual layer will not obscure the original layer; when the face replacement demand is received, adjusting the transparency of the virtual layer to a second transparency, wherein the first transparency is less than the second transparency, and when the transparency of the virtual layer is the second transparency, the virtual layer obscures the original layer.

[0008] Optionally, before constructing the virtual layer, the method also includes: in the case where there are multiple face replacement requirements, constructing multiple virtual layers, wherein the face replacement requirements and the virtual layers correspond one-to-one; according to the multiple face replacement requirements, replacing the original face image with multiple target face images corresponding to the multiple face replacement requirements, and displaying the corresponding target face images on all the virtual layers, wherein the face replacement requirements and the target face images correspond one-to-one.

[0009] Optionally, after constructing multiple virtual layers, the method also includes: obtaining a layer range corresponding to the virtual layer, wherein the layer range is an effective range of the face replacement layer for the original video; according to the layer range and the face replacement requirements, replacing the original face image with the target face image, and displaying the corresponding target face image on the virtual layer.

[0010] Optionally, after constructing multiple virtual layers, the method further includes: obtaining the priority corresponding to the virtual layer, wherein the priority is the display level between the multiple virtual layers; when at least two of the virtual layers are superimposed, displaying the virtual layers according to the priority, wherein the upper virtual layer obscures the lower virtual layer.

[0011] According to another aspect of the present application, a display device for layer-based face replacement is provided, including: a first acquisition unit, used to acquire an original video; a first construction unit, used to construct a virtual layer, wherein the virtual layer is used to display a replaced face image; a replacement unit, used to extract the original face image in the original video when a face replacement requirement is received, and replace the extracted original face image with a target face image corresponding to the face replacement requirement; a first display unit, used to display the original video in the original layer, and display the target face image in the virtual layer, wherein the target face image of the virtual layer covers the original face image of the original layer.

[0012] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any one of the layer-based face replacement display methods.

[0013] According to another aspect of the present application, a face replacement display system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a display method for executing any one of the layer-based face replacement methods.

[0014] By applying the technical solution of this application, a software layer (i.e., virtual layer) dedicated to face replacement is designed. The software layer will overlay the target face image after face replacement on the original face image. By superimposing the virtual layer, the virtual layer can process face replacement independently of the underlying content, so as to replace the face in a specific area without affecting other content. The software layer technology will not modify the data stream of the video source. The original video stream remains unchanged, so as to achieve face replacement without intervening in the original video stream, ensuring the stability of the original video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for performing a display method for layer-based face replacement provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic flow chart of a method for displaying face replacement based on layers provided according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic flow chart of another method for displaying face replacement based on layers is shown;

[0019] Figure 4 A structural block diagram of a display device for layer-based face replacement provided according to an embodiment of the present application is shown.

[0020] The above drawings include the following reference numerals:

[0021] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0026] Software layer: refers to a virtual layer superimposed on the display interface of a display device, which is used to independently process and display specific images or effects without affecting the underlying content.

[0027] Face replacement: Face replacement technology is a technology that replaces the original face in a video or image with a target face through an algorithm.

[0028] As introduced in the background technology, the face replacement scheme in the prior art processes the original video stream, resulting in poor stability of the original video stream. To solve the above problem, the embodiments of the present application provide a layer-based face replacement display method, device, computer program product and face replacement display system.

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal for displaying a method for face replacement based on layers according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] In this embodiment, a layer-based face replacement display method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 It is a flowchart of a method for displaying layer-based face replacement according to an embodiment of the present application.

[0034] like Figure 2 As shown, the method comprises the following steps:

[0035] Step S201, obtaining the original video;

[0036] Specifically, the raw video data without any processing can be obtained from the video source (such as network streaming media, local files, etc.). The video can be a live stream, a recorded video file, or any visual content containing a face. At this stage, the software does not make any modifications to the video content, but only obtains the video for subsequent processing.

[0037] Step S202, constructing a virtual layer, wherein the virtual layer is used to display the replaced face image;

[0038] Specifically, one or more virtual software layers are created on the video playback interface. These layers are independent of the original video stream. Their function is to display the face images processed by the algorithm in real time while the video is playing, rather than directly modifying the video stream itself. Each layer can independently process the faces of different people in the video, realizing multi-task parallel processing.

[0039] Step S203, when a face replacement requirement is received, extracting an original face image from the original video, and replacing the extracted original face image with a target face image corresponding to the face replacement requirement;

[0040] Specifically, when it is detected that the user has a need for face replacement (for example, the user selects the face to be replaced or uploads the target face image), the original video is analyzed. Using face detection and recognition technology, the image area containing the face is extracted from the video frame. Then, the extracted original face image is replaced with the target face image using a face replacement algorithm. This process involves matching facial features, simulating expressions, and blending with the background to ensure that the replaced face looks natural and consistent with the video background.

[0041] Step S204, displaying the original video on the original layer, and displaying the target face image on the virtual layer, wherein the target face image on the virtual layer covers the original face image on the original layer.

[0042] Specifically, on the video playback interface, the original video continues to play in the original layer, ensuring that the unprocessed part of the video is displayed normally. At the same time, the target face image that has been processed by face replacement is displayed on the constructed virtual layer. The virtual layer is designed to cover the original layer at a specific location (i.e., where the original face is located), so that in the video seen by the user, the original face is replaced by the target face image, while the other parts of the video content remain unchanged. The transparency and position of the virtual layer can be adjusted to ensure that the replaced face blends perfectly with the background and other elements of the original video stream.

[0043] Through this embodiment, a software layer (i.e., virtual layer) dedicated to face replacement is designed. The software layer will overlay the target face image after face replacement on the original face image. By superimposing the virtual layer, the virtual layer processes face replacement independently of the underlying content, and replaces the face of a specific area without affecting other content. The software layer technology will not modify the data stream of the video source. The original video stream remains unchanged, thereby achieving face replacement without intervening in the original video stream, ensuring good stability of the original video.

[0044] Specifically, traditional face replacement technology needs to be processed at the video source end, which involves directly modifying the video stream data, increasing the difficulty of implementation and maintenance costs, and may affect the stability of the video stream and the compatibility of existing software. This solution implements face replacement by adding a software layer (i.e., a virtual layer) on the terminal display interface, avoiding direct modification of the original video stream and improving flexibility and compatibility.

[0045] For high-resolution videos, processing face replacement for the entire frame will result in a lot of computing power consumption. This solution limits the layer processing range and only detects and replaces the face area defined by the layer, reducing the number of pixels that need to be processed, thereby reducing computing power consumption and improving processing efficiency.

[0046] In addition, the software layer-based implementation method seems to "move" the processing process to the display layer on the surface, but the core difference is that this solution does not directly modify or replace the data in the original video stream. In traditional solutions, face replacement occurs in the encoding or transmission stage of video data, which requires face-changing software or technology to directly intervene in the source of video data, process and replace the entire frame, and then re-encode or directly play the modified video stream. This operation may destroy the original format of the video stream, increase the encoding burden, and it is difficult to ensure real-time performance in scenarios such as live broadcasts.

[0047] In this solution, the key points to achieve the effect of "realizing face replacement without intervening in the original video stream" are:

[0048] 1. No modification of data stream: The software layer technology in this solution does not modify the data stream of the video source. The original video stream remains unchanged and is directly transmitted to the display device without any changes at the data level.

[0049] 2. Independent processing: The software layer is independent of the video stream processing, only capturing images from the display interface in real time, and then performing face detection, replacement and display on the virtual layer. This processing method will not affect the transmission and display of the video stream, and maintains the original quality and stability of the video.

[0050] 3. Layer overlay display: The replaced face image is only displayed on the software layer. Through the layer overlay, the face part in the original video frame is covered, while the underlying video stream remains unchanged. This means that video playback and transmission are not affected, and the display properties of the layer (such as position, size, transparency, etc.) can be flexibly adjusted.

[0051] 4. Real-time and efficiency: Since the software layer processes the real-time image of the display interface instead of each frame of the video source, more efficient algorithms and processing technologies can be used to reduce the amount of calculation and improve real-time processing capabilities. At the same time, the range of the layer can further limit the processing area and reduce the overall computing power consumption.

[0052] 5. Privacy protection and flexibility: Face replacement is achieved through software layers on local terminals. Data processing is completed locally and is not uploaded to the network or cloud server, protecting user privacy. At the same time, users can adjust the position of the software layer and the target face at any time to achieve a high degree of personalization and control.

[0053] In summary, this solution achieves real-time replacement of faces in videos by adding a software layer to the display interface of the terminal device. During the whole process, the original video stream remains unchanged and the video data is not directly modified. Therefore, it can achieve the effect of "realizing face replacement without intervening in the original video stream" while also providing higher flexibility and privacy protection.

[0054] Specifically, in order to improve efficiency and provide a more flexible solution, this solution proposes to add one or more software layers on the display device to achieve real-time replacement of detected faces without redrawing the entire screen. This method not only reduces computing power consumption, but also allows multiple layers to independently handle different face replacements, providing higher flexibility and personalized customization capabilities. This solution provides higher flexibility and privacy protection by using software layers on terminal devices without intervening in the image source data stream. At the same time, the solution can reduce computing power consumption by limiting the layer range and only detecting and replacing faces within the range of the layer, and can stack multiple layers to independently handle different face replacements, improving flexibility and personalized customization capabilities.

[0055] In addition, it should be noted that before using anyone's portrait for replacement, their explicit authorization and consent must be obtained. This includes the consent of the original person in the video and the consent of the person used as the replacement target. Face replacement will not be performed without proper authorization to ensure that the data and privacy of the person in the original video, the provider of the replaced face, and the end user are protected during the face replacement process. Avoid improper collection, use, and leakage of facial data.

[0056] Users should be aware of whether the video content has been processed with face replacement to avoid confusion or misleading. In some situations, it may be necessary to mark "This video has been processed with face replacement using AI technology" when the video is displayed.

[0057] In the technical implementation of this solution, the following mechanisms must be added to ensure legality and compliance:

[0058] (1) User Authorization System: Develop a system or interface that requires users to upload the portrait authorization of the person being replaced and the target before using the face replacement function. The system verifies the authorization information.

[0059] (2) Privacy protection mechanism: Ensure that all facial data is processed locally and not uploaded to the cloud. After processing, all temporary data is immediately cleared, including but not limited to facial images and recognition results.

[0060] (3) Warnings and prompts: Add obvious warnings and prompts to the software to inform users of the potential legal risks of face replacement and the terms of use that must be followed.

[0061] (4) Restricted functions: Limit the use of software in sensitive scenarios or for specific people to avoid possible legal disputes.

[0062] Therefore, all facial images in the present application, whether original facial images or target facial images, are facial images that have been authorized by the user. When a face replacement request is received, the face replacement request will include the facial image that needs to be replaced, and a prompt box will pop up on the interface to display whether the user is allowed to grant the right to use the portrait. If the user agrees, the original facial image will be extracted and replaced with the target facial image.

[0063] Of course, if it is a public figure or celebrity, you can make a judgment based on the facial image. If it is determined that the facial image is a public figure or celebrity, the interface will display "This is an image that does not allow face replacement" and then no face replacement will be performed.

[0064] Additionally, you can add irremovable watermarks or metadata to videos or images, marking that the content has been processed to increase transparency and prevent abuse.

[0065] When users register or use the face-swapping function for the first time, a detailed and clear user agreement should be provided, clearly stating the purpose, method, scope of use of facial images and how user privacy will be protected. The agreement should emphasize that facial images may not be used for other purposes or shared with third parties without the user's consent.

[0066] Design a dedicated authorization interface in the software, requiring users to confirm authorization before uploading facial images. The interface should be simple and clear, informing users that they are authorizing the software to use their facial images for face swapping, and clearly displaying the options to agree and refuse. Users must actively select "agree" to proceed to the next step.

[0067] In the authorization interface, provide authorization instructions, including but not limited to:

[0068] 1. Confirmation of portrait rights: Users must confirm that they have the right to upload and use the facial image. If the image contains the portrait of other people, they must provide corresponding authorization proof.

[0069] 2. Data usage restrictions: Users need to understand that their data will only be used for the requested face-swapping operation, and the image data will be deleted immediately after the operation is completed.

[0070] 3. Revocation of authorization: Inform users how to revoke authorization in the future and how the software will handle stored personal information and image data after revocation.

[0071] In high-risk or sensitive scenarios, a two-factor verification process can be used, such as SMS verification code, email confirmation, or biometrics (such as fingerprint, facial recognition) to further confirm the user's identity and authorization willingness.

[0072] Keep user authorization records, including authorization time, authorization content, and user confirmation operations. These records should be stored securely so that they can be used as a basis for legal requirements or dispute resolution.

[0073] For example, by adding layers, the flexibility of face replacement is mainly improved. For example, if someone visits a shopping website and wants to replace the faces of the models on the web pages he sees with his own face (showing himself wearing these clothes), in this case, the layer can quickly achieve the replacement, while copying the image and then replacing it is cumbersome and not real-time;

[0074] This solution not only realizes face replacement on the layer by adding a layer, but also involves, when processing real-time video streams, using image processing technologies such as face tracking, timestamp synchronization, change detection and incremental calculation to ensure the real-time replacement of faces. For example, when watching a certain live broadcast, if you want to replace the face of the live broadcaster with a specified face, you can simply record the video first and then replace it. Instead, you need to analyze the face position in real time and then replace the face status synchronously to ensure the consistency of the replaced face and the facial expression of the host in the live video stream.

[0075] In the specific implementation process, extracting the original face image in the above-mentioned original video can be achieved through the following steps: constructing a face recognition model, wherein the above-mentioned face recognition model is trained by a target recognition algorithm using multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes historical original videos obtained within a historical time period and historical original face images corresponding to the above-mentioned historical original videos, wherein the above-mentioned target recognition algorithm is one or more of MobileNet, SSD and YOLO; inputting the above-mentioned original video into the above-mentioned face recognition model to obtain the above-mentioned original face image corresponding to the above-mentioned original video.

[0076] In this solution, the use of MobileNet, SSD and YOLO algorithms enables face detection and recognition to be performed at low latency and high frame rate, and even on resource-constrained terminal devices, real-time processing can be achieved, improving user experience. By using multiple sets of training data, the model can learn the diversity and complexity of faces, so that it can accurately detect and recognize faces under different lighting conditions, facial expressions, head angles, etc., reducing false detections and missed detections.

[0077] Specifically, first, a special face recognition model needs to be built. This model is trained by the target recognition algorithm in deep learning. The training data contains multiple sets of historical original videos obtained in historical time periods and historical original face images corresponding to these videos. Through a large amount of training data, the model can learn the characteristics of the face, such as the layout of key points such as contours, eyes, nose, mouth, as well as various facial expressions such as facial texture, lighting, and angle.

[0078] The object recognition algorithm used when training the face recognition model can be one or more of MobileNet, SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once). These algorithms are widely used in object detection due to their efficiency and accuracy, especially MobileNet and YOLO, which perform well in processing real-time video streams and face detection on lightweight devices.

[0079] Input the original video that needs face replacement into the trained face recognition model. The model will analyze each frame in the video stream in real time, detect and identify the face images in it. The model accurately locates and extracts the images of the face area from the video frame, which will be used as the raw material for the subsequent face replacement operation.

[0080] MobileNet is a convolutional neural network (CNN) designed specifically for mobile devices and edge computing. MobileNet uses depthwise separable convolution and inverted residuals, which can significantly reduce the number of network parameters and computational complexity while maintaining high accuracy. MobileNet is very suitable for resource-constrained environments such as smartphones and embedded devices, and is often used for tasks such as real-time image analysis, face recognition, and object recognition.

[0081] SSD is an algorithm for object detection that can simultaneously detect multiple objects and their locations in an image in a single forward propagation. Compared with previous object detection methods (such as R-CNN and Fast R-CNN), SSD can significantly improve detection speed while maintaining relatively high detection accuracy. SSD makes predictions on multiple different feature maps, each corresponding to a different scale and ratio to accommodate objects of different sizes and shapes. It uses a series of default boxes to predict the bounding box and category of the object. SSD's real-time processing capabilities and high detection accuracy make it an ideal choice for scenarios such as video surveillance, autonomous vehicle object detection, and real-time analysis on mobile devices.

[0082] YOLO is an end-to-end object detection system that treats object detection as a regression problem and directly predicts the location and category of the object without generating candidate regions first. YOLO is one of the most advanced algorithms in the field of real-time object detection, which can maintain high accuracy while ensuring speed. YOLO divides the input image into multiple grids, each of which is responsible for detecting the object in its area. It predicts the bounding box and category probability at each position in the image, and then performs global non-maximum suppression (NMS) to remove duplicate bounding boxes, and finally outputs the detection results. YOLO's high-speed detection capability and good generalization ability make it widely used in video processing, game development, robot vision and other fields. Its real-time performance is particularly suitable for application scenarios that require instant feedback, such as video stream analysis.

[0083] For example, video clips containing human faces can be collected from publicly available video resource libraries or from historical videos uploaded by users through legal channels. These video clips should cover different lighting conditions, background environments, facial expressions, and angles to ensure the generalization ability of the model.

[0084] For each historical original video, use annotation tools to manually or semi-automatically annotate the face position in the video frame, that is, generate a bounding box containing the face position information. These annotated data will be used as label information when training the model to guide the model to learn the characteristics of the face.

[0085] The YOLO algorithm was selected as the target recognition algorithm because YOLO has high speed and accuracy in real-time video stream processing and face detection. YOLO can detect multiple targets at a time and is very suitable for processing complex scenes containing multiple faces.

[0086] The YOLO model is trained using multiple sets of prepared training data, including historical original videos and the annotated information of the corresponding historical original face images. The training process involves adjusting the parameters of the model to minimize the difference between the predicted results and the annotated data until the model can accurately detect faces on unseen data.

[0087] After training is complete, an independent test dataset is used to verify the performance of the model. The test data should be different from the training data to ensure the generalization ability of the model. The model's accuracy, recall, detection speed and other indicators will be evaluated, and the model will be fine-tuned if necessary.

[0088] Get the raw video data stream from the video source (such as live webcast, local file, etc.). Input the real-time video frames into the trained YOLO model, which analyzes each frame in real time to detect and locate the faces in it. The output includes the bounding box coordinates of the detected face and the corresponding confidence score. Based on the detection results, crop the image area containing the face from the original video frame. This process requires precise positioning of the bounding box to ensure that the extracted face image is complete and the background interference is minimized.

[0089] In some embodiments, the extracted original facial image is replaced with the target facial image corresponding to the face replacement requirement, which can be specifically achieved through the following steps: using a face replacement tool to replace the extracted original facial image with the target facial image corresponding to the face replacement requirement, wherein the face replacement tool is DeepFaceLab or rope.

[0090] In this solution, by using advanced face replacement tools such as DeepFaceLab or rope, this solution can replace the faces in the original video while maintaining high precision, real-time and natural fusion effects, and this process is completed on the terminal device. Tools such as DeepFaceLab and rope usually optimize computing efficiency and can process face data from video streams in real time, which is crucial for live or real-time video applications because they require instant feedback and processing speed.

[0091] After face detection and positioning are completed, the next step is to use a dedicated face replacement tool to achieve face replacement. In this scenario, DeepFaceLab and rope are two popular face replacement technologies. These tools are usually based on deep learning and can learn and imitate the details of the face, such as facial features, texture, expression, and lighting, to achieve natural and realistic face replacement effects.

[0092] The face replacement tool replaces the original face image detected and extracted from the original video with the target face image specified by the user. This process involves fine facial feature matching, expression synchronization, and fusion with the background to ensure that the target face in the original video is both realistic and natural, and coordinated with the video background.

[0093] DeepFaceLab is a face replacement software based on deep learning. It uses the techniques of Generative Adversarial Networks (GANs) and Autoencoders to achieve face replacement.

[0094] Rope is a relatively new face replacement tool. Its core is to use deep learning technology to achieve the migration of facial expressions and features.

[0095] In the specific implementation process, after constructing the virtual layer, the method further includes the following steps: when the face replacement demand is not received, adjusting the transparency of the virtual layer to a first transparency, wherein when the transparency of the virtual layer is the first transparency, the virtual layer will not obstruct the original layer; when the face replacement demand is received, adjusting the transparency of the virtual layer to a second transparency, wherein the first transparency is less than the second transparency, and when the transparency of the virtual layer is the second transparency, the virtual layer obstructs the original layer.

[0096] In this solution, by adjusting the transparency, the solution achieves flexible response to the needs of face replacement. When face replacement is not needed, the virtual layer becomes completely transparent, which does not affect the user's experience of watching the original video. When face replacement is required, the opacity of the virtual layer increases to ensure the clear presentation of the replacement effect. This mechanism ensures the viewing experience of users under different needs. When face replacement is not needed, users can watch the original video without interference; when face replacement is required, it can respond quickly and provide a clear face replacement effect.

[0097] Specifically, in the absence of face replacement requirements, the transparency of the virtual layer is adjusted to the "first transparency" (usually 100% transparent or very close), which means that the virtual layer will hardly or completely block the underlying original layer, that is, the original picture of the video stream. Users will see the original video content without any processing.

[0098] When it is detected that the user has a specific face replacement requirement, the transparency of the virtual layer will be adjusted to a "second transparency" (e.g. 0%). This transparency is usually low enough to allow the content on the virtual layer (such as the replaced face) to show through, thereby covering or replacing the original face image at the corresponding position on the original layer.

[0099] Specifically, the virtual layer is fully or partially transparent by default, which means it can be superimposed on the underlying video without blocking anything. When a face that needs to be replaced is detected, the transparency of the virtual layer will be adjusted to opaque only in the area where the face is located to display the replaced face image. In areas outside the face, the virtual layer remains transparent and the underlying video is still clearly visible.

[0100] The face replacement position and size in the software layer need to accurately match the face position and size in the underlying video. This means that the replacement face displayed on the virtual layer will accurately cover the corresponding position in the underlying video without exceeding the range, avoiding unnecessary overlap and occlusion.

[0101] A virtual layer is designed to be the topmost or specific layer of the overlay, which means it will be displayed before other content in the underlying video. By controlling the hierarchy and overlay order of the layers, you can ensure that the replaced face is only displayed in the corresponding position without affecting other parts of the video.

[0102] The face replacement in the software layer is not only performed at the initial moment, but also tracks the position and size changes of the face in real time, and dynamically updates the replacement area. This ensures that even if the face in the video moves, the replacement effect can keep up in real time, always maintaining accurate coverage and replacement.

[0103] At the edge of the face, this solution may use edge fusion technology to smoothly transition the replaced face image with the underlying video background, preventing obvious edge overlap or fragmentation, making the replacement effect more natural and seamless.

[0104] In some embodiments, before constructing the virtual layer, the method further includes the following steps: in the case where there are multiple face replacement requirements, constructing multiple virtual layers, wherein the face replacement requirements and the virtual layers correspond one-to-one; according to the multiple face replacement requirements, replacing the original face image with the multiple target face images corresponding to the multiple face replacement requirements, and displaying the corresponding target face images in all the virtual layers, wherein the face replacement requirements and the target face images correspond one-to-one.

[0105] In this solution, users can specify different replacement targets for each different face in the video, greatly improving the personalization and customization of face-changing applications. Users can choose different target faces to replace according to their preferences or specific needs. The independence of each virtual layer means that users can adjust the parameters of each replacement effect separately, such as target face image, transparency, position, etc., to achieve the best visual effect.

[0106] Specifically, when multiple independent face replacement requirements are detected, a virtual layer corresponding to each face replacement requirement is dynamically created. Each virtual layer is dedicated to handling one of the requirements, which means that each layer can independently perform face detection, feature analysis, and replacement processing without interfering with each other.

[0107] The face replacement requirement on each virtual layer is matched with a specific target face image, that is, a target face image is selected from the target face library specified by the user or provided by the system according to the face replacement requirement corresponding to each layer.

[0108] On each virtual layer, the faces detected in the original video are replaced with the corresponding target face images, and these replaced target faces are displayed on their respective layers. Since each virtual layer is independent, even if there are multiple faces that need to be replaced in the video, the corresponding replacement effect can be accurately displayed on each layer.

[0109] Specifically, on the display interface, the software can create multiple independent layers, each of which can be used to process different characters or tasks in the video. For example, layer 1 can be used to replace the face of character A in the video, layer 2 can be used to replace the face of character B, and so on. Each layer is independent of other layers, and face detection, tracking, and replacement can be performed without interfering with the underlying video content and other layers.

[0110] Specifically, software layers are similar to transparent films superimposed on the screen. Each layer can independently display images, videos or specific effects without affecting other layers or the underlying display content. In this solution, multiple software layers are created, each of which specializes in processing the face replacement or personalized effects of a person in the video, keeping the original video stream unmodified.

[0111] During the specific implementation process, after constructing multiple of the above-mentioned virtual layers, the above-mentioned method also includes the following steps: obtaining the layer range corresponding to the above-mentioned virtual layer, wherein the above-mentioned layer range is the effective range of the face replacement layer for the above-mentioned original video; according to the above-mentioned layer range and the above-mentioned face replacement requirements, replacing the above-mentioned original face image with the above-mentioned target face image, and displaying the corresponding above-mentioned target face image on the above-mentioned virtual layer.

[0112] In this solution, by limiting the layer range of the virtual layer, it can be ensured that the face replacement operation is only performed in the specific area containing the face, avoiding unnecessary processing of other parts of the video, reducing computing power consumption and processing time. Limiting the layer range can reduce interference with the video background or other non-face content, ensure the natural fusion of the replaced face image with the video background, and enhance the visual effect and viewing experience of the final video. When there are multiple faces that need to be replaced in the video, each virtual layer can independently handle different face replacement requirements, thereby achieving accurate replacement of multiple target faces while maintaining the original appearance of other areas, improving the complexity and flexibility of processing.

[0113] Specifically, before performing face replacement, it is necessary to determine the layer range of each virtual layer, that is, the specific area where the virtual layer is effective. This layer range is usually based on the results of face detection, ensuring that the virtual layer only covers the part of the image area that contains the original face, without affecting other non-face parts.

[0114] Once the layer range is determined, the original face image detected in the specified area will be replaced according to the face replacement requirements proposed by the user. This includes using face replacement tools (such as DeepFaceLab, rope, etc.) to replace the original face with the face image of the target person.

[0115] After the replacement is completed, the generated target face image is displayed on the corresponding virtual layer to cover and replace the original face image in the original video. Since the coverage of the virtual layer is limited to the area containing the face, the image content in other areas is not affected, ensuring the integrity and originality of the video background.

[0116] Specifically, the software layer not only supports independent creation, but also allows flexible delimitation of processing range, that is, face replacement for specific areas in the video stream. Each layer can set its effective area, for example, the face of person A can appear in the upper left corner of the video, and the face of person B can appear in the lower right corner. By delimiting the display range of the layer, it can be ensured that each layer only processes its own target person, avoiding useless processing of non-relevant areas in the video.

[0117] In some embodiments, after constructing the above-mentioned multiple virtual layers, the above-mentioned method also includes the following steps: obtaining the priority corresponding to the above-mentioned virtual layer, wherein the priority is the display level between the above-mentioned multiple virtual layers; when at least two of the above-mentioned virtual layers are superimposed, displaying the above-mentioned virtual layers according to the priority, wherein the above-mentioned virtual layer of the upper layer obscures the above-mentioned virtual layer of the lower layer.

[0118] In this solution, by setting priorities, the display order of multiple virtual layers can be effectively controlled to ensure that important or higher-priority replacement content can occupy a dominant position in the video frame and avoid being covered by other layers. When there are multiple faces that need to be replaced in the video, and these replacement virtual layers overlap in space, the priority mechanism can automatically resolve content display conflicts and ensure that the replacement effects set by the user are displayed correctly in priority order. The priority setting supports dynamic adjustment. Users can change the priority of the layer according to real-time needs. The priority mechanism enhances interactivity. Users can flexibly adjust the display order of each virtual layer according to viewing needs or creative purposes, thereby creating a more personalized and demand-oriented viewing experience.

[0119] Specifically, when there are multiple virtual layers, each layer can be assigned a priority, which determines the layer order when the final image is displayed. Layers with higher priorities are considered "upper layers", while layers with lower priorities are considered "lower layers".

[0120] When at least two virtual layers are superimposed, they are displayed in the order of their priority. Specifically, a virtual layer with a higher priority will appear on top of a virtual layer with a lower priority, which means that the "upper" virtual layer will obscure the "lower" virtual layer, especially when the areas covered by the two layers overlap.

[0121] Specifically, when multiple layers exist at the same time, they can be superimposed in a specific order to form an overall display effect. The superposition order of the layers can be set by the user or the system to determine which layer is displayed on the top. For example, if a person's face is blocked by another person, the priority of the layer can be intelligently adjusted to ensure that the blocked face can also display the replacement result at the appropriate time.

[0122] In addition, each layer can be customized independently, including the target face to be replaced, the adjustment of the replacement effect (such as expression, lighting, color, etc.), the transparency and display range setting of the layer, etc. This means that users can apply different face-changing effects or personalized processing to different characters in the video to meet diverse needs.

[0123] In addition, since each layer is processed independently, independent computing resources can be allocated to different layers and dynamically adjusted according to the complexity and priority of the task. This not only improves processing efficiency, but also ensures flexible response when processing multiple target people, avoiding processing delays or failures caused by resource competition.

[0124] Specifically, the current mainstream solutions are to replace faces at the source of the video stream, and then push the replaced face video stream to the streaming media. If you want to replace faces in real-time video streams on the local side, you need to directly modify the video stream data, which requires deep involvement in the data transmission process of the video software, increasing the complexity of implementation and maintenance, and may affect the stability of the video stream. At the same time, if you need to replace faces on video streams from different platforms, you need to develop them separately for each platform, which increases development costs and time, and it is difficult to achieve consistent results on different operating systems and devices.

[0125] Therefore, the solution of this application is mainly innovative in three dimensions. First, by developing a layer software on the display terminal, the software can obtain the interface image of the display terminal in real time from the screen capture API of the operating system, and detect and identify the face in the video stream in real time from the interface image, and then through a virtual layer superimposed on the top of the display interface, the target face image after replacement is finally displayed on this virtual layer, covering the original face image, so as to achieve the effect of face replacement without intervening in the original video stream; second, the software can support face detection and recognition within the range of the layer, without processing the entire frame image of the complete video stream, thereby reducing the pixel content processed by the program to a certain extent and reducing computing power consumption; third, by providing a method of superimposing multiple layers of software layers, it supports the replacement or personalized processing of the faces of one or more specific people in the video image, thereby realizing more flexible processing logic.

[0126] In summary, the solution of this application mainly focuses on adding one or more software layers dedicated to face replacement to the display interface of the terminal. The software layer will overlay the target face image after face replacement on the original face image, thereby achieving the effect of face replacement without intervening in the original video stream. The software monitors, identifies and replaces faces locally on the terminal without directly modifying the original video stream, which improves the flexibility of processing and privacy protection. Based on the flexible definition of the layer coverage range, it can detect and replace faces only within the limited range of the layer, avoiding the complete processing of the entire frame of video, and significantly reducing computing power consumption. At the same time, it allows the superposition of multiple layers to process different face replacements separately, achieving higher flexibility and personalized effects, and the processing between each layer does not interfere with each other.

[0127] This solution proposes a targeted solution to the real-time and synchronization issues between layers and original video streams. Through image processing technologies such as face tracking, timestamp synchronization, change detection and incremental calculation, higher real-time and synchronization can be achieved, and visual delays and asynchrony problems can be reduced.

[0128] In summary, compared to the current solution, this solution does not directly modify the original video stream or data, and will not affect the original data. This non-invasive method is more compatible with existing applications and systems, reducing the complexity of integration with existing software. This solution utilizes the screen capture API and layer technology of the operating system. By adding software layers, it can run on various devices and operating systems and achieve consistent effects. There is no need to develop separately for the source platform of each video stream. This method makes it easier to achieve cross-platform compatibility. Through the multi-layer design, this solution allows users to adjust the replacement effect in real time according to their needs. For example, different layers can switch different target faces, adjust the replacement intensity, etc. This method provides higher user customization and control capabilities, improving the user experience.

[0129] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the layer-based face replacement display method of the present application will be described in detail below in combination with specific embodiments.

[0130] This embodiment relates to a specific layer-based face replacement display method, which mainly involves three modules: "image capture module", "layer setting module" and "face replacement module". The processing flow is as follows: Figure 3 As shown:

[0131] First, when the user of the video receiving terminal is watching the video, the "image capture module" calls the screen capture API of the video receiving terminal itself to realize real-time capture of the initial display image of the video stream (screen capture software APIs such as Windows' DirectX, macOS's Quartz Display Services, Android's MediaProjection API, iOS's ReplayKit, etc.).

[0132] The second step is to pass the real-time screen image obtained by the "image capture module" to the "layer setting module". The "layer setting module" supports the creation of one or more transparent overlay layers on the terminal device as the display area of ​​the target face image after face replacement. At the same time, the "layer setting module" supports the management of multiple layers, including defining the scope of each layer for targeted face image processing, as well as the layer stacking order, transparency adjustment, display range, etc., to ensure that the processing between each layer does not interfere with each other.

[0133] In the third step, taking layer 1 as an example, the image of the delimited area of ​​layer 1 specified by the "layer setting module" is input into the face detection and face tracking model of the "face replacement module", so as to quickly detect the face within the limited range of the layer (a lightweight face detection model such as MobileNet SSD or YOLO model can be used). The same operation is performed on layers 1-N.

[0134] In the fourth step, the face image of the demarcated area in the detected layer 1 is input into the "Face Replacement Tool 1" of the "Face Replacement Module" (open source face replacement tools such as DeepFaceLab and rope can be used), and the detected original face is replaced with the specified target face. At the same time, edge smoothing, lighting adjustment, and color matching techniques are used to merge the generated target face image with the background in the original video stream to ensure the natural fusion of the replaced face and the background. In order to solve the problem of real-time display of replaced faces, the face position is continuously tracked during video playback to ensure that the replaced target face can always be aligned with the original face position, and a timestamp is added to each frame of video and layer processing results. According to the timestamp, the layer processing results are ensured to be displayed synchronously with the original video frame to avoid visual delay. At the same time, through the change detection algorithm, updates are only performed when changes in the face area are detected. The incremental calculation technology is used to calculate only the changed part based on the similarity between the previous and next frames, reducing repeated calculations and further improving real-time performance. Finally, the replaced target face output by the "Face Replacement Tool 1" is drawn in the designated face area of ​​layer 1, thereby covering the face area in the original image.

[0135] The fifth step is to overlay layer 1 to layer N so that the final overlaid face replacement image is overlaid on the original video stream image (at the same time, the layer only covers the image near the face area, and the layers in other areas remain transparent). The image finally displayed to the terminal user is the video image with the specified face replaced.

[0136] The embodiment of the present application also provides a display device for layer-based face replacement. It should be noted that the display device for layer-based face replacement in the embodiment of the present application can be used to execute the display method for layer-based face replacement provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0137] The following is an introduction to the display device for layer-based face replacement provided in an embodiment of the present application.

[0138] Figure 4 is a structural block diagram of a display device for face replacement based on layers according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0139] A first acquisition unit 10, used to acquire an original video;

[0140] A first constructing unit 20 is used to construct a virtual layer, wherein the virtual layer is used to display the replaced face image;

[0141] The replacement unit 30 is used to extract the original face image in the original video when receiving the face replacement requirement, and replace the extracted original face image with the target face image corresponding to the face replacement requirement;

[0142] The first display unit 40 is used to display the above-mentioned original video on the original layer, and display the above-mentioned target face image on the above-mentioned virtual layer, wherein the above-mentioned target face image on the above-mentioned virtual layer covers the above-mentioned original face image on the above-mentioned original layer.

[0143] Through this embodiment, a software layer (i.e., virtual layer) dedicated to face replacement is designed. The software layer will overlay the target face image after face replacement on the original face image. By superimposing the virtual layer, the virtual layer processes face replacement independently of the underlying content, and replaces the face of a specific area without affecting other content. The software layer technology will not modify the data stream of the video source. The original video stream remains unchanged, thereby achieving face replacement without intervening in the original video stream, ensuring good stability of the original video.

[0144] During the specific implementation process, the replacement unit includes a construction module and a processing module. The construction module is used to construct a face recognition model, wherein the face recognition model is trained by a target recognition algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical original videos obtained within a historical time period and historical original face images corresponding to the historical original videos, wherein the target recognition algorithm is one or more of MobileNet, SSD and YOLO; the processing module is used to input the original video into the face recognition model to obtain the original face image corresponding to the original video.

[0145] In this solution, the use of MobileNet, SSD and YOLO algorithms enables face detection and recognition to be performed at low latency and high frame rate, and even on resource-constrained terminal devices, real-time processing can be achieved, improving user experience. By using multiple sets of training data, the model can learn the diversity and complexity of faces, so that it can accurately detect and recognize faces under different lighting conditions, facial expressions, head angles, etc., reducing false detections and missed detections.

[0146] In some embodiments, the replacement unit includes a replacement module, which is used to use a face replacement tool to replace the extracted original face image with the target face image corresponding to the face replacement requirement, wherein the face replacement tool is DeepFaceLab or rope.

[0147] In this solution, by using advanced face replacement tools such as DeepFaceLab or rope, this solution can replace the faces in the original video while maintaining high precision, real-time and natural fusion effects, and this process is completed on the terminal device. Tools such as DeepFaceLab and rope usually optimize computing efficiency and can process face data from video streams in real time, which is crucial for live or real-time video applications because they require instant feedback and processing speed.

[0148] During the specific implementation process, the above-mentioned device also includes a first adjustment unit and a second adjustment unit. The first adjustment unit is used to adjust the transparency of the above-mentioned virtual layer to a first transparency after constructing the virtual layer and when the above-mentioned face replacement demand is not received, wherein when the transparency of the above-mentioned virtual layer is the first transparency, the above-mentioned virtual layer will not obstruct the above-mentioned original layer; the second adjustment unit is used to adjust the transparency of the above-mentioned virtual layer to a second transparency when the above-mentioned face replacement demand is received, wherein the above-mentioned first transparency is less than the above-mentioned second transparency, and when the transparency of the above-mentioned virtual layer is the second transparency, the above-mentioned virtual layer obstructs the above-mentioned original layer.

[0149] In this solution, by adjusting the transparency, the solution achieves flexible response to the needs of face replacement. When face replacement is not needed, the virtual layer becomes completely transparent, which does not affect the user's experience of watching the original video. When face replacement is required, the opacity of the virtual layer increases to ensure the clear presentation of the replacement effect. This mechanism ensures the viewing experience of users under different needs. When face replacement is not needed, users can watch the original video without interference; when face replacement is required, it can respond quickly and provide a clear face replacement effect.

[0150] In some embodiments, the device further includes a second construction unit and a first processing unit, wherein the second construction unit is used to construct multiple virtual layers before constructing the virtual layer in the case where there are multiple face replacement requirements, wherein the face replacement requirements and the virtual layers correspond one-to-one; and the first processing unit is used to replace the original face image with the multiple target face images corresponding to the multiple face replacement requirements according to the multiple face replacement requirements, and display the corresponding target face images in all the virtual layers, wherein the face replacement requirements and the target face images correspond one-to-one.

[0151] In this solution, users can specify different replacement targets for each different face in the video, greatly improving the personalization and customization of face-changing applications. Users can choose different target faces to replace according to their preferences or specific needs. The independence of each virtual layer means that users can adjust the parameters of each replacement effect separately, such as target face image, transparency, position, etc., to achieve the best visual effect.

[0152] During the specific implementation process, the above-mentioned device also includes a second acquisition unit and a second processing unit. The second acquisition unit is used to obtain the layer range corresponding to the above-mentioned virtual layer after constructing multiple of the above-mentioned virtual layers, wherein the above-mentioned layer range is the effective range of the face replacement layer for the above-mentioned original video; the second processing unit is used to replace the above-mentioned original face image with the above-mentioned target face image according to the above-mentioned layer range and the above-mentioned face replacement requirements, and display the corresponding above-mentioned target face image on the above-mentioned virtual layer.

[0153] In this solution, by limiting the layer range of the virtual layer, it can be ensured that the face replacement operation is only performed in the specific area containing the face, avoiding unnecessary processing of other parts of the video, reducing computing power consumption and processing time. Limiting the layer range can reduce interference with the video background or other non-face content, ensure the natural fusion of the replaced face image with the video background, and enhance the visual effect and viewing experience of the final video. When there are multiple faces that need to be replaced in the video, each virtual layer can independently handle different face replacement requirements, thereby achieving accurate replacement of multiple target faces while maintaining the original appearance of other areas, improving the complexity and flexibility of processing.

[0154] In some embodiments, the above-mentioned device also includes a third acquisition unit and a second display unit, the third acquisition unit is used to obtain the priority corresponding to the above-mentioned virtual layer after constructing the above-mentioned multiple virtual layers, wherein the priority is the display level between the multiple virtual layers; the second display unit is used to display the above-mentioned virtual layer according to the priority when at least two of the above-mentioned virtual layers are superimposed, wherein the above-mentioned virtual layer of the upper layer obscures the above-mentioned virtual layer of the lower layer.

[0155] In this solution, by setting priorities, the display order of multiple virtual layers can be effectively controlled to ensure that important or higher-priority replacement content can occupy a dominant position in the video frame and avoid being covered by other layers. When there are multiple faces that need to be replaced in the video, and these replacement virtual layers overlap in space, the priority mechanism can automatically resolve content display conflicts and ensure that the replacement effects set by the user are displayed correctly in priority order. The priority setting supports dynamic adjustment. Users can change the priority of the layer according to real-time needs. The priority mechanism enhances interactivity. Users can flexibly adjust the display order of each virtual layer according to viewing needs or creative purposes, thereby creating a more personalized and demand-oriented viewing experience.

[0156] The above-mentioned display device for face replacement based on layers includes a processor and a memory. The above-mentioned first acquisition unit, the first construction unit, the replacement unit and the first display unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0157] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of poor stability of the original video stream caused by the face replacement solution in the prior art processing the original video stream is solved by adjusting the kernel parameters.

[0158] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0159] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned layer-based face replacement display method.

[0160] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned layer-based face replacement display method when running.

[0161] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the steps of a display method for face replacement based on a layer are implemented. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0162] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method of layer-based face replacement in each embodiment of the present application are implemented.

[0163] The present application also provides a face replacement display system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a display method for executing any one of the above-mentioned layer-based face replacement.

[0164] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0169] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0170] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0171] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0172] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0173] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A display method for face replacement based on layers, characterized in that: include: Get the original video; Constructing a virtual layer, wherein the virtual layer is used to display the replaced face image; When a face replacement requirement is received, extracting an original face image in the original video, and replacing the extracted original face image with a target face image corresponding to the face replacement requirement; The original video is displayed on the original layer, and the target face image is displayed on the virtual layer, wherein the target face image on the virtual layer covers the original face image on the original layer.

2. The method according to claim 1, characterized in that Extracting the original face image in the original video includes: Constructing a face recognition model, wherein the face recognition model is trained by a target recognition algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes a historical original video acquired within a historical time period and a historical original face image corresponding to the historical original video, wherein the target recognition algorithm is one or more of MobileNet, SSD and YOLO; The original video is input into the face recognition model to obtain the original face image corresponding to the original video.

3. The method according to claim 1, characterized in that Replacing the extracted original face image with a target face image corresponding to the face replacement requirement, comprising: Use a face replacement tool to replace the extracted original face image with the target face image corresponding to the face replacement requirement, wherein the face replacement tool is DeepFaceLab or rope.

4. The method according to claim 1, characterized in that After constructing the virtual layer, the method further includes: When the face replacement requirement is not received, adjusting the transparency of the virtual layer to a first transparency, wherein when the transparency of the virtual layer is the first transparency, the virtual layer will not cover the original layer; When the face replacement requirement is received, the transparency of the virtual layer is adjusted to a second transparency, wherein the first transparency is less than the second transparency, and when the transparency of the virtual layer is the second transparency, the virtual layer obscures the original layer.

5. The method according to claim 1, characterized in that Before constructing the virtual layer, the method further includes: In the case where there are multiple face replacement requirements, construct multiple virtual layers, wherein the face replacement requirements correspond to the virtual layers one by one; According to the multiple face replacement requirements, the original face image is replaced with the multiple target face images corresponding to the multiple face replacement requirements, and the corresponding target face images are displayed on all the virtual layers, wherein the face replacement requirements and the target face images correspond one to one.

6. The method according to claim 5, characterized in that After constructing the plurality of virtual layers, the method further comprises: Obtaining a layer range corresponding to the virtual layer, wherein the layer range is an effective range of the face replacement layer for the original video; According to the layer range and the face replacement requirement, the original face image is replaced with the target face image, and the corresponding target face image is displayed on the virtual layer.

7. The method according to claim 5, characterized in that After constructing the plurality of virtual layers, the method further comprises: Obtaining a priority corresponding to the virtual layer, wherein the priority is a display level among the multiple virtual layers; In the case where at least two of the virtual layers are superimposed, the virtual layers are displayed according to priorities, wherein the upper virtual layer obscures the lower virtual layer.

8. A display device for face replacement based on layers, characterized in that: include: A first acquisition unit, used to acquire an original video; A first construction unit is used to construct a virtual layer, wherein the virtual layer is used to display the replaced face image; A replacement unit, configured to extract an original face image from the original video upon receiving a face replacement requirement, and replace the extracted original face image with a target face image corresponding to the face replacement requirement; The first display unit is used to display the original video on the original layer and display the target face image on the virtual layer, wherein the target face image on the virtual layer covers the original face image on the original layer.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the display method of layer-based face replacement described in any one of claims 1 to 7 are implemented.

10. A face replacement display system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a display method for executing the layer-based face replacement described in any one of claims 1 to 7.

Images