Method for facial image processing of a video stream and electronic device

Abstract

Embodiments of the present application disclose a method for facial image processing of a video stream and an electronic device. The method comprises: in a process of collecting a video stream on a client side, determining a target image to be processed according to a plurality of video image frames in the video stream; on the client side, processing a face region in the target image by using a target algorithm model adapted to a data processing capability of a terminal device where the client is located, wherein the target algorithm model is generated based on a training set, the training set comprising a plurality of image pairs, the image pairs comprising a first image with facial defects and a second image without the facial defects, so as to train the target algorithm model with the goal of removing facial defects in an input image; and on the client side, displaying the video stream according to the processed image. Through the embodiments of the present application, a more optimal facial image processing result can be obtained while meeting real-time requirements.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to methods and electronic devices for facial image processing of video streams. Background Technology

[0002] In daily life, a large number of videos or images containing faces require facial enhancement. Facial enhancement is also an essential function of live streaming, short video, and facial image processing software. Traditional facial enhancement methods mainly use edge-preserving filtering or image processing algorithms to smooth the skin of the facial area and the entire image while keeping the edges of the face intact.

[0003] Edge-preserving filtering, in particular, preserves high-frequency edges in an image while smoothing low-frequency flat areas. Applying edge-preserving filtering to skin smoothing algorithms can smooth the skin while preserving facial contours. Specifically, the main method for smoothing skin is to average the pixels in the center of the skin area with the surrounding pixels, resulting in a more refined appearance. However, some people may have noticeable blemishes such as pimples or spots. Because the color of pimples or spots is significantly different from the surrounding area, the aforementioned smoothing algorithms struggle to handle such cases. Summary of the Invention

[0004] This application provides a method and electronic device for facial image processing of video streams, which can obtain better facial image processing results while meeting real-time requirements.

[0005] This application provides the following solution: A method for facial image processing of a video stream includes: During the acquisition of video streams on the client side, the target image to be processed is determined based on multiple video image frames in the video stream; On the client side, a target algorithm model adapted to the data processing capabilities of the client's terminal device is used to process the face region in the target image. The target algorithm model is generated after training on a training set, which includes multiple image pairs. Each image pair includes a first image with facial blemishes and a second image after removing the facial blemishes, so that the target algorithm model is trained with the goal of removing facial blemishes from the input image. On the client side, a video stream is displayed based on the processed image.

[0006] Specifically, on the client side, processing the face region in the target image using a target algorithm model adapted to the data processing capabilities of the client's terminal device includes: If the terminal device associated with the client is equipped with a graphics processing unit (GPU) chip or a neural network processing unit (NPU) chip, the task of executing the target algorithm model to process the face region in the target image will be assigned to the GPU chip or NPU chip, so as to utilize the hardware processing capabilities of the GPU chip or NPU chip to complete the processing of the face region in the target image.

[0007] This also includes: The target image is subjected to face region detection to segment the face region image located within the rectangular detection box; The process of processing the face region in the target image using a target algorithm model adapted to the data processing capabilities of the client's terminal device includes: The face region image located within the rectangular detection box is input into the target algorithm model so that the target algorithm model can be used to process the face region image located within the rectangular detection box. The method further includes: The face region image processed by the target algorithm model is used to cover or overlay the original face region image in the target image so as to display a video stream based on the covered or overlaid image.

[0008] This also includes: Multiple facial key points are identified from the target image, and a facial key point map and a binarized mask map are generated to determine the location of the region within the facial contour area from the target image. The process of overlaying or superimposing the original face region image in the target image onto the face region image processed by the target algorithm model includes: Based on the facial key point map and the binarized mask map, the face region image processed by the target algorithm model is used to cover or overlay the image within the facial contour range of the target image.

[0009] The step of identifying multiple facial key points from the target image includes: Using a facial landmark recognition algorithm, the first number of basic facial landmarks are identified; Based on the basic facial key points, expand outwards to determine a second number of expanded facial key points; The first number of basic facial landmarks and the second number of extended facial landmarks are determined as the facial landmarks.

[0010] This also includes: After processing the face region in the target image using the target algorithm model, the processed face region is then smoothed to preserve its edges, so that a video stream can be displayed based on the image after edge-preserving smoothing.

[0011] The step of performing edge-preserving smoothing on the processed face area includes: If the terminal device associated with the client is equipped with a GPU chip, the edge-preserving smoothing process is assigned to the GPU so as to utilize the hardware processing power of the GPU chip to perform edge-preserving smoothing on the processed face area.

[0012] This also includes: Determine whether the target image belongs to a keyframe in the video stream; If it is a keyframe, the step of processing the keyframe using the target algorithm model is triggered; otherwise, the processing result of the non-keyframe is obtained by performing image deformation processing on the processing result corresponding to the historical frame.

[0013] A method for facial image processing of a video stream includes: During the acquisition of video streams on the client side, the target image to be processed is determined based on multiple video image frames in the video stream; A first task is generated and assigned to the CPU chip in the terminal device associated with the client for execution; the first task includes: using a first algorithm model to perform face region detection on the target image, so as to segment the face region image located within a rectangular detection box; A second task is generated and assigned to the GPU chip or NPU chip in the terminal device; the second task includes: processing the face region image located within the rectangular detection box using a second algorithm model; wherein, the second algorithm model is generated after training on a training set, the training set including multiple image pairs, the image pairs including a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; the second algorithm model is adapted to the data processing capabilities of the terminal device associated with the client; A third task is generated and assigned to the GPU chip for execution; the third task includes: using the processing results returned by the second algorithm model to cover or overlay the face region in the target image so that the processed image can be displayed as a video stream on the client side.

[0014] The first task further includes: The third algorithm model is used to identify facial key points in the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range in the target image. The third task specifically includes: Based on the facial key point map and the binarized mask map, the processing results returned by the second algorithm model are used to cover or overlay the facial region in the target image, so as to generate an image after processing the region within the facial contour range in the target image.

[0015] This also includes: A fourth task is generated and assigned to the GPU chip for execution. The fourth task includes rendering the binarized mask and displaying it on the screen. The displaying on the screen includes displaying the binarized mask on the screen corresponding to the offline buffer to limit the number of faces to be processed.

[0016] A face image processing method, comprising: Identify the target image to be processed; The first algorithm model is used to detect face regions in the target image to segment the face region image located within the rectangular detection box; The face region in the target image is processed using a second algorithm model, wherein the second algorithm model is generated after training on a training set, the training set includes multiple image pairs, the image pairs include a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; The third algorithm model is used to identify facial key points in the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range in the target image. Based on the facial key point map and the binarized mask map, the facial region in the target image is covered or overlaid using the processing results returned by the second algorithm model, so as to generate an image after processing the region within the facial contour range of the target image.

[0017] An apparatus for facial image processing of a video stream, comprising: The target image determination unit is used to determine the target image to be processed based on multiple video image frames in the video stream during the process of acquiring the video stream on the client side. The inference computing unit is used on the client side to process the face region in the target image using a target algorithm model adapted to the data processing capabilities of the terminal device where the client is located. The target algorithm model is generated after training on a training set, which includes multiple image pairs. The image pairs include a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image. The display unit is used on the client side to display a video stream based on the processed image.

[0018] An apparatus for facial image processing of a video stream, comprising: The target image determination unit is used to determine the target image to be processed based on multiple video image frames in the video stream during the process of acquiring the video stream on the client side. A face detection unit is used to generate a first task and assign the first task to the central processing unit (CPU) chip in the terminal device associated with the client for execution; the first task includes: using a first algorithm model to perform face region detection on the target image to segment the face region image located within a rectangular detection box; An inference computing unit is used to generate a second task and assign the second task to a GPU chip or NPU chip in the terminal device; the second task includes: processing the face region image located within the rectangular detection box using a second algorithm model; wherein, the second algorithm model is generated after training on a training set, the training set including multiple image pairs, the image pairs including a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; the second algorithm model is adapted to the data processing capabilities of the terminal device associated with the client; An overlay or superimposition processing unit is used to generate a third task and assign the third task to the GPU chip for execution; the third task includes: using the processing result returned by the second algorithm model to overlay or superimpose the face region in the target image so that the processed image can be displayed as a video stream on the client side.

[0019] A face image processing device, comprising: The target image determination unit is used to determine the target image to be processed; The face detection unit is used to perform face region detection on the target image using a first algorithm model, so as to segment the face region image located within the rectangular detection box; The inference computing unit is used to process the face region in the target image using a second algorithm model. The second algorithm model is generated after training on a training set, which includes multiple image pairs. The image pairs include a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image. The facial key point recognition unit is used to perform facial key point recognition on the target image using a third algorithm model, and generate a facial key point map and a binarized mask map, so as to determine the location of the region within the facial contour range in the target image. The overlay or overlay processing unit is used to overlay or overlay the face region in the target image based on the face key point bitmap and the binarized mask map, using the processing result returned by the second algorithm model, so as to generate an image after processing the region within the face contour range in the target image.

[0020] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0021] An electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the preceding descriptions.

[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed: Through the embodiments of this application, during the acquisition of video streams on the client side (e.g., the client on the broadcaster's side in a live streaming scenario), a target image to be processed can be determined based on multiple video image frames in the video stream. The facial region in the target image can then be processed directly on the client side using a target algorithm model. Specifically, the target algorithm model can be adapted to the data processing capabilities of the client's terminal device. It can be generated after training on a training set, which includes multiple image pairs. Each image pair includes a first image with facial blemishes and a second image after removing the blemishes, so that the target algorithm model is trained with the goal of removing facial blemishes from the input image. After processing the facial region in the target image, a video stream can be displayed on the client side based on the processed image. In this way, because a pre-trained algorithm model is used to remove blemishes in the face, the blemish processing effect is improved compared to the existing blemish processing method using edge-preserving filtering. Furthermore, since face region processing can be performed directly on the client side of image frames in the video stream without requiring network transmission or transcoding, real-time processing is guaranteed. Therefore, the solution provided in this application embodiment can balance processing effectiveness and real-time performance when processing face images, achieving better facial image processing results in scenarios such as live video streaming while meeting real-time requirements.

[0023] Furthermore, in a preferred embodiment, based on the inference calculation using the target algorithm model described above, it can be integrated with various other technologies to further ensure the real-time performance and processing effect. In terms of algorithms, preprocessing for face detection in the target image can be included to reduce the complexity of the target algorithm model's inference calculations, thereby accelerating the process. Additionally, by generating a face keypoint bitmap and a binarized mask image from the target image, the processing area can be controlled within the facial contour, preventing the clarity of other areas from being affected. Moreover, it can be integrated with edge-preserving filtering techniques to achieve further skin smoothing, thereby further improving the removal effect of facial blemishes.

[0024] On the hardware side, the capabilities of chips such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and NPU (Neural Networks Processing Unit) can be fully utilized to further accelerate the algorithm, thereby ensuring the real-time performance of facial image processing.

[0025] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram comparing the processing effects of the original image with those of existing technologies and the processing effects in the embodiments of this application; Figure 2 This is a schematic diagram of the system architecture provided in the embodiments of this application; Figure 3 This is a flowchart of the first method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the processing flow provided in the embodiments of this application; Figure 5 This is a flowchart of the second method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the software and hardware interaction process in the solution provided in the embodiments of this application; Figure 7 This is a flowchart of the third method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the first device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the second device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the third device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] In this embodiment, to more effectively remove obvious blemishes such as acne and spots encountered during facial image processing, a pre-trained deep learning model can be used. It should first be noted that currently available professional facial image processing software does not perform ideally on blemishes like acne and spots, especially in scenarios like live video streaming, where the processing effect is even worse when acne and spots are densely packed. For example, as... Figure 1 As shown, the left image (A) is the original image of a frame captured during a live video stream, and the middle image (B) is the result after processing using existing processing techniques. It is evident that although the "pimple" problem has been somewhat improved compared to the original image, obvious "pimple" blemishes are still visible.

[0030] In implementing the embodiments of this application, the inventors discovered that a major reason for the aforementioned problems is that image processing algorithms generally have high complexity. Therefore, these algorithms typically run on the server side, with the terminal side only used to display the processing results. The above solution is feasible for processing static images (e.g., beautifying a photograph in facial image processing software) because users are not time-sensitive in such cases, allowing ample time for the server to complete processing. However, for scenarios like live video streaming, it's not only necessary to push the stream to the viewers but also to display the live video stream on the streaming end. The broadcaster typically needs to see their own beautification effects in real-time, thus requiring more real-time facial image processing. If the broadcaster first pushes the live video stream to the server for beautification processing before pushing it to both the broadcaster and viewers, it inevitably creates a significant delay for the broadcaster, impacting their experience. It is evident that for scenarios with higher real-time requirements for facial image processing, such as live video streaming, achieving ideal processing results for blemishes like pimples and spots is quite difficult. Therefore, current technologies for live video streaming scenarios must choose between real-time performance and processing quality. For example, edge-preserving filtering can be used on the client side. Since edge-preserving filtering is relatively simple to implement, this method can generally guarantee real-time performance; however, the processing quality may vary. Figure 1 As shown in (B), it may not be ideal.

[0031] To address the above issues, this application provides corresponding solutions. Firstly, it offers deep learning models more suitable for operation on mobile devices. Specifically, since deep learning models can have various structures, a GAN (Generative Adversarial Network) is preferred. Secondly, to make the algorithm model more suitable for mobile devices and to accelerate its operation, inference can be accelerated. For example, inter-frame strategies such as image warping can be used to reduce the computational load of inter-frame models, thereby accelerating inference. Modifications can also be made to the standard GAN network (e.g., reducing the number of layers) or model sparsification can be performed (e.g., setting some parameters to fixed values, where the fixed values ​​can be 0 or other values, etc.). This reduces computational load or complexity while maintaining the GAN structure, thus achieving better suitability for operation on mobile devices and faster performance.

[0032] After determining the algorithm model structure, the model can be trained using a training set. The training set can include multiple image pairs, each pair containing two corresponding images. One image shows a face with blemishes such as acne, while the other image represents the face after blemish removal using various methods. In other words, the two images in the same pair are identical except for one having blemishes and the other being flawless. By using multiple such image pairs as training data to train the deep learning model, it gains the ability to remove facial blemishes from input images. Therefore, this model can be used to process blemishes such as acne and spots in facial images from live video streams. Compared to edge-preserving filtering methods in existing technologies, this approach effectively improves the processing of facial blemishes. Furthermore, since it can run directly on the client side, avoiding network transmission and transcoding steps, real-time performance is better guaranteed.

[0033] Furthermore, in the preferred embodiment, to further improve the effect and real-time performance of facial image processing in the video stream, the beautification effect of the above scheme can be further optimized through multi-technology fusion. For example, in terms of algorithms, pre-executed face detection processing can be combined to further reduce the algorithm complexity of the aforementioned GAN network, achieving further speedup; by recognizing facial key points and generating a binary mask (which can be called a Mask) map, it is possible to process only the image within the facial contour range, avoiding the impact on images in other areas, which can also improve the processing effect; optionally, the basic facial key points can be expanded so that the facial key point bitmap can more completely cover the image within the facial contour range. In addition, edge-preserving filtering technology can be combined for further skin smoothing processing, thereby further improving the processing effect. For example, as Figure 1 As shown in Figure (C) on the right, it is the processing effect after combining the above-mentioned technologies in the embodiment of this application. It can be seen that the "pimples" in the processed image have been basically completely removed, and the processing effect is significantly better than the processing effect shown in Figure (B).

[0034] In terms of hardware, the capabilities of various chips integrated into the terminal device can be fully utilized. The collaboration of these chips can accelerate the algorithm processing flow and facilitate support for more complex but effective algorithms, thus improving processing performance. For example, current mobile terminal devices such as smartphones may include not only CPUs (Central Processing Units) but also GPUs (Graphics Processing Units), NPUs (Neural Networks Processing Units), and so on. GPUs are large-scale parallel computing architectures composed of numerous computing units, better suited for processing image-parallel computing data. NPUs are specifically optimized for matrix operations, solving the problem of low efficiency in neural network operations using traditional chips. Therefore, in this embodiment, to further improve real-time performance, the aforementioned hardware resources in the terminal device can be utilized more fully, reducing the CPU load and achieving acceleration. For example, tasks involving GAN network algorithm calculations can be assigned to GPUs or NPUs for execution. Since NPU chips are better suited for neural network computations, they should be prioritized if the terminal device has one. Otherwise, the GPU can be used for processing. Tasks such as face detection and facial landmark recognition can be performed by the CPU, GPU, or NPU. Multithreading technology in the CPU can be used to parallelize facial landmark detection with face detection, further accelerating the process. Furthermore, post-processing tasks such as edge-preserving filtering can be performed by the GPU, and final rendering can also be done directly on the GPU. This collaboration between the CPU, GPU, and NPU ensures real-time performance in facial image processing, making it more suitable for live video streaming scenarios.

[0035] From a system architecture perspective, such as Figure 2As shown, in a typical application scenario, real-time facial image processing with superior processing effects can be provided in a video system (e.g., a live video streaming system). Specifically, the aforementioned video system can be provided by a dedicated short video or live streaming platform, or by a product information service system or platform, and so on. The specific video stream can be of various types. For example, in one application scenario, it could be a live video stream related to a host explaining or demonstrating products; of course, it could also be a video stream with other content, as long as it includes facial images and requires real-time processing of the facial images in the video stream at the acquisition end before display. The facial image processing function can be implemented on the client side of the video system, that is, on the client side that acquires the video stream. After identifying the target image to be processed from the acquired video stream, the face region in the target image can be processed on the client side using a pre-trained target algorithm model (e.g., the GAN network mentioned above), and then rendered and displayed on the client. Alternatively, the stream can be pushed to the server so that the viewer client can watch the live content by pulling the stream from the server.

[0036] As mentioned earlier, if the client's terminal device is equipped with chips such as GPUs and NPUs, the facial image processing process can be accelerated by fully utilizing the capabilities of these chips. Specifically, the aforementioned GAN network can leverage the capabilities of GPUs and NPUs (in...) Figure 2 In this example, the NPU (using a processor as an example) is used to run the processing. The results can be returned to the CPU for data distribution and scheduling. For instance, the NPU can return the processing results to the CPU, which can then provide the original image and the NPU's processing results to the GPU. The GPU then performs image overlay (overlaying the original image with the residual image output by the GAN network; residual images will be discussed later) or overlay processing, followed by rendering and display. If facial landmark recognition and mask image generation are included, this part can be done on the CPU. The CPU can provide the mask image, the original image, and the NPU's processing results to the GPU for image overlay or overlay processing. Furthermore, further edge-preserving filtering can be performed on the GPU to further improve the processing effect. After edge-preserving filtering, the GPU can render the image for display on a screen. In addition, the CPU's multi-threading and batch processing capabilities can achieve more comprehensive acceleration.

[0037] Of course, in practical applications, the specific facial image processing function provided in this application embodiment can be applied not only to the broadcaster's client but also to the server or cloud. For example, in a specific live video scenario, facial image processing can be performed on the broadcaster's client side. However, to achieve speed improvements, the specific GAN network may be simplified, and its processing effect may not be as good as that of a standard GAN network. Therefore, a simplified version of the GAN network can also be used for processing on the broadcaster's side. However, when pushing the stream to the server, the original live video stream can be pushed so that the server can process it based on a standard GAN network (and further ensure real-time performance and processing effect through multi-technology fusion), thereby enabling the live video stream pulled by the viewer's client to have a better beautification effect, and so on. In addition, this function can also be applied to facial image processing tools and other related software, and so on.

[0038] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0039] Example 1 First, this first embodiment provides a method for facial image processing of images in a live video stream from the perspective of the broadcaster's client in a live video streaming scenario. See [link to previous document]. Figure 3 The method may include: S301: During the process of acquiring the video stream on the client side, the target image to be processed is determined based on multiple video image frames in the video stream.

[0040] As mentioned above, the client described in this application embodiment can specifically be a client on the broadcaster's side in a live video streaming scenario, or it can be a client for capturing short videos, etc. The specific video stream can vary depending on the specific video content. For example, in one application scenario, it can specifically include a live video stream in which a person explains or demonstrates a product, etc.

[0041] During the acquisition of video streams via a client, the target image to be processed can be determined from multiple video image frames in the video stream. In other words, since the video stream includes multiple video image frames, each image frame can correspond to one image. Specifically, when performing facial image processing, it can be done on a frame-by-frame basis.

[0042] Ideally, facial beautification could be performed on each image frame in the video stream. However, since the solution in this embodiment is performed locally on the client's terminal device, and the data processing capabilities of different users' terminal devices vary, the specific implementation can be determined based on the data processing capabilities of the specific terminal device. Specifically, if the terminal device's data processing capability is insufficient, frame extraction can be performed on the image frames in the acquired video stream so that only the extracted image frames are used as the target image, while other image frames are discarded. That is, assuming the video stream's frame rate is 60 frames per second, but the terminal device's current data processing capability can only process 20 frames per second, frame extraction can be performed, meaning 20 frames per second are extracted as the target image, and other frames are discarded. Of course, in practical implementation, factors such as video smoothness can also be considered. For example, suppose that a certain terminal device can only process 10 frames per second when executing the algorithm in this embodiment. However, since a video that can guarantee smoothness for human visual recognition requires at least 15 frames per second, if frame extraction is performed at 10 frames per second, it will not meet the smoothness requirement for human visual recognition. In this case, the user can be prompted that the current terminal device does not support the current algorithm, and so on.

[0043] In addition, in a specific implementation, a switch option can be provided in the client interface, allowing users to choose to enable the facial image processing function provided in this application embodiment. Once enabled, facial image processing can be performed on image frames in the captured video stream. Furthermore, the intensity of facial image processing can be selected through the client interface.

[0044] S302: On the client side, a target algorithm model adapted to the data processing capabilities of the client's terminal device is used to process the face region in the target image. The target algorithm model is generated after training on a training set, which includes multiple image pairs. Each image pair includes a first image with facial blemishes and a second image after removing the facial blemishes, so that the target algorithm model is trained with the goal of removing facial blemishes from the input image.

[0045] The target algorithm model in this step can be a deep learning model, specifically the GAN network mentioned earlier. To make the specific algorithm model more suitable for execution on the client side, in optional implementations, the standard GAN network can be simplified by modifying the network and / or performing model sparsity processing. Since data processing on different terminal devices will vary, and there may even be significant differences, multiple versions of the algorithm model can be provided. These versions may have different algorithmic complexities and processing effects, and correspondingly, different requirements for the data processing capabilities of the terminal devices, thus adapting to terminal devices with different data processing capabilities.

[0046] After determining the structure of the algorithm model, it can be trained. Specifically, training the algorithm model involves first constructing a training dataset. In this embodiment, the training dataset may include multiple image pairs, each pair containing a first image with facial blemishes and a second image after removing the blemishes. For example, in practice, first images with facial blemishes (including pimples, spots, wrinkles, especially large clusters of pimples and spots) can be collected. Then, by manually retouching images using image processing software, the facial blemishes are removed, resulting in second images corresponding to each first image. This ensures that the first and second images are identical except for the presence or absence of blemishes and a smoother, more detailed face; other background information remains identical. This allows the algorithm model to clearly identify the specific areas requiring processing and the appropriate level of processing during the learning process. After training the algorithm model in this way, it gains the necessary inference and computational capabilities to remove facial blemishes from the output image. Compared with the simple edge-preserving filtering process in existing technologies, this method can achieve better processing results and can remove facial blemishes more cleanly.

[0047] After training the algorithm model, it can be deployed to the client. Then, after capturing a live video stream through the client, the algorithm model can be used to perform facial image processing on the target images within it.

[0048] While the complexity of the standard GAN network has been controlled to a level suitable for operation on terminal devices and speeds up execution through network modifications and model sparsification, in the preferred embodiment, to better meet the real-time and processing quality requirements of facial image processing in live video streaming, the hardware capabilities of the terminal device can be fully utilized to further improve real-time performance and processing quality. Specifically, since the specific algorithm model is usually based on a deep learning model of artificial neural networks, the task of processing the facial region in the target image using the target algorithm model can be assigned to the GPU chip or NPU chip in the terminal device associated with the client. Since NPU chips are better at processing artificial neural network algorithms than GPU chips, if the terminal device is equipped with an NPU chip, it can be used first to complete the execution of the aforementioned GAN network and other algorithms; if no NPU chip is available, then the GPU chip can be considered. In this way, since NPUs are more efficient and perform better than CPUs in processing neural network algorithms, and given the large number of algorithms running in live streaming scenarios, resulting in a relatively high CPU load, offloading some of the computation to GPUs or NPUs can effectively reduce the CPU load, thereby improving overall performance. This approach can achieve speedups without changing the complexity of the algorithm model itself, or it can improve edge processing performance by appropriately increasing the computational complexity of the algorithm model.

[0049] In practice, since the CPU is usually the core of data processing, the program in the CPU can assign the specific tasks to be processed to the GPU or NPU chip. The GPU or NPU then returns the processed results to the CPU, which then performs the next task scheduling.

[0050] Specifically, the processing result of a GAN network on an input image can be either a beautified image or a residual image, which is the difference between the images before and after beautification. For example, assuming the image before processing is image A and the image after processing is image B, the GAN network can directly output image B, or it can output a residual image between B and A, i.e., BA. If the GAN network outputs image B, the processed image can be obtained by overlaying image B onto image A. If the GAN network outputs the residual image BA, the processed image can be obtained by superimposing image BA onto image A.

[0051] Specifically, further speed-up can be achieved through frame overlay. Specifically, the target image can be judged to determine if it is a keyframe in the live video stream. If it is a keyframe, the facial region in the current target image can be processed using a GAN network, followed by the aforementioned overlay or overlay processing. If it is not a keyframe, the processing results of cached historical frames (e.g., the previous frame) can be retrieved, and image warping processing can be applied to those historical frames before overlaying or overlaying the current target image. Specifically, the image warping processing can be implemented based on the differences between the current target image and the historical frames, etc., which will not be detailed here. In this way, not all target images need to undergo inference calculations through the GAN network, thus further improving processing speed and ensuring real-time performance. Regarding the identification of keyframes, they can be determined based on their similarity to the previous frame. Simply put, if a target image is very similar to the image in its previous frame, it is usually not a keyframe. Conversely, if the similarity is low, it may indicate a change in camera angles or a change in the subject being explained, thus making it a keyframe.

[0052] It should be noted that when processing the face region image in the target image using algorithms such as GAN networks, the original target image can be directly input into the GAN network for processing. Alternatively, in a preferred embodiment, to further reduce the processing difficulty of algorithms such as GAN networks, some preprocessing can be performed on the target image. This preprocessing can specifically include face detection in the target image, and then the detected face region image can be input into the GAN network for inference calculation. In this way, since the influence of non-face regions in the target image can be removed, the inference calculation efficiency of the GAN network can be improved. In addition, it can also avoid the blurring caused by the misprocessing of non-face regions. In this way, since the GAN network only processes the face region within the face detection box, the original face region image in the target image is overlaid or superimposed using the face region image processed by the GAN network to complete the processing of the target image.

[0053] From a hardware perspective, face detection algorithms can be performed on a CPU, and their output can be an image located within a rectangular face detection box. After obtaining the rectangular face detection box, the task of inference calculation based on GAN networks can be assigned to an NPU or GPU chip to remove blemishes such as "pimples" and spots.

[0054] Furthermore, since face detection bounding boxes are typically rectangular, and the actual face region is smaller than the area of ​​this rectangle, directly overlaying or replacing the image within the rectangular face detection bounding box may result in unexpected overlay effects on non-face regions within the bounding box. Therefore, to further improve the algorithm's processing performance and reduce the impact on image content in non-face regions, multiple facial key points can be identified from the target image, generating a facial key point map and a binary mask image. This mask image is used to determine the location of regions within the face contour area in the target image. In other words, by recognizing facial key points, the specific areas to be covered or overlaid can be precisely located within the facial contour area, which is smaller than the aforementioned face detection bounding box, thus further avoiding the impact on non-face regions. With the generated facial key point map and mask image, the processed face region image can be used to cover or overlay images within the face contour area of ​​the target image.

[0055] Specifically, after obtaining the facial landmark map and mask image, the mask image and the face in the original image can be mapped together through image rendering, that is, the mask image displayed on the screen is obtained through rendering. The facial landmark map and mask image primarily indicate the more precise location of the facial region in each frame of the image. Based on these facial landmark map and mask image, the specific areas for facial beautification processing can be well controlled, ensuring that facial beautification only optimizes the facial skin areas that require optimization. Rendering and displaying the facial mask image on the screen (i.e., displaying the mask image on the screen, but not directly to the user; "displaying on the screen" here mainly refers to displaying the mask image on an offline screen (FBO, Frame Buffer Object, offline buffer) for the program to read data from) is mainly to optimize the computing efficiency of mobile devices and limit the maximum number of faces that can be processed. In practice, it can be any number of faces, or there can be no specific limit, depending on the computing efficiency requirements.

[0056] It's important to note that the set of facial landmarks S output by common machine learning models typically only covers the facial features below the eyebrows and the facial edges. To cover the entire face, set S can be expanded into a set S_ext. In other words, during facial landmark recognition, a first set of basic facial landmarks is identified using a facial landmark recognition algorithm. Then, based on these basic landmarks, a second set of extended facial landmarks is determined. Finally, the first set of basic landmarks and the second set of extended facial landmarks are defined as the facial landmarks. Specifically, the facial landmark segmentation map is formed by connecting points in set S_ext into triangles. The landmark expansion can generally be generated using mathematical calculations or topological rules; the specific generation method is not limited.

[0057] In practice, the generation of the facial landmark subdivision map and mask map can be performed on the CPU, and this step can be processed in parallel with face detection and GAN network inference computation. The part that renders the mask map can be implemented on the GPU to speed up the process.

[0058] It should be noted that, in the case of generating a facial landmark bitmap, the aforementioned inter-frame overlay processing method can be used to overlay the facial landmark bitmap of the current target image with the residual image of the previous frame after the image bending and deformation processing to obtain the processed image.

[0059] Furthermore, to further improve the processing effect and make the face in the target image more delicate and smooth, after processing the face region in the target image using the target algorithm model, a further edge-preserving smoothing process can be applied to the processed face region, thereby producing a further skin-smoothing effect. Specifically, to improve efficiency, the task of performing the above-mentioned edge-preserving smoothing process on the processed face region can also be assigned to a GPU chip for execution.

[0060] There are various specific edge-preserving smoothing algorithms. For example, one approach involves first extracting and enhancing edges. Specifically, the image can be first subjected to Gaussian blurring, then the variance map before and after blurring can be calculated. Noise suppression is then performed based on this variance map to obtain a smoothed variance map. This smoothed map is then fused with a face mask image to achieve a smoothed "skin-smoothing" effect. To achieve a better "skin-smoothing" effect, the smoothing process can be performed by region, i.e., different smoothing processes can be applied to different areas of the face, such as the eyes, nose, and mouth. In practice, when recognizing facial landmarks, region labels can be added to specific landmarks to distinguish different regions during the smoothing process, and so on.

[0061] In practical implementation, to maximize processing efficiency, the above algorithms can be combined. For example, in a real-world application, such as... Figure 4 As shown, the specific steps may include: 1. First, face detection can be performed on the target image (optionally, correction processing can also be performed to improve accuracy) to obtain the face image within the rectangular detection box; 2. The residual map is obtained by performing inference calculations on the image located within the rectangular face detection box using a GAN network; 3. Using predefined face segmentation and pre-made mask images, perform facial landmark detection on the target image to generate a facial landmark bitmap and a mask image; 4. Display the Mask image on the screen; 5. Based on the facial landmark map and mask image, overlay the original image and the residual image; 6. Post-processing, for example, mainly involves further edge-preserving filtering of the overlaid images to achieve a "skin smoothing" effect. This step can include two sub-modules for better results: an edge extraction and enhancement sub-module, and a region-specific "skin smoothing" extraction sub-module.

[0062] S303: On the client side, a video stream is displayed based on the processed image.

[0063] After processing the target image, a video stream can be displayed on the client side based on the processed image. For example, the target image can be overlaid or superimposed based on the processing results output by the GAN network before direct rendering and display. Alternatively, it can be post-processed using the aforementioned algorithms before rendering and display, and so on. This allows the user at the acquisition end to view the processed effect on the display screen. Because this embodiment of the application ensures real-time processing through the fusion of multiple technologies, the client side will not experience significant processing delays, thus guaranteeing a good user experience.

[0064] It's important to note that in live video streaming scenarios, there's also the process of the client pushing the live video stream to the server. Specifically, the client can directly push the processed live video stream to the server. Alternatively, since the server equipment typically has stronger data processing capabilities and can deploy more powerful algorithms, potentially achieving better processing results, in an optional real-time mode, the raw live video stream generated on the client side can also be pushed to the server. This allows the server to process the facial regions in the raw live video stream before providing it to the viewer's client.

[0065] In summary, through the embodiments of this application, during the acquisition of video streams on the client side (e.g., the client on the broadcaster's side in a live streaming scenario), the target image to be processed can be determined based on multiple video image frames in the video stream, and the face region in the target image can be processed directly on the client side using a target algorithm model. Specifically, the target algorithm model can be adapted to the data processing capabilities of the client's terminal device, and can be generated after training on a training set. The training set includes multiple image pairs, each including a first image with facial blemishes and a second image after removing the facial blemishes, so that the target algorithm model is trained with the goal of removing facial blemishes from the input image. After processing the face region in the target image, the video stream can be displayed on the client side based on the processed image. In this way, since a pre-trained algorithm model is used to remove blemishes in the face, the blemish processing effect can be improved compared to the blemish processing method of edge-preserving filtering in the prior art. Furthermore, since face region processing can be performed directly on the client side of image frames in the video stream without requiring network transmission or transcoding, real-time processing is guaranteed. Therefore, the solution provided in this application embodiment can balance processing effectiveness and real-time performance when processing face images, achieving better facial image processing results in scenarios such as live video streaming while meeting real-time requirements.

[0066] Furthermore, in a preferred embodiment, based on the inference calculation using the target algorithm model described above, it can be integrated with various other technologies to further ensure the real-time performance and processing effect. In terms of algorithms, preprocessing for face detection in the target image can be included to reduce the complexity of the target algorithm model's inference calculations, thereby accelerating the process. Additionally, by generating a face keypoint bitmap and a binarized mask image from the target image, the processing area can be controlled within the facial contour, preventing the clarity of other areas from being affected. Moreover, it can be integrated with edge-preserving filtering techniques to achieve further skin smoothing, thereby further improving the removal effect of facial blemishes.

[0067] On the hardware side, the capabilities of chips such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and NPU (Neural Networks Processing Unit) can be fully utilized to further accelerate the algorithm, thereby ensuring the real-time performance of facial image processing.

[0068] Example 2 In the aforementioned Embodiment 1, facial image processing is primarily performed by a pre-trained algorithm model, directly on the client side, ensuring both real-time processing and quality. Subsequently, various hardware and software technologies can be integrated to further improve efficiency or enhance processing performance. In other processing methods, image preprocessing can reduce the algorithm's complexity, while fully utilizing the capabilities of NPU, GPU, and other chips in the terminal device to achieve a balance between processing quality and real-time performance. Therefore, Embodiment 2 primarily provides a method for facial image processing of video streams from this perspective. For details, see [link to documentation]. Figure 5 The method may include: S501: During the acquisition of video stream on the client side, the target image to be processed is determined based on multiple video image frames in the video stream; S502: Generate a first task and assign the first task to the central processing unit (CPU) chip in the terminal device associated with the client for execution; the first task includes: using a first algorithm model to perform face region detection on the target image to segment the face region image located within a rectangular detection box; S503: Generate a second task and assign it to the GPU chip or NPU chip in the terminal device; the second task includes: processing the face region image located within the rectangular detection box using a second algorithm model; wherein, the second algorithm model is generated after training on a training set, the training set including multiple image pairs, the image pairs including a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; the second algorithm model is adapted to the data processing capabilities of the terminal device associated with the client; S504: Generate a third task and assign the third task to the GPU chip for execution; the third task includes: using the processing results returned by the second algorithm model to cover or overlay the face region in the target image so that the processed image can be displayed as a video stream on the client side.

[0069] In a specific implementation, the first task may further include: using a third algorithm model to perform facial key point recognition on the target image, and generating a facial key point map and a binarized mask map, so as to determine the location of the region within the facial contour range in the target image; thus, the third task may specifically include: based on the facial key point map and the binarized mask map, using the processing results returned by the second algorithm model to cover or overlay the facial region in the target image, so as to generate an image after processing the region within the facial contour range in the target image.

[0070] In addition, a fourth task can be generated and assigned to the GPU chip for execution. The fourth task includes rendering the binarized mask and displaying it on the screen. The displaying on the screen includes displaying the binarized mask on the screen corresponding to the offline buffer to limit the number of faces to be processed, thereby further speeding up the process.

[0071] Furthermore, the steps of generating the facial key point map and mask map can be processed in parallel with the face detection step by starting multi-threading based on the CPU, which is also conducive to speeding up the process.

[0072] Furthermore, after the overlay processing is performed based on the GPU chip, a fifth task can be generated, which involves using a fourth algorithm model to perform edge-preserving smoothing processing on the face region in the overlaid or overlaid image, and then rendering and displaying the edge-preserving smoothed image. This task is then assigned to the GPU chip for execution.

[0073] To better understand the solutions provided in the embodiments of this application, the interaction process between multiple chips and other terminals within a terminal device is described below using an example from a specific application. Specifically, for example... Figure 6 As shown, the specific processing interaction flow may include: 1. First, the specific target image to be processed is generated in the video live streaming and other clients; 2. Generate a task to perform face detection on the target image and assign it to the CPU chip for execution; 3. Generate a task that uses algorithms such as GAN networks to process the image content located within the face detection bounding box in the target image, and assign it to the NPU; 4. The NPU chip performs inference calculations by executing algorithms such as GAN networks to process facial blemishes and generate a residual map between the images before and after processing. 5. The NPU side returns the residual map to the CPU side; 6. Generate a task to detect facial landmarks in the target image, so as to generate a facial landmark bitmap and a mask image, and assign it to the CPU chip for execution; this step can be performed in parallel with step 2. 7. Through the CPU-side data scheduling module, a task is generated to cover or overlay the face region in the target image using the processing result returned by the second algorithm model, and assigned to the GPU chip for execution. The original image of the target image, the residual image, the face key point bitmap, and the mask image are provided to the GPU. 8. Generate a task that overlays the original image of the target image with the residual image based on the facial key point bitmap and the mask image, and assign it to the GPU for execution; 9. Generate a task to perform edge-preserving filtering on the overlayed image and assign it to the GPU for execution; 10. Render the processed image and output it to the display screen for display.

[0074] Example 3 Regarding the multi-technology fusion processing scheme at the algorithm level mentioned in the foregoing embodiments, it can be applied not only to the client side of video live streaming scenarios, but also to the server side, the cloud side, or facial image processing software, etc. When the real-time requirements for processing are not high, it is not necessary to integrate with hardware technologies to achieve the same goal of improving the processing effect of facial blemishes such as acne and spots. Therefore, this third embodiment also provides a facial image processing method, in which the executing entity of each step can be any client, server, cloud side, or dedicated facial image processing software with a facial image processing system, etc. Specifically, such as... Figure 7 As shown, the method may include: S701: Determine the target image to be processed; S702: Use the first algorithm model to detect the face region in the target image, so as to segment the face region image located within the rectangular detection box; S703: The face region in the target image is processed using a target algorithm model, wherein the target algorithm model is generated after training on a training set, the training set includes multiple image pairs, the image pairs include a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; S704: Use the third algorithm model to perform facial key point recognition on the target image, and generate a facial key point map and a binary mask map, so as to determine the location of the region within the facial contour range from the target image. S705: Based on the facial key point map and the binarized mask map, the face region in the target image is covered or overlaid using the processing result returned by the second algorithm model, so as to generate an image after processing the region within the facial contour range in the target image.

[0075] In an optional implementation, a fourth algorithm model can be used to perform edge-preserving smoothing on the face region, and the image after edge-preserving smoothing can be rendered and displayed.

[0076] For the parts of Embodiments 2 and 3 that are not described in detail above, please refer to Embodiment 1 and other parts of this specification. They will not be repeated here.

[0077] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0078] Corresponding to Embodiment 1, this application also provides an apparatus for facial image processing of a video stream, see [link to embodiment]. Figure 8 The device may include: The target image determination unit 801 is used to determine the target image to be processed based on multiple video image frames in the video stream during the process of acquiring the video stream on the client side. The inference computing unit 802 is used on the client side to process the face region in the target image using a target algorithm model adapted to the data processing capabilities of the terminal device where the client is located. The target algorithm model is generated after training on a training set, which includes multiple image pairs. The image pairs include a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image. The display unit 803 is used to display a video stream based on the processed image on the client side.

[0079] The target algorithm model may include a simplified generative adversarial network model obtained by modifying the network and / or sparsifying the model based on the standard generative adversarial network model.

[0080] Specifically, the inference computing unit can be used for: If the terminal device associated with the client is equipped with a graphics processing unit (GPU) chip or a neural network processing unit (NPU) chip, the task of executing the target algorithm model to process the face region in the target image will be assigned to the GPU chip or NPU chip, so as to utilize the hardware processing capabilities of the GPU chip or NPU chip to complete the processing of the face region in the target image.

[0081] In an optional implementation, to reduce the difficulty of inference computation processing, the device may further include: A face detection unit is used to detect face regions in the target image to segment the face region image located within a rectangular detection box; At this point, the inference calculation unit can specifically be used for: The face region image located within the rectangular detection box is input into the target algorithm model so that the target algorithm model can be used to process the face region image located within the rectangular detection box. The device may further include: The overlay or overlay processing unit is used to overlay or overlay the original face region image in the target image using the face region image processed by the target algorithm model, so as to display the video stream based on the overlay or overlay processed image.

[0082] The output information of the target algorithm model includes: residual images of the face region before and after processing.

[0083] Additionally, the device may also include: A facial key point recognition unit is used to identify multiple facial key points from the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range from the target image. At this time, the overlay or superimposition processing unit can specifically be used for: Based on the facial key point map and the binarized mask map, the face region image processed by the target algorithm model is used to cover or overlay the image within the facial contour range of the target image.

[0084] Specifically, the facial key point recognition unit can be used for: Using a facial landmark recognition algorithm, the first number of basic facial landmarks are identified; Based on the basic facial key points, expand outwards to determine a second number of expanded facial key points; The first number of basic facial landmarks and the second number of extended facial landmarks are determined as the facial landmarks.

[0085] Furthermore, the device may also include: The edge-preserving smoothing processing unit is used to perform edge-preserving smoothing processing on the face region in the target image after processing it using the target algorithm model, so as to display the video stream based on the image after edge-preserving smoothing processing.

[0086] In an optional manner, the edge-preserving smoothing processing unit can specifically be used for: If the terminal device associated with the client is equipped with a GPU chip, the edge-preserving smoothing process is assigned to the GPU so as to utilize the hardware processing power of the GPU chip to perform edge-preserving smoothing on the processed face area.

[0087] In a specific implementation, the device may further include: A keyframe determination unit is used to determine whether the target image belongs to a keyframe in the live video stream; The processing unit is configured to, if it is a keyframe, trigger the execution of a step to process the keyframe using the target algorithm model; otherwise, obtain the processing result of the non-keyframe by performing image deformation processing on the processing result corresponding to the historical frame.

[0088] The live video stream includes live video streams in which people explain or demonstrate products.

[0089] The target image determination unit can specifically be used for: Based on the data processing capability information of the terminal device associated with the client, the image frames in the video stream are extracted so that the extracted image frames are used as the target image, and the other image frames are discarded.

[0090] Corresponding to Embodiment 2, this application also provides an apparatus for facial image processing of a video stream, see [link to embodiment]. Figure 9 The device may include: The target image determination unit 901 is used to determine the target image to be processed based on multiple video image frames in the video stream during the process of acquiring the video stream on the client side. The face detection unit 902 is used to generate a first task and assign the first task to the central processing unit (CPU) chip in the terminal device associated with the client for execution; the first task includes: using a first algorithm model to perform face region detection on the target image to segment the face region image located within a rectangular detection box; The inference computing unit 903 is used to generate a second task and assign the second task to the GPU chip or NPU chip in the terminal device; the second task includes: processing the face region image located within the rectangular detection box using a second algorithm model; wherein, the second algorithm model is generated after training on a training set, the training set including multiple image pairs, the image pairs including a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image; the second algorithm model is adapted to the data processing capabilities of the terminal device associated with the client; The overlay or superimposition processing unit 904 is used to generate a third task and assign the third task to the GPU chip for execution; the third task includes: using the processing result returned by the second algorithm model to overlay or superimpose the face region in the target image so that the processed image can be displayed as a video stream on the client side.

[0091] Specifically, the first task may also include: The third algorithm model is used to identify facial key points in the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range in the target image. The third task may specifically include: When performing the overlay processing based on the graphics processor chip, the face region in the target image is covered or overlaid using the processing result returned by the second algorithm model according to the face key point bitmap and the binarized mask map, so as to generate an image after processing the region within the face contour range in the target image.

[0092] Additionally, the device may also include: The binarized mask image display unit is used to generate a fourth task and assign the fourth task to the GPU chip for execution. The fourth task includes rendering the binarized mask image and displaying it on the screen. The display includes displaying the binarized mask image on the screen corresponding to the offline buffer to limit the number of faces to be processed.

[0093] Additionally, the device may also include: The edge-preserving smoothing processing unit is used to generate a fifth task after the graphics processor chip performs the overlay or overlay processing, and assign it to the GPU chip. The fifth task includes: using a fourth algorithm model to perform edge-preserving smoothing processing on the face region in the overlay or overlay processing image, and rendering and displaying the edge-preserving smoothed image.

[0094] Corresponding to Embodiment 3, this application also provides a face image processing device, see [link to embodiment]. Figure 10 The device may include: The target image determination unit 1001 is used to determine the target image to be processed; The face detection unit 1002 is used to perform face region detection on the target image using a first algorithm model, so as to segment the face region image located within the rectangular detection box; The inference computing unit 1003 is used to process the face region in the target image using a second algorithm model. The second algorithm model is generated after training on a training set. The training set includes multiple image pairs, including a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image. The facial key point recognition unit 1004 is used to perform facial key point recognition on the target image using a third algorithm model, and generate a facial key point map and a binary mask map, so as to determine the location of the region within the facial contour range in the target image. The overlay or overlay processing unit 1005 is used to overlay or overlay the face region in the target image based on the face key point bitmap and the binarized mask map, using the processing result returned by the second algorithm model, so as to generate an image after processing the region within the face contour range in the target image.

[0095] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0096] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0097] in, Figure 11 The architecture of an electronic device is illustrated by example. For example, device 1100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, aircraft, etc.

[0098] Reference Figure 11 The device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power supply component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0099] Processing component 1102 typically controls the overall operation of device 1100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1102 may include one or more processors 1120 to execute instructions to complete all or part of the steps of the methods provided in this disclosure. For example, in embodiments of this application, the processor 1120 in processing component 1102 may specifically include a CPU, GPU, NPU, etc. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0100] Memory 1104 is configured to store various types of data to support the operation of device 1100. Examples of this data include instructions for any application or method operating on device 1100, contact data, phonebook data, messages, pictures, videos, etc. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0101] Power supply component 1106 provides power to various components of device 1100. Power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1100.

[0102] Multimedia component 1108 includes a screen that provides an output interface between device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0103] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0104] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0105] Sensor assembly 1114 includes one or more sensors for providing status assessments of various aspects of device 1100. For example, sensor assembly 1114 may detect the on / off state of device 1100, the relative positioning of components such as the display and keypad of device 1100, changes in the position of device 1100 or a component of device 1100, the presence or absence of user contact with device 1100, the orientation or acceleration / deceleration of device 1100, and temperature changes of device 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0106] Communication component 1116 is configured to facilitate wired or wireless communication between device 1100 and other devices. Device 1100 can access wireless networks based on communication standards, such as WiFi, or mobile communication networks such as 2G, 3G, 4G / LTE, and 5G. In one exemplary embodiment, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1116 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0107] In an exemplary embodiment, device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0108] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, which can be executed by a processor 1120 of device 1100 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0109] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0110] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0111] The foregoing has provided a detailed description of the method and electronic device for facial image processing of video streams provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for facial image processing of a video stream, characterized in that, include: During the process of acquiring video streams on the broadcaster's client side in a live video streaming scenario, the target image to be processed is determined based on multiple video image frames in the video stream. When the terminal device where the broadcaster's client is located is equipped with a graphics processing unit (GPU) chip or a neural network processing unit (NPU) chip, a target algorithm model adapted to the data processing capabilities of the client's terminal device is determined. It is then determined whether the target image is a keyframe in the video stream. If so, the task of processing the face region in the target image using the target algorithm model is assigned to the GPU chip or NPU chip. This allows the hardware processing capabilities of the GPU chip or NPU chip to complete the processing of the face region in the target image, resulting in a processed face region image. Based on this processed face region image, a processed image is obtained through overlay or superposition processing. The target algorithm model... It is a simplified version of a standard deep learning model obtained by modifying the network and / or sparsifying the model, and is pre-trained on a training set including multiple image pairs. Each image pair includes a first image with facial blemishes and a second image after removing the facial blemishes, so as to train the target algorithm model with the goal of removing facial blemishes from the input image. The processed image includes: an image obtained by overlaying the processed face region image on the original face region image in the target image, or an image obtained by overlaying a residual image on the original face region image in the target image, wherein the residual image is the difference image between the processed face region image and the original image. On the broadcaster's client, a video stream is displayed based on the processed image to show the broadcaster the facial image processing effect in real time. When pushing the stream to the live video server, the original captured video stream is pushed so that the server processes the facial region in the target image based on a standard deep learning model and provides it to the viewer's client.

2. The method according to claim 1, characterized in that, Also includes: The target image is subjected to face region detection to segment the face region image located within the rectangular detection box; The task of processing the face region in the target image using the target algorithm model is assigned to the GPU chip or NPU chip, so that the hardware processing capabilities of the GPU chip or NPU chip can be used to complete the processing of the face region in the target image, including: The face region image located within the rectangular detection box is input into the target algorithm model so that the target algorithm model can be used to process the face region image located within the rectangular detection box.

3. The method according to claim 2, characterized in that, Also includes: Multiple facial key points are identified from the target image, and a facial key point map and a binarized mask map are generated to determine the location of the region within the facial contour area from the target image. The process of obtaining the processed image based on the processed face region image through overlay or superposition includes: Based on the facial key point map and the binarized mask map, the processed facial region image is used to cover or overlay the image within the facial contour range of the target image.

4. The method according to claim 3, characterized in that, The identification of multiple facial key points from the target image includes: Using a facial landmark recognition algorithm, the first number of basic facial landmarks are identified; Based on the basic facial key points, expand outwards to determine a second number of expanded facial key points; The first number of basic facial landmarks and the second number of extended facial landmarks are determined as the facial landmarks.

5. The method according to claim 1, characterized in that, Also includes: After processing the face region in the target image, the processed face region is smoothed with edge preservation so that a video stream can be displayed based on the image after edge preservation and smoothing.

6. The method according to claim 5, characterized in that, The process of smoothing the processed face area while preserving its edges includes: If the terminal device associated with the client is equipped with a GPU chip, the edge-preserving smoothing process is assigned to the GPU so as to utilize the hardware processing power of the GPU chip to perform edge-preserving smoothing on the processed face area.

7. The method according to claim 1, characterized in that, Also includes: If the target image is not a key frame in the video stream, the processing result of the non-key frame is obtained by performing image deformation processing on the processing result corresponding to the historical frame.

8. A method for facial image processing of a video stream, characterized in that, include: During the process of acquiring video streams on the broadcaster's client side in a live video streaming scenario, the target image to be processed is determined based on multiple video image frames in the video stream. A first task is generated and assigned to the central processing unit (CPU) chip in the terminal device associated with the client for execution. The first task includes: using a first algorithm model to detect face regions in the target image, so as to segment the face region image located within a rectangular detection box; When the terminal device where the broadcaster's client is located is equipped with a graphics processing unit (GPU) chip or a neural network processing unit (NPU) chip, a second task is generated and assigned to the GPU chip or NPU chip in the terminal device. The second task includes: determining whether the target image is a keyframe in the video stream; if so, processing the face region image located within the rectangular detection box using a second algorithm model to obtain a processed face region image. The second algorithm model is generated after training on a training set, which includes multiple image pairs, including a first image with facial blemishes and a second image after removing the facial blemishes, so that the second algorithm model is trained with the goal of removing facial blemishes from the input image. The second algorithm model is adapted to the data processing capabilities of the terminal device associated with the client. The second algorithm model is simplified based on a standard deep learning model by modifying the network and / or performing model sparsity processing. A third task is generated and assigned to the GPU chip for execution. The third task includes: using the processing results returned by the second algorithm model, obtaining a processed image through overlay or superposition processing, so that the video stream can be displayed on the broadcaster's client based on the processed image, thereby showing the facial image processing effect to the broadcaster in real time. When pushing the stream to the live video server, the original captured video stream is pushed, so that the server processes the face region in the target image based on a standard deep learning model and provides it to the viewer's client. The processed image includes: an image obtained by overlaying the processed face region image onto the original face region image in the target image, or an image obtained by superimposing a residual image onto the original face region image in the target image, wherein the residual image is the difference image between the processed face region image and the original image.

9. The method according to claim 8, characterized in that, The first task also includes: The third algorithm model is used to identify facial key points in the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range in the target image. The third task specifically includes: Based on the facial key point map and the binarized mask map, the processing results returned by the second algorithm model are used to cover or overlay the facial region in the target image, so as to generate an image after processing the region within the facial contour range in the target image.

10. The method according to claim 9, characterized in that, Also includes: A fourth task is generated and assigned to the GPU chip for execution. The fourth task includes rendering the binarized mask and displaying it on the screen. The displaying on the screen includes displaying the binarized mask on the screen corresponding to the offline buffer to limit the number of faces to be processed.

11. A method for processing human face images, characterized in that, include: Identify the target image to be processed; The first algorithm model is used to detect face regions in the target image to segment the face region image located within the rectangular detection box; If the target image is a keyframe in the video stream, then the face region in the target image is processed using a second algorithm model to obtain a processed face region image. The second algorithm model is generated after training on a training set, which includes multiple image pairs. Each image pair includes a first image with facial blemishes and a second image after removing the facial blemishes, so that the second algorithm model is trained with the goal of removing facial blemishes from the input image. The second algorithm model is simplified based on a standard deep learning model by modifying the network and / or performing model sparsity processing. The third algorithm model is used to identify facial key points in the target image and generate a facial key point map and a binarized mask map to determine the location of the region within the facial contour range in the target image. Based on the facial key point map and the binarized mask map, using the processing results returned by the second algorithm model, an image is generated by processing the region within the facial contour range of the target image through overlay or superposition processing. The processed image includes: an image obtained by overlaying the processed facial region image on the original facial region image in the target image, or an image obtained by superimposing a residual image on the original facial region image in the target image, wherein the residual image is the difference image between the processed facial region image and the original image.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program performs the steps of the method described in any one of claims 1 to 11.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 11.

Images