Image processing method and apparatus, electronic device

Abstract

The application relates to an image processing method and device, electronic equipment and a computer readable storage medium, comprising: a second thread acquires a current to-be-processed image and a current hair-filament-level portrait segmentation mask corresponding to the current to-be-processed image, and performs image processing on the current to-be-processed image according to the current hair-filament-level portrait segmentation mask to obtain a corresponding current target image; meanwhile, a first thread acquires a next image of the current to-be-processed image, performs hair-filament-level portrait segmentation on the next image, and obtains a hair-filament-level portrait segmentation mask corresponding to the next image; the hair-filament-level portrait segmentation mask corresponding to the next image is used for input to the second thread, the processing efficiency of hair-filament-level image processing on a video is improved, and the fluency of the video is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method and apparatus, electronic devices, and computer-readable storage media. Background Technology

[0002] Current methods for generating portrait masks typically employ matting to extract the portrait image, then use the results for various processing applications such as blurring, exposure adjustment, and focus adjustment. With increasing demands for image quality, portrait masks often need to accurately extract even individual hair strands, requiring significant computation and thus consuming considerable time. In addition to hair-level portrait matting, post-processing is also necessary. Therefore, the time consumed per frame is the sum of the time spent on portrait matting and post-processing, further increasing the overall time consumption of the solution. Summary of the Invention

[0003] This application provides an image processing method and apparatus, an electronic device, and a computer-readable storage medium. By generating a hair-level portrait segmentation mask and performing image post-processing through multi-threaded parallel processing, it saves time, improves the processing efficiency of hair-level image processing on videos, and ensures the smoothness of the videos.

[0004] An image processing method, comprising:

[0005] The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and performs image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image;

[0006] Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread.

[0007] An image processing apparatus, comprising:

[0008] The second thread processing module is used to obtain the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and to perform image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image;

[0009] The first thread processing module is used to obtain the next image of the current image to be processed while the second thread processing module is processing, perform hair-level portrait segmentation on the next image, and obtain the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used to input to the second thread.

[0010] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the following steps:

[0011] The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and performs image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image;

[0012] Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread.

[0013] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor, causing the processor to perform the following steps:

[0014] The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and performs image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image;

[0015] Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread.

[0016] The aforementioned image processing method, apparatus, electronic device, and computer-readable storage medium acquire the current image to be processed and the corresponding current hair-level portrait segmentation mask through a second thread. Based on the current hair-level portrait segmentation mask, image processing is performed on the current image to be processed to obtain the corresponding current target image. Simultaneously, the first thread acquires the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the corresponding hair-level portrait segmentation mask. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread. The second thread and the first thread can complete the image processing tasks in parallel, enabling image rendering and image segmentation tasks to be processed in parallel. By generating the hair-level portrait segmentation mask and performing post-processing through multi-threaded parallel processing, time is saved, the processing efficiency of hair-level image processing on video is improved, and the smoothness of the video is guaranteed. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0019] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0020] Figure 3 This is a schematic diagram of the image processing flow where thread 1 and thread 2 perform image processing in parallel in one embodiment;

[0021] Figure 4 This is a schematic diagram of the process for obtaining a hair-level portrait segmentation mask in one embodiment;

[0022] Figure 5 This is a flowchart illustrating an explicit ternary image matting algorithm in one embodiment;

[0023] Figure 6 This is a flowchart illustrating an implicit ternary image matting algorithm in one embodiment;

[0024] Figure 7 This is a schematic diagram of a single codec structure in one embodiment;

[0025] Figure 8 This is a comparative diagram of the input original image, the corresponding ternary portrait segmentation mask, and the hair-level portrait segmentation mask in one embodiment;

[0026] Figure 9 This is a flowchart illustrating the image processing method in another embodiment;

[0027] Figure 10 This is a structural block diagram of an image processing device in one embodiment;

[0028] Figure 11 This is a structural block diagram of an image processing apparatus in another embodiment;

[0029] Figure 12 This is a block diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] Figure 1 This is a diagram illustrating the application environment of an image processing method in one embodiment. For example... Figure 1 As shown, the application environment includes a terminal 110 and a server 120. The terminal 110 and server 120 can independently or collaboratively complete the image processing method. The terminal 110 or server 120 obtains the current image to be processed and the corresponding hair-level portrait segmentation mask through a second thread. Based on the current hair-level portrait segmentation mask, image processing is performed on the current image to obtain the corresponding current target image. Simultaneously, the first thread obtains the next image from the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the corresponding hair-level portrait segmentation mask. This mask is input to the second thread, which can be used to perform background blurring, portrait extraction, color retention, neon portrait effects, and other image processing on the image to be processed to obtain the target image. The terminal 110 can be a mobile phone, tablet computer, PDA (Personal Digital Assistant), in-vehicle computer, wearable device, or other terminal device. The server 120 can be a single server or a server cluster. Server 120 can also obtain the video to be processed from terminal 110 and return the processed target image to terminal 110 for display.

[0032] Figure 2 This is a flowchart of an image processing method in one embodiment. Figure 2 The image processing method shown can be applied to the aforementioned terminal 110 or server 120, including:

[0033] Step 202: The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and performs image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image.

[0034] The image to be processed can be an image captured by an electronic device, an image stored locally on the electronic device, or an image downloaded from the internet, and is not limited to these. The image to be processed can be a raw image (original image), a YUV image, or an RGB (Red, Green, Blue) image. An RGB image to be processed refers to an image in RGB format; raw images and YUV images can be converted to obtain the RGB image to be processed. A hair-level portrait segmentation mask is an image filter template used to identify the human figure in an image. It can obscure other parts of the image and filter out the human figure. When the human figure contains hair, and the hair strands are relatively clear, the hair-level portrait segmentation mask can clearly segment the hair ends, ensuring that the hair edge segmentation is consistent with the desired result.

[0035] The current image to be processed and the corresponding hair-level portrait segmentation mask can be normalized images. A normalized image can mean that all pixel values ​​in the image are in the range of 0 to 1, or that all pixel values ​​are in the range of -1 to 1, etc., and is not limited to these. Normalization can be achieved by subtracting the mean from the RGB three-channel values ​​of each pixel in the image, and then dividing by the variance of the pixel values ​​in the image.

[0036] The image to be processed can be an image of a preset size after being scaled up. The preset size can be customized or it can be a size that matches the hair-level portrait segmentation model.

[0037] Specifically, the second thread and the first thread are two parallel and different threads. The acquisition of raw image data can be performed through one or both of the first and second threads. If only one thread acquires the raw image data, the acquired raw image data needs to be passed to the other thread for use. After the first thread acquires the current image to be processed, it processes the current image to obtain the current hair-level portrait segmentation mask corresponding to the current image to be processed. Then, it passes the current hair-level portrait segmentation mask corresponding to the current image to be processed to the second thread. While the second thread processes the current image to be processed based on the current hair-level portrait segmentation mask to obtain the corresponding current target image, the first thread acquires the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. Thus, while the second thread is performing image rendering processing on the current image to be processed, the first thread is also synchronously and in parallel performing hair-level portrait segmentation on the next image of the current image to be processed.

[0038] Step 204: Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread.

[0039] Here, "simultaneously" refers to the parallel overlap between the first and second threads during image processing. This overlap can be either a complete overlap of their start and end times, or a partial overlap. The next image to be processed can be a temporally adjacent image acquired in real-time after the current image, or it can be the next image played sequentially within the same video.

[0040] Specifically, the first thread performs hair-level portrait segmentation on the next image. A hair-level portrait segmentation mask corresponding to the output image can be obtained through a hair-level portrait segmentation model. There are several different ways to perform hair-level portrait segmentation. In one embodiment, a ternary portrait segmentation mask is generated based on a portrait segmentation algorithm, and then combined with the original image input for hair-level portrait matting, i.e., an explicit ternary image portrait matting algorithm. The RGB image after image preprocessing is obtained and processed by the portrait segmentation algorithm to obtain a ternary portrait segmentation mask. After normalization, the ternary portrait segmentation mask is concatenated with the channels of the preprocessed RGB image and then input into the hair-level portrait matting algorithm module for processing, finally obtaining the hair-level portrait segmentation mask. The ternary portrait segmentation mask has three pixel values, such as 0, 1, and -1, or other three custom values. Each value represents a region, including the portrait region, background region, and transition region. The transition region is the area between the portrait region and the background region, and the transition region is generally the hair region.

[0041] In one embodiment, an implicit ternary image portrait matting algorithm is employed. This algorithm does not rely on a ternary portrait segmentation mask; instead, it directly performs hair-level portrait matting on the original image, obtaining a pre-processed RGB image. This image is then directly processed by a hair-level portrait matting model containing an encoder-decoder structure to output a hair-level portrait segmentation mask. Pixels in the hair-level mask have only two values, such as 0 and 1, or 0 and 255. 0 can represent the pixel value of the portrait contour region, where "hair-level" indicates that the hair contour of the portrait contour region can be accurately identified, with the hair portion identified as a pixel within the portrait contour region. 255 represents the pixel value of other regions in the mask besides the portrait contour region.

[0042] The first thread can proactively return the hair-level portrait segmentation mask corresponding to the next image to the second thread, or the second thread can send a retrieval request to the first thread when the time period arrives, and the first thread will then return the hair-level portrait segmentation mask corresponding to the next image to the second thread based on the retrieval request.

[0043] It's understandable that when the current image to be processed is the first frame, the first thread performs hair-level portrait segmentation on the first frame to obtain the corresponding hair-level portrait segmentation mask. At this time, the second thread, since it hasn't yet obtained any hair-level portrait segmentation masks for any image, can be idle or handle other tasks. After the first thread obtains the hair-level portrait segmentation mask for the first frame, the second thread acquires the first frame and its corresponding hair-level portrait segmentation mask for rendering. Simultaneously, the first thread performs hair-level portrait segmentation on the second frame, allowing the first and second threads to work in parallel, thus accelerating the output speed of the target image. Figure 3 The diagram shows a process flow diagram of image processing performed in parallel by two threads, Thread 1 and Thread 2.

[0044] In this embodiment, the image processing method involves a second thread acquiring the current image to be processed and its corresponding hair-level portrait segmentation mask. Based on this mask, the second thread performs image processing on the current image to obtain the corresponding current target image. Simultaneously, the first thread acquires the next image from the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains its corresponding hair-level portrait segmentation mask. This mask is then input to the second thread. The second and first threads can complete image processing tasks in parallel, enabling image rendering and image segmentation tasks to be processed concurrently. This multi-threaded parallel processing generates the hair-level portrait segmentation mask and performs post-processing, saving time, improving the efficiency of hair-level image processing on videos, and ensuring video smoothness.

[0045] In one embodiment, the second thread obtaining the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed includes: the second thread obtaining the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed from the first thread.

[0046] Specifically, the acquisition of raw image data is performed solely by the first thread. The first thread is used to acquire the raw image data and combine the raw image with the corresponding hair-level portrait segmentation mask obtained through processing. Figure 1 The process then returns to the second thread. The first thread can proactively return the next image and its corresponding hair-level portrait segmentation mask to the second thread, or the second thread can send a retrieval request to the first thread when the time period arrives. Based on the retrieval request, the first thread then returns the next image and its corresponding hair-level portrait segmentation mask to the second thread. Since each image and its corresponding hair-level portrait segmentation mask are transmitted to the second thread together by the first thread, mismatches between the image and the hair-level portrait segmentation mask are avoided, thus improving the accuracy of image processing.

[0047] In this embodiment, the first thread uniformly acquires the original image data and combines the original image with the corresponding hair-level portrait segmentation mask obtained through processing. Figure 1 The process returns to the second thread, ensuring synchronization between the hair-level portrait segmentation mask and the corresponding original image, further improving the accuracy of image processing.

[0048] In one embodiment, the acquisition of the current image to be processed includes at least one of the following methods:

[0049] Method 1: Obtain the image to be processed by acquiring the real-time image from the real-time acquisition device.

[0050] Specifically, images acquired by the acquisition device can be obtained in real time, such as the current image to be processed from the sensor or adjacent images acquired after the current image to be processed to obtain the next image. When in preview mode, real-time preview images can be continuously acquired through a first thread or a second thread.

[0051] Method 2: Obtain the decoded image from the decoding device to get the current image to be processed.

[0052] Specifically, the stored video can be decoded. The first thread or the second thread can obtain the decoded image from the decoding device as the image to be processed. After the current image to be processed is processed, the next image is obtained by obtaining the adjacent image decoded after the current image to be processed.

[0053] In this embodiment, real-time image processing and image processing of saved images can be achieved by continuously acquiring real-time images or decoded images.

[0054] In one embodiment, the method further includes: performing image preprocessing on the acquired image to obtain a corresponding adjusted image, wherein the acquired image is the current image to be processed or the next image of the current image to be processed. Image preprocessing includes at least one of the following steps: scaling the input image to a preset size; rotating the orientation of the input image to a target orientation; and normalizing the input image.

[0055] Specifically, the input image is scaled to the size required by the hair-level portrait matting model, for example, but not limited to, an image size of 800x600. The input image is rotated to the orientation required by the hair-level portrait matting model, such as a 90-degree clockwise rotation. The normalization method can be customized, such as subtracting the mean (typically 127.5) from the RGB values ​​of each pixel in the input image, and then dividing by the variance of the pixel values ​​in the image to obtain the normalized RGB image to be processed. If the input image pixel value is X, the normalized image pixel value is calculated by (X-127.5) / 127.5. Other normalization methods can also be used, such as directly dividing the input image pixel value by 255 to obtain the normalized image pixel value. It can be understood that the above three different preprocessing methods can be combined arbitrarily in different orders. In one embodiment, the above three preprocessing methods are performed in sequence: image scaling first, then orientation rotation, and finally normalization.

[0056] In this embodiment, the image is adjusted to obtain an adjusted image, which facilitates the formation of input data that meets the input requirements of the hair-level portrait matting model. Establishing a unified standard for image processing also makes the hair-level portrait matting model more widely applicable and improves image processing efficiency.

[0057] In one embodiment, such as Figure 4 As shown, step 204 includes:

[0058] Step 204A: Perform portrait segmentation on the adjusted image corresponding to the next image to obtain a ternary portrait segmentation mask. The ternary portrait segmentation mask includes a portrait region, a background region, and an edge region. The edge region is the transition region between the portrait region and the background region.

[0059] Specifically, a ternary portrait segmentation mask can be obtained by inputting the image into a ternary portrait segmentation model. This model can employ algorithms used in Convolutional Neural Networks (CNNs), including but not limited to DeepLab segmentation algorithms, U-Net algorithms, and Fully Convolutional Neural Networks (FCNs). It typically includes an Encoder module for feature encoding and a Decoder module for target template generation. The Encoder module uses backbones such as MobileNet, ResNet, and VGG for encoding; this module is also the feature extraction module, used for feature extraction. The Decoder module processes the image features to obtain the ternary portrait segmentation mask.

[0060] The ternary portrait segmentation model is trained by pre-collecting a large amount of training data, which is then input into the model containing initial network parameters. The network parameters are continuously adjusted through training loss. Each set of training data includes a training RGB image containing a portrait and an annotated portrait mask. The training RGB image serves as the input to the trained ternary portrait segmentation model, while the annotated ternary portrait training mask represents the expected ground truth output. In the annotated ternary portrait training mask, the body represents one category, the background another, and the edge between the body and background yet another. After training, the ternary portrait segmentation model directly generates the corresponding ternary portrait segmentation mask, which contains three different numerical values, each representing a category. The ternary human image segmentation model can be obtained using deep learning algorithms, such as CNN (Convolutional Neural Networks), U-Net algorithm, and FCN (Fully Convolutional Neural Networks).

[0061] In one embodiment, the ternary portrait segmentation mask can also be obtained by first inputting a binary portrait segmentation model to obtain a binary portrait segmentation mask, and then determining transition regions at the edges to obtain a third-class region. The training data for the binary portrait segmentation model consists of a training RGB image containing a portrait and an annotated binary portrait training mask. In the annotated binary portrait training mask, the human body is classified as one category, and the background as another. After training, the binary portrait segmentation model generates the corresponding binary portrait segmentation mask, and then, by setting a third-class region for the edge transition regions, the corresponding ternary portrait segmentation mask is generated.

[0062] Step 204B: The adjusted image corresponding to the next image is stitched together with the ternary portrait segmentation mask and then input into the first hair-level portrait matting model for processing to obtain the hair-level portrait segmentation mask.

[0063] Specifically, the first hair-level portrait matting model is obtained by pre-collecting a large amount of training data, inputting the training data into the first hair-level portrait matting model containing initial network parameters, and continuously adjusting the network parameters through training loss. Each set of training data includes a stitched image obtained by concatenating the training image with a ternary portrait segmentation training mask, and an annotated hair-level portrait segmentation training mask. The stitched image serves as the input to the trained hair-level portrait matting model, while the annotated hair-level portrait segmentation training mask represents the expected ground truth output of the trained hair-level portrait matting model. In the annotated hair-level portrait segmentation training mask, the human body is classified as one category, and the background as another. The human body includes the hair region, with accurate and clear hair ends. After training, the hair-level portrait matting model directly generates the corresponding hair-level portrait segmentation mask. Because the training data includes a ternary face segmentation training mask, the hair-level face matting model can identify transition regions, typically hair regions. Therefore, it assigns higher complexity to the hair region, enabling more accurate identification of individual hair strands and resulting in a more precise hair-level face segmentation mask. The first hair-level face matting model can be obtained using deep learning algorithms, such as CNN (Convolutional Neural Networks), U-Net, and FCN (Fully Convolutional Neural Networks). This model typically includes an Encoder feature encoding module and a Decoder target template generation module. The Encoder module uses, but is not limited to, backbones such as MobileNet, ResNet, and VGG for encoding; this module is also the feature extraction module. The Decoder module processes the image features to obtain the hair-level face segmentation mask. Figure 5 The diagram shown is a flowchart of the explicit ternary image portrait matting algorithm.

[0064] In this embodiment, a ternary portrait segmentation mask is first obtained, and then the adjusted original image is stitched together with it and input into the first hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask. The accurate ternary portrait segmentation mask helps to obtain a more accurate hair-level portrait segmentation mask, which is suitable for scenarios and platforms where effect is the priority.

[0065] In one embodiment, step 204 includes: inputting the adjusted image corresponding to the next image into the second hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask.

[0066] Specifically, the second hair-level portrait matting model is obtained by pre-collecting a large amount of training data, inputting the training data into the second hair-level portrait matting model containing initial network parameters, and continuously adjusting the network parameters through training loss. Each set of training data includes training images and labeled hair-level portrait segmentation training masks. The training images serve as the input to the trained hair-level portrait matting model, and the labeled hair-level portrait segmentation training masks represent the expected ground truth output of the trained hair-level portrait matting model. In the labeled hair-level portrait segmentation training masks, the human body is classified as one category, and the background as another. The human body includes the hair region, with accurate and clear hair ends. After training, the hair-level portrait matting model directly generates the corresponding hair-level portrait segmentation mask. Since the original image can be directly input into the hair-level portrait matting model to identify the hair strands without the need for ternary portrait segmentation masks, the complexity is low, making it convenient and efficient. The second-level hair-scale portrait matting model can be obtained using deep learning algorithms, such as CNN (Convolutional Neural Networks), U-Net, and FCN (Fully Convolutional Neural Networks). This model typically includes an Encoder module for feature encoding and a Decoder module for target template generation. The Encoder module uses backbones such as MobileNet, ResNet, and VGG for encoding; this module is also the feature extraction module. The Decoder module processes the image features to obtain a hair-scale portrait segmentation mask. Figure 6 The diagram shown is a flowchart of the implicit ternary image portrait matting algorithm.

[0067] In this embodiment, the original image can be directly input into the second hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask. This does not require relying on a ternary portrait segmentation mask, has low complexity, and is suitable for performance-priority scenarios and platforms.

[0068] In one embodiment, the aforementioned ternary portrait segmentation model, the first hair-level portrait matting model, and the second hair-level portrait matting model all include an enc-dec structure, i.e., an encoding / decoding structure. A single enc-dec structure is as follows: Figure 7As shown. The input is the normalized image. The Encoder module is generally composed of basic backbones such as MobileNet, ShuffleNet, and ResNet. The Decoder module is generally composed of deconvolution or interpolation upsampling. Skip connections are used to fuse shallow features into deep features, increasing the generalization ability of the algorithm model. The output is a ternary segmentation mask of the segmented human face, or a hair-level segmentation mask of the human face.

[0069] In one embodiment, processing the current image to be processed based on the current hair-level portrait segmentation mask to obtain the corresponding current target image includes: determining the background region and portrait region of the current image to be processed based on the current hair-level portrait segmentation mask; performing a first rendering process on the background region; and performing a second rendering process on the portrait region.

[0070] Specifically, the electronic device determines the background region in the portrait mask based on the pixel value of each pixel. For example, all pixel values ​​in the background region are 0. Based on the position of the background region in the RGB image to be processed, the background region undergoes a first rendering process, while the portrait region outside the background region undergoes a second rendering process. Different first and second rendering processes correspond to different image processing effects. Rendering processes include, but are not limited to, blurring, color retention, and neon portrait. Blurring keeps the human body sharp while blurring the background. Color retention retains color in the human body while processing the background into black and white. The rendered image is then output to the device for display. For example, for background blurring, the background region is blurred while the portrait region remains sharp to obtain the target image. Neon portrait is a "movie bokeh" effect mode that increases the background blur intensity and bokeh, strongly blurring the background region while keeping the portrait region sharp to obtain the target image.

[0071] In this embodiment, since the hair-level portrait mask effectively distinguishes between the background area and the foreground portrait area containing hair, different rendering processes can be applied to the background area and the portrait area in the RGB image to be processed, achieving precision down to the hair level and resulting in better effects. Furthermore, while the second thread is performing rendering, the first thread is simultaneously performing image matting, further improving image processing efficiency.

[0072] In one embodiment, the method further includes: a second thread outputting and displaying the current target image.

[0073] Specifically, after the second thread completes the rendering operation, it outputs and displays the current target image. The rendering process of the second thread lags behind that of the first thread in the portrait keying step. Figure 1Frame. While the first thread performs portrait keying on the second frame, the second thread renders and displays the first frame.

[0074] In this embodiment, the current target image is output and displayed through a second thread, making it easier for users to view the image processing effect.

[0075] In one specific embodiment, an image processing method is provided, which specifically includes the following processes:

[0076] 1. The first thread acquires the first frame of the raw image to be processed through the image sensor. The raw image to be processed is not limited to raw image, YUV image, RGB image, etc. When the raw image to be processed is not an RGB image, it is converted into an RGB image to be processed. The first thread uses the first frame of the RGB image to be processed as the current image to be processed. After the first thread finishes processing the current image to be processed, it acquires the next image as the current image to be processed.

[0077] 2. The first thread scales the current image to be processed to the size required by the hair-level portrait matting model, rotates the scaled image to the orientation required by the hair-level portrait matting model, and normalizes the image after orientation adjustment.

[0078] 3. The first thread determines the current platform mode. If the current platform mode is effect-priority mode, then the adjusted image is segmented to obtain a ternary portrait segmentation mask. The adjusted image and the ternary portrait segmentation mask are then concatenated and input into the first hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask. For example... Figure 8 The image shown is a schematic diagram of the input original image, the corresponding ternary portrait segmentation mask, and the hair-level portrait segmentation mask in one embodiment.

[0079] 4. If the current platform mode is performance-first mode, the first thread directly inputs the adjusted image into the second hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask.

[0080] 5. If the first thread is currently processing the first frame image, the second thread can be idle or handle other tasks before generating a hair-level portrait segmentation mask. If the first thread is currently processing the second frame image, the second thread obtains the first frame image and its corresponding hair-level portrait segmentation mask from the first thread. While the first thread processes the second frame image, the second thread simultaneously performs rendering image processing on the first frame image based on the first frame image and its corresponding hair-level portrait segmentation mask to obtain the corresponding current target image. Rendering processing includes, but is not limited to, blurring, color retention, and neon portrait effects. The second thread then outputs and displays the processed current target image. This process continues, with the second thread's rendering image processing lagging behind the first thread's portrait segmentation. Figure 1 frame.

[0081] In this embodiment, the first thread first performs image preprocessing required for the current frame's matting, and then performs hair-level portrait matting. The second thread obtains the original image of the previous frame that the first thread has already processed and the corresponding hair-level portrait segmentation mask, performs rendering processing on the previous frame image, and outputs the rendering result. The second thread and the first thread can complete the image processing tasks in parallel, enabling image rendering and image segmentation tasks to be processed in parallel. By generating the hair-level portrait segmentation mask and image post-processing through multi-threaded parallel processing, time is saved. Furthermore, the first thread uniformly obtains the original image and passes the current image to be processed and the corresponding hair-level portrait segmentation mask to the second thread, ensuring the synchronization of the hair-level portrait segmentation mask and the corresponding original image, improving the processing efficiency of hair-level image processing on the video, and ensuring the smoothness of the video.

[0082] In one embodiment, such as Figure 9 As shown, an image processing method is provided, including the following steps:

[0083] Step 302: The second thread obtains the next image of the current image to be processed and passes it to the first thread. The second thread obtains the current hair-level portrait segmentation mask of the current image to be processed from the first thread, and performs image processing on the current image to be processed based on the current hair-level portrait segmentation mask to obtain the corresponding current target image. The current image to be processed is directly obtained by the second thread.

[0084] Step 304: Simultaneously, the first thread performs hair-level portrait segmentation on the next image to obtain a hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread, where the first thread obtains the next image of the current image to be processed from the second thread.

[0085] Specifically, the acquisition of the original image is performed by the second thread. After acquiring the original image, the second thread passes it to the first thread, which performs hair-level portrait segmentation on the original image, obtaining the corresponding hair-level portrait segmentation mask, which is then returned to the second thread. If it is the first frame image, the second thread is idle or handles other tasks. If it is the second frame image, while the first thread is performing hair-level portrait segmentation on the second frame image, the second thread performs rendering processing based on the hair-level portrait segmentation mask corresponding to the first frame image returned by the first thread. This achieves task parallelism between threads. By generating the hair-level portrait segmentation mask and performing image post-processing through multi-threaded parallel processing, time is saved, and the synchronization between the hair-level portrait segmentation mask and the corresponding original image is ensured, improving the processing efficiency of hair-level image processing on video and ensuring video smoothness.

[0086] In this embodiment, the original image data is acquired by the second thread and passed to the first thread. The first thread performs hair-level portrait segmentation on the original image to obtain a corresponding hair-level portrait segmentation mask. This mask is then input to the second thread. While the first thread processes the image, the second thread performs rendering based on the hair-level portrait segmentation mask returned by the first thread and the previous original image. This allows the second and first threads to complete image processing tasks in parallel, enabling image rendering and image segmentation tasks to be processed concurrently. By generating the hair-level portrait segmentation mask and performing post-processing through multi-threaded parallel processing, time is saved, the processing efficiency of hair-level image processing on video is improved, and the smoothness of the video is guaranteed. The second thread uniformly acquires the original image data and passes it to the first thread, ensuring the synchronization of the hair-level portrait segmentation mask and the corresponding original image, further improving the accuracy of image processing.

[0087] It is understood that the process of hair-level portrait segmentation and image processing of the current image to be processed based on the current hair-level portrait segmentation mask to obtain the corresponding current target image in this embodiment can refer to the steps in the above embodiments.

[0088] It should be understood that, although Figure 2 , Figure 4 , Figure 9 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 4 , Figure 9At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0089] Figure 10 This is a structural block diagram of an image processing apparatus 500 according to one embodiment. Figure 10 As shown, an image processing device 500 includes: a second thread processing module 502 and a first thread processing module 504, wherein:

[0090] The second thread processing module 502 is used to obtain the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and to perform image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image.

[0091] The first thread processing module 504 is used to obtain the next image of the current image to be processed while the second thread processing module 502 is processing, perform hair-level portrait segmentation on the next image, obtain the hair-level portrait segmentation mask corresponding to the next image, and input the hair-level portrait segmentation mask corresponding to the next image to the second thread.

[0092] The image processing device 500 in this embodiment obtains the current image to be processed and the corresponding current hair-level portrait segmentation mask through a second thread. Based on the current hair-level portrait segmentation mask, it performs image processing on the current image to be processed to obtain the corresponding current target image. Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the corresponding hair-level portrait segmentation mask. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread. The second thread and the first thread can complete the image processing tasks in parallel, so that the image rendering task and the image segmentation task can be processed in parallel. By generating the hair-level portrait segmentation mask and image post-processing through multi-threaded parallel processing, time is saved, the processing efficiency of hair-level image processing of video is improved, and the smoothness of video is guaranteed.

[0093] In one embodiment, the second thread processing module 502 is further configured to obtain the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed from the first thread.

[0094] In this embodiment, the first thread uniformly acquires the original image data and combines the original image with the corresponding hair-level portrait segmentation mask obtained through processing. Figure 1The process returns to the second thread, ensuring synchronization between the hair-level portrait segmentation mask and the corresponding original image, further improving the accuracy of image processing.

[0095] In one embodiment, the second thread processing module 502 is further configured to directly obtain the current image to be processed through the second thread, and the first thread processing module 504 is further configured to obtain the next image of the current image to be processed from the second thread through the first thread.

[0096] In this embodiment, the second thread uniformly acquires the original image data and transmits it to the first thread. Only one thread acquires the original image data, ensuring the synchronization of the hair-level portrait segmentation mask and the corresponding original image, further improving the accuracy of image processing.

[0097] In one embodiment, the first thread processing module 504 is further configured to obtain the current image to be processed by at least one of the following methods: method one: obtaining the image acquired in real time from the real-time acquisition device to obtain the current image to be processed; method two: obtaining the image decoded from the decoding device to obtain the current image to be processed.

[0098] The image processing device 500 in this embodiment can continuously acquire real-time captured images or decoded images, and then process the acquired images sequentially to achieve real-time image processing and image processing of saved images.

[0099] In one embodiment, the first thread processing module 504 is further configured to perform image preprocessing on the acquired image to obtain a corresponding adjusted image. The acquired image is the current image to be processed or the next image of the current image to be processed. The image preprocessing includes at least one of the following steps: scaling the input image to adjust it to a preset size; rotating the direction of the input image to the target direction; and normalizing the input image.

[0100] The image processing device 500 in this embodiment adjusts the image to obtain an adjusted image, which facilitates the generation of input data that meets the input requirements of the hair-level portrait matting model. Establishing a unified standard for image processing also makes the hair-level portrait matting model more widely applicable and improves image processing efficiency.

[0101] In one embodiment, the first thread processing module 504 is further configured to perform portrait segmentation on the adjusted image corresponding to the next image to obtain a ternary portrait segmentation mask. The ternary portrait segmentation mask includes a portrait region, a background region, and an edge region. The edge region is the transition region between the portrait region and the background region. The adjusted image corresponding to the next image is then stitched together with the ternary portrait segmentation mask and input into the first hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask.

[0102] In this embodiment, a ternary portrait segmentation mask is first obtained, and then the adjusted original image is stitched together with it and input into the first hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask. The accurate ternary portrait segmentation mask helps to obtain a more accurate hair-level portrait segmentation mask, which is suitable for scenarios and platforms where effect is the priority.

[0103] In one embodiment, the first thread processing module 504 is further configured to input the adjusted image corresponding to the next image into the second hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask.

[0104] In this embodiment, the original image can be directly input into the second hair-level portrait matting model for processing to obtain a hair-level portrait segmentation mask. This does not require relying on a ternary portrait segmentation mask, has low complexity, and is suitable for performance-priority scenarios and platforms.

[0105] In one embodiment, the second thread processing module 502 is further configured to determine the background region and the portrait region of the current image to be processed based on the current hair-level portrait segmentation mask; perform a first rendering process on the background region; and perform a second rendering process on the portrait region.

[0106] In this embodiment, since the hair-level portrait mask effectively distinguishes between the background area and the foreground portrait area containing hair, different rendering processes can be applied to the background area and the portrait area in the RGB image to be processed, achieving precision down to the hair level and resulting in better effects. Furthermore, while the second thread is performing rendering, the first thread is simultaneously performing image matting, further improving image processing efficiency.

[0107] In one embodiment, the second thread processing module 502 is further configured to output and display the current target image via a second thread.

[0108] In this embodiment, the current target image is output and displayed through a second thread, making it easier for users to view the image processing effect.

[0109] In one embodiment, such as Figure 11 As shown, an image processing apparatus 600 is provided, comprising:

[0110] The second thread processing module 602 is used to obtain the next image of the current image to be processed and pass the next image of the current image to be processed to the first thread. The second thread obtains the current hair-level portrait segmentation mask of the current image to be processed from the first thread and performs image processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image. The current image to be processed is directly obtained by the second thread.

[0111] The first thread processing module 604 is used to simultaneously perform hair-level portrait segmentation on the next image to obtain a hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread.

[0112] In this embodiment, the acquisition of the original image is performed by the second thread. After acquiring the original image, the second thread passes it to the first thread, which performs hair-level portrait segmentation on the original image, obtains the corresponding hair-level portrait segmentation mask, and then returns it to the second thread. If it is the first frame image, the second thread is idle or handles other tasks. If it is the second frame image, while the first thread is performing hair-level portrait segmentation on the second frame image, the second thread performs rendering processing based on the hair-level portrait segmentation mask corresponding to the first frame image returned by the first thread. This achieves task parallelism between threads. By generating the hair-level portrait segmentation mask and image post-processing through multi-threaded parallel processing, time is saved, the processing efficiency of hair-level image processing on video is improved, and the smoothness of the video is guaranteed. The second thread uniformly acquires the original image data and passes it to the first thread, ensuring the synchronization of the hair-level portrait segmentation mask and the corresponding original image.

[0113] For specific limitations regarding the image processing apparatus, please refer to the limitations on the image processing method above, which will not be repeated here. Each module in the aforementioned image processing apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0114] Figure 12 This is a schematic diagram of the internal structure of an electronic device in one embodiment. For example... Figure 12 As shown, the electronic device includes a processor and a memory connected via a system bus. The processor provides computing and control capabilities to support the operation of the entire electronic device. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. These computer programs can be executed by the processor to implement the image processing methods provided in the various embodiments described above. The internal memory provides a cached runtime environment for the operating system computer programs in the non-volatile storage media. This electronic device can be a mobile phone, a server, etc.

[0115] The various modules in the image processing apparatus provided in this application embodiment can be implemented in the form of computer programs. These computer programs can run on a terminal or server. The program modules constituted by these computer programs can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements the image processing method described in this application embodiment.

[0116] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the image processing method described in this application.

[0117] A computer program product containing instructions that, when run on a computer, causes the computer to perform the image processing method described in the embodiments of this application.

[0118] Any references to memory, storage, databases, or other media used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and performs image rendering processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image; the current image to be processed is a real-time acquired image or a decoded video image; Simultaneously, the first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used as input to the second thread. The hair-level portrait segmentation uses a hair-level portrait matting model to identify the hair region. The first thread passes the next image and the corresponding hair-level portrait segmentation mask to the second thread; the first thread and the second thread are parallel threads. The second thread's rendering process lags behind the first thread's portrait cutout by one frame. The first thread obtains the next image of the current image to be processed, performs hair-level portrait segmentation on the next image, and obtains the hair-level portrait segmentation mask corresponding to the next image, including: If the current platform mode is effect-first mode, then the adjusted image corresponding to the next image is segmented to obtain a ternary portrait segmentation mask. The ternary portrait segmentation mask includes a portrait region, a background region, and an edge region. The edge region is the transition region between the portrait region and the background region. The adjusted image corresponding to the next image is then stitched together with the ternary portrait segmentation mask and input into the first hair-level portrait matting model for processing to obtain the hair-level portrait segmentation mask. If the current platform mode is performance-first mode, the adjusted image corresponding to the next image is input into the second hair-level portrait matting model for processing to obtain the hair-level portrait segmentation mask.

2. The method according to claim 1, characterized in that, The second thread obtains the current image to be processed and the corresponding current hair-level portrait segmentation mask image, including: The second thread obtains the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed from the first thread.

3. The method according to claim 1, characterized in that, The current image to be processed is directly obtained by the second thread, and the first thread obtains the next image after the current image to be processed from the second thread.

4. The method according to claim 1, characterized in that, The acquisition of the current image to be processed includes at least one of the following methods: The current image to be processed is obtained by acquiring real-time images from a real-time acquisition device; The decoded image is obtained from the decoding device to obtain the current image to be processed.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The acquired image is preprocessed to obtain the corresponding adjusted image, wherein the acquired image is the current image to be processed or the next image of the current image to be processed; The image preprocessing includes at least one of the following steps: The input image is scaled and adjusted to a preset size. Rotate the input image to the target orientation; The input image is normalized.

6. The method according to claim 1, characterized in that, The step of performing image processing on the current image to be processed based on the current hair-level portrait segmentation mask to obtain the corresponding current target image includes: The background region and the portrait region of the current image to be processed are determined based on the current hair-level portrait segmentation mask. The background area is subjected to a first rendering process; The portrait area is then subjected to a second rendering process.

7. The method according to any one of claims 1 to 4 or 6, characterized in that, The method further includes: The second thread outputs and displays the current target image.

8. An image processing apparatus, characterized in that, include: The second thread processing module is used to obtain the current image to be processed and the current hair-level portrait segmentation mask corresponding to the current image to be processed, and to perform image rendering processing on the current image to be processed according to the current hair-level portrait segmentation mask to obtain the corresponding current target image. The image to be processed is a real-time acquired image or a decoded video image; The first thread processing module is used to obtain the next image of the current image to be processed while the second thread processing module is processing, perform hair-level portrait segmentation on the next image, and obtain a hair-level portrait segmentation mask corresponding to the next image. The hair-level portrait segmentation mask corresponding to the next image is used to input to the second thread. The hair-level portrait segmentation uses a hair-level portrait matting model to identify the hair region. The first thread passes the next image and the corresponding hair-level portrait segmentation mask to the second thread; the first thread and the second thread are parallel threads. The second thread's rendering process lags behind the first thread's portrait cutout by one frame. The first thread processing module is further configured to, if the current platform mode is effect-priority mode, perform portrait segmentation on the adjusted image corresponding to the next image to obtain a ternary portrait segmentation mask, wherein the ternary portrait segmentation mask includes a portrait region, a background region, and an edge region, and the edge region is a transition region between the portrait region and the background region; concatenate the adjusted image corresponding to the next image with the ternary portrait segmentation mask and input it into a first hair-level portrait matting model for processing to obtain the hair-level portrait segmentation mask; if the current platform mode is performance-priority mode, input the adjusted image corresponding to the next image into a second hair-level portrait matting model for processing to obtain the hair-level portrait segmentation mask.

9. An electronic device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images