Image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose an image processing method and device, electronic equipment and storage medium, the method comprising: identifying a portrait region of an original image to be segmented to obtain a preliminary portrait segmentation result of the original image, the preliminary portrait segmentation result comprising a first probability of each pixel point in the original image being identified as the portrait region; selecting N target pixel points to be optimized from the preliminary portrait segmentation result, N being a positive integer greater than or equal to 1, the first probability of the target pixel points being identified as the portrait region satisfying a selection condition; optimizing the N target pixel points according to image features corresponding to the N target pixel points to update the first probability of the N target pixel points in the preliminary portrait segmentation result being identified as the portrait region to a second probability; and generating a portrait mask corresponding to the original image according to the updated preliminary portrait segmentation result. The embodiments of the present application can improve the segmentation accuracy of segmenting a portrait region from an image.

Description

Technical Field

[0001] This application relates to the field of imaging technology, specifically to an image processing method, an electronic device, and a storage medium. Background Technology

[0002] Currently, applications installed on smartphones and other electronic devices offer functions such as automatic portrait cutout, automatic background removal, and motion tracking of portraits. To achieve these functions, the prerequisite is that the computer can accurately segment the portrait area from the image. However, in practice, it has been found that current image processing methods are not accurate enough in segmenting portrait details such as hair or the edges of the portrait contour, resulting in low segmentation accuracy. Summary of the Invention

[0003] This application discloses an image processing method, apparatus, electronic device, and storage medium, which can improve the segmentation accuracy of human image regions from images and improve the accuracy of human image segmentation.

[0004] This application discloses an image processing method, the method comprising: identifying human image regions in an original image to be segmented, obtaining a preliminary human image segmentation result of the original image, the preliminary human image segmentation result including a first probability that each pixel in the original image is identified as a human image region; selecting N target pixels to be optimized from the preliminary human image segmentation result, where N is a positive integer greater than or equal to 1, and the first probability that the target pixels are identified as human image regions satisfies the selection condition; optimizing the N target pixels according to the image features corresponding to each of the N target pixels, so as to update the first probability that the N target pixels in the preliminary human image segmentation result are identified as human image regions to a second probability; and generating a human image mask corresponding to the original image based on the updated preliminary human image segmentation result.

[0005] This application discloses an image processing apparatus, comprising: a segmentation module for identifying human image regions in an original image to be segmented, obtaining a preliminary human image segmentation result of the original image, the preliminary human image segmentation result including a first probability that each pixel in the original image is identified as a human image region; an identification module for selecting N target pixels to be optimized from the preliminary human image segmentation result, where N is a positive integer greater than or equal to 1, and the first probability that the target pixels are identified as human image regions satisfies the selection condition; an optimization module for optimizing the N target pixels according to the image features corresponding to each of the N target pixels, so as to update the first probability that the N target pixels in the preliminary human image segmentation result are identified as human image regions to a second probability; and a generation module for generating a human image mask corresponding to the original image based on the updated preliminary human image segmentation result.

[0006] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor implements any of the image processing methods disclosed in this application.

[0007] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the image processing methods disclosed in this application.

[0008] Compared with related technologies, the embodiments of this application have the following beneficial effects: An electronic device identifies the human figure region in the original image to be segmented, obtaining a preliminary human figure segmentation result. This preliminary result includes the first probability that each pixel in the original image is identified as a human figure region. N target pixels that satisfy the selection criteria based on the first probability are selected from the preliminary result, and these N target pixels are optimized based on their respective image features. Compared to the first probability of the N target pixels before optimization, the second probability of the optimized N target pixels being identified as human figure regions has a higher accuracy, thus improving the segmentation accuracy. Therefore, after updating the first probability of the N target pixels being identified as human figure regions in the preliminary result to the second probability, the human figure mask generated based on the updated preliminary result can retain more human figure details such as hair and contour edges, which helps improve the segmentation accuracy of human figure regions from the image. Simultaneously, selecting some pixels from the preliminary result as target pixels for optimization can reduce the computational load required for optimization and improve image processing efficiency. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the structure of an image processing circuit disclosed in an embodiment of this application; Figure 2 This is a schematic flowchart of an image processing method disclosed in an embodiment of this application; Figure 3A This is an example diagram of a preliminary portrait segmentation result disclosed in an embodiment of this application; Figure 3B This is an example image of a target pixel to be optimized, as disclosed in an embodiment of this application. Figure 4 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application; Figure 5 This is a schematic diagram of a process for processing an original image using a first segmentation model, as disclosed in an embodiment of this application. Figure 6 This is an example diagram showing the coordinate position of a target pixel in the preliminary portrait segmentation result and the first feature map, as disclosed in an embodiment of this application. Figure 7 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application; Figure 8A This is an example diagram of performing a rotation operation on an original image as disclosed in an embodiment of this application; Figure 8B This is another example diagram of performing a rotation operation on the original image as disclosed in the embodiments of this application; Figure 9A This is a schematic diagram of a process for optimizing target pixels disclosed in an embodiment of this application; Figure 9B This is a schematic diagram of another process for optimizing target pixels disclosed in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an image processing apparatus disclosed in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0013] In related technologies, due to the limited computing resources of electronic devices such as smartphones, tablets, or embedded devices, the computational load of electronic devices is often reduced by lowering the image resolution to reduce the time consumption and power consumption of image processing methods such as portrait segmentation. However, reducing image resolution can easily lead to the loss of image information, making it difficult to handle edge details when performing portrait segmentation, resulting in a low overall segmentation accuracy. For example, when performing portrait segmentation, the edges of the hair in the portrait may be blurry or hair details may not be preserved. This application discloses an image processing method, apparatus, electronic device, and storage medium that can improve the segmentation accuracy of portrait regions segmented from images. These will be described in detail below.

[0014] Please see Figure 1 , Figure 1 This is a schematic diagram of an image processing circuit disclosed in an embodiment of this application. This image processing circuit can be applied to electronic devices such as smartphones, tablets, and smartwatches, but is not limited thereto. Figure 1 As shown, the image processing circuit may include an imaging device (camera) 110, an attitude sensor 120, an image memory 130, an image signal processing (ISP) processor 140, a control logic unit 150, and a display 160.

[0015] The image processing circuitry includes an ISP processor 140 and a control logic unit 150. Image data captured by the imaging device 110 is first processed by the ISP processor 140, which analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (such as a Bayer filter), and can acquire light intensity and wavelength information captured by each imaging pixel, providing a set of raw image data that can be processed by the ISP processor 140. An attitude sensor 120 (such as a three-axis gyroscope, Hall sensor, accelerometer, etc.) can provide the ISP processor 140 with parameters for image processing (such as image stabilization parameters) based on the attitude sensor 120 interface type. The attitude sensor 120 interface can be an SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0016] In addition, image sensor 114 can also send raw image data to attitude sensor 120. Attitude sensor 120 can provide raw image data to ISP processor 140 based on attitude sensor 120 interface type, or attitude sensor 120 can store raw image data in image memory 130.

[0017] The ISP processor 140 processes raw image data pixel by pixel in various formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits. The ISP processor 140 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed with the same or different bit depth precision.

[0018] The ISP processor 140 can also receive image data from the image memory 130. For example, the attitude sensor 120 interface sends raw image data to the image memory 130, and the raw image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and may include DMA (Direct Memory Access) features.

[0019] Upon receiving raw image data from image sensor 114 interface, attitude sensor 120 interface, or image memory 130, ISP processor 140 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to image memory 130 for further processing before display. ISP processor 140 receives processed data from image memory 130 and performs image data processing on the processed data in the raw domain and in the RGB and YCbCr color spaces. The processed image data may be output to display 160 for user viewing and / or further processed by a graphics engine or GPU (Graphics Processing Unit). Furthermore, the output of ISP processor 140 may also be sent to image memory 130, and display 160 may read image data from image memory 130. In one embodiment, image memory 130 may be configured to implement one or more frame buffers.

[0020] The statistical data determined by the ISP processor 140 can be sent to the control logic unit 150. For example, the statistical data may include image sensor 114 statistics such as gyroscope vibration frequency, auto exposure, auto white balance, auto focus, flicker detection, black level compensation, and lens 112 shading correction. The control logic unit 150 may include a processor and / or microcontroller executing one or more routines (such as firmware) that determine control parameters for the imaging device 110 and the ISP processor 140 based on the received statistical data. For example, the control parameters for the imaging device 110 may include attitude sensor 120 control parameters (e.g., gain, integration time for exposure control, image stabilization parameters, etc.), camera flash control parameters, camera image stabilization shift parameters, lens 112 control parameters (e.g., focal length for focusing or zooming), or combinations of these parameters. The ISP control parameters may include gain levels and color correction matrices for auto white balance and color adjustment (e.g., during RGB processing), and lens 112 shading correction parameters.

[0021] In one embodiment, the original image to be acquired and segmented is obtained through the lens 112 and image sensor 114 in the imaging device (camera) 110, and the original image is sent to the ISP processor 140. The ISP processor 140 identifies the human image region in the original image to obtain a preliminary human image segmentation result. The ISP processor 140 selects N target pixels from the preliminary human image segmentation result that satisfy the selection condition with a first probability of being identified as a human image region, and optimizes the N target pixels according to the image features corresponding to each of the N target pixels; N is a positive integer greater than or equal to 1. After optimization, the ISP processor 140 updates the first probability of the N target pixels in the preliminary human image segmentation result being identified as a human image region to a second probability, and generates a human image mask corresponding to the original image based on the updated preliminary human image segmentation result.

[0022] In some embodiments, the ISP processor 140 may further perform rendering processing such as hair-level blurring on the original image based on the generated portrait mask, and output the rendered image to the display 160 for the user to view.

[0023] Please see Figure 2 , Figure 2 This is a schematic flowchart of an image processing method disclosed in an embodiment of this application. Figure 2 The method shown can be applied to electronic devices that include the image processing circuit described above. For example... Figure 2 As shown, the method may include: 210. Identify the human image region in the original image to be segmented, and obtain the preliminary human image segmentation result of the original image.

[0024] In this embodiment, the original image to be segmented can be an image captured by the imaging device of an electronic device, or it can be transmitted to the electronic device via wired or wireless communication, without any specific limitation. The original image may include various different image data, including but not limited to: color data based on the YUV color space, color data based on the RGB color space, and texture data.

[0025] In this embodiment of the application, after acquiring the original image, the electronic device can identify the human image region in the original image through a human image segmentation method or a human image matting method. The human image region can refer to the region in the original image composed of pixels identified as human images. Wherein: Human image segmentation methods may include, but are not limited to, graph-based segmentation methods, clustering-based segmentation methods, semantic-based segmentation methods, and instance-based segmentation methods, which identify human image regions from the original image. For example, when using a graph-based segmentation method, the electronic device can divide the original image into several sub-images. Sub-images classified as human image regions maintain maximum similarity internally, while minimizing similarity between sub-images. Graph-based segmentation methods may include, but are not limited to, NormalizedCut and GraphCut algorithms.

[0026] For example, when using a clustering-based segmentation method, the electronic device can first initialize a coarse cluster, and then iteratively cluster pixels with similar features in the original image into the same superpixel, iterating until convergence, thereby obtaining multiple pixels clustered into the portrait region. Clustering-based segmentation methods may include, but are not limited to, k-means clustering and Simple Linear Iterative Clustering (SLIC) algorithms.

[0027] For example, when using a semantic-based segmentation method, an electronic device can employ a convolutional neural network to perform normalized (Softmax) cross-entropy classification on each pixel in the original image to achieve segmentation of the human image region. The convolutional neural network can include, but is not limited to, fully convolutional networks (FCNs) and any type of convolutional network included in the DeepLab series.

[0028] For example, when using an instance-based segmentation algorithm, an electronic device can first segment the human image region from the original image using a semantic segmentation method, and then perform bounding box regression detection on the segmented human image region to distinguish human image regions belonging to different instances.

[0029] Human face matting methods may include, but are not limited to: traditional matting methods that do not use deep learning (DL) technology and human face matting methods that use DL.

[0030] For example, traditional matting methods may include Bayesian matting algorithms, data-driven machine learning (non-deep learning) matting algorithms, and closed-surface matting algorithms. Bayesian matting solves the Maximum A Posteriori (MAP) problem based on Bayesian theory, thus segmenting the human image region from the original image. Data-driven machine learning matting algorithms establish a mapping between alpha values ​​and pixel colors, where the alpha of a pixel indicates the proportion of the human image region involved in that pixel. Closed-surface matting algorithms assume that the local color distribution follows a linear color model and solve in a closed-form solution without explicitly estimating the foreground and background colors, thus segmenting the human image region.

[0031] For example, the portrait matting method using deep learning can be regarded as a regression problem. By learning from a large amount of data, image features are extracted and the alpha value of the original image is output, thereby segmenting the portrait region based on the alpha value of each pixel.

[0032] It should be noted that the embodiments of this application do not limit the implementation method for identifying human image regions from the original image, including but not limited to the aforementioned human image segmentation method or human image matting method. The process by which an electronic device identifies human image regions from the original image can be viewed as a process of calculating the first probability that each pixel in the original image is identified as a human image region. The higher the value of the first probability, the more likely that the pixel belongs to the human image region. Therefore, the electronic device can obtain a preliminary human image segmentation result including the first probability that each pixel in the original image is identified as a human image region.

[0033] For example, the preliminary portrait segmentation result can be represented by a grayscale image. The grayscale value of a pixel in the grayscale image can be used to indicate the first probability that the pixel is identified as a portrait region in the original image, and the grayscale value is directly proportional to the first probability. Please refer to [link / reference]. Figure 3A , Figure 3A This is an example diagram of a preliminary portrait segmentation result disclosed in an embodiment of this application. Figure 3AAs shown, after performing portrait region recognition on the original image 310, a preliminary portrait segmentation result 320 is obtained. For ease of display, the first probability of each pixel in the preliminary portrait segmentation result 320 can be multiplied by 255 to obtain the grayscale image corresponding to the preliminary portrait segmentation result. Therefore, in the preliminary portrait segmentation result 320, the grayscale value corresponding to the pixel with a first probability of 1 identified as a portrait region in the grayscale image can be 255 (white), and the grayscale value corresponding to the pixel with a first probability of 0 identified as a portrait region in the preliminary portrait segmentation result 320 can be 0 (black). From Figure 3A It is clear that white pixels correspond to the portrait area, while black pixels correspond to the background area.

[0034] 220. Select N target pixels to be optimized from the preliminary portrait segmentation results.

[0035] In this embodiment, N is a positive integer greater than or equal to 1, and the first probability that a target pixel is identified as a portrait region satisfies the selection criteria. The selection criteria can be set according to actual business needs; pixels that are easily missegmented can be selected as target pixels, and there is no specific limitation. For example, pixels located at the boundary edge between the portrait region and the background region are often easily missegmented, so pixels at the boundary edge between the portrait region and the background region can be selected as target pixels to be optimized. For an example, please refer to... Figure 3B , Figure 3B This is an example image of a target pixel to be optimized, as disclosed in an embodiment of this application. Figure 3B As shown, white pixel 321 is the target pixel selected from the preliminary portrait segmentation results.

[0036] In this embodiment, the first probability value of each pixel in the preliminary segmentation result being identified as a human portrait region is within the range of [0,1]. The first probability corresponding to pixels that are easily missegmented is often in the middle of [0,1]. Therefore, the selection condition can be set according to the probability threshold located in the middle of [0,1]. The specific value of the probability threshold can be set with reference to business requirements, and this embodiment does not limit it.

[0037] In one embodiment, step 220 may include: selecting pixels identified as human portrait regions with a first probability falling within a probability range from the preliminary portrait segmentation results as target pixels to be optimized; the probability range is set based on a probability threshold. For example, the probability threshold may be set to 0.5, and the probability range may be set to [0.4, 0.6]. In this embodiment, the value of N depends on the number of pixels with a first probability falling within the probability range, and the value of N may differ for preliminary portrait segmentation results obtained from different original images.

[0038] In one embodiment, step 220 may include: determining the difference between the first probability that each pixel in the preliminary portrait segmentation result is identified as a portrait region and a probability threshold, and selecting the top N pixels arranged in ascending order of the difference as target pixels to be optimized. That is, selecting the N pixels whose first probability is closest to the probability threshold as target pixels, where the value of N can be set according to actual business needs. For example, the probability threshold can be set to 0.5, and the three pixels whose first probability is closest to 0.5 can be selected as target pixels. Finally, pixels with first probabilities of 0.51, 0.49, and 0.505 can be selected as target pixels.

[0039] It should be noted that the value of N can affect the accuracy and computational efficiency of the image processing algorithm for portrait segmentation. A larger N results in more target pixels being selected for optimization, leading to a greater improvement in portrait segmentation accuracy, but also increasing computational load. Conversely, a smaller N results in fewer target pixels being selected for optimization, leading to a smaller improvement in portrait segmentation accuracy, but also reducing computational load, which is beneficial for the application of image processing algorithms on small mobile terminals such as smartphones and smartwatches. Therefore, in this embodiment, the value of N can be set by comprehensively considering the accuracy requirements for portrait segmentation and the computational capabilities of the electronic device.

[0040] 230. Optimize the N target pixels according to the image features corresponding to each of the N target pixels, so as to update the first probability of the N target pixels in the preliminary portrait segmentation result being identified as portrait regions to the second probability.

[0041] In this embodiment, for the selected N target pixels, optimization can be performed using either the aforementioned portrait segmentation method or the portrait matting method to update the probability that the N target pixels are identified as portrait regions. During optimization, the electronic device can further identify the target pixels as portrait regions based on the image features of each target pixel, including but not limited to: color features, texture features, and position features.

[0042] The second probability of a target pixel being identified as a human portrait region may differ from the first probability. This means that for N target pixels, the result of the second portrait segmentation is used instead of the initial portrait segmentation result. In this embodiment, a subset of pixels are selected as target pixels for optimization from the initial portrait segmentation result, rather than optimizing the entire initial portrait segmentation result. This reduces the computational load required for optimization and improves image processing efficiency.

[0043] Optionally, during optimization, the electronic device may use a different method than step 210 to determine a second probability that a target pixel is identified as a human image region. For example, if a graph-based segmentation method is used to identify the original image and obtain a preliminary human image segmentation result, a clustering-based segmentation method can be used to optimize the N target pixels; or, if a semantic-based segmentation method is used to identify the original image and obtain a preliminary human image segmentation result, the BayesMatting algorithm can be used to optimize the N target pixels; or, if a deep learning-based human image matting method is used to identify the original image and obtain a preliminary human image segmentation result, the NormalizedCut algorithm can be used to optimize the target pixels.

[0044] 240. Generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

[0045] In this embodiment, the updated preliminary portrait segmentation result may include a second probability that N target pixels are identified as portrait regions, and a first probability that other pixels besides the N target pixels are identified as portrait regions. An electronic device may compare the probability of each pixel in the updated preliminary portrait segmentation result being identified as a portrait region with a mask threshold to generate a portrait mask of the original image based on the comparison result. The portrait mask is used to indicate portrait regions in the original image. For example, the mask threshold may be set to 'a', where 'a' ∈ [0,1]. In the updated preliminary portrait segmentation result, pixels with a probability greater than 'a' are represented by a value of 255 in the portrait mask; pixels with a probability less than or equal to 'a' are represented by a value of 0 in the portrait mask.

[0046] After generating a portrait mask, the electronic device multiplies the original image with the portrait mask. The resulting multiplication preserves the portrait area in the original image, while eliminating the background area, thus extracting the portrait area from the original image. Because the updated preliminary portrait segmentation results are optimized for target pixels that are easily missegmented, especially increasing the probability that pixels at the boundary between the portrait and background areas are accurately segmented into portrait areas, the generated portrait mask can retain more portrait details such as hair and contour edges.

[0047] After the human figure region is extracted, the electronic device may further perform one or more image processing operations on the human figure region, including but not limited to: rendering processing such as hair-level blurring on the extracted human figure region, or performing motion tracking on the human figure in the original image based on the extracted human figure region.

[0048] In summary, in the aforementioned embodiments, the electronic device can first perform preliminary segmentation on the original image to be segmented to obtain preliminary portrait segmentation results; and select N target pixels that need to be optimized from the preliminary portrait segmentation results, optimize the target pixels to update the preliminary portrait segmentation results, thereby improving the portrait segmentation accuracy of the target pixels, and thus improving the accuracy of portrait segmentation, making the edges of the portrait region extracted from the original image clear and accurate.

[0049] Please see Figure 4 , Figure 4 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application. Figure 4 The method shown can be applied to the aforementioned electronic devices. For example... Figure 4 As shown, the method may include: 410. The image features in the original image to be segmented are extracted layer by layer through the M cascaded network layers in the first segmentation model. Based on the image features output by each of the M network layers, the first probability of each pixel in the original image being identified as a human portrait region is determined, thus obtaining the preliminary human portrait segmentation result of the original image.

[0050] In this embodiment, the first segmentation model can be any neural network model including M network layers, where M is a positive integer greater than or equal to 2. For example, the first segmentation model may include, but is not limited to, a Fully Convolutional Network (FCN), any convolutional network included in the DeepLab series, or a U-Net. Regardless of the model structure used, the first segmentation model can be trained using the first portrait dataset, enabling the trained model to learn the ability to segment portraits, perform preliminary portrait segmentation on the input original image, and output preliminary portrait segmentation results. The first portrait dataset includes multiple first sample portrait images with labeled portrait regions. The electronic device can use supervised learning, unsupervised learning, or semi-supervised learning methods to train the first segmentation model; this embodiment does not limit the specific methods used. In this embodiment, the original image can be preliminarily segmented by the mapping relationship between image features and portrait segmentation results learned by the neural network model during training. This avoids the limitations of manually designed image features for portrait segmentation and improves the generalization ability of the image processing method disclosed in this embodiment when applied to portrait segmentation scenarios.

[0051] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating a process for processing an original image using a first segmentation model, as disclosed in an embodiment of this application. Figure 5As shown, the first segmentation model 510 includes M network layers, which may include at least an encoder 511, a decoder 512, and a softmax normalization layer 513. The encoder 511 and decoder may each include m1 and m2 network layers, respectively, where m1 and m2 are positive integers greater than or equal to 2 and less than M. The encoder 511 may include multiple convolutional layers; optionally, any neural network model from the MobileNet series or ShuffleNet system can be used as the encoder 511. The decoder 512 may include, but is not limited to, convolutional layers, deconvolutional layers, upsampling layers, batch normalization (BN) layers, and rectified linear unit (ReLU) layers.

[0052] The first network layer of encoder 511 receives the input raw image and extracts image features from the raw image layer by layer through the m1 network layers included in encoder 511. Each network layer of encoder 511, after extracting image features, generates a feature map including the image features. For example, Figure 5 As shown, the image size of the feature map output by each network layer in encoder 511 can be different, and the image size of the feature map output by the network layer in the later position can be smaller than the image size of the feature map in the earlier position.

[0053] The first network layer of decoder 512 processes the feature map output by the m1-th network layer of encoder, and restores the image features extracted by encoder 511 layer by layer through the m2-th network layers of decoder 512, as well as recovering the spatial information of pixels. Furthermore, the network layers of decoder 512 and encoder 511 can be skipped to fuse the image features extracted by the network layers of encoder 511 into the image features extracted by the network layers of decoder 512. This improves the generalization ability of the first segmentation model and enhances its accuracy in portrait segmentation.

[0054] The Softmax layer 513 can be used to normalize the predicted value of each pixel in the original image being identified as a human portrait region. That is, it is used to map the predicted value of each pixel in the original image being identified as a human portrait region to the probability range of [0,1], and finally obtain the preliminary human portrait segmentation result output by the Softmax layer 513.

[0055] 420. Select N target pixels to be optimized from the preliminary portrait segmentation results.

[0056] In this embodiment of the application, the electronic device can select N target pixels from the preliminary portrait segmentation results according to the selection conditions. The setting of the selection conditions can be referred to the foregoing embodiments, and will not be repeated below.

[0057] 430. Obtain the first image feature corresponding to each target pixel from the first feature map output by the Xth network layer in the first segmentation model, and obtain the second image feature corresponding to each target pixel from the second feature map output by the Yth network layer in the first segmentation model.

[0058] In this embodiment, X is a positive integer greater than or equal to 1 and less than M; Y is a positive integer greater than X and less than M. That is, the Yth network layer is arranged after the Xth network layer. The electronic device can extract the first image feature corresponding to the target pixel from the shallow network layer of the first segmentation model, and then extract the second image feature corresponding to the target pixel from the higher network layer of the first segmentation model.

[0059] In this embodiment, the Xth and Yth network layers can be any two network layers selected from the first segmentation model, and can be selected with reference to the characteristics of each network layer in the first segmentation model. Optionally, when the first segmentation model includes an encoder, a decoder, and a normalization (Softmax) layer, the Xth network layer can be any network layer in the encoder, the Yth network layer can be any network layer in the decoder, or the Yth network layer can be a Softmax layer. It should be noted that in the first segmentation model, the decoder can be connected to the encoder, and the Softmax layer can be connected to the decoder layer. For example, the Xth network layer can be the fourth network layer in the encoder, and the Yth network layer can be a Softmax layer.

[0060] In this embodiment, the image features extracted by each network layer of the first segmentation model can be output in the form of feature maps. The first feature map output by the Xth network and the second feature map output by the Yth network can be tensors with a channel dimension of not 1, and the electronic device can represent the first and second feature maps using multidimensional arrays. Therefore, the electronic device can obtain the first image feature corresponding to each of the N target pixels from the first feature map by retrieving the value at the coordinate position corresponding to each target pixel from the first feature map. Similarly, the electronic device can obtain the second image feature corresponding to each of the N target pixels from the second feature map by retrieving the value at the coordinate position corresponding to each target pixel from the second feature map.

[0061] For example, please refer to Figure 6 , Figure 6This is an example diagram showing the coordinate position of a target pixel in the preliminary portrait segmentation result and the first feature map, as disclosed in an embodiment of this application. Figure 6 As shown, the preliminary portrait segmentation result 610 can be a two-dimensional image, and the first feature map 620 can be a three-dimensional tensor. The coordinate position of the target pixel in the preliminary portrait segmentation result 610 can be as follows: Figure 6 As shown at midpoint 611, the coordinate position of the target pixel in the first feature map 620 can be as follows: Figure 6 As shown at midpoint 621, Figure 6 The dashed line can be used to indicate the mapping relationship between the coordinate position of the target pixel and the preliminary portrait segmentation result and the first feature map. The electronic device obtains the first image feature corresponding to the target pixel from the first feature map output by the Xth network, which can refer to obtaining the value at the coordinate position of point 621 in the first feature map.

[0062] 440. Optimize the N target pixels based on the first image features and the second image features corresponding to each of the N target pixels, so as to update the first probability of the N target pixels in the preliminary portrait segmentation result being identified as portrait regions to the second probability.

[0063] In this embodiment, the image features extracted by different network layers of the first segmentation model may not be the same. The electronic device optimizes the target pixel to be optimized based on the first image features obtained from the shallow network and the second image features obtained from the high-level network. Different image features can be used to determine whether the target pixel should be identified as a human portrait region, thereby improving the accuracy of human portrait segmentation of the target pixel.

[0064] In one embodiment, the electronic device optimizes N target pixels based on the first and second image features corresponding to each of the N target pixels. This optimization may include: first, fusing the first and second image features of each of the N target pixels to obtain a fused image feature for each target pixel; and then optimizing the N target pixels based on the fused image feature. During optimization, the electronic device may use any of the aforementioned portrait segmentation methods or portrait matting methods to process the fused image features, without specific limitations. Based on the fusion-then-optimization approach, the electronic device can combine the image features extracted from the shallow and high-level networks of the first segmentation model for optimization of the target pixels; and this fusion-then-optimization approach effectively reduces computational load, which is beneficial for improving the portrait segmentation accuracy of the target pixels while keeping computational load relatively low.

[0065] The fusion of the first image feature and the second image feature of each target pixel may include, but is not limited to: concatenating the first image feature and the second image feature of each target pixel along the channel dimension, and using the concatenated image feature as the fused image feature; or, adding the first image feature and the second image feature of each target pixel, and using the added image feature as the fused image feature.

[0066] It should be noted that since the second image feature is obtained from the Y-th network layer, which is closer to the output layer of the first segmentation model, the number of channels in the Y-th network layer may be less than that in the X-th network layer. Therefore, the channel dimension of the second image feature may be less than that of the first image feature. If the channel dimensions of the first and second image features for each target pixel are different before addition, a convolution operation can be performed on the first and / or second image features to change their channel dimensions. Then, the two features can be added together when their channel dimensions are relative.

[0067] In one embodiment, the electronic device optimizes N target pixels based on first and second image features corresponding to each of the N target pixels. This may further include: the electronic device identifying a human figure region in each target pixel based on the first and second image features corresponding to each of the N target pixels, to obtain a first optimization result and a second optimization result for the N target pixels. Both the first and second optimization results may include an optimized probability that each target pixel is identified as a human figure region. The first optimization result may be based on the first image features, and the second optimization result may be based on the second image features. The electronic device may fuse the first and second optimization results. For example, it may average the optimized probabilities of each target pixel being identified as a human figure region in the first and second optimization results, and use the average optimized probability of each target pixel as the second probability that each target pixel is identified as a human figure region.

[0068] 450. Generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

[0069] In this application embodiment, the implementation of step 450 by the electronic device can be found in the foregoing embodiments, and will not be repeated below.

[0070] In the foregoing embodiments, the electronic device can use the trained first segmentation model to perform preliminary portrait segmentation on the original image to obtain preliminary portrait segmentation results. This avoids the limitations of manually designed image features for portrait segmentation and improves the generalization ability of the image processing method disclosed in this application when applied to portrait segmentation scenarios. Furthermore, when optimizing the selected target pixels, the probability of the target pixels being identified as portrait regions is optimized based on image features extracted from the shallow and high-level networks of the first segmentation model. This helps to improve the portrait segmentation accuracy of target pixels based on different image features, thereby improving portrait segmentation precision.

[0071] Please see Figure 7 , Figure 7 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application. Figure 7 The method shown can be applied to the aforementioned electronic devices. For example... Figure 7 As shown, the method may include: 710. Preprocess the original image to be segmented so that the image size of the preprocessed original image is consistent with the image size of the input image of the first segmentation model.

[0072] In this embodiment, the original image to be segmented may be captured by an electronic device or transmitted to the electronic device from another device. Therefore, the image specifications of different original images may not be consistent, and the image specifications of each frame of the original image may also differ from the image specifications of the input image of the first segmentation model. The image specifications of the input image of the first segmentation model can be determined based on the image specifications of the input image received by the first network layer of the first segmentation model.

[0073] In this embodiment of the application, in order for the first segmentation model to perform portrait segmentation on the original image, the original image can be preprocessed first. The preprocessing operation may include one or more of the following: rotation operation, scaling operation, and normalization operation.

[0074] A rotation operation refers to rotating a pixel in the original image by a certain angle. The electronic device can determine the shooting direction of the original image based on its width and height. For example, if the width is greater than the height, the original image is shot horizontally; if the height is greater than the width, the original image is shot vertically. Alternatively, the shooting direction of the original image can be determined based on the shooting direction value recorded by the shooting device that captured the original image. The shooting direction can include either horizontal or vertical. When it is determined that the shooting direction of the original image is inconsistent with the shooting direction of the input image of the first segmentation model, the electronic device can perform a rotation operation on the original image to make the rotated original image consistent with the shooting direction of the input image. The direction of rotation is not limited and can include clockwise or counterclockwise rotation.

[0075] For example, please refer to Figure 8A , Figure 8A This is an example diagram of performing a rotation operation on the original image disclosed in an embodiment of this application. Assuming that the input image of the first segmentation model is shot vertically; if it is detected that the original image 810 before preprocessing is shot horizontally, while the input image is shot vertically, the original image 810 can be rotated 90° counterclockwise to obtain the original image 820 shot vertically.

[0076] For example, please refer to Figure 8B , Figure 8B This is another example diagram of performing a rotation operation on the original image disclosed in this application. Assume that the input image of the first segmentation model is taken horizontally; if it is detected that the original image 830 obtained before preprocessing is taken vertically, while the input image is taken horizontally, the original image can be rotated 90° counterclockwise to obtain the horizontal original image 840.

[0077] Scaling operations refer to operations that reduce or enlarge the size of the original image. When the size of the original image is smaller than the size of the input image of the first segmentation model, the electronic device can enlarge the original image; when the size of the original image is larger than the size of the input image, the electronic device can reduce the size of the original image. For example, if the size of the input image of the first segmentation model is 640*480, then the size of the original image needs to be reduced or enlarged to 640*480.

[0078] Normalization refers to mapping the image data values ​​of each pixel in the original image to the range [0,1]. Normalization can include subtracting the mean from the RGB channel values ​​of each pixel in the original image and then dividing by the variance. For example, assuming the mean is 127.5, the subtraction of the mean and division by the variance for the RGB channel value X of any pixel in the original image can be expressed by the following formula: (X-127.5) / 127.5. Alternatively, normalization can include directly dividing the RGB channel values ​​of each pixel in the original image by 255. For example, dividing the RGB channel value X of any pixel in the original image by 255 can be expressed by the following formula: X / 255.

[0079] 720. The image features in the original image to be segmented are extracted layer by layer by the M cascaded network layers in the first segmentation model, and the first probability of each pixel in the original image being identified as a human portrait region is identified based on the image features output by each of the M network layers, so as to obtain the preliminary human portrait segmentation result of the original image.

[0080] In this embodiment, the first segmentation model may include an encoder, a decoder connected to the encoder, and a Softmax layer connected to the decoder. The original image is input to the encoder, and after layer-by-layer processing by the encoder, decoder, and Softmax layer, a preliminary portrait segmentation result of the original image is obtained from the output of the Softmax layer. Optionally, the first segmentation model may be a lightweight segmentation model. A lightweight segmentation model refers to a neural network model with fewer parameters and lower computational cost. When the first segmentation model is a lightweight segmentation model, the encoder may use a lightweight backbone network, such as networks from the MobileNet series or ShuffleNet series. Furthermore, the decoder may use convolutional layers with a relatively small number of channels. The lightweight segmentation model can be deployed on electronic devices with limited computing resources, such as smartphones, tablets, and embedded devices. Using a lightweight first segmentation model to perform portrait segmentation on the original image allows for the utilization of the advantages of deep learning models on electronic devices with limited computing resources.

[0081] 730. Select N target pixels to be optimized from the preliminary portrait segmentation results.

[0082] 740. Obtain the first image feature corresponding to each target pixel from the first feature map output by the Xth network layer in the first segmentation model, and obtain the second image feature corresponding to each target pixel from the second feature map output by the Yth network layer in the first segmentation model.

[0083] In this application embodiment, the implementation of steps 730-740 by the electronic device can be found in the foregoing embodiments, and will not be repeated below.

[0084] 750. Fuse the first image features and the second image features of each target pixel to obtain the fused image features of each target pixel.

[0085] In this embodiment, fusing the first image feature and the second image feature of each target pixel may include, but is not limited to: concatenating the first image feature and the second image feature of each target pixel along the channel dimension, and using the concatenated image feature as the fused image feature; or, adding the first image feature and the second image feature of each target pixel, and using the added image feature as the fused image feature. It should be noted that in this embodiment, the fusion operation is performed on the target pixel, rather than fusing the first feature map and the second feature map as a whole.

[0086] For example, the first image feature corresponding to the N target pixels obtained from the Xth network layer is N×d1 dimensional, where d1 is a positive integer greater than or equal to 1; the second image feature corresponding to the N target pixels obtained from the Yth network layer is N×d2 dimensional, where d2 is a positive integer greater than or equal to 1. Since the Yth network layer is closer to the output layer of the first segmentation model, the number of channels in the feature map output by the Yth network layer may be less, and d2 can be less than d1. By fusing the first and second image features of each target pixel, an N×D dimensional fused image feature can be obtained, where D can be the sum of d1 and d2.

[0087] 760. By analyzing the fused image features of N target pixels using the second segmentation model, the second probability of N target pixels being identified as a human portrait region is obtained.

[0088] In this embodiment, the second segmentation model can be any deep learning-based neural network model or machine learning model, without specific limitations. For example, a deep learning-based neural network model can include convolutional layers, fully connected layers, batch normalization (BN) layers, and ReLU layers. A machine learning model can include support vector machines (SVM), random forest classifiers, etc. Optimizing the target pixels using the second segmentation model can avoid the limitations of manually designed image features and improve the generalization ability of image processing methods applied to portrait segmentation. Meanwhile, due to the low interpretability of the first segmentation model, if traditional feature matching methods are used for optimization, the information contained in the image features extracted by the first segmentation model may not be fully utilized, leading to a decrease in the accuracy of portrait segmentation of the target pixels. In this embodiment, optimization using the second segmentation model can address the problem caused by the low interpretability of the first segmentation model. The second segmentation model can fully mine the information in the image features extracted by the first segmentation model through machine learning or deep learning, which is beneficial to improving the accuracy of portrait segmentation and achieving optimization of the target pixels.

[0089] In this embodiment, the second segmentation model can also be trained using a second portrait dataset, which includes multiple second sample portrait images with labeled portrait regions. The electronic device can train the second segmentation model using supervised learning, unsupervised learning, or semi-supervised learning methods; this embodiment does not impose any limitations. Furthermore, the portrait datasets used to train the second segmentation model and the first segmentation model can be the same or different; no specific limitations are imposed.

[0090] For example, please refer to Figure 9A , Figure 9A This is a schematic diagram illustrating a process for optimizing target pixels as disclosed in an embodiment of this application. Figure 9AAs shown, the system includes a first segmentation model 910 and a second segmentation model 920. The first segmentation model 910 includes an encoder 911, a decoder 912, and a Softmax layer 913. The second segmentation model 920 includes at least three cascaded network layers. The electronic device can obtain first image features corresponding to N target pixels from a first feature map output by network layer 911a of the encoder 911, and second image features corresponding to the N target pixels from a second feature map output by the Softmax layer 913. The electronic device fuses the first and second image features to obtain N×D dimensional fused image features. The N×D dimensional fused image features are input to the second segmentation model 920, and after step-by-step processing by each network layer of the second segmentation model 920, an N×2 dimensional segmentation result is obtained from the output of the second segmentation model 920. This N×2 dimensional segmentation result includes a second probability that each of the N target pixels is identified as a human image region.

[0091] For example, please refer to Figure 9B , Figure 9B This is a schematic diagram illustrating another process for optimizing target pixels disclosed in an embodiment of this application. For example... Figure 9B As shown, it includes a first segmentation model 910 and a second segmentation model. In Figure 9B In the middle, the model structure of the first segmentation model 910 can be compared with... Figure 9A Same as in the text. (and) Figure 9A The difference is, Figure 9B The second segmentation model can be an SVM classifier 930. (And...) Figure 9A Similarly, the electronic device can fuse the image features obtained from the shallow network (network layer 911a in the encoder) and the high-level network (Softmax layer 913) of the first segmentation model 910 for N target pixels, and input the fused N×D dimensional image features into the SVM classifier 930. After the SVM classifier 930 processes the N×D dimensional fused image features, an N×2 dimensional segmentation result is obtained from the output of the SVM classifier 930. This N×2 dimensional segmentation result includes the second probability that each of the N target pixels is identified as a human image region.

[0092] While the lightweight first segmentation model reduces the computational load of portrait segmentation, it may sacrifice segmentation accuracy, resulting in poor segmentation accuracy for image details such as region edges. Therefore, in this embodiment, after the first segmentation model processes the original image to obtain a preliminary portrait segmentation result, a second segmentation model is used to perform secondary segmentation on the target pixels to be optimized in the preliminary portrait segmentation result. This helps to compensate for the shortcomings of the lightweight first segmentation model in detail segmentation, improves the portrait segmentation accuracy of target pixels, and thus improves the overall portrait segmentation accuracy of the image.

[0093] 770. Based on the image coordinates of each target pixel in the preliminary portrait segmentation result, replace the first probability of each target pixel in the preliminary portrait segmentation result being identified as a portrait region with the second probability to obtain the updated preliminary portrait segmentation result.

[0094] In this embodiment, when the electronic device selects image features corresponding to target pixels from the first feature map and the second feature map for fusion, the coordinate position of the target pixel in the fused image features may not be the same as the coordinate position of the target pixel in the preliminary portrait segmentation result. Therefore, when the electronic device selects N target pixels to be optimized in the aforementioned step 730, it can record the index position of each target pixel in the preliminary portrait segmentation result. After obtaining the second probability of the N target pixels being identified as portrait regions from the output of the second segmentation model in step 760, each target pixel can be replaced from the first probability to the second probability in the preliminary portrait segmentation result according to the recorded index position.

[0095] 780. Generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

[0096] As can be seen, in the aforementioned embodiments, a lightweight first segmentation model is used to perform initial segmentation on the original image to generate preliminary segmentation results; then, target pixels to be optimized are selected from the preliminary segmentation results, and the trained second segmentation model is used to optimize the target pixels to update the preliminary segmentation results. This reduces the overall computational load of the image processing method while ensuring the overall portrait segmentation accuracy of the image, enabling the image processing method proposed in this application to achieve better performance on electronic devices with limited computing resources.

[0097] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an image processing apparatus disclosed in an embodiment of this application. This image processing apparatus can be applied to the aforementioned electronic devices. For example... Figure 10 As shown, the image processing device 1000 may include: a segmentation module 1010, a recognition module 1020, an optimization module 1030, and a generation module 1040.

[0098] The segmentation module 1010 can be used to identify the human image region of the original image to be segmented, and obtain the preliminary human image segmentation result of the original image. The preliminary human image segmentation result includes the first probability that each pixel in the original image is identified as a human image region. The recognition module 1020 can be used to select N target pixels to be optimized from the preliminary portrait segmentation results, where N is a positive integer greater than or equal to 1, and the target pixels are identified as the first probability of the portrait region satisfying the selection condition. The optimization module 1030 can be used to optimize N target pixels based on the image features corresponding to each of the N target pixels, so as to update the first probability of the N target pixels in the preliminary portrait segmentation result being identified as a portrait region to the second probability. The generation module 1040 can be used to generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

[0099] In one embodiment, the segmentation module 1010 can be used to extract image features from the original image to be segmented layer by layer through the M cascaded network layers in the first segmentation model, and identify the first probability that each pixel in the original image is identified as a human portrait region based on the image features output by each of the M network layers, thereby obtaining the preliminary human portrait segmentation result of the original image; the first segmentation model is trained using a human portrait dataset including multiple human portrait images.

[0100] In one embodiment, the optimization module 1030 may include a selection unit and an optimization unit.

[0101] The selected unit can be used to obtain the first image feature corresponding to each target pixel from the first feature map output by the Xth network layer in the first segmentation model; X is a positive integer greater than or equal to 1 and less than M; and, It can be used to obtain the second image features corresponding to each target pixel from the second feature map output by the Y-th network layer in the first segmentation model; Y is a positive integer greater than X.

[0102] The optimization unit can be used to optimize N target pixels based on the first image features and the second image features corresponding to each of the N target pixels.

[0103] In one embodiment, the first segmentation model includes an encoder, a decoder connected to the encoder, and a normalization layer connected to the decoder; the Xth network layer is any network layer included in the encoder; the Yth network layer is any network layer included in the decoder; or, the Yth network layer is a normalization layer.

[0104] In one embodiment, the optimization unit can be used to fuse the first image features and the second image features of each target pixel to obtain the fused image features of each target pixel; and to optimize the N target pixels based on the fused image features of the N target pixels.

[0105] In one embodiment, the optimization unit can be used to analyze the fused image features of N target pixels using a second segmentation model to obtain a second probability that the N target pixels are identified as a human portrait region; the second segmentation model is trained using a human portrait dataset including multiple human portrait images; and it can be used to replace the first probability that each target pixel in the preliminary human portrait segmentation result is identified as a human portrait region with the second probability based on the image coordinates of each target pixel in the preliminary human portrait segmentation result.

[0106] In one embodiment, the image processing apparatus 1000 may further include a preprocessing module.

[0107] The preprocessing module can be used to preprocess the original image to be segmented before the segmentation module 1010 extracts image features from the original image to be segmented through the M-layer network cascaded in the first segmentation model, so that the image specifications of the preprocessed original image are consistent with the image specifications of the input image of the first segmentation model; the preprocessing operation includes at least one or more of the following: rotation operation, scaling operation, and normalization operation.

[0108] In one embodiment, the recognition module 1020 can be used to select, from the preliminary portrait segmentation results, the pixels that are identified as portrait regions and whose probability falls within the probability range as target pixels to be optimized, wherein the probability range is set based on a probability threshold.

[0109] In one embodiment, the recognition module 1020 can be used to determine the difference between the first probability and the probability threshold of each pixel in the preliminary portrait segmentation result being identified as a portrait region, and select the first N pixels arranged in ascending order of the difference as the target pixels to be optimized.

[0110] As can be seen, in the aforementioned embodiments, the image processing device can first perform preliminary segmentation on the original image to be segmented to obtain preliminary portrait segmentation results; and select N target pixels that need to be optimized from the preliminary portrait segmentation results, optimize the target pixels to update the preliminary portrait segmentation results, thereby improving the portrait segmentation accuracy of the target pixels, and thus improving the accuracy of portrait segmentation, making the edges of the portrait region extracted from the original image clear and accurate.

[0111] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. For example... Figure 11 As shown, the electronic device 1100 may include: Memory 1110 storing executable program code; Processor 1120 coupled to memory 1110; Specifically, the processor 1120 calls the executable program code stored in the memory 1110 to execute any of the image processing methods disclosed in the foregoing embodiments. It should be noted that... Figure 11 The electronic device shown may also include components not shown, such as a power supply, input buttons, camera, speaker, screen, RF circuit, Wi-Fi module, Bluetooth module, and sensors, which will not be described in detail in this embodiment.

[0112] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any of the image processing methods disclosed in the foregoing embodiments.

[0113] This application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform any of the image processing methods disclosed in the foregoing embodiments.

[0114] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0115] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0116] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0119] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0120] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, The method includes: The first segmentation model extracts image features from the original image to be segmented layer by layer through M cascaded network layers, and identifies the first probability that each pixel in the original image is identified as a human portrait region based on the image features output by each of the M network layers, thus obtaining a preliminary human portrait segmentation result for the original image. The first segmentation model is trained using a first human portrait dataset, which includes multiple first sample human portrait images with human portrait regions annotated. The preliminary human portrait segmentation result includes the first probability that each pixel in the original image is identified as a human portrait region. From the preliminary portrait segmentation results, N target pixels to be optimized are selected, where N is a positive integer greater than or equal to 1, and the first probability of the target pixels being identified as portrait regions satisfies the selection criteria. The N target pixels are optimized based on the image features corresponding to each of the N target pixels, so as to update the first probability of the N target pixels in the preliminary portrait segmentation result being identified as a portrait region to a second probability. Generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

2. The method according to claim 1, characterized in that, The optimization of the N target pixels based on the image features corresponding to each target pixel includes: The first image feature corresponding to each target pixel is obtained from the first feature map output by the Xth network layer in the first segmentation model; X is a positive integer greater than or equal to 1 and less than M. The second image feature corresponding to each target pixel is obtained from the second feature map output by the Y-th network layer in the first segmentation model; Y is a positive integer greater than X and less than or equal to M. The N target pixels are optimized based on the first image features and the second image features corresponding to each of the N target pixels.

3. The method according to claim 2, characterized in that, The first segmentation model includes an encoder, a decoder connected to the encoder, and a normalization layer connected to the decoder; The Xth network layer is any one of the network layers included in the encoder; The Yth network layer is any one of the network layers included in the decoder; or, the Yth network layer is the normalization layer.

4. The method according to claim 2, characterized in that, The optimization of the N target pixels based on the first image features and the second image features corresponding to each of the N target pixels includes: The first image feature and the second image feature of each target pixel are fused to obtain the fused image feature of each target pixel; The N target pixels are optimized based on the fused image features of the N target pixels.

5. The method according to claim 4, characterized in that, The optimization of the N target pixels based on the fused image features of the N target pixels includes: The fused image features of the N target pixels are analyzed by the second segmentation model to obtain the second probability that the N target pixels are identified as human portrait regions. The second segmentation model is trained using a second human portrait dataset that includes multiple human portrait images. The second human portrait dataset includes multiple second sample human portrait images with human portrait regions annotated. Based on the image coordinates of each target pixel in the preliminary portrait segmentation result, the first probability that each target pixel in the preliminary portrait segmentation result is identified as a portrait region is replaced with the second probability.

6. The method according to claim 1, characterized in that, Before extracting image features from the original image to be segmented through the cascaded M-layer network in the first segmentation model, the method further includes: The original image to be segmented is preprocessed to make the image size of the preprocessed original image consistent with the image size of the input image of the first segmentation model; the preprocessing operation includes at least one or more of the following: rotation operation, scaling operation, and normalization operation.

7. The method according to any one of claims 1-6, characterized in that, The step of selecting N target pixels to be optimized from the preliminary portrait segmentation results includes: From the preliminary portrait segmentation results, the pixels that are identified as portrait regions and whose probability falls within the probability range are selected as target pixels to be optimized. The probability range is set based on a probability threshold.

8. The method according to any one of claims 1-6, characterized in that, The step of selecting N target pixels to be optimized from the preliminary portrait segmentation results includes: The difference between the first probability and the probability threshold of each pixel in the preliminary portrait segmentation result being identified as a portrait region is determined, and the first N pixels arranged in ascending order of the difference are selected as the target pixels to be optimized.

9. An image processing apparatus, characterized in that, The device includes: The segmentation module is used to extract image features from the original image to be segmented layer by layer through M cascaded network layers in the first segmentation model, and to identify the first probability that each pixel in the original image is identified as a human portrait region based on the image features output by each of the M network layers, thereby obtaining a preliminary human portrait segmentation result of the original image; the first segmentation model is trained using a first human portrait dataset, which includes multiple first sample human portrait images with human portrait regions labeled, and the preliminary human portrait segmentation result includes the first probability that each pixel in the original image is identified as a human portrait region; The recognition module is used to select N target pixels to be optimized from the preliminary portrait segmentation results, where N is a positive integer greater than or equal to 1, and the target pixels are identified as portrait regions with a first probability that they meet the selection criteria. An optimization module is used to optimize the N target pixels according to the image features corresponding to each of the N target pixels, so as to update the first probability of the N target pixels in the preliminary portrait segmentation result being identified as a portrait region to a second probability; The generation module is used to generate a portrait mask corresponding to the original image based on the updated preliminary portrait segmentation results.

10. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images