Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium. The image processing method comprises the following steps: acquiring a first gray image to be colored, wherein the first gray image comprises a portrait region to be colored; inputting the first gray image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the gray value of the portrait region in the portrait mask image is different from the gray value of a background region except the portrait region; acquiring a second gray image corresponding to the portrait region based on the first gray image and the first mask image; and inputting the first gray image and the second gray image into a pre-trained coloring model to obtain a first portrait image after coloring the portrait region in the first gray image. The method can realize targeted coloring of the portrait region in the image and reduce the consumption of computing power.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid advancement of technology and living standards, electronic devices (such as smartphones and tablets) have become one of the most commonly used electronic products in people's lives. People can use electronic devices to take and save a large number of photos, and in some cases, there is a need to colorize saved black and white photos, converting them into color. However, the technology for colorizing black and white photos consumes a lot of computing power and has a slow processing speed. Summary of the Invention

[0003] This application proposes an image processing method, apparatus, electronic device, and storage medium that can perform targeted colorization of human portrait areas in an image, reducing computational power consumption.

[0004] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring a first grayscale image to be colored, wherein the first grayscale image includes a portrait region to be colored; inputting the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region; acquiring a second grayscale image corresponding to the portrait region based on the first grayscale image and the first mask image; and inputting the first grayscale image and the second grayscale image into a pre-trained coloring model to obtain a first portrait image after coloring the portrait region.

[0005] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising: a first acquisition module, a portrait segmentation module, a second acquisition module, and an image coloring module, wherein the first acquisition module is used to acquire a first grayscale image to be colorized, wherein the first grayscale image includes a portrait region to be colorized; the portrait segmentation module is used to input the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region; the second acquisition module is used to acquire a second grayscale image corresponding to the portrait region based on the first grayscale image and the first mask image; the image coloring module is used to input the first grayscale image and the second grayscale image into a pre-trained coloring model to obtain a first portrait image after coloring the portrait region.

[0006] Thirdly, embodiments of this application provide an electronic device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the image processing method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the image processing method provided in the first aspect above.

[0008] The solution provided in this application involves acquiring a first grayscale image to be colored, wherein the first grayscale image includes a portrait region to be colored; inputting the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the grayscale values ​​of the portrait region in the portrait mask image are different from the grayscale values ​​of the background region other than the portrait region; based on the first grayscale image and the first mask image, acquiring a second grayscale image corresponding to the portrait region; and inputting the first grayscale image and the second grayscale image into a pre-trained colorization model to obtain a first portrait image after colorizing the portrait region in the first grayscale image. Therefore, targeted colorization of the portrait region in an image can be achieved, thereby reducing computational power consumption and increasing the speed of image colorization. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of an image processing method according to an embodiment of this application is shown.

[0011] Figure 2 This illustration shows a schematic diagram of a portrait mask image provided in an embodiment of this application.

[0012] Figure 3 A flowchart of an image processing method according to another embodiment of this application is shown.

[0013] Figure 4 This illustration shows a structural diagram of a portrait matting model provided in an embodiment of this application.

[0014] Figure 5A schematic diagram of a coloring model provided in an embodiment of this application is shown.

[0015] Figure 6 A flowchart of an image processing method according to yet another embodiment of this application is shown.

[0016] Figure 7 A block diagram of an image processing apparatus according to one embodiment of the present application is shown.

[0017] Figure 8 This is a block diagram of an electronic device for performing an image processing method according to an embodiment of this application.

[0018] Figure 9 This is a storage unit in this application embodiment for storing or carrying program code that implements the image processing method according to this application embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0020] In related technologies, colorization schemes for black and white images are mainly divided into two types: one is based on traditional image processing methods, and the other is based on deep learning. Traditional image processing methods suffer from poor stability and robustness; while deep learning-based methods colorize the entire image, covering a wide area and requiring significant computation, making them unsuitable for electronic devices. Furthermore, when colorizing portrait images, the complex background necessitates large-scale models to achieve a seamless background colorization effect.

[0021] To address the aforementioned problems, the inventors have proposed an image processing method, apparatus, electronic device, and storage medium as described in the embodiments of this application. These methods enable targeted colorization of human figures in images, thereby reducing computational power consumption and increasing the speed of image colorization. The specific image processing methods are described in detail in the subsequent embodiments.

[0022] The image processing method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0023] Please see Figure 1 , Figure 1 A schematic flowchart of an image processing method according to an embodiment of this application is shown. In a specific embodiment, the image processing method is applied to, for example... Figure 7 The image processing apparatus 400 shown and the electronic device 100 on which the image processing apparatus 400 is disposed ( Figure 8The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment can be a smartphone, tablet computer, smartwatch, e-reader, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown will be described in detail. The image processing method may specifically include the following steps:

[0024] Step S110: Obtain the first grayscale image to be colored, wherein the first grayscale image includes the portrait area to be colored.

[0025] In this embodiment, the first grayscale image to be colored can be a grayscale image corresponding to a black and white portrait image. The electronic device can acquire the first grayscale image to be colored. The first grayscale image is an image containing a portrait area, which is the area to be colored. That is, the first grayscale image includes the portrait area and the background area excluding the portrait area. The portrait area refers to the area containing the human body, which may include the body area, head area, hair area, etc. Understandably, many older photographs are black and white portrait images. The visual effect of black and white portrait images is difficult to compare with color portrait images. Alternatively, some devices support shooting in black and white mode, and the stored image is black and white. However, it is necessary to colorize the portrait area in the black and white image later. Therefore, there is a need for users to convert black and white portrait images to color portrait images.

[0026] In some implementations, the electronic device can acquire a first grayscale image to be colored from a local file. Optionally, the electronic device can acquire a black-and-white portrait image to be colored from a locally stored file, and then convert the black-and-white portrait image into a grayscale image. The stored black-and-white portrait image can be an image captured by a mobile terminal in black-and-white image shooting mode, or it can be downloaded from a server via a network; this is not limited. For example, when the electronic device is a mobile terminal, its photo album may contain black-and-white portrait images. When a user needs to colorize a portrait area in a black-and-white portrait image, they can select a black-and-white portrait image from the album based on user input and convert it into a grayscale image.

[0027] Step S120: Input the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as the first mask image, wherein the grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region.

[0028] In this embodiment of the application, when the electronic device colors the portrait area in the first grayscale image, it can input the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait area output by the portrait matting model, and use the obtained portrait mask image as the first mask image, and the grayscale value of the portrait area in the portrait mask image output by the portrait matting model is different from the grayscale value of the background area.

[0029] In one possible implementation, the above portrait mask image can be a binary mask image. A binary mask image is a binary image obtained by segmenting the portrait region based on the pixel values ​​of the image. In the portrait mask image, the pixel values ​​of the portrait region are different from the pixel values ​​of other regions, thereby distinguishing the portrait region in the portrait mask image. For example, please refer to... Figure 2 , Figure 2 In the portrait mask image shown, the pixel value corresponding to the portrait area A1 can be 255 (white), while the pixel value of the background area can be 0 (black).

[0030] In another possible implementation, the grayscale value of the background area other than the portrait area in the above portrait mask image can be 0, while the grayscale value of the portrait area can be greater than 0, and the grayscale value of the portrait area is less than or equal to 255. This allows the grayscale value of the background area to differ from that of the portrait area, enabling the portrait mask image to distinguish between the portrait area and the background area.

[0031] Step S130: Based on the first grayscale image and the first mask image, obtain the second grayscale image corresponding to the portrait region.

[0032] In this embodiment, after obtaining the first mask image, a second grayscale image corresponding to the portrait region can be determined based on the first grayscale image and the first mask image. It is understood that since the portrait region and the background region can be distinguished in the first mask image, the portrait region to be colored can be determined from the first grayscale image based on the first mask image, and a second grayscale image corresponding to the portrait region can be obtained. The obtained second grayscale image can be an image with the grayscale value of the background region in the first grayscale image set to 0. Therefore, the second grayscale image can only include image information of the portrait region. That is, the second grayscale image has the same size as the first grayscale image, includes the portrait region in the first grayscale image, and the position and size of the portrait region in the second grayscale image are the same as the position and size of the portrait region in the first grayscale image. Furthermore, the grayscale value of the region in the second grayscale image other than the portrait region (i.e., the region corresponding to the background region in the first grayscale image) is 0.

[0033] In some embodiments, the grayscale value of the background area other than the portrait area in the first mask image is 0. The electronic device can multiply the first mask image by the first grayscale image to obtain the second grayscale image. For example... Figure 2 As shown, since the grayscale value of the background area in the first mask image is 0, the human figure area in the first grayscale image can be retained after multiplying the first mask image with the first grayscale image, while the grayscale value of the background area in the first grayscale image is 0. Thus, the human figure area to be colored can be clearly identified in the second grayscale image.

[0034] Step S140: Input the first grayscale image and the second grayscale image into the pre-trained colorization model to obtain the first portrait image after colorization of the portrait region.

[0035] In this embodiment, after acquiring the second grayscale image, the electronic device can input the first grayscale image and the second grayscale image into a pre-trained colorization model, thereby obtaining a first portrait image with the portrait region colorized, output by the colorization model. The colorization model can extract image features from both the first and second grayscale images, and then output the first portrait image with the portrait region colorized based on the extracted image features. Thus, it can combine the image features of the original first grayscale image and the corresponding second grayscale image to colorize the portrait region, improving the colorization effect.

[0036] In some implementations, the colorization model can be an encoder-decoder model. The encoder in the colorization model extracts image features from a first grayscale image and a second grayscale image, fuses these features, and then the decoder in the colorization model decodes the fused features, outputting a first portrait image with the portrait region colorized. Of course, the specific type of colorization model is not limited.

[0037] The image processing method provided in this application embodiment obtains a portrait mask image corresponding to the portrait area in a grayscale image. Based on the portrait mask image, the portrait area that needs to be colored can be determined. Then, the original portrait area is colored based on the portrait mask image. This enables targeted coloring of the portrait area in the image, thereby reducing the consumption of computing power and increasing the speed of coloring the image.

[0038] Please see Figure 3 , Figure 3A flowchart illustrating an image processing method according to another embodiment of this application is shown. In this embodiment, the portrait matting model includes a first encoding module and a first decoding module, and the coloring model includes a second encoding module, a third encoding module, and a second decoding module. This image processing method is applied to the aforementioned electronic device, and will be discussed below. Figure 3 The process shown will be described in detail. The image processing method may specifically include the following steps:

[0039] Step S210: Obtain a first grayscale image to be colored, wherein the first grayscale image includes the portrait area to be colored.

[0040] In this embodiment, step S210 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0041] Step S220: Input the first grayscale image into the first encoding module, and decode the image features output by the first decoding module to obtain the portrait mask image corresponding to the portrait area output by the first decoding module as the first mask image, wherein the grayscale value of the portrait area in the portrait mask image is different from the grayscale value of the background area other than the portrait area.

[0042] In this embodiment, the portrait matting model includes a first encoding module and a first decoding module. That is, the portrait matting model can be designed with an Encoder-Decoder structure. The Encoder stage continuously downsamples the input grayscale image to extract deep feature information, while the Decoder stage decodes and upsamples the image, and finally performs binary classification processing to output the detection result of the portrait region, i.e., the portrait mask image corresponding to the portrait region. Since the portrait matting model needs to determine whether each pixel is a portrait region or not, using an encoding-decoding network structure allows the output image and input image to be the same size, enabling pixel-by-pixel judgment and thus improving the accuracy of portrait region detection.

[0043] In some implementations, the first encoding module includes N encoding layers, the first decoding module includes N decoding layers, the N encoding layers are cascaded in sequence, and the N encoding layers correspond one-to-one with the N decoding layers. The first grayscale image can be input to the first encoding module, and the intermediate features extracted from the Nth encoding layer can be input to the decoding layer corresponding to the Nth encoding layer. For each encoding layer from the 4th to the (N-1)th encoding layer, the intermediate features extracted from each encoding layer, the intermediate features decoded by the decoding layer corresponding to the next encoding layer of each encoding layer, and the intermediate features extracted from the next encoding layer of each encoding layer after target processing are fused and input to the decoding layer corresponding to each encoding layer. The target processing includes convolution processing and upsampling processing. The intermediate features extracted from the 3rd encoding layer are fused with the intermediate features decoded by the decoding layer corresponding to the 4th encoding layer and input to the decoding layer corresponding to the 3rd encoding layer. The intermediate features decoded by the decoding layer corresponding to the 3rd encoding layer are input to the decoding layer corresponding to the 2nd encoding layer. The intermediate features decoded by the decoding layer corresponding to the 2nd encoding layer are input to the decoding layer corresponding to the 1st encoding layer, and the portrait mask image corresponding to the portrait region output by the first decoding module is used as the first mask image. Where N is a positive integer greater than 3; the fusion of intermediate features can be performed according to the channel dimension. Using the above method, image features extracted from multiple coding layers can be fused and then decoded by the decoding layer to finally output a portrait mask image, thereby ensuring and refining the segmentation of the portrait region, achieving hair-level portrait region segmentation.

[0044] In one possible implementation, N can be 6. For example... Figure 4 As shown, the first encoding module includes 6 encoding layers, and the first decoding module includes 6 decoding layers. The 6 encoding layers are cascaded sequentially, with a one-to-one correspondence between the 6 encoding layers and the 6 decoding layers. The 6 encoding layers can be: layer 1, layer 2, layer 3, layer 4, layer 5, and layer 6. The 6 decoding layers can be: layer 1 corresponding to layer 1, layer 2 corresponding to layer 2, layer 3 corresponding to layer 3, layer 4 corresponding to layer 4, layer 5 corresponding to layer 5, and layer 6 corresponding to layer 6.

[0045] In the above method, after the first grayscale image is input to the first coding layer, the intermediate features extracted by the first coding layer are input to the second coding layer; the second coding layer then performs further feature extraction based on the input intermediate features; the intermediate features extracted by the second coding layer are then input to the third coding layer, where further feature extraction is performed; similarly, the third to sixth coding layers all acquire the intermediate features input from the previous coding layer and perform feature extraction. Each coding layer can extract semantic features and perform downsampling; the intermediate features extracted by the first coding layer can be primary semantic features, while the intermediate features extracted by subsequent coding layers can be high-level semantic features.

[0046] For the 6th encoding layer, the intermediate features extracted from the 6th encoding layer are convolved by a 1x1 convolutional layer and then fused with the intermediate features extracted from the 6th encoding layer. The fused features are then input to the 6th decoding layer. For the 5th encoding layer, the intermediate features extracted from the 6th encoding layer are convolved by a 1x1 convolutional layer and then upsampled by an upsampling layer. They are then fused with the intermediate features extracted from the 5th encoding layer for the first time. The features fused after the first time are then fused with the intermediate features extracted from the 5th encoding layer and the features decoded by the 6th decoding layer for the second time. The features fused after the second time are then input to the 5th decoding layer. The features decoded by the 6th decoding layer are upsampled by an upsampling layer before the second fusion. For the 4th encoding layer, the intermediate features extracted from the 5th encoding layer are convolved by a 1x1 convolutional layer and then upsampled by an upsampling layer. After sampling, the features are first fused with the intermediate features extracted from the 4th encoding layer. The features after the first fusion are then fused a second time with the intermediate features extracted from the 4th encoding layer and the features decoded from the 5th decoding layer. The features after the second fusion are then input into the 4th decoding layer. Before the features decoded from the 5th decoding layer are fused a second time, they are upsampled by an upsampling layer. For the 3rd encoding layer, the features decoded from the 4th decoding layer are upsampled by an upsampling layer and then fused with the intermediate features extracted from the 3rd encoding layer. The fused features are then input into the 3rd decoding layer. The features decoded from the 3rd decoding layer are upsampled by an upsampling layer and then input into the 2nd decoding layer. The features decoded from the 2nd decoding layer are upsampled by an upsampling layer and then input into the 1st decoding layer. Finally, the 1st decoding layer outputs the portrait mask image corresponding to the portrait region. In other words, for the 4th and 5th decoding layers, three types of feature information will be fed into the corresponding decoding layers: one type is the overall image feature information extracted by the current encoding layer (i.e., its corresponding encoding layer); another type is the feature information decoded by the next decoding layer; and the third type is the fusion feature information of the relatively high-level overall image feature information extracted by the next encoding layer and the overall image feature information extracted by the current encoding layer. Furthermore, for the features input into each decoding layer, the features extracted by the corresponding encoding layer can compensate for the information loss caused by subsequent encoding and decoding layers. This avoids information loss caused by multiple layers of encoding and decoding, ensuring the segmentation accuracy in the final output portrait mask image and achieving hair-level portrait region segmentation.

[0047] Optionally, the first encoding layer and the first decoding layer can be 3*3 convolutional layers; the second to Nth encoding layers are MobileNet v3 encoders. This allows for the implementation of a lightweight portrait matting model, facilitating the deployment of the image processing method provided in this embodiment in electronic devices. Furthermore, the upsampling layers can be implemented using BottleNeck blocks (residual blocks).

[0048] In one possible implementation, the grayscale value of the portrait region in the portrait mask image output by the portrait matting model can range from (0, 255). The grayscale value of the portrait region can be greater than 0 and less than or equal to 255, while the grayscale value of the background region can be 0. Specifically, when the first decoding module outputs the final portrait mask image, it determines whether each pixel is a portrait region and obtains the probability that each pixel is a portrait region. When the probability is 0, the grayscale value of that pixel in the output portrait mask image is 0. When the probability is not 0, the grayscale value is positively correlated with the probability value. Thus, the grayscale value of most of the portrait region can be 255, while the grayscale value of some pixels in the hair region at the edge is greater than 0 but less than 255, thereby achieving fine portrait region segmentation, i.e., hair-level portrait region segmentation.

[0049] In some implementations, when training the above portrait matting model, a sample grayscale image set can be obtained. The sample grayscale image set includes multiple sample grayscale images, each of which contains a portrait region. In addition, each sample grayscale image is labeled with its corresponding portrait mask image. The sample grayscale images are input into the above untrained portrait matting model to obtain the output result of the portrait matting model. Then, based on the output result of the portrait matting model and the portrait mask image labeled on the sample grayscale images, the total loss value is determined. Based on the total loss value, the portrait matting model is iteratively trained until the portrait matting model meets the training termination condition, thereby obtaining the trained portrait matting model.

[0050] In one possible implementation, the Adam optimizer can be used to iteratively train the portrait matting model based on the total loss value until the loss value of the output of the portrait matting model converges. The model at this point is then saved, resulting in the trained portrait matting model. The Adam optimizer combines the advantages of both AdaGra (Adaptive Gradient) and RMSProp optimization algorithms, comprehensively considering the first moment estimation (mean of the gradient) and the second moment estimation (uncentered variance of the gradient) to calculate the update step size.

[0051] In one possible implementation, the training termination condition may include: the number of training iterations reaches the target number; or the total loss value of the portrait matting model's output meets a set condition. Optionally, the convergence condition is to minimize the total loss value, using an initial learning rate of 1e-3, with the learning rate decreasing cosine with the number of steps, batch_size = 8, and training for 16 epochs, after which convergence is considered complete. Here, batch_size can be understood as a batch processing parameter, its limit being the total number of training set samples, and epoch refers to the number of times the entire dataset is trained using all samples in the training set. In simple terms, the value of epoch is the number of times the entire dataset is processed; one epoch is equivalent to training once using all samples in the training set. Optionally, the total loss value meeting the set condition may include: the total loss value is less than a set threshold. Of course, the specific set condition is not limited.

[0052] Step S230: Based on the first grayscale image and the first mask image, obtain the second grayscale image corresponding to the portrait region.

[0053] In this embodiment, step S230 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0054] Step S240: Input the first grayscale image into the second encoding module to obtain the first image feature output by the second encoding module.

[0055] Step S250: Input the second grayscale image into the third encoding module to obtain the second image feature output by the third encoding module.

[0056] Step S260: Fuse the first image feature with the second image feature to obtain the third image feature.

[0057] Step S270: Input the third image feature into the second decoding module to obtain the first portrait image output by the second decoding module. The first portrait image is a portrait image after coloring the portrait area in the first grayscale image.

[0058] In the embodiments of this application, such as Figure 5As shown, the colorization model includes a second encoding module, a third encoding module, and a second decoding module. In other words, the colorization model can also be designed as an Encoder-Decoder structure. The Encoder stage continuously downsamples the input grayscale image to extract deep feature information, while the Decoder stage decodes and upsamples the image, ultimately outputting a colorized image of the portrait region. Understandably, the first grayscale image is the original image; therefore, the first image features extracted by the second encoder can be understood as the image features of the original image, including high-level semantic features of the portrait foreground (i.e., the portrait region). The grayscale value of the background area outside the portrait region in the second grayscale image is 0, meaning the second grayscale image only includes grayscale information of the portrait region. Therefore, the second image features extracted by the second encoding module can be understood as refined features of the portrait foreground (i.e., the portrait region). Thus, by fusing the first and second image features and then outputting the colorized portrait image through the second decoding module, the edges of the portrait region can be made more refined and softer when colorizing it.

[0059] In some implementations, fusing the first image feature and the second image feature can be achieved by stitching the first image feature and the second image feature together according to the channel dimension to obtain the fused image feature as the third image feature.

[0060] In some implementations, when training the above colorization model, a sample image set can be obtained. This set includes multiple first sample grayscale images and corresponding second sample grayscale images. The first sample grayscale images include a portrait region and a background region, while the second sample grayscale images include the portrait region from the first sample grayscale images, with the background region having a grayscale value of 0. Furthermore, each second sample grayscale image is labeled with a colorized sample image showing the portrait region within it. Each first sample grayscale image and its corresponding second sample grayscale image are input into the untrained colorization model to obtain its output. The total loss value is then determined based on the colorization model's output and the labeled colorized image on the second sample grayscale image. The colorization model is iteratively updated based on the total loss value until it meets the training termination condition, resulting in the trained colorization model. Therefore, by learning only the colorization of the portrait region, the training difficulty is reduced, the colorization effect is guaranteed, and the model size is effectively controlled.

[0061] The image processing method provided in this application segmentes the portrait region using an encoding-decoding structure portrait matting model to obtain a portrait mask image. Then, based on the portrait mask image, the portrait region that needs to be colored can be determined. Furthermore, the coloring model with an encoding-decoding structure is used to perform targeted coloring on the portrait region in the image, thereby reducing the consumption of computing power and improving the speed of coloring the image.

[0062] Please see Figure 6 , Figure 6 A schematic flowchart of another embodiment of the image processing method provided in this application is shown. This image processing method is applied to the aforementioned electronic device, and will be discussed below. Figure 6 The process shown will be described in detail. The image processing method may specifically include the following steps:

[0063] Step S310: Obtain the first grayscale image to be colored, wherein the first grayscale image includes the portrait area to be colored.

[0064] Step S320: Input the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as the first mask image, wherein the grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region.

[0065] Step S330: Based on the first grayscale image and the first mask image, obtain the second grayscale image corresponding to the portrait region.

[0066] Step S340: Input the first grayscale image and the second grayscale image into the pre-trained colorization model to obtain a first portrait image after colorization of the portrait area. The pixel value of the background area other than the colorized portrait area in the first portrait image is 0.

[0067] In the embodiments of this application, steps S310 to S340 can be referred to the content of the foregoing embodiments, and will not be repeated here.

[0068] Step S350: Based on the first grayscale image and the first mask image, obtain the third grayscale image corresponding to the background region.

[0069] In this embodiment, the background area in the first portrait image obtained above, excluding the colored portrait area, has a pixel value of 0. That is, the first portrait image obtained after colorizing the portrait area in the second grayscale image does not include the background area. Therefore, after obtaining the first portrait image, it can be further processed to obtain a final portrait image that is the result of colorizing the portrait area in the first grayscale image. The electronic device can obtain a third grayscale image corresponding to the background area based on the first grayscale image and the first mask image. This third grayscale image can be understood as a grayscale image obtained by setting the grayscale value of the portrait area in the first grayscale image to 0.

[0070] In some implementations, the electronic device can invert the grayscale values ​​of the portrait region and the background region in the first mask image to obtain a second mask image; multiply the first grayscale image and the second mask image to obtain a third grayscale image corresponding to the background region. Specifically, if the first mask image is a binary image, i.e., the grayscale value of the background region is 0 and the grayscale value of the portrait region is 255, then the grayscale values ​​of the portrait region and the background region in the first mask image can be directly inverted to obtain the second mask image; if the first mask image is a binary image, i.e., the grayscale value of the background region is 0 and the grayscale value of the portrait region is (0, 255), then all grayscale values ​​of the portrait region can be set to 255, and then the grayscale values ​​of the portrait region and the background region in the first mask image can be inverted to obtain the second mask image.

[0071] Step S360: Based on the first portrait image and the first mask image, obtain the second portrait image after colorization.

[0072] In this embodiment of the application, the electronic device can also obtain a colorized second portrait image based on the first portrait image and the first mask image. Optionally, the first portrait image can be multiplied by the first mask image to obtain the colorized second portrait image.

[0073] Step S370: Based on the third grayscale image and the second portrait image, obtain a colorized third portrait image, wherein the third portrait image includes the background area in the first grayscale image and the colorized portrait area.

[0074] In this embodiment, after obtaining the second portrait image, the electronic device can acquire a colorized third portrait image based on the third grayscale image and the second portrait image. This third portrait image can be understood as an image after colorizing the portrait area in the first grayscale image. In other words, the third portrait image includes the background area in the first grayscale image and the colorized portrait area. This achieves the goal of colorizing the portrait area in a grayscale image that includes both the portrait area and the background area, satisfying the user's need to colorize the subject in a black and white photograph to highlight the subject. Optionally, the third grayscale image can be added to the second portrait image, that is, the pixel values ​​of corresponding pixels in the third grayscale image and the second portrait image can be added together to obtain the third portrait image.

[0075] The image processing method provided in this application embodiment can perform targeted colorization of the portrait area in an image, thereby reducing the consumption of computing power and increasing the speed of colorization. In addition, the obtained colorized portrait image is further processed to obtain a portrait image after colorization of the portrait area in the original grayscale image, which meets the user's need to highlight the subject.

[0076] Please see Figure 7 This document illustrates a structural block diagram of an image processing apparatus 400 provided in an embodiment of this application. The image processing apparatus 400 utilizes the aforementioned electronic device and includes: a first acquisition module 410, a portrait segmentation module 420, a second acquisition module 430, and an image coloring module 440. Specifically, the first acquisition module 410 acquires a first grayscale image to be colorized, wherein the first grayscale image includes a portrait region to be colorized; the portrait segmentation module 420 inputs the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the grayscale value of the portrait region in the portrait mask image differs from the grayscale value of the background region other than the portrait region; the second acquisition module 430 acquires a second grayscale image corresponding to the portrait region based on the first grayscale image and the first mask image; and the image coloring module 440 inputs the first grayscale image and the second grayscale image into a pre-trained coloring model to obtain a first portrait image after coloring the portrait region.

[0077] In some embodiments, the portrait matting model 400 includes a first encoding module and a first decoding module. The portrait segmentation module 420 can be specifically used to: input the first grayscale image into the first encoding module, and decode the image features output by the first decoding module to obtain a portrait mask image corresponding to the portrait region output by the first decoding module as the first mask image.

[0078] In one possible implementation, the first encoding module includes N encoding layers, and the first decoding module includes N decoding layers. The N encoding layers are cascaded sequentially, and each of the N encoding layers corresponds one-to-one with the N decoding layers. The portrait segmentation module 420 can also be specifically used to: input the first grayscale image into the first encoding module, and input the intermediate features extracted from the Nth encoding layer into the decoding layer corresponding to the Nth encoding layer; for each encoding layer from the 4th to the (N-1)th encoding layers, fuse the extracted intermediate features from each encoding layer, the decoded intermediate features from the next encoding layer, and the extracted intermediate features from the next encoding layer after target processing, and then input this fusion into the decoding layer corresponding to each encoding layer. The target processing includes convolution processing and upsampling processing. The intermediate features extracted by the third coding layer are fused with the intermediate features decoded by the decoding layer corresponding to the fourth coding layer, and then input to the decoding layer corresponding to the third coding layer. The intermediate features decoded by the decoding layer corresponding to the third coding layer are input to the decoding layer corresponding to the second coding layer. The intermediate features decoded by the decoding layer corresponding to the second coding layer are input to the decoding layer corresponding to the first coding layer, so as to obtain the portrait mask image corresponding to the portrait region output by the first decoding module as the first mask image.

[0079] In some embodiments, the colorization model includes a second encoding module, a third encoding module, and a second decoding module. The image colorization module 440 can be specifically used to: input the first grayscale image to the second encoding module to obtain a first image feature output by the second encoding module; input the second grayscale image to the third encoding module to obtain a second image feature output by the third encoding module; fuse the first image feature and the second image feature to obtain a third image feature; and input the third image feature to the second decoding module to obtain the first portrait image output by the second decoding module.

[0080] In one possible implementation, the image coloring module 440 fuses the first image feature and the second image feature to obtain a third image feature, which may include: stitching the first image feature and the second image feature according to the channel dimension to obtain the fused image feature as the third image feature.

[0081] In some embodiments, the pixel value of the background area other than the colored portrait area in the first portrait image is 0. The image processing apparatus 400 may further include a third acquisition module, a fourth acquisition module, and a fifth acquisition module. The third acquisition module is used to acquire a third grayscale image corresponding to the background area based on the first grayscale image and the first mask image after the first grayscale image and the second grayscale image are input into a pre-trained colorization model to obtain a first portrait image with the colored portrait area; the fourth acquisition module is used to acquire a second colored portrait image based on the first portrait image and the first mask image; the fifth acquisition module is used to acquire a third colored portrait image based on the third grayscale image and the second portrait image, wherein the third portrait image includes the background area in the first grayscale image and the colored portrait area.

[0082] In one possible implementation, the third acquisition module may be specifically used to: invert the grayscale values ​​of the portrait region and the background region in the first mask image to obtain a second mask image; and multiply the first grayscale image and the second mask image to obtain a third grayscale image corresponding to the background region.

[0083] In some implementations, the grayscale value of the background area other than the portrait area in the first mask image is 0. The second acquisition module 430 may be specifically used to: multiply the first grayscale image with the first mask image to obtain a second grayscale image corresponding to the portrait area.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0086] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0087] In summary, the solution provided in this application involves: acquiring a first grayscale image to be colored, wherein the first grayscale image includes the portrait region to be colored; inputting the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as a first mask image, wherein the grayscale values ​​of the portrait region in the portrait mask image are different from the grayscale values ​​of the background region other than the portrait region; based on the first grayscale image and the first mask image, acquiring a second grayscale image corresponding to the portrait region; and inputting the first grayscale image and the second grayscale image into a pre-trained colorization model to obtain a first portrait image after colorizing the portrait region in the first grayscale image. Therefore, targeted colorization of the portrait region in an image can be achieved, thereby reducing computational power consumption and improving the speed of image colorization.

[0088] Please refer to Figure 8 This document illustrates a structural block diagram of an electronic device according to an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, smartwatch, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by the one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0089] Processor 110 may include one or more processing cores. Processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0090] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0091] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 500 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0092] The computer-readable storage medium 500 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 500 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has storage space for program code 510 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 510 may be compressed, for example, in a suitable form.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain a first grayscale image to be colored, wherein the first grayscale image includes the portrait area to be colored; The first grayscale image is input into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as the first mask image. The grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region. The portrait matting model includes a first encoding module and a first decoding module. The first encoding module includes N encoding layers and the first decoding module includes N decoding layers. The N encoding layers are cascaded in sequence, and the N encoding layers and the N decoding layers correspond one-to-one. Based on the first grayscale image and the first mask image, obtain the second grayscale image corresponding to the portrait region; The first grayscale image and the second grayscale image are input into a pre-trained colorization model to obtain a first portrait image after colorization of the portrait region; The step of inputting the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as the first mask image includes: The first grayscale image is input to the first encoding module, and the intermediate features extracted from the Nth encoding layer are input to the decoding layer corresponding to the Nth encoding layer. For each coding layer from the 4th coding layer to the (N-1)th coding layer, the intermediate features extracted from each coding layer, the intermediate features decoded by the decoding layer corresponding to the next coding layer of each coding layer, and the intermediate features extracted from the next coding layer of each coding layer after target processing are fused and then input into the decoding layer corresponding to each coding layer. The target processing includes convolution processing and upsampling processing. The intermediate features extracted from the third coding layer are fused with the intermediate features decoded by the corresponding decoding layer of the fourth coding layer, and then input into the corresponding decoding layer of the third coding layer. The intermediate features decoded by the decoding layer corresponding to the third encoding layer are input into the decoding layer corresponding to the second encoding layer. The intermediate features decoded by the decoding layer corresponding to the second encoding layer are input to the decoding layer corresponding to the first encoding layer to obtain the portrait mask image corresponding to the portrait region output by the first decoding module as the first mask image.

2. The method according to claim 1, characterized in that, The colorization model includes a second encoding module, a third encoding module, and a second decoding module. The step of inputting the first grayscale image and the second grayscale image into the pre-trained colorization model to obtain a first portrait image after colorizing the portrait region includes: The first grayscale image is input into the second encoding module to obtain the first image feature output by the second encoding module; The second grayscale image is input into the third encoding module to obtain the second image feature output by the third encoding module; The first image feature is fused with the second image feature to obtain the third image feature; The third image feature is input into the second decoding module to obtain the first portrait image output by the second decoding module.

3. The method according to claim 2, characterized in that, The step of fusing the first image features with the second image features to obtain the third image features includes: The first image feature and the second image feature are concatenated according to the channel dimension to obtain the fused image feature as the third image feature.

4. The method according to any one of claims 1-3, characterized in that, The pixel value of the background area other than the colored portrait area in the first portrait image is 0. After inputting the first grayscale image and the second grayscale image into the pre-trained colorization model to obtain the first portrait image after colorizing the portrait area, the method further includes: Based on the first grayscale image and the first mask image, obtain the third grayscale image corresponding to the background region; Based on the first portrait image and the first mask image, obtain the colorized second portrait image; Based on the third grayscale image and the second portrait image, a third portrait image after colorization is obtained. The third portrait image includes the background area in the first grayscale image and the colorized portrait area.

5. The method according to claim 4, characterized in that, The step of obtaining a third grayscale image corresponding to the background region based on the first grayscale image and the first mask image includes: The grayscale values ​​of the portrait area and the background area in the first mask image are inverted to obtain the second mask image; Multiply the first grayscale image with the second mask image to obtain the third grayscale image corresponding to the background region.

6. The method according to any one of claims 1-3, characterized in that, The grayscale value of the background area other than the portrait area in the first mask image is 0. The step of obtaining the second grayscale image corresponding to the portrait area based on the first grayscale image and the first mask image includes: Multiply the first grayscale image with the first mask image to obtain the second grayscale image corresponding to the portrait area.

7. An image processing apparatus, characterized in that, The device includes: a first acquisition module, a portrait segmentation module, a second acquisition module, and an image colorization module, wherein, The first acquisition module is used to acquire a first grayscale image to be colored, wherein the first grayscale image includes a portrait area to be colored; The portrait segmentation module is used to input the first grayscale image into a pre-trained portrait matting model to obtain a portrait mask image corresponding to the portrait region as the first mask image. The grayscale value of the portrait region in the portrait mask image is different from the grayscale value of the background region other than the portrait region. The portrait matting model includes a first encoding module and a first decoding module. The first encoding module includes N encoding layers, and the first decoding module includes N decoding layers. The N encoding layers are cascaded in sequence, and the N encoding layers and the N decoding layers correspond one-to-one. The second acquisition module is used to acquire a second grayscale image corresponding to the portrait region based on the first grayscale image and the first mask image; The image colorization module is used to input the first grayscale image and the second grayscale image into a pre-trained colorization model to obtain a first portrait image after colorization of the portrait region; The portrait segmentation module is specifically used to input the first grayscale image into the first encoding module, and input the intermediate features extracted from the Nth encoding layer into the decoding layer corresponding to the Nth encoding layer; for each encoding layer from the 4th to the (N-1)th encoding layer, the intermediate features extracted from each encoding layer, the intermediate features decoded by the decoding layer corresponding to the next encoding layer of each encoding layer, and the intermediate features extracted from the next encoding layer of each encoding layer after target processing are fused and then input into the decoding layer corresponding to each encoding layer. The target processing includes convolution processing and upsampling processing; the intermediate features extracted by the third encoding layer are fused with the intermediate features decoded by the decoding layer corresponding to the fourth encoding layer, and then input into the decoding layer corresponding to the third encoding layer; the intermediate features decoded by the decoding layer corresponding to the third encoding layer are input into the decoding layer corresponding to the second encoding layer; the intermediate features decoded by the decoding layer corresponding to the second encoding layer are input into the decoding layer corresponding to the first encoding layer, so as to obtain the portrait mask image corresponding to the portrait region output by the first decoding module as the first mask image.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-6.

Images