An image processing method and apparatus

Abstract

The embodiment of the application discloses an image processing method, according to the classification label obtained by performing semantic segmentation on the to-be-processed image, determining the subject region, the edge undetermined region and the background region corresponding to the target object in the to-be-processed image, setting the identification value of the subject region as a first identification value, and setting the identification value of the background region as a second identification value. According to the pixel points adjacent to the edge undetermined region in the subject region and the background region, the pixel feature similarity between the pixel points in the edge undetermined region and the adjacent pixel points is calculated, and according to the similarity calculation result and the first and second identification values, the identification value of the pixel points in the edge undetermined region is determined. The closeness of the identification value to the first identification value and the second identification value can determine whether the pixel points belong to the target object, so that the edge part of the target object can be accurately determined, the determination accuracy of the region where the target object is located in the to-be-processed image is improved, and the image background removal scene with high precision requirement is suitable.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201910364702.0, filed on April 30, 2019, entitled "An Image Processing Method and Apparatus". Technical Field

[0002] This application relates to the field of image processing, and in particular to an image processing method and apparatus. Background Technology

[0003] Image background removal mainly refers to removing the background from an image except for the target object. For example, in a portrait image, if the target object is a human figure, then image background removal for that image means removing all pixels in the image except for the human figure as background.

[0004] Neural network-based semantic segmentation technology can identify the classification label of each pixel in an image. The classification label can identify, for example, a person, a cat, or a bicycle. Therefore, most commonly used image background removal techniques employ image semantic segmentation technology. This technology identifies the classification label of each pixel in the image and removes pixels (i.e., background) whose classification labels do not match the category of the target object.

[0005] However, the problem with traditional semantic segmentation technology is that the accuracy of identifying the classification labels of edge pixels of the target object is not high. That is, the semantic labels of the edge pixels of the target object often do not match the actual classification of the object. For example, the classification labels of the edge pixels of a portrait do not indicate the category of portrait, which leads to the removal of pixels that should actually belong to the portrait as background pixels.

[0006] It is evident that traditional methods of image background removal using semantic segmentation techniques have low edge recognition accuracy for target objects in images, making them unsuitable for high-precision image background removal scenarios. Summary of the Invention

[0007] To address the aforementioned technical problems, this application provides an image processing method that can accurately determine the edge portion of a target object, improving the accuracy of determining the region where the target object is located in the image to be processed, thus making it applicable to high-precision image background removal scenarios.

[0008] The embodiments of this application disclose the following technical solutions:

[0009] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0010] Based on the classification labels of pixels obtained through semantic segmentation of the image to be processed, the main body region, edge undetermined region, and background region corresponding to the target object in the image to be processed are determined; the identifier value of the main body region is set as a first identifier value, and the identifier value of the background region is set as a second identifier value;

[0011] Based on the pixels in the main region and the background region that are adjacent to the edge undetermined region, calculate the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels;

[0012] Based on the similarity calculation results of the pixels in the edge undetermined region, and the first and second identifier values, the identifier values ​​of the pixels in the edge undetermined region are determined;

[0013] The edge contour of the target object in the image to be processed is determined based on the identifier value of the pixel in the edge undetermined region.

[0014] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising a first determining unit, a calculation unit, a second determining unit, and a third determining unit:

[0015] The first determining unit is configured to determine the main body region, edge undetermined region, and background region of the target object in the image to be processed based on the classification labels of the pixels obtained by semantic segmentation of the image to be processed; the identifier value of the main body region is set to a first identifier value, and the identifier value of the background region is set to a second identifier value;

[0016] The calculation unit is used to calculate the pixel feature similarity between adjacent pixels in the undetermined edge region based on the pixels in the main body region and the background region that are adjacent to the undetermined edge region.

[0017] The second determining unit is used to determine the identifier value of the pixel in the edge-to-edge region based on the similarity calculation result of the pixel in the edge-to-edge region, and the first identifier value and the second identifier value;

[0018] The third determining unit is used to determine the edge contour of the target object in the image to be processed based on the identifier value of the pixel in the edge-to-determined region.

[0019] Thirdly, embodiments of this application provide an apparatus for image processing, the apparatus including a processor and a memory:

[0020] The memory is used to store program code and transmit the program code to the processor;

[0021] The processor is used to execute the image processing method described in the first aspect according to the instructions in the program code.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium for storing program code for executing the image processing method described in the first aspect.

[0023] As can be seen from the above technical solution, for an image to be processed that includes a target object, the classification labels obtained from semantic segmentation of the image can determine the main body region, edge undetermined region, and background region corresponding to the target object in the image. The identifier value of the main body region is set as the first identifier value, and the identifier value of the background region is set as the second identifier value. Based on the pixels adjacent to the edge undetermined region in the main body region and background region, the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels is calculated. Based on the similarity calculation result of the pixels in the edge undetermined region and their adjacent pixels, as well as the first and second identifier values, the identifier value of the pixels in the edge undetermined region is determined. Since the pixel features of pixels belonging to the target object are more similar to those of pixels in the background region, the similarity calculation result determined for any pixel in the edge undetermined region can reflect the probability that the pixel belongs to the target object or the background. Since different identifier values ​​are set for pixels in the background area and the main body area, the degree of similarity between the identifier value determined for the pixel in the edge undetermined area and the first identifier value and the second identifier value can determine whether the pixel belongs to the target object. This can accurately determine the edge part of the target object, improve the determination accuracy of the target object area in the image to be processed, and thus be applicable to high-precision image background removal scenarios. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 An example of edge recognition using traditional semantic segmentation techniques;

[0026] Figure 2 An example diagram illustrating an application scenario of an image processing method provided in this application embodiment;

[0027] Figure 3 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0028] Figure 4 Example diagram of FCN structure provided in the embodiments of this application;

[0029] Figure 5 Example diagram of intermediate processed images provided in embodiments of this application;

[0030] Figure 6a Example diagrams illustrating the process of generating ID photos based on determined edge contours provided in this application embodiment;

[0031] Figure 6b A flowchart illustrating a method for determining an intermediate processing image corresponding to an image to be processed, provided in an embodiment of this application;

[0032] Figure 7 Example diagram of the first image provided for embodiments of this application;

[0033] Figure 8 Example of a second image provided for an embodiment of this application;

[0034] Figure 9 Example diagram of the second pixel region provided in the embodiments of this application;

[0035] Figure 10 An example image of the second pixel region after dilation processing provided in an embodiment of this application;

[0036] Figure 11 A flowchart for generating ID photos using social software is provided as an embodiment of this application;

[0037] Figure 12 Example diagram of an interface for generating ID photos using social software, provided for embodiments of this application;

[0038] Figure 13a A structural diagram of an image processing apparatus provided in an embodiment of this application;

[0039] Figure 13b A structural diagram of an image processing device provided in an embodiment of this application;

[0040] Figure 14 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application will now be described with reference to the accompanying drawings.

[0042] Traditional semantic segmentation techniques have low accuracy in recognizing the classification labels of edge pixels of target objects. That is, the semantic labels of the edge pixels of the target object often do not match the actual classification of the object. As a result, when removing the background, pixels that should actually belong to the target object are identified as background and removed as well.

[0043] For example Figure 1 As shown, Figure 1 The image shows the results of edge recognition using traditional semantic segmentation techniques. The first column shows the original image containing different target objects (e.g., airplanes, birds, chairs). The second column shows the results of edge recognition using Fully Convolutional Networks-8s (FCN-8s). The third column shows the results of edge recognition using a dilated convolutional network (Dilated Conv). The fourth column shows the actual edge of the target object. As can be seen from the image, the accuracy of edges recognized using traditional semantic segmentation techniques differs significantly from the actual edge. For example, in the case of a chair, the semantic labels of the identified edge pixels do not match the actual classification of the object, leading to the armrest pixels being identified as background. Therefore, during background removal, the chair armrests are removed as background as well.

[0044] It's worth noting that image background removal can be applied in many scenarios, such as when users generate personal ID photos by removing the background. In this scenario, ID photos often have backgrounds of different colors, and users may sometimes need to temporarily change the background, or they may want to generate ID photos based on personal snapshots. To make it easier for users to generate ID photos and avoid the need to go to a photo studio, the original background can be removed and a new background overlaid, thus quickly obtaining an ID photo that meets the user's needs.

[0045] However, generating ID photos by removing the background of an image requires very high accuracy in recognizing the edge pixels of the target object. To solve the above technical problems and meet the high accuracy requirements, this application provides an image processing method. Based on semantic segmentation technology, this method further determines the edge region to be determined, so as to further accurately identify the edge contour of the target object by calculating the pixel feature similarity between the pixels in the edge region to be determined and the adjacent pixels, thereby accurately determining the edge contour of the target object in the image to be processed.

[0046] This method can be applied to image processing devices, which can be terminal devices, such as smart terminals, computers, personal digital assistants (PDAs), tablet computers, and other devices with image processing capabilities.

[0047] The image processing device can also be a server, which provides image processing services to terminal devices. The terminal device can upload the image to be processed to the server, and the server uses the method provided in this application embodiment to obtain the edge contour of the target object in the image to be processed, and returns the processing result to the terminal device. The server can be a standalone server or a server in a cluster.

[0048] To facilitate understanding of the technical solution of this application, the image processing method provided in the embodiments of this application will be introduced below, taking a terminal device as an example, in conjunction with a practical application scenario.

[0049] See Figure 2 , Figure 2 The diagram illustrates an application scenario of an image processing method. This scenario includes a terminal device 201, which acquires an image to be processed. The image includes a target object, which is the object desired by the user within the image. For example, the image may include a portrait, flowers, trees, etc. If the user requires a portrait, then the portrait is the target object, while the flowers, trees, etc., can serve as the background.

[0050] Terminal device 201 performs semantic segmentation on the image to be processed to obtain pixel classification labels. Based on these labels, it can determine the main body region, edge-to-be-determined region, and background region corresponding to the target object in the image to be processed. The main body region and background region are precisely determined regions. The main body region is the area in the image to be processed containing the main body of the target object, excluding its edges. The background region is the area in the image to be processed containing the background that needs to be removed relative to the target object. The identifier value of the main body region is set to a first identifier value, and the identifier value of the background region is set to a second identifier value. The first and second identifier values ​​are used to determine the identifier values ​​of pixels in the edge-to-be-determined region.

[0051] The edge-to-be-determined region is the area in the image to be processed where the edge portion corresponding to the target object is located. This edge region includes not only pixels that actually belong to the edge portion of the target object, but may also include pixels that actually belong to the background. Therefore, the terminal device 201 needs to further identify the pixels in the edge-to-be-determined region to determine which pixels actually belong to the target object and which pixels actually belong to the background, thereby accurately determining the edge contour of the target object.

[0052] Specifically, the terminal device 201 calculates the pixel feature similarity between pixels in the undetermined edge region and their adjacent pixels based on pixels in the main body region and background region of the image to be processed that are adjacent to the undetermined edge region. Then, based on the similarity calculation result and the first and second identifier values, it determines the identifier value of the pixels in the undetermined edge region. Here, adjacent pixels are pixels in the main body region and background region that are relatively close to the undetermined edge region. In this embodiment, adjacent pixels are not strictly adjacent to a pixel in the undetermined edge region; the adjacent pixels of a pixel in the undetermined edge region can be pixels surrounding that pixel in the undetermined edge region.

[0053] Since the pixel features of pixels belonging to the target object are more similar to those of pixels in the background region, the similarity calculation result of any pixel in the edge-to-edge region can reflect the probability that the pixel belongs to the target object or the background. The higher the similarity of pixel features between a pixel and its adjacent pixels in the main body region, the higher the probability that the pixel actually belongs to the target object; similarly, the higher the similarity of pixel features between a pixel and its adjacent pixels in the background region, the higher the probability that the pixel actually belongs to the background.

[0054] The known subject and background regions are accurate and each has a corresponding identifier value (first identifier value and second identifier value). The identifier value of pixels in the edge region to be determined is generated based on the similarity calculation results between that pixel and its neighboring pixels in the subject and background regions, as well as the identifier values ​​corresponding to the subject and background regions, respectively. Therefore, the closeness of the identifier value determined for a pixel in the edge region to the first and second identifier values ​​can reflect whether the pixel in the edge region actually belongs to the target object or the background. If the pixel's identifier value is the first identifier value or close to the first identifier value, then the pixel actually belongs to the target object; if the pixel's identifier value is the second identifier value or close to the second identifier value, then the pixel actually belongs to the background.

[0055] Next, the image processing method provided in the embodiments of this application will be described in conjunction with the accompanying drawings.

[0056] See Figure 3 , Figure 3 A flowchart of an image processing method is shown, which includes:

[0057] S301. Based on the classification labels of the pixels obtained by semantic segmentation of the image to be processed, determine the main body region, edge undetermined region, and background region of the target object in the image to be processed.

[0058] The terminal device acquires the image to be processed and performs semantic segmentation on the image to obtain classification labels for each pixel. These classification labels represent the probability that a pixel belongs to a certain category of object; for example, they might represent the probability that a pixel belongs to a person, a bicycle, or a chair.

[0059] The method of acquiring the image to be processed can be varied. For example, the image to be processed can be obtained by the user taking a picture through the terminal device, or it can be obtained by the user selecting a picture from the local (album, storage, etc.).

[0060] In this embodiment, FCN can be used for semantic segmentation of the image to be processed. (See FCN for details.) Figure 4 As shown, the last three layers of a Convolutional Neural Network (CNN) (corresponding to 4096, 4096, 21 in the diagram above) are replaced with multi-channel convolutional layers (i.e., fully convolutional layers) with 1*1 kernels of the same vector length, thus obtaining an FCN. Here, 21 represents the dimension of the final generated classification labels. Of course, other semantic segmentation networks can be used for semantic segmentation; this embodiment does not limit this.

[0061] Semantic segmentation can yield a probability matrix with the same resolution as the image to be processed. The probability value of each point in the matrix represents the classification label of the corresponding pixel in the image to be processed. When the probability corresponding to a certain classification label of a pixel reaches a threshold, it can be considered that the pixel corresponds to the category of that classification label.

[0062] In one possible implementation, S201 can be implemented by determining the intermediate processing image corresponding to the image to be processed based on the classification labels of the pixels obtained by semantic segmentation of the image to be processed.

[0063] The intermediate processed image is used to identify the subject region, edge-to-be-determined region, and background region corresponding to the target object in the image to be processed. The identifier value of the subject region is set to a first identifier value, and the identifier value of the background region is set to a second identifier value. The first identifier value and the second identifier value are different identifier values, thereby distinguishing the subject region and the background region through different identifier values.

[0064] The identifier value can be represented in different forms. For example, the identifier value can be represented by the transparency value (alpha, i.e., α). In order to clearly distinguish between the subject area and the background area, the first identifier value can be represented by the maximum transparency value, such as the first identifier value being 1, and the second identifier value can be represented by the minimum transparency value, such as the second identifier value being 0.

[0065] In this case, the determined intermediate processing image can be seen from Figure 5 As shown, the intermediate processed image can be represented using a trimap. In Figure 5 In the diagram, the white area represents the main body area 501, the gray area represents the edge area to be determined 502, and the black area represents the background area 503.

[0066] Of course, the identifier value can also be represented by a number without special meaning. For example, the first identifier value can be 50, the second identifier value can be 0, and so on.

[0067] S302. Based on the pixels in the main area and the background area that are adjacent to the edge undetermined area, calculate the pixel feature similarity between the pixels in the edge undetermined area and the adjacent pixels.

[0068] For each pixel in the region to be determined by the edge, its neighboring pixels are found in the main body and background regions of the image to be processed. The pixel feature similarity between the pixel in the region to be determined and its neighboring pixels is calculated. Neighboring pixels can be, for example, multiple pixels closely surrounding the pixel in the region to be determined by the edge. Pixel features identify the relevant characteristics of the content represented by the pixel in the image to be processed.

[0069] It should be noted that, in order to calculate pixel feature similarity, pixel features can be constructed for each pixel in the region to be determined. Pixel features can include different information. In one possible implementation, pixel features can include color information and position information. Specifically, the color information can be the red (R), green (G), and blue (B) color values ​​of the pixel in the image to be processed; the position information can be the x and y coordinates of the pixel in the image to be processed.

[0070] Color information can be in addition to R, G, and B color values, as well as color spaces such as hue, saturation, and value (HSV).

[0071] Taking pixel features including color values ​​R, G, and B, and coordinate values ​​x and y as an example, we construct pixel features X(r,g,b,x,y) and define the kernel function k(i,j) as k(i,j)=1-||X(i)-X(j)|| / C. The similarity calculation result can be represented by a similarity matrix A, where C is the maximum feature distance of X, and || indicates the use of the feature Euclidean distance L2 norm. Then, the similarity calculation result of pixel points in the edge region to be determined can be expressed as: A ij = k(i,j), where i represents the pixel position of a pixel in the region to be determined on the edge, and j represents the pixel position of the adjacent pixel of the pixel position corresponding to i.

[0072] S303. Based on the similarity calculation results of the pixels in the edge undetermined region, and the first and second identifier values, determine the identifier values ​​of the pixels in the edge undetermined region.

[0073] Understandably, the proximity of a pixel's identifier value in the undetermined edge region to its first and second identifier values ​​can determine whether the pixel belongs to the target object, thus accurately identifying the edge portion of the target object. In determining the identifier value of a pixel in the undetermined edge region, the proximity of the identifier values ​​of adjacent pixels is related to the similarity of their pixel features. For example, the higher the similarity of the pixel features of adjacent pixels, the closer the identifier value of a pixel in the undetermined edge region is to the identifier values ​​of the regions containing its adjacent pixels.

[0074] For example, if the first identifier value is 1 and the second identifier value is 0, the higher the similarity of pixel features between a pixel in the edge region and its adjacent pixels in the main region, the closer the identifier value of that pixel in the edge region is to the first identifier value of 1. For example, the identifier value is set to 0.9. Similarly, if the higher the similarity of pixel features between a pixel in the edge region and its adjacent pixels in the background region, the closer the identifier value of that pixel in the edge region is to the second identifier value of 0. For example, the identifier value is set to 0.1.

[0075] In some cases, if the similarity of pixel features between adjacent pixels is high enough, for example, if a target condition is met, it can be directly determined that the identifier value of a pixel in the edge region to be determined is the same as the identifier value of its adjacent pixels. The target condition can be that the similarity reaches a first preset threshold.

[0076] For example, if the first identifier value is 1 and the second identifier value is 0, and the similarity of pixel features between a pixel in the edge region and its adjacent pixels in the main body region meets the target condition (the target condition is a similarity of 90%), then the identifier value of the pixel in the edge region is the same as the identifier value of its adjacent pixels in the main body region, and the identifier value of that pixel in the edge region is determined to be the first identifier value 1. If the similarity of pixel features between a pixel in the edge region and its adjacent pixels in the background region meets the target condition (the target condition is a similarity of 90%), then the identifier value of the pixel in the edge region is the same as the identifier value of its adjacent pixels in the background region, and the identifier value of that pixel in the edge region is determined to be the second identifier value 0.

[0077] S304. Determine the edge contour of the target object in the image to be processed based on the identifier value of the pixel in the edge undetermined region.

[0078] The known subject and background regions are accurate and each has a corresponding identifier value. The identifier value of pixels in the edge region to be determined is generated based on the similarity calculation results between that pixel and its neighboring pixels in the subject and background regions, as well as the identifier values ​​corresponding to the subject and background regions, respectively. Therefore, the closeness of the identifier value determined for a pixel in the edge region to the first and second identifier values ​​can reflect whether the pixel in the edge region actually belongs to the target object or the background. This allows for the accurate determination of the target object's edge portion, and thus the determination of the target object's edge contour in the image to be processed.

[0079] If the proximity of the identifier value of a pixel in the edge undetermined region to the first identifier value is greater than the proximity of the identifier value of the pixel to the second identifier value, then the pixel in the edge undetermined region is determined to actually belong to the target object; otherwise, the pixel in the edge undetermined region is determined to actually belong to the background.

[0080] For example, the first identifier value is 1, the second identifier value is 0, and the edge region to be determined includes three pixels: A, B, and C. Pixel A has an identifier value of 0.9, pixel B has an identifier value of 0.7, and pixel C has an identifier value of 0.1. Pixel A's identifier value of 0.9 is closer to the first identifier value of 1 than to the second identifier value of 0; therefore, pixel A can be determined to belong to the target object. Similarly, pixel B's identifier value of 0.7 is closer to the first identifier value of 1 than to the second identifier value of 0; therefore, pixel B can be determined to belong to the target object. Pixel C's identifier value of 0.1 is less close to the first identifier value of 1 than to the second identifier value of 0; therefore, pixel C can be determined to belong to the background.

[0081] After determining the edge contour of the target object in the image to be processed, the original background can be removed based on the edge contour. The image with the determined edge contour is then overlaid with the new background required by the user, thereby generating an image that meets the user's needs, such as generating an ID photo with a new background. The process of generating an ID photo based on the determined edge contour can be found in [link to documentation]. Figure 6a As shown, 601 is the image to be processed, 602 is the image with the edge contour determined, i.e., the Alpha Mask, 603 is the new background actually needed by the user, and 604 is the final generated ID photo.

[0082] As can be seen from the above technical solution, for an image to be processed that includes a target object, the classification labels obtained from semantic segmentation of the image can determine the main body region, edge undetermined region, and background region corresponding to the target object in the image. The identifier value of the main body region is set as the first identifier value, and the identifier value of the background region is set as the second identifier value. Based on the pixels adjacent to the edge undetermined region in the main body region and background region, the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels is calculated. Based on the similarity calculation result of the pixels in the edge undetermined region and their adjacent pixels, as well as the first and second identifier values, the identifier value of the pixels in the edge undetermined region is determined. Since the pixel features of pixels belonging to the target object are more similar to those of pixels in the background region, the similarity calculation result determined for any pixel in the edge undetermined region can reflect the probability that the pixel belongs to the target object or the background. Since different identifier values ​​are set for pixels in the background area and the main body area, the degree of similarity between the identifier value determined for the pixel in the edge undetermined area and the first identifier value and the second identifier value can determine whether the pixel belongs to the target object. This can accurately determine the edge part of the target object, improve the determination accuracy of the target object area in the image to be processed, and thus be applicable to high-precision image background removal scenarios.

[0083] Next, we will introduce in detail the method of generating intermediate processing images in S301.

[0084] See Figure 6b , Figure 6b A flowchart of a method for determining an intermediate processing image corresponding to an image to be processed is shown, the method comprising:

[0085] S601. Determine the region to be determined in the image to be processed, corresponding to the target object, based on the classification label.

[0086] Category labels indicate the probability that a pixel belongs to a certain category of object. That is, based on the category labels, we can identify whether a pixel in the image to be processed belongs to a person, bicycle, chair, etc. If the target object is a person, then the area containing the person is the undefined region for the target object.

[0087] S602. By performing binarization processing on the image to be processed, a first image is obtained.

[0088] The image to be processed may include a region to be determined and other regions outside the region. To distinguish between these two regions, the image to be processed can be binarized by setting different pixel values ​​for the pixels in the region to be determined and the pixels outside the region in the first image. For example, binarization can be used to set the pixel values ​​of the pixels in the image to 0 or 255, thus obtaining the first image, which is a black and white image.

[0089] In some cases, the pixel values ​​of pixels in the region to be determined can be set to 255, and the pixel values ​​of pixels outside the region to be determined can be set to 0. See the first image obtained. Figure 7 As shown, the white area is the undetermined area 701, and the black area is the area 702 outside the undetermined area.

[0090] S603. A second image is obtained by blurring the pixel values ​​of the edge portion of the region to be determined in the first image.

[0091] The blurring process can be Gaussian blurring or other blurring methods, and this embodiment does not limit it. By blurring the pixel values, the pixel values ​​of pixels near the edge of the region to be determined can show a gradual change. For example, the pixel value of the pixels in the region to be determined is a first value, and the pixel value of the pixels outside the region to be determined (e.g., non-region to be determined) is a second value. Then, the pixel value of the pixels in the blurred region (near the edge of the region to be determined) is between the first value and the second value. Moreover, the closer the pixels are to the region to be determined, the closer their pixel values ​​are to the first value, and the closer the pixels are to the region to non-region to be determined, the closer their pixel values ​​are to the second value.

[0092] The second image obtained after blurring can be seen in [reference]. Figure 8 As shown, at this point, the pixel values ​​of the pixels in the second image are no longer just two values, but may also include other pixel values. If the pixel value of the pixel in the region to be determined is 255, Figure 8 The white area represents region 801, which contains pixel value 255. Region 801 is the location of the main target object, equivalent to... Figure 7 The undetermined region is 701; pixels outside the undetermined region have a pixel value of 0. Figure 8 The black area in the middle represents region 803, where pixels with a value of 0 are located. Region 803 is the location of the background, which is equivalent to... Figure 7 The undetermined region 702 in the non-central region; the pixel values ​​of the pixels in the blurred second image may also include other pixel values ​​between 0 and 255, causing the pixel values ​​of pixels near the edge of the undetermined region to show a gradual change. Figure 8The gradient near the edge of the undetermined region represents the area containing pixels with pixel values ​​between 0 and 255 (the blurred area) 802.

[0093] After obtaining the second image, a second pixel region can be determined within the first pixel region that has undergone pixel value blurring, based on the pixel value distribution of the pixels in the second image. The first pixel region is the blurred region 802, for example, the region containing pixels with pixel values ​​between 0 and 255; the second pixel region is the region containing pixels whose pixel values ​​meet certain conditions, for example, the region containing pixels with pixel values ​​greater than 10 and less than 245 is determined as the second pixel region. See [link to documentation for the second pixel region]. Figure 9 As shown, the white area is the second pixel area 902, and the black area includes areas 901 and 903. Area 901 is the location of the main target object, equivalent to... Figure 8 The middle area is 801. Area 903 is the background location, equivalent to... Figure 8 Central area 803.

[0094] S604. Determine the intermediate processing image based on the pixel value distribution of the pixels in the second image.

[0095] It should be noted that if a second pixel region is determined in the first pixel region after pixel value blurring, one possible implementation of S604 is to determine the intermediate processed image based on the pixel value distribution of the pixels in the second image and the second pixel region.

[0096] It should be noted that the second pixel region determined from the first pixel region after pixel value blurring may be relatively narrow. This could lead to some edge pixels that actually belong to the target object but are outside the second pixel region being directly identified as background. In this case, to ensure that the second pixel region covers as many edge pixels as possible, dilation can be applied to the second pixel region before determining the intermediate processed image based on the pixel value distribution in the second image and the second pixel region.

[0097] Dilation refers to expanding the original pixel area (e.g., the second pixel area) outwards, resulting in a larger pixel area than the original. See [example image for dilated second pixel area]. Figure 10 As shown, the white area is the second pixel area 1002 after dilation, and the black areas include areas 1001 and 1003. Area 1001 is the location of the main target object, equivalent to... Figure 9 The middle area is 901. Area 1003 is the background location, equivalent to... Figure 9The middle region is 903. The second pixel region after dilation is 1002. Figure 9 The second pixel region 902 is larger.

[0098] In the case of dilating the second pixel region, in order to avoid the second pixel region being too wide and causing excessive computation, the intermediate processing image is determined based on the pixel value distribution of the pixels in the second image and the second pixel region. For example, the pixel value distribution of the second image can be mixed with the second pixel region after dilation, and the intermediate processing image can be determined based on the mixed pixel value distribution.

[0099] Next, we will introduce how to determine the identifier value of pixels in the edge-to-determined region in S303.

[0100] If the identifier value is the alpha value of a pixel in the region to be determined on the edge, then the alpha value of each determined pixel must satisfy the following formula: I = αF + (1-α)B

[0101] Where I is the pixel value of a pixel in the image to be processed, F is the pixel value of a pixel in the main region, B is the pixel value of a pixel in the background region, and α is the alpha (identifier value) of a pixel.

[0102] Calculating alpha is equivalent to solving the formula I = αF + (1-α)B.

[0103] The original formula I = αF + (1-α)B can be transformed into solving a matrix equation, as shown below:

[0104]

[0105] Where, similarity matrix A ij The diagonal matrix D can be obtained through the method described in the foregoing embodiments. ii =∑ j A ij L=D ii -A ij , m is the first identifier of the main area or the second identifier of the background area, and λ is the constraint parameter.

[0106] Since A can be calculated through the aforementioned similarity calculation steps... ij According to formula D ii =∑ j A ij D can be calculated ii (D), according to the formula L=D ii -A ij Furthermore, if L, m, and λ are known (for example, they can be set to 100), then the equation can be solved. The alpha value (α) of each pixel.

[0107] It is understandable that in the scenario of generating a personal ID photo based on a single photo, since the personal ID photo is required to be an image showing the front of the face, the image to be processed in S301 can be an image showing the front of the face.

[0108] To avoid obtaining an invalid ID photo due to an image that does not meet the requirements, before executing S301, it is possible to check whether the image to be processed is compliant (e.g., whether it is an image of a frontal face). If it is not compliant, a corresponding prompt will pop up, such as prompting the user to "upload a frontal face photo".

[0109] Specifically, during the image acquisition process, face detection is first used to check if the image contains a face. If there is no face or the face exceeds a bounding box, an error is returned, prompting the user to take a separate frontal photo. Then, face orientation detection and a predefined orientation threshold are used to determine if the face in the image meets the requirement of a frontal orientation for ID photos. If not, an error is returned, and the user is prompted to take a separate frontal photo. Generally, if face orientation detection and the predefined orientation threshold determine that the x / y / z axis orientation angles of the face do not exceed 5 degrees, the image is considered to meet the requirement of a frontal orientation for ID photos.

[0110] It should be noted that ID photos may have certain requirements regarding the proportion and position of the face within the photo. In such cases, the face frame of the original photo can be centered and expanded proportionally to generate a cropped image that conforms to the proportion and position of the face in the ID photo. This cropped image can then be used as the image to be processed.

[0111] Next, the method provided in this application embodiment will be described in conjunction with a practical application scenario. In this application scenario, a user wants to remove the background from their personal photo to generate a blue background ID photo. Specifically, an application on the terminal device has an ID photo generation function, which uses the method provided in this application embodiment. For example, the ID photo generation function can be implemented through social software, such as WeChat, WeChat Work, QQ, etc.

[0112] In this embodiment, the generation of ID photos using social media software will be used as an example. See [link to documentation]. Figure 11 The method includes:

[0113] S1101, The user opens a social media app.

[0114] S1102. Users select original images from their albums via the image selection entry in social media software.

[0115] The original image selected can be found here. Figure 12 As shown in 1201.

[0116] S1103. Social media apps check whether the original photos are compliant. If compliant, proceed to S1104.

[0117] S1104. Social media software crops the original photo according to the proportion requirements of the portrait to obtain the image to be processed.

[0118] The target object in the image to be processed is a user's portrait.

[0119] S1105. The social software determines the edge contour corresponding to the target object based on the image to be processed, and removes the original background of the image to be processed based on the edge contour.

[0120] The method by which social software determines the edge contour of a target object based on the image to be processed can be found in [reference needed]. Figure 3 The description of the corresponding embodiments will not be repeated here.

[0121] S1106. Social media apps overlay backgrounds onto ID photos to generate ID photos.

[0122] See the screenshot of the interface for generating ID photos. Figure 12 As shown in Figure 1202.

[0123] It should be noted that the background color value used when overlaying ID photos can be the default background color value. If the default background color value does not meet the user's needs, the user can choose a color value. For example, the user can choose white, blue, red, etc. in the interface shown in 1202.

[0124] S1107. Users can select different beautification parameters to adjust the generated ID photo.

[0125] S1108. Output the adjusted ID photo.

[0126] See Figure 12 As shown in Figure 1203, the adjusted ID photo can be found here. Figure 12 As shown in Figure 1204.

[0127] S1109. Users save or share the adjusted ID photo through social media.

[0128] S1103-S1105 can be executed by social software (client) or by server; this embodiment does not limit this. S1106-S1107 can be executed by social software.

[0129] Based on the image processing method provided in the foregoing embodiments, this embodiment provides an image processing apparatus, see [link to previous embodiment]. Figure 13a The device includes a first determining unit 1301, a calculation unit 1302, a second determining unit 1303, and a third determining unit 1304.

[0130] The first determining unit 1301 is used to determine the main body region, edge undetermined region, and background region of the target object in the image to be processed based on the classification labels of the pixels obtained by semantic segmentation of the image to be processed; the identifier value of the main body region is set to a first identifier value, and the identifier value of the background region is set to a second identifier value;

[0131] The calculation unit 1302 is used to calculate the pixel feature similarity between adjacent pixels in the undetermined edge region based on the pixels in the main body region and the background region that are adjacent to the undetermined edge region.

[0132] The second determining unit 1303 is used to determine the identifier value of the pixel in the edge-to-edge region based on the similarity calculation result of the pixel in the edge-to-edge region, as well as the first identifier value and the second identifier value.

[0133] The third determining unit 1304 is used to determine the edge contour of the target object in the image to be processed based on the identification value of the pixel in the edge undetermined region.

[0134] In one possible implementation, during the process of determining the identifier value of a pixel in the edge region to be determined, the proximity of the identifier values ​​of adjacent pixels is related to the similarity of the pixel features of the adjacent pixels.

[0135] In one possible implementation, if the similarity of pixel features of adjacent pixels meets the target condition, the identifier values ​​of adjacent pixels are the same.

[0136] In one possible implementation, the pixel features include color information and position information.

[0137] In one possible implementation, the first determining unit 1301 is configured to:

[0138] Based on the classification labels of the pixels obtained by semantic segmentation of the image to be processed, an intermediate processing image corresponding to the image to be processed is determined. The intermediate processing image is used to identify the main body region, edge undetermined region, and background region of the target object in the image to be processed.

[0139] In one possible implementation, the first determining unit 1301 is configured to:

[0140] Based on the classification labels, determine the undefined region of the corresponding target object in the image to be processed;

[0141] By binarizing the image to be processed, a first image is obtained; the pixel values ​​of the pixels in the region to be determined in the first image are different from the pixel values ​​of the pixels outside the region to be determined.

[0142] The second image is obtained by blurring the pixel values ​​of the edge portion of the region to be determined in the first image;

[0143] The intermediate processed image is determined based on the pixel value distribution of the pixels in the second image.

[0144] In one possible implementation, the first determining unit 1301 is further configured to:

[0145] Based on the pixel value distribution of pixels in the second image, a second pixel region is determined in the first pixel region after the pixel value blurring process.

[0146] The intermediate processing image is determined based on the pixel value distribution of the pixels in the second image and the second pixel region.

[0147] In one possible implementation, the first determining unit 1301 is further configured to:

[0148] The second pixel region is then dilated.

[0149] As can be seen from the above technical solution, for an image to be processed that includes a target object, the classification labels obtained from semantic segmentation of the image can determine the main body region, edge undetermined region, and background region corresponding to the target object in the image. The identifier value of the main body region is set as the first identifier value, and the identifier value of the background region is set as the second identifier value. Based on the pixels adjacent to the edge undetermined region in the main body region and background region, the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels is calculated. Based on the similarity calculation result of the pixels in the edge undetermined region and their adjacent pixels, as well as the first and second identifier values, the identifier value of the pixels in the edge undetermined region is determined. Since the pixel features of pixels belonging to the target object are more similar to those of pixels in the background region, the similarity calculation result determined for any pixel in the edge undetermined region can reflect the probability that the pixel belongs to the target object or the background. Since different identifier values ​​are set for pixels in the background area and the main body area, the degree of similarity between the identifier value determined for the pixel in the edge undetermined area and the first identifier value and the second identifier value can determine whether the pixel belongs to the target object. This can accurately determine the edge part of the target object, improve the determination accuracy of the target object area in the image to be processed, and thus be applicable to high-precision image background removal scenarios.

[0150] This application also provides an image processing device, which will be described below with reference to the accompanying drawings. Please refer to... Figure 13b As shown, this application embodiment provides an image processing device 1300, which can be a terminal device. This terminal device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0151] Figure 13b This diagram illustrates a partial structural design of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 13b The mobile phone includes components such as a radio frequency (RF) circuit 1310, a memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a Wi-Fi module 1370, a processor 1380, and a power supply 1390. Those skilled in the art will understand that... Figure 13b The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0152] The following is combined Figure 13b A detailed introduction to each component of a mobile phone:

[0153] The RF circuit 1310 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1380; additionally, it transmits uplink data to the base station. The memory 1320 can be used to store software programs and modules. The processor 1380 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1320. The memory 1320 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, phonebook, etc.).

[0154] The input unit 1330 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332.

[0155] The display unit 1340 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1340 may include a display panel 1341. The mobile phone may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors.

[0156] The audio circuit 1360, speaker 1361, and microphone 1362 provide an audio interface between the user and the mobile phone.

[0157] The processor 1380 is the control center of the mobile phone. It connects various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1320 and calling data stored in the memory 1320.

[0158] In this embodiment, the processor 1380 included in the terminal device also has the following functions:

[0159] Based on the classification labels of pixels obtained through semantic segmentation of the image to be processed, the main body region, edge undetermined region, and background region corresponding to the target object in the image to be processed are determined; the identifier value of the main body region is set as a first identifier value, and the identifier value of the background region is set as a second identifier value;

[0160] Based on the pixels in the main region and the background region that are adjacent to the edge undetermined region, calculate the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels;

[0161] Based on the similarity calculation results of the pixels in the edge undetermined region, and the first and second identifier values, the identifier values ​​of the pixels in the edge undetermined region are determined;

[0162] The edge contour of the target object in the image to be processed is determined based on the identifier value of the pixel in the edge undetermined region.

[0163] The image processing device provided in this application embodiment can be a server; please refer to [link / reference]. Figure 14 As shown, Figure 14The diagram illustrates the structure of a server 1400 provided in this embodiment. The server 1400 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 1422 (e.g., one or more processors) and a memory 1432, and one or more storage media 1430 (e.g., one or more mass storage devices) for storing application programs 1442 or data 1444. The memory 1432 and storage media 1430 may be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations stored in the storage media 1430 on the server 1400.

[0164] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0165] The steps performed by the server in the above embodiments can be based on this Figure 14 The server structure shown.

[0166] CPU 1422 is used to perform the following steps:

[0167] Based on the classification labels of pixels obtained through semantic segmentation of the image to be processed, the main body region, edge undetermined region, and background region corresponding to the target object in the image to be processed are determined; the identifier value of the main body region is set as a first identifier value, and the identifier value of the background region is set as a second identifier value;

[0168] Based on the pixels in the main region and the background region that are adjacent to the edge undetermined region, calculate the pixel feature similarity between the pixels in the edge undetermined region and their adjacent pixels;

[0169] Based on the similarity calculation results of the pixels in the edge undetermined region, and the first and second identifier values, the identifier values ​​of the pixels in the edge undetermined region are determined;

[0170] The edge contour of the target object in the image to be processed is determined based on the identifier value of the pixel in the edge undetermined region.

[0171] This application provides a computer-readable storage medium for storing program code for executing the image processing method described in the foregoing embodiments.

[0172] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises: determining a to-be-determined region of a target object in the to-be-processed image according to a classification label of a pixel point obtained by semantic segmentation on the to-be-processed image; obtaining a first image by performing binaryzation processing on the to-be-processed image; pixel values of pixel points in the to-be-determined region in the first image are different from pixel values of pixel points outside the to-be-determined region; obtaining a second image by performing pixel value blurring processing on an edge part of the to-be-determined region in the first image; determining a second pixel region in the first pixel region on which the pixel value blurring processing is performed according to a pixel value distribution of pixel points in the second image; determining an intermediate processing image according to the pixel value distribution of pixel points in the second image and the second pixel region; the intermediate processing image is used to identify a main body region, an edge to-be-determined region and a background region of the target object in the to-be-processed image; the main body region is a region in the to-be-processed image where a main body of the target object except for an edge part is located; the background region is a region in the to-be-processed image where a background relative to the target object needs to be removed is located; the edge to-be-determined region is a region in the to-be-processed image where the edge part corresponding to the target object is located; an identification value of the main body region is set as a first identification value; an identification value of the background region is set as a second identification value; the identification values are represented by transparency; the first identification value is represented by a maximum transparency value 1; and the second identification value is represented by a minimum transparency value 0; calculating a pixel feature similarity between pixel points in the edge to-be-determined region and adjacent pixel points according to pixel points in the main body region and the background region adjacent to the edge to-be-determined region; According to the similarity calculation result of the pixel points in the edge undetermined region, and the first and second identification values, the identification value of the pixel points in the edge undetermined region is determined by using a formula , wherein, in the process of determining the identification value of the pixel points in the edge undetermined region, the proximity degree of the identification values of adjacent pixel points is related to the similarity degree of the pixel features of the adjacent pixel points, and if the similarity degree of the pixel features of the adjacent pixel points meets a target condition, the identification values of the adjacent pixel points are the same, , is a diagonal matrix, , is a similarity matrix used for representing the similarity calculation result of the pixel points in the edge undetermined region, is the first identification of the subject region or the second identification of the background region, is a constraint parameter, D is a diagonal matrix, is the identification value of the pixel points in the edge undetermined region. determining an edge contour of the target object in the to-be-processed image according to an identification value of a pixel point in the edge to-be-determined region.

2. The method of claim 1, wherein, The pixel feature comprises color information and position information.

3. The method of claim 1, wherein, Before the intermediate processing image is determined according to the pixel value distribution of pixel points in the second image and the second pixel region, the method further comprises: performing inflation processing on the second pixel region.

4. An image processing apparatus characterized by comprising: The device comprises a first determining unit, a calculating unit, a second determining unit and a third determining unit: the first determining unit is configured to determine a to-be-determined region of a target object in a to-be-processed image according to a classification label of a pixel point obtained by semantic segmentation on the to-be-processed image; and obtain a first image by performing binaryzation processing on the to-be-processed image; pixel values of pixel points in the to-be-determined region in the first image are different from pixel values of pixel points outside the to-be-determined region; and obtain a second image by performing pixel value blurring processing on an edge part of the to-be-determined region in the first image; determine a second pixel region in the first pixel region on which the pixel value blurring processing is performed according to a pixel value distribution of pixel points in the second image; According to a pixel value distribution of the pixels in the second image and the second pixel region, an intermediate processing image is determined, the intermediate processing image is used to identify a main body region, an edge undetermined region and a background region of the target object in the image to be processed, the main body region is a region of the main body of the target object in the image to be processed except for the edge part, the background region is a region of the background to be removed relative to the target object in the image to be processed, and the edge undetermined region is a region of the edge part of the target object in the image to be processed, an identification value of the main body region is set as a first identification value, an identification value of the background region is set as a second identification value, the identification value is represented by transparency, the first identification value is represented by a maximum transparency value 1, and the second identification value is represented by a minimum transparency value 0; The computing unit is configured to calculate pixel feature similarity between adjacent pixels in the edge undetermined region according to the pixels adjacent to the edge undetermined region in the main body region and the background region. The second determining unit is configured to determine the identification value of the pixel point in the edge undetermined region according to the similarity calculation result of the pixel point in the edge undetermined region and the first identification value and the second identification value, by using a formula , wherein, in the process of determining the identification value of the pixel point in the edge undetermined region, the proximity degree of the identification values of adjacent pixel points is related to the similarity degree of the pixel features of the adjacent pixel points, and if the similarity degree of the pixel features of the adjacent pixel points meets a target condition, the identification values of the adjacent pixel points are the same, , is a diagonal matrix, , is a similarity matrix used for representing the similarity calculation result of the pixel point in the edge undetermined region, is the first identification of the subject region or the second identification of the background region, is a constraint parameter, D is a diagonal matrix, is the identification value of the pixel point in the edge undetermined region. The third determining unit is configured to determine an edge contour of the target object in the image to be processed according to the identification values of the pixels in the edge undetermined region.

5. The apparatus of claim 4, wherein, The first determining unit is further configured to: perform inflation processing on the second pixel region.

6. An apparatus for image processing, characterized by The device includes a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the image processing method according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is used to execute the image processing method according to any one of claims 1-3. The computer readable storage medium is configured to store program code, and the program code is used to execute the image processing method according to any one of claims 1-3.

Images