Image segmentation method and device

Abstract

The present application discloses an image segmentation method and apparatus. The method comprises: losslessly downsampling a segmentation map to obtain a first sub-map with a P channel; losslessly downsampling a coordinate map to obtain a second sub-map with a Q channel; the coordinate map and the segmentation map have the same resolution, and the coordinate map is used to indicate the position information of each pixel in the segmentation map; performing a first convolution operation on the first and second sub-maps to obtain a third sub-map with a K channel; K is less than the sum of P and Q; determining a first sub-mask map with a K channel corresponding to the third sub-map through an image segmentation network; performing a second convolution operation on the first sub-mask map to obtain a second sub-mask map with an L channel; L is greater than K; concatenating the second sub-mask map into a mask map in a lossless manner; the mask map and the segmentation map have the same resolution; and performing image segmentation on the segmentation map using the mask map. This method can accurately segment ultra-high-definition images under low device performance conditions.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image segmentation method and device. Background Art

[0002] In the field of image processing technology, the ability to accurately segment the foreground and background of an image is extremely necessary in some scenarios. For example, in high-definition video calls, segmenting a person's image from the background and then blurring or replacing the segmented image or background effectively protects user privacy.

[0003] Currently, when segmenting ultra-high-definition (4K) images, the image can be directly fed into a deep neural network for image segmentation. This method uses ultra-high-definition images, which have a resolution of 3840 x 2160 pixels. This large image data requires both tremendous image processing power and storage capacity when fed into a deep neural network, placing high demands on graphics cards.

[0004] To address this issue, pooling operations can be used to significantly downsample the UHD image to be segmented. Pooling-based downsampling operations, such as average pooling and max pooling, retain the average or maximum value within a neighborhood while removing other values, thereby extracting features and reducing computational complexity. However, this approach inevitably loses detailed information in the UHD image, resulting in less refined image segmentation results, such as unclear foreground and background edges.

[0005] In summary, there is an urgent need for an image segmentation method that does not have very high requirements on graphics cards and can accurately segment ultra-high-definition images. Summary of the Invention

[0006] The present application provides an image segmentation method and apparatus for accurately segmenting ultra-high-definition images under conditions of low device performance.

[0007] In a first aspect, an embodiment of the present application provides an image segmentation method, which includes: downsampling the segmentation map in a lossless manner to obtain a first sub-map with a P channel; downsampling the coordinate map in the lossless manner to obtain a second sub-map with a Q channel; the coordinate map has the same resolution as the segmentation map, and the coordinate map is used to indicate the position information of each pixel in the segmentation map; performing a first convolution operation on the first sub-map and the second sub-map to obtain a third sub-map with a K channel; K is less than the sum of P and Q; determining, through an image segmentation network, a first sub-mask map with a K channel corresponding to the third sub-map; performing a second convolution operation on the first sub-mask map to obtain a second sub-mask map with an L channel; L is greater than K; splicing the second sub-mask map into a mask map in the lossless manner; the mask map has the same resolution as the segmentation map; and performing image segmentation on the segmentation map through the mask map.

[0008] Based on this solution, for the segmentation map, by constructing a corresponding coordinate map and losslessly downsampling the two respectively, and for the results after lossless downsampling - the first sub-map with P channel and the second sub-map with Q channel, a first convolution operation can be performed, which fuses each pixel on the segmentation map with the position information of the pixel, so that in the process of image segmentation, no detail information of the segmentation map is lost, ensuring the smoothness of the foreground and background edges of the image during segmentation; in addition, compared to directly inputting the segmentation map into the image segmentation network, the third sub-map with K channel obtained by the first convolution operation of this application will occupy a relatively small amount of data processing resources of the image segmentation network. In this way, the image segmentation network of this application does not have high requirements for device performance, that is, this application can support image segmentation on a device with low graphics card performance. In this way, it is possible to accurately segment ultra-high-definition images under low device performance conditions.

[0009] In one possible implementation method, the segmentation map is downsampled in a lossless manner to obtain a first sub-map with P channels, including: determining each sampling neighborhood in the segmentation map according to a preset sampling step size; each sampling neighborhood has P pixels at different positions; and extracting pixels at the same position in each sampling neighborhood, and combining the extracted pixels into the first sub-map with P channels in a preset combination order.

[0010] Based on this scheme, in the process of segmenting the segmentation map, in order to improve the smoothness of the edges of the foreground and background, the segmentation map can be downsampled in a lossless manner, that is, every pixel in the segmentation map needs to be sampled without omission. To this end, the various sampling neighborhoods in the segmentation map can be determined according to a preset sampling step size, wherein each sampling neighborhood includes P pixels at different positions; then, the pixels at the same position in each sampling neighborhood are extracted, and for each pixel at the same position in any group, the pixels can be arranged according to a preset combination order, so that for each pixel at the same position in each group, the result of the arrangement can be called 1 channel; however, since each sampling neighborhood includes P pixels at different positions, the result of extracting the pixels at the same position in each sampling neighborhood of the segmentation map and arranging them according to the preset combination order can be called the first sub-image with P channels.

[0011] In one possible implementation method, the step of stitching the second sub-mask images into a mask image in the lossless manner includes: for any channel in the second sub-mask image, extracting the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel into corresponding sampling neighborhoods in the segmentation image, thereby obtaining the mask image.

[0012] Based on this solution, when segmenting the foreground and background of an image, it can be done based on a mask image, wherein the resolution of the mask image needs to be consistent with the resolution of the segmentation image. To this end, for any channel in the second sub-mask image, the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel can be extracted into the corresponding sampling neighborhoods in the segmentation image, thereby obtaining a mask image for segmenting the segmentation image. This process of obtaining a mask image for segmenting the segmentation image can be understood as an inverse process of the process of lossless downsampling of the segmentation image. This method is based on the fact that the sampling method of the segmentation image is lossless, that is, the sampling process covers any pixel point in the segmentation image. Therefore, when the segmentation image is segmented for foreground and background according to the mask image obtained in this way, the smoothness of the edges of the obtained foreground and background will be greatly improved.

[0013] In one possible implementation method, the first convolution operation has K 1*1*(P+Q) first convolution kernels; the first convolution operation is performed on the first sub-image and the second sub-image to obtain a third sub-image with K channels, including: for any first convolution kernel, performing a convolution operation on the first sub-image and the second sub-image through the first convolution kernel to obtain a single-channel third sub-image.

[0014] Based on this scheme, the first sub-image with P channels and the second sub-image with Q channels obtained by losslessly downsampling the segmentation map and the coordinate map can be spliced ​​along the channel direction. Therefore, the position information of each pixel point lost when the segmentation map is losslessly downsampled can be compensated by losslessly downsampling the coordinate map with the same resolution as the segmentation map. This is mainly compensated by splicing the first sub-image and the second sub-image after lossless downsampling along the channel direction. Since the number of channels of the feature map generated by splicing is large, it is not friendly to use the image segmentation network to perform further feature extraction on the spliced ​​feature map, which requires a large amount of data processing power and data storage capacity. To this end, the feature map generated by splicing the first sub-image and the second sub-image along the channel direction can be reduced in terms of the number of channels, for example, reduced to K channels. Then, the feature map with fewer channels, such as the third sub-image with K channels, can be input into the image segmentation network, and the image segmentation network will perform further feature extraction on it. Among them, when performing the channel reduction operation on the feature map generated by splicing the first sub-image and the second sub-image along the channel direction, K 1*1*(P+Q) first convolution kernels can be used. Specifically, for the feature map generated by splicing the first sub-image and the second sub-image along the channel direction, using any one of the K 1*1*(P+Q) first convolution kernels can generate a single-channel third sub-image. Since there are K 1*1*(P+Q) first convolution kernels, a third sub-image with K channels can be finally generated.

[0015] In one possible implementation method, the second convolution operation has L 1*1*K second convolution kernels; performing the second convolution operation on the first sub-mask image to obtain a second sub-mask image with L channels includes: for any second convolution kernel, performing a convolution operation on the first sub-mask image through the second convolution kernel to obtain a single-channel second sub-mask image.

[0016] Based on this scheme, when downsampling the segmentation map using a lossless method, a first sub-image with P channels can be obtained. When the segmentation map is a grayscale image, the first sub-image includes P low-resolution images, and when the segmentation map is an RGB color image, the first sub-image includes P / 3 low-resolution images. Since when using a mask image to segment the segmentation map, the mask image must be obtained based on the same number of sub-mask images as the low-resolution image generated after lossless downsampling of the segmentation map, the number of channels of the third sub-image entering the image segmentation network may not be the same as the number of channels in the low-resolution image. Therefore, before generating the mask image, it is necessary to restore the channels of the first sub-mask image with K channels, the result of the image segmentation network, to restore the channels to the number of channels in the low-resolution image. When performing a channel recovery operation on the first sub-mask image with K channels, L 1*1*K second convolution kernels can be used. Specifically, for the first sub-mask image with K channels, using any 1*1*K second convolution kernel among the L 1*1*K first convolution kernels can generate a single-channel second sub-mask image. Since there are L 1*1*K second convolution kernels, a second sub-mask image with L channels can be ultimately generated. For example, when the segmentation image is a grayscale image, L=P, and when the segmentation image is an RGB color image, L=P / 3.

[0017] In one possible implementation method, the first convolution operation, the second convolution operation, and the image segmentation network are trained through a unified loss function.

[0018] Based on this solution, since the segmentation of the foreground and background of the segmentation map involves the first convolution operation, the second convolution operation and the image segmentation network, this application uses a unified loss function to achieve simultaneous training of the first convolution operation, the second convolution operation and the image segmentation network, thereby improving the efficiency of image segmentation.

[0019] In one possible implementation method, the formula corresponding to the loss function is:

[0020]

[0021] Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask image under the i-th channel, P represents the number of channels of the second sub-mask image, and λ represents a preset coefficient.

[0022] Based on this scheme, since the loss function used to train the first convolution operation, the second convolution operation and the image segmentation network takes into account the loss of the mask image and the loss of the second sub-mask image in the determination process, the determined loss function can more comprehensively characterize the difference between the actual segmentation result and the true segmentation result in the image segmentation process. According to the difference, the first convolution operation, the second convolution operation and the image segmentation network can be better trained. In this way, better image segmentation effect can be achieved.

[0023] In a second aspect, an embodiment of the present application provides an image segmentation device, which includes: a lossless downsampling unit for downsampling the segmentation map in a lossless manner to obtain a first sub-image with a P channel; the lossless downsampling unit is also used to downsample the coordinate map in the lossless manner to obtain a second sub-image with a Q channel; the coordinate map has the same resolution as the segmentation map, and the coordinate map is used to indicate the position information of each pixel in the segmentation map; a convolution operation unit is used to perform a first convolution operation on the first sub-image and the second sub-image to obtain a second sub-image with a K channel. a third sub-image of the channel; K is less than the sum of P and Q; a first sub-mask image determining unit, used to determine, through an image segmentation network, a first sub-mask image with a K channel corresponding to the third sub-image; the convolution operation unit, further used to perform a second convolution operation on the first sub-mask image to obtain a second sub-mask image with an L channel; L is greater than K; a mask image generating unit, used to splice the second sub-mask image into a mask image in the lossless manner; the mask image and the segmentation image have the same resolution; an image segmentation unit, used to perform image segmentation on the segmentation image through the mask image.

[0024] Based on this solution, for the segmentation map, by constructing a corresponding coordinate map and losslessly downsampling the two respectively, and for the results after lossless downsampling - the first sub-map with P channel and the second sub-map with Q channel, a first convolution operation can be performed, which fuses each pixel on the segmentation map with the position information of the pixel, so that in the process of image segmentation, no detail information of the segmentation map is lost, ensuring the smoothness of the foreground and background edges of the image during segmentation; in addition, compared to directly inputting the segmentation map into the image segmentation network, the third sub-map with K channel obtained by the first convolution operation of this application will occupy a relatively small amount of data processing resources of the image segmentation network. In this way, the image segmentation network of this application does not have high requirements for device performance, that is, this application can support image segmentation on a device with low graphics card performance. In this way, it is possible to accurately segment ultra-high-definition images under low device performance conditions.

[0025] In one possible implementation method, the lossless downsampling unit is specifically used to: determine each sampling neighborhood in the segmentation image according to a preset sampling step size; each sampling neighborhood has P pixels at different positions; extract pixels at the same position in each sampling neighborhood, and combine the extracted pixels into a first sub-image with P channels in a preset combination order.

[0026] Based on this scheme, in the process of segmenting the segmentation map, in order to improve the smoothness of the edges of the foreground and background, the segmentation map can be downsampled in a lossless manner, that is, every pixel in the segmentation map needs to be sampled without omission. To this end, the various sampling neighborhoods in the segmentation map can be determined according to a preset sampling step size, wherein each sampling neighborhood includes P pixels at different positions; then, the pixels at the same position in each sampling neighborhood are extracted, and for each pixel at the same position in any group, the pixels can be arranged according to a preset combination order, so that for each pixel at the same position in each group, the result of the arrangement can be called 1 channel; however, since each sampling neighborhood includes P pixels at different positions, the result of extracting the pixels at the same position in each sampling neighborhood of the segmentation map and arranging them according to the preset combination order can be called the first sub-image with P channels.

[0027] In one possible implementation method, the mask image generation unit is specifically used to: for any channel in the second sub-mask image, extract the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel into the corresponding sampling neighborhoods in the segmentation image, thereby obtaining the mask image.

[0028] Based on this solution, when segmenting the foreground and background of an image, it can be done based on a mask image, wherein the resolution of the mask image needs to be consistent with the resolution of the segmentation image. To this end, for any channel in the second sub-mask image, the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel can be extracted into the corresponding sampling neighborhoods in the segmentation image, thereby obtaining a mask image for segmenting the segmentation image. This process of obtaining a mask image for segmenting the segmentation image can be understood as an inverse process of the process of lossless downsampling of the segmentation image. This method is based on the fact that the sampling method of the segmentation image is lossless, that is, the sampling process covers any pixel point in the segmentation image. Therefore, when the segmentation image is segmented for foreground and background according to the mask image obtained in this way, the smoothness of the edges of the obtained foreground and background will be greatly improved.

[0029] In one possible implementation method, the first convolution operation has K 1*1*(P+Q) first convolution kernels; the convolution operation unit is specifically used to: for any first convolution kernel, perform a convolution operation on the first sub-image and the second sub-image through the first convolution kernel to obtain a single-channel third sub-image.

[0030] Based on this scheme, the first sub-image with P channels and the second sub-image with Q channels obtained by losslessly downsampling the segmentation map and the coordinate map can be spliced ​​along the channel direction. Therefore, the position information of each pixel point lost when the segmentation map is losslessly downsampled can be compensated by losslessly downsampling the coordinate map with the same resolution as the segmentation map. This is mainly compensated by splicing the first sub-image and the second sub-image after lossless downsampling along the channel direction. Since the number of channels of the feature map generated by splicing is large, it is not friendly to use the image segmentation network to perform further feature extraction on the spliced ​​feature map, which requires a large amount of data processing power and data storage capacity. To this end, the feature map generated by splicing the first sub-image and the second sub-image along the channel direction can be reduced in terms of the number of channels, for example, reduced to K channels. Then, the feature map with fewer channels, such as the third sub-image with K channels, can be input into the image segmentation network, and the image segmentation network will perform further feature extraction on it. Among them, when performing the channel reduction operation on the feature map generated by splicing the first sub-image and the second sub-image along the channel direction, K 1*1*(P+Q) first convolution kernels can be used. Specifically, for the feature map generated by splicing the first sub-image and the second sub-image along the channel direction, using any one of the K 1*1*(P+Q) first convolution kernels can generate a single-channel third sub-image. Since there are K 1*1*(P+Q) first convolution kernels, a third sub-image with K channels can be finally generated.

[0031] In one possible implementation method, the second convolution operation has L 1*1*K second convolution kernels; the convolution operation unit is specifically used to: for any second convolution kernel, perform a convolution operation on the first sub-mask image through the second convolution kernel to obtain a single-channel second sub-mask image.

[0032] Based on this scheme, when downsampling the segmentation map using a lossless method, a first sub-image with P channels can be obtained. When the segmentation map is a grayscale image, the first sub-image includes P low-resolution images, and when the segmentation map is an RGB color image, the first sub-image includes P / 3 low-resolution images. Since when using a mask image to segment the segmentation map, the mask image must be obtained based on the same number of sub-mask images as the low-resolution image generated after lossless downsampling of the segmentation map, the number of channels of the third sub-image entering the image segmentation network may not be the same as the number of channels in the low-resolution image. Therefore, before generating the mask image, it is necessary to restore the channels of the first sub-mask image with K channels, the result of the image segmentation network, to restore the channels to the number of channels in the low-resolution image. When performing a channel recovery operation on the first sub-mask image with K channels, L 1*1*K second convolution kernels can be used. Specifically, for the first sub-mask image with K channels, using any 1*1*K second convolution kernel among the L 1*1*K first convolution kernels can generate a single-channel second sub-mask image. Since there are L 1*1*K second convolution kernels, a second sub-mask image with L channels can be ultimately generated. For example, when the segmentation image is a grayscale image, L=P, and when the segmentation image is an RGB color image, L=P / 3.

[0033] In one possible implementation method, the first convolution operation, the second convolution operation, and the image segmentation network are trained through a unified loss function.

[0034] Based on this solution, since the segmentation of the foreground and background of the segmentation map involves the first convolution operation, the second convolution operation and the image segmentation network, this application uses a unified loss function to achieve simultaneous training of the first convolution operation, the second convolution operation and the image segmentation network, thereby improving the efficiency of image segmentation.

[0035] In one possible implementation method, the formula corresponding to the loss function is:

[0036]

[0037] Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask image under the i-th channel, P represents the number of channels of the second sub-mask image, and λ represents a preset coefficient.

[0038] Based on this scheme, since the loss function used to train the first convolution operation, the second convolution operation and the image segmentation network takes into account the loss of the mask image and the loss of the second sub-mask image in the determination process, the determined loss function can more comprehensively characterize the difference between the actual segmentation result and the true segmentation result in the image segmentation process. According to the difference, the first convolution operation, the second convolution operation and the image segmentation network can be better trained. In this way, better image segmentation effect can be achieved.

[0039] In a third aspect, an embodiment of the present application provides a computing device, including:

[0040] memory for storing computer programs;

[0041] The processor is configured to call the computer program stored in the memory and execute the method according to the first aspect according to the obtained program.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 An image segmentation method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of lossless downsampling of a 4*4 grayscale image provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of mask image generation provided in an embodiment of the present application;

[0047] Figure 4 An image segmentation device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0049] At present, accurate segmentation of foreground and background in ultra-high-definition images is of great significance for privacy protection in some scenarios.

[0050] However, if deep neural networks are used directly to segment ultra-high-definition images, very high requirements will be placed on the graphics card of the device; if a pooling operation is used to significantly downsample the ultra-high-definition images to be segmented before using deep neural networks to segment ultra-high-definition images, so as to reduce the amount of calculation, this image segmentation method is prone to lose detailed information of the image to be segmented, resulting in the final image segmentation result being not refined enough, that is, the edges of the foreground and background generated by image segmentation are not smooth enough.

[0051] In order to solve the above technical problems, the present invention provides an image segmentation method. Figure 1 As shown, the method includes the following steps:

[0052] Step 101 : Downsample the segmentation image in a lossless manner to obtain a first sub-image with P channels.

[0053] In this step, the segmentation map is the image that needs to be segmented into foreground and background. Its type can be a single-channel grayscale image or a multi-channel color image, such as an RGB image with three channels of red, green, and blue. This application does not limit the type of segmentation map. In addition, this application does not limit the size of the segmentation map. The segmentation map can be an image of various resolutions. For example, this application will use the segmentation of a 4K RGB image as an example to illustrate.

[0054] In one implementation of this step, the segmentation image is downsampled in a lossless manner to obtain a first subimage with P channels, including: determining each sampling neighborhood in the segmentation image according to a preset sampling step size; each sampling neighborhood has P pixels at different positions; and extracting pixels at the same position in each sampling neighborhood, and combining the extracted pixels into a first subimage with P channels in a preset combination order.

[0055] For example, assume that a 4K RGB image with a resolution of 3840*2160 is downsampled in a lossless manner.

[0056] Before introducing the downsampling of 4K RGB images in a lossless manner, let's first briefly introduce the lossless downsampling. Among them, the key to the implementation of lossless downsampling is: taking a neighborhood as a unit, traversing the segmentation map according to a preset translation method; so that for each neighborhood traversed on the segmentation map, the pixels at the same position in each neighborhood can be taken and combined into a new image. Assuming that the neighborhood used when downsampling the 4K RGB image in a lossless manner has N pixels, the result after downsampling is that N new images can be formed, and the N new images can be spliced ​​along the channel direction to form an image group. Among them, the side length of the neighborhood is equal to the sampling step size, and the sampling step size can be set by those skilled in the art according to actual operational requirements.

[0057] The implementation of lossless downsampling has been explained in the previous paragraph. As a visual example, the following will further illustrate lossless downsampling by performing lossless downsampling on a 4*4 grayscale image.

[0058] It should be noted that in the example process, the reason why the lossless downsampling of a 4K RGB image is not performed directly, but a 4*4 grayscale image is first losslessly downsampled is mainly because the resolution of a 4K RGB image is very large, 3840*2160, which is obviously not conducive to a clear explanation of the implementation process of lossless downsampling in this application. Therefore, this application first uses the lossless downsampling of a 4*4 grayscale image to illustrate the implementation process of lossless downsampling, thereby providing a reference basis for the implementation of lossless downsampling of 4K RGB images.

[0059] like Figure 2 FIG. 1 is a schematic diagram of a method for losslessly downsampling a 4*4 grayscale image provided by an embodiment of the present application. Figure 2 For the 4*4 grayscale image shown in (a), the sampling step size can be set to 2, meaning that every two pixels are sampled once, resulting in a neighborhood size of 2*2. When the 4*4 grayscale image is losslessly downsampled based on this neighborhood, the neighborhood can be matched to the 2*2 regions located in the upper left corner, the upper right corner, the lower left corner, and the lower right corner of the 4*4 grayscale image, respectively.

[0060] The pixels in the same position in these four regions can be extracted to form four new images. For example, the pixels in the same position in these four regions include: four pixels marked as "1", four pixels marked as "2", four pixels marked as "3" and four pixels marked as "4". Then:

[0061] For the four pixels marked as "1", they can be recombined into a new image;

[0062] For the four pixels marked as "2", they can be recombined into a new image;

[0063] For the four pixels marked as "3", they can be recombined into a new image;

[0064] For the four pixels marked as "4", they can be recombined into a new image.

[0065] Therefore, we can finally get 4 downsampled low-resolution images, where the resolution of each low-resolution image is 2*2; by splicing these 4 low-resolution images in the channel direction, we can get a lossless downsampled image group, which is as follows: Figure 2 (b) As shown. Since the four downsampled low-resolution images are also grayscale images, grayscale images are single-channel, therefore, Figure 2 The number of channels of the lossless downsampled image group shown in (b) is 4, which corresponds to the size of the neighborhood of 4 (2*2=4).

[0066] Given that the above has been described in detail about the process of downsampling a 4*4 grayscale image in a lossless manner, the same downsampling method can be used to downsample a 4K (3840*2160) RGB image. It should be noted that since the 4K RGB image has three channels: red, green, and blue, this application will be able to losslessly downsample the 4K RGB image in any color channel. For example, the following example illustrates the downsampling of the 4K RGB image in a lossless manner when the color channel is red:

[0067] When the color channel is red, assuming the sampling step size for losslessly downsampling the 4K RGB image is 16, the neighborhood size is 16*16. When losslessly downsampling the 4K RGB image based on this neighborhood, the neighborhood can match a total of 32,400 regions (3840 / 16=240, 2160 / 16=135, 240*135=32,400) located on the RGB image. After extracting pixels with the same position in these 32,400 regions, 256 new images can be formed, each with a resolution of 240*135. By splicing these 256 new images together in the channel direction, a lossless downsampled image group can be obtained. Since the obtained lossless downsampled image group consisting of 256 new images is based on a single color channel, red, the number of channels of the image group is 256, which corresponds to a neighborhood size of 256 (16*16=256).

[0068] Similarly, for the 4K RGB image, when the color channel is green, the sampling step is 16, and the neighborhood is 16*16, a lossless downsampled image group consisting of 256 new images can also be obtained. The number of channels in this image group is also 256, and the number of channels 256 corresponds to the neighborhood size of 256 (16*16=256).

[0069] Similarly, for the 4K RGB image, when the color channel is blue, the sampling step is 16, and the neighborhood is 16*16, a lossless downsampled image group consisting of 256 new images can also be obtained. The number of channels in this image group is also 256, and the number of channels 256 corresponds to a neighborhood size of 256 (16*16=256).

[0070] The above discusses the case of losslessly downsampling a 4K RGB image in three single channels: red, green, and blue. Then, an image group consisting of 256 new images of the 4K RGB image when the color channels are red, green, and blue can be merged to obtain a new three-channel image group composed of 256 low-resolution images. The number of channels in this image group is 768 (256*3=768). This image group with 768 channels is the first sub-image.

[0071] Step 102: downsample the coordinate image in the lossless manner to obtain a second sub-image with a Q channel.

[0072] In this step, the coordinate map is a two-channel matrix with the same resolution as the segmentation map in step 101, wherein the first channel represents the horizontal coordinate of each pixel point on the RGB image, and the second channel represents the vertical coordinate of each pixel point on the RGB image. That is, the coordinate map can be used to indicate the position information of each pixel point in the segmentation map.

[0073] For example, based on the fact that the segmentation map in the aforementioned example is a 4K RGB image, this step takes downsampling the 4K coordinate matrix using a lossless method as an example to illustrate how to obtain the second sub-image.

[0074] Among them, the sampling step size for lossless downsampling of the 4K coordinate matrix will also be 16, and the neighborhood will be 16*16; the result of lossless downsampling of the first channel of the 4K coordinate matrix is: 256 new coordinate matrices are formed, and the resolution of each new coordinate matrix is ​​240*135; by splicing these 256 new coordinate matrices in the channel direction, a lossless downsampling coordinate matrix group can be obtained, and the number of channels of this lossless downsampling coordinate matrix group is 256.

[0075] Similarly, the result of lossless downsampling of the second channel of the 4K coordinate matrix according to the same sampling step size of 16 is: 256 new coordinate matrices are formed, and the resolution of each new coordinate matrix is ​​240*135; by splicing these 256 new coordinate matrices in the channel direction, a lossless downsampled coordinate matrix group can be obtained, and the number of channels of this lossless downsampled coordinate matrix group is 256.

[0076] When the coordinate matrix group formed by the 256 new coordinate matrices of the 4K coordinate matrix in the first channel and the second channel is merged, a new two-channel coordinate matrix group formed by 256 new coordinate matrices can be obtained. The number of channels of this coordinate matrix group is 512 (256*2=512). The coordinate matrix group with 512 channels is the second sub-image.

[0077] Step 103: Perform a first convolution operation on the first subgraph and the second subgraph to obtain a third subgraph with K channels; K is smaller than the sum of P and Q.

[0078] In the above example of segmenting a 4K RGB image, the first sub-image obtained has 768 channels, and the second sub-image obtained has 512 channels. In order to overcome the problem of missing position information of each pixel in the lossless downsampling process of RGB images, before inputting the first sub-image generated by lossless downsampling into the image segmentation network, it is also necessary to merge the second sub-image containing the position information into the first sub-image to generate a new feature map. Among them, by splicing the first sub-image and the second sub-image along the channel direction, a feature map with 1280 channels (768+512=1280) can be generated. This feature map as a whole can illustrate that the purpose of truly lossless segmentation of RGB images can be achieved, and the smoothness of the edges of the foreground and background generated by the segmentation can be greatly improved.

[0079] In one implementation of this step, the first convolution operation has K 1*1*(P+Q) first convolution kernels; the first convolution operation is performed on the first sub-image and the second sub-image to obtain a third sub-image with K channels, including: for any first convolution kernel, performing a convolution operation on the first sub-image and the second sub-image through the first convolution kernel to obtain a single-channel third sub-image.

[0080] For example, for a feature map with 1280 channels generated by splicing the first sub-image with 768 channels and the second sub-image with 512 channels along the channel direction, if this feature map is directly input into the image segmentation network, then since the number of channels of this feature map is very large, 1280, this places extremely high requirements on the processing power of the image segmentation network, greatly affecting the efficiency of image segmentation.

[0081] In order to solve the above technical problems, the present application proposes that before the feature map with 1280 channels generated by channel splicing is input into the image segmentation network, a convolution operation can be performed on the feature map to achieve the effect of reducing the number of channels of the feature map while further extracting features. For example, the number of channels of the feature map to be input into the image segmentation network can be set to 32, and then 32 1*1*1280 convolution kernels can be used to perform a convolution operation on the feature map with 1280 channels generated by channel splicing. Among them, for any of the 32 1*1*1280 convolution kernels, a convolution operation will be performed on the feature map with 1280 channels generated by channel splicing, thereby obtaining a third sub-image of a single channel; since there are 32 convolution kernels, a third sub-image with 32 channels can be generated after this convolution operation.

[0082] It should be noted that in this application, the number of channels of the feature map to be input into the image segmentation network can be set to 32, or other numbers such as 64. The specific number of channels of the feature map to be input into the image segmentation network needs to be controlled to a certain value. This will be set by those skilled in the art according to actual operational requirements and is not limited in this application.

[0083] Step 104: Determine a first sub-mask image with a K channel corresponding to the third sub-image through an image segmentation network.

[0084] After the first convolution operation in step 103, a third sub-image with K channels can be obtained. Since the number of channels of the feature map is reduced, the third sub-image with K channels can be input into the image segmentation network in this step, and the image segmentation network performs image segmentation processing on it, thereby obtaining the first sub-mask image with K channels.

[0085] Step 105 : Perform a second convolution operation on the first sub-mask image to obtain a second sub-mask image with L channels; L is greater than K.

[0086] In one implementation of this step, the second convolution operation has L 1*1*K second convolution kernels; performing the second convolution operation on the first sub-mask image to obtain a second sub-mask image with L channels includes: for any second convolution kernel, performing a convolution operation on the first sub-mask image through the second convolution kernel to obtain a single-channel second sub-mask image.

[0087] In the previous example, when downsampling a 4K RGB image using a lossless method, a lossless downsampled image group consisting of 256 low-resolution images is obtained. Therefore, when using the mask image to segment the 4K RGB image, 256 masks are required. Since one mask corresponds to one channel, the number of channels in the first sub-mask image, 32, does not meet the required 256 channels corresponding to 256 masks in the mask image.

[0088] In order to solve the above technical problems, the present application proposes that before generating the mask map, the number of feature map channels can be restored by performing a convolution operation on the first sub-mask map with 32 channels. In the example of the present application, since 256 masks are needed, L=256, which means that 256 1*1*32 convolution kernels can be used to perform a convolution operation on the first sub-mask map with 32 channels. Among them, for any one of the 256 1*1*32 convolution kernels, a convolution operation will be performed on the first sub-mask map with 32 channels, thereby obtaining a single-channel second sub-mask map; since there are 256 convolution kernels, this convolution operation will generate a second sub-mask map with 256 channels.

[0089] Step 106: splice the second sub-mask images into a mask image in the lossless manner; the mask image and the segmentation image have the same resolution.

[0090] In one implementation of this step, the second sub-mask image is spliced ​​into a mask image in the lossless manner, including: for any channel in the second sub-mask image, the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel are extracted into the corresponding sampling neighborhoods in the segmentation image, thereby obtaining the mask image.

[0091] Take the above example to illustrate the lossless downsampling of a 4*4 grayscale image and the need to generate a mask map for it. Figure 3 As shown, a schematic diagram of mask image generation provided by an embodiment of the present application shows the process of generating a single-channel 4*4 mask image from a 4-channel 2*2 sub-mask image. Black represents the background, white represents the foreground, and the numbers indicate the corresponding positions of the pixels before and after the mask image is generated. First, the pixel point in the upper left corner of each 2*2 mask is sequentially generated into a 2*2 neighborhood in the upper left corner of the 4*4. Then the remaining three neighborhoods are generated in sequence to obtain a single-channel 4*4 mask image. The process of generating the mask image is equivalent to the inverse process of downsampling the segmentation image in a lossless manner.

[0092] Regarding the aforementioned example of segmenting a 4K RGB image, due to the large image resolution, a schematic diagram is no longer used here to illustrate the process of generating a single-channel 3840*2160 mask image from a 256-channel 240*135 sub-mask image.

[0093] Step 107: performing image segmentation on the segmentation image using the mask image.

[0094] In this step, the segmentation image can be segmented into foreground and background according to the mask image generated in step 106 .

[0095] Based on this solution, for the segmentation map, by constructing a corresponding coordinate map and losslessly downsampling the two respectively, and for the results after lossless downsampling - the first sub-map with P channel and the second sub-map with Q channel, a first convolution operation can be performed, which fuses each pixel on the segmentation map with the position information of the pixel, so that in the process of image segmentation, no detail information of the segmentation map is lost, ensuring the smoothness of the foreground and background edges of the image during segmentation; in addition, compared to directly inputting the segmentation map into the image segmentation network, the third sub-map with K channel obtained by the first convolution operation of this application will occupy a relatively small amount of data processing resources of the image segmentation network. In this way, the image segmentation network of this application does not have high requirements for device performance, that is, this application can support image segmentation on a device with low graphics card performance. In this way, it is possible to accurately segment ultra-high-definition images under low device performance conditions.

[0096] Since the present application involves the first convolution operation, the second convolution operation and the use of the image segmentation network when segmenting the segmentation map, all three can be trained through the same loss function.

[0097] In some implementations of this application, the formula corresponding to the loss function is:

[0098]

[0099] Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask under the i-th channel, P represents the number of channels of the second sub-mask, such as P=256 in the embodiment of the present application, and λ represents a preset coefficient for balancing the focus of network learning. f and L i It can be calculated using traditional segmentation loss functions, such as cross entropy loss, Dice loss, etc.

[0100] In some implementations of this application, the formula corresponding to the loss function is:

[0101] L T =L f

[0102] Among them, L T Represents the total loss, L f represents the loss of the mask image, L f It can be calculated using traditional segmentation loss functions, such as cross entropy loss, Dice loss, etc.

[0103] Based on the same concept, the embodiment of the present application also provides an image segmentation device, such as Figure 4 As shown, the device includes:

[0104] A lossless downsampling unit 401 is configured to downsample the segmentation image in a lossless manner to obtain a first sub-image having P channels;

[0105] The lossless downsampling unit 401 is further configured to downsample the coordinate map in the lossless manner to obtain a second sub-map having a Q channel; the coordinate map has the same resolution as the segmentation map, and the coordinate map is used to indicate position information of each pixel in the segmentation map;

[0106] A convolution operation unit 402 is configured to perform a first convolution operation on the first sub-graph and the second sub-graph to obtain a third sub-graph having K channels, where K is less than the sum of P and Q.

[0107] A first sub-mask image determining unit 403 is configured to determine a first sub-mask image having a K channel corresponding to the third sub-image through an image segmentation network;

[0108] The convolution operation unit 402 is further configured to perform a second convolution operation on the first sub-mask image to obtain a second sub-mask image having L channels; L is greater than K;

[0109] The mask image generating unit 404 is configured to stitch the second sub-mask images into a mask image in the lossless manner; the mask image and the segmentation image have the same resolution;

[0110] The image segmentation unit 405 is configured to perform image segmentation on the segmentation image using the mask image.

[0111] Furthermore, for the device, the lossless downsampling unit 401 is specifically used to: determine each sampling neighborhood in the segmentation image according to a preset sampling step; each sampling neighborhood has P pixels at different positions; extract the pixels at the same position in each sampling neighborhood, and combine the extracted pixels into a first sub-image with P channels in a preset combination order.

[0112] Furthermore, for the device, the mask image generation unit 404 is specifically used to: for any channel in the second sub-mask image, extract the pixel values ​​of each pixel point in the sub-mask image corresponding to the channel into the corresponding sampling neighborhoods in the segmentation image, thereby obtaining the mask image.

[0113] Furthermore, for the device, the first convolution operation has K 1*1*(P+Q) first convolution kernels; the convolution operation unit 402 is specifically used to: for any first convolution kernel, perform a convolution operation on the first sub-image and the second sub-image through the first convolution kernel to obtain a single-channel third sub-image.

[0114] Furthermore, for the device, the second convolution operation has L 1*1*K second convolution kernels; the convolution operation unit 402 is specifically used to: for any second convolution kernel, perform a convolution operation on the first sub-mask image through the second convolution kernel to obtain a single-channel second sub-mask image.

[0115] Furthermore, for the device, the first convolution operation, the second convolution operation and the image segmentation network are trained by a unified loss function.

[0116] Furthermore, for this device, the formula corresponding to the loss function is:

[0117]

[0118] Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask image under the i-th channel, P represents the number of channels of the second sub-mask image, and λ represents a preset coefficient.

[0119] An embodiment of the present application provides a computing device, which may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), a memory, input / output devices, etc. The input devices may include a keyboard, a mouse, a touch screen, etc., and the output devices may include a display device such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0120] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be used to store program instructions for the image segmentation method;

[0121] The processor is used to call the program instructions stored in the memory and execute the image segmentation method according to the obtained program.

[0122] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute an image segmentation method.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0128] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image segmentation method, characterized in that: include: Downsample the segmentation image in a lossless manner to obtain a first sub-image with P channels; Downsampling the coordinate map in the lossless manner to obtain a second sub-map having a Q channel; the coordinate map has the same resolution as the segmentation map, and the coordinate map is used to indicate position information of each pixel in the segmentation map; Performing a first convolution operation on the first subgraph and the second subgraph to obtain a third subgraph having K channels; K is less than the sum of P and Q; Determine, through an image segmentation network, a first sub-mask image with a K channel corresponding to the third sub-image; Performing a second convolution operation on the first sub-mask image to obtain a second sub-mask image having L channels; L is greater than K; splicing the second sub-mask images into a mask image in the lossless manner; the mask image and the segmentation image have the same resolution; Performing image segmentation on the segmentation map using the mask map; The first convolution operation, the second convolution operation, and the image segmentation network are trained by a unified loss function; the formula corresponding to the loss function is: Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask image under the i-th channel, P represents the number of channels of the second sub-mask image, and λ represents a preset coefficient.

2. The method according to claim 1, wherein The segmentation image is downsampled in a lossless manner to obtain a first sub-image having P channels, comprising: Determine each sampling neighborhood in the segmentation image according to a preset sampling step size; each sampling neighborhood has P pixels at different positions; Pixels at the same position in each sampling neighborhood are extracted, and the extracted pixels are combined into a first sub-image with P channels according to a preset combination order.

3. The method according to claim 2, wherein The step of stitching the second sub-mask images into a mask image in the lossless manner includes: For any channel in the second sub-mask image, pixel values ​​of each pixel point in the sub-mask image corresponding to the channel are extracted into corresponding sampling neighborhoods in the segmentation image, thereby obtaining the mask image.

4. The method according to claim 1, wherein The first convolution operation includes K first convolution kernels of 1*1*(P+Q); The performing a first convolution operation on the first subgraph and the second subgraph to obtain a third subgraph having K channels includes: For any first convolution kernel, a convolution operation is performed on the first sub-image and the second sub-image using the first convolution kernel to obtain a single-channel third sub-image.

5. The method according to claim 1, wherein The second convolution operation has L 1*1*K second convolution kernels; The performing a second convolution operation on the first sub-mask image to obtain a second sub-mask image having an L channel includes: For any second convolution kernel, a convolution operation is performed on the first sub-mask image through the second convolution kernel to obtain a single-channel second sub-mask image.

6. An image segmentation device, characterized in that: include: a lossless downsampling unit, configured to downsample the segmentation image in a lossless manner to obtain a first sub-image having P channels; The lossless downsampling unit is further configured to downsample the coordinate map in the lossless manner to obtain a second sub-map having a Q channel; the coordinate map has the same resolution as the segmentation map, and the coordinate map is used to indicate position information of each pixel in the segmentation map; a convolution operation unit, configured to perform a first convolution operation on the first subgraph and the second subgraph to obtain a third subgraph having K channels; K is less than the sum of P and Q; a first sub-mask image determining unit, configured to determine a first sub-mask image having a K channel corresponding to the third sub-image through an image segmentation network; The convolution operation unit is further configured to perform a second convolution operation on the first sub-mask image to obtain a second sub-mask image having L channels; L is greater than K; a mask image generating unit, configured to stitch the second sub-mask images into a mask image in the lossless manner; the mask image and the segmentation image have the same resolution; an image segmentation unit, configured to perform image segmentation on the segmentation image using the mask image; The first convolution operation, the second convolution operation, and the image segmentation network are trained by a unified loss function; the formula corresponding to the loss function is: Among them, L T Represents the total loss, L f represents the loss of the mask image, L i represents the loss of the second sub-mask image under the i-th channel, P represents the number of channels of the second sub-mask image, and λ represents a preset coefficient.

7. A computer device, characterized in that: include: memory for storing computer programs; A processor is configured to call a computer program stored in the memory and execute the method according to any one of claims 1 to 5 according to the obtained program.

8. A computer-readable storage medium, characterized in that The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images