A learnable sampling image segmentation method and a computer-readable storage medium

Abstract

A learnable sampling image segmentation method and a computer-readable storage medium are disclosed. The method constructs an image segmentation network based on deep learning technology. The image segmentation network includes a learnable non-uniform image sampling module and a segmentation module. The learnable non-uniform image sampling module predicts the importance index of each pixel in the image to be segmented after sampling to a first resolution and generates a pixel-wise sampling density map. The image to be segmented is then sampled according to the pixel-wise sampling density map, where the sampling density of each pixel is proportional to its importance index. The segmentation module performs image segmentation on the sampled image and restores the segmented image to its original resolution to complete the image segmentation. The learnable non-uniform image sampling module can perform deep learning, learning to oversample regions with few target pixels and difficult segmentation, and downsample regions with many pixels and easy segmentation, thereby resolving the imbalance between categories and improving the segmentation accuracy of the image segmentation network.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a learnable sampling image segmentation method and a computer-readable storage medium. Background Technology

[0002] In industrial production processes, it is often necessary to image industrial products and use the obtained industrial images for product inspection. Industrial image segmentation is often an important part of this process.

[0003] Class imbalance is a common pain point in industrial image segmentation, and it is even more serious in high-resolution industrial images. In actual production processes, the segmentation targets in industrial images are mostly characterized by being "small and weak," with the background area accounting for the vast majority of the image content. The number of foreground target pixels that need to be segmented is relatively small, and the proportion of each foreground target category is also significantly different, resulting in severe class imbalance and affecting the final segmentation performance of the segmentation model.

[0004] Existing technologies offer various solutions to class imbalance, such as assigning different weights to different classes, combining loss functions, and improving network models. These methods can alleviate class imbalance to some extent, but they are ineffective when class imbalance is severe, and the segmentation accuracy is even worse on high-resolution images. Summary of the Invention

[0005] This invention primarily addresses the technical problem of low segmentation accuracy in images with class imbalance.

[0006] According to a first aspect, one embodiment provides a learnable sampling image segmentation method, comprising:

[0007] Obtain the image to be segmented and its initial resolution;

[0008] A pre-trained image segmentation network is used to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with the first resolution;

[0009] The initial segmented image is backsampled to restore it to the same resolution as the image to be segmented, thus obtaining the final segmented image;

[0010] The image segmentation network includes a learnable non-uniform image sampling module and a segmentation module. The step of using a pre-trained image segmentation network to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with the first resolution includes:

[0011] The learnable non-uniform image sampling module is used to predict the importance index of each pixel after sampling the image to be segmented to the first resolution, and a pixel-wise sampling density map is generated. The image to be segmented is then sampled to the first resolution according to the pixel-wise sampling density map to obtain a non-uniform sampling image, wherein the sampling density of each pixel is proportional to its importance index.

[0012] The non-uniformly sampled image is input into the segmentation module to perform image segmentation, thereby obtaining the initial segmented image.

[0013] In some embodiments, the learnable non-uniform image sampling module includes a localization network, a compression network, and a non-uniform consistency sampler. The step of using the learnable non-uniform image sampling module to predict the importance index of each pixel after sampling the image to be segmented to the first resolution, and generating a pixel-by-pixel sampling density map, and sampling the image to be segmented to the first resolution according to the pixel-by-pixel sampling density map to obtain a non-uniform sampled image, includes:

[0014] A scaling operation is performed on the image to be segmented to obtain a scaled image with the first resolution;

[0015] The scaled image is input into the localization network to determine the importance of each pixel in the scaled image, thereby obtaining a localization map of important regions with the first resolution;

[0016] The important region location map is input into the compression network to reduce the number of channels to 1, thereby obtaining a region saliency map. The pixel value of each pixel in the region saliency map represents the saliency of that pixel.

[0017] A pixel-by-pixel sampling density map is generated based on the region saliency map, wherein the sampling density of each pixel is proportional to its saliency;

[0018] The image to be segmented and the pixel-by-pixel sampling density map are input into the non-uniform consistency sampler to sample the pixel values ​​of the image to be segmented to the corresponding positions of the non-uniform sampling image according to the pixel-by-pixel sampling density map.

[0019] In some embodiments, the localization network includes multiple standard convolutional layers, and the number of output channels of the first standard convolutional layer is greater than the number of input channels. The number of output channels of each standard convolutional layer is first increased or remains unchanged and then decreased. The standard convolutional layer includes a convolutional layer, a batch normalization layer and an activation layer connected in sequence.

[0020] In some embodiments, the localization network includes three standard convolutional layers, and the number of output channels of the three standard convolutional layers are μ×c, μ×c and c, respectively, where c represents the number of channels of the scaled image, and μ is a preset value and is an integer greater than 0.

[0021] In some embodiments, the compressed network includes a 1×1 standard convolutional layer, wherein the 1×1 standard convolutional layer includes a 1×1 convolutional layer, a batch normalization layer and an activation layer connected in sequence.

[0022] In some embodiments, generating a pixel-by-pixel sampling density map based on the region saliency map includes: normalizing the region saliency map using a softmax function in the channel dimension to obtain the pixel-by-pixel sampling density map.

[0023] In some embodiments, the image segmentation network is trained according to the following steps:

[0024] Obtain the training sample images and their corresponding segmentation labels, as well as the second resolution;

[0025] The learnable non-uniform image sampling module is used to predict the importance index of each pixel after the training sample image is sampled to the second resolution, and a pixel-by-pixel sampling density map of the training sample is generated. The training sample image is sampled to the second resolution according to the pixel-by-pixel sampling density map of the training sample to obtain a non-uniform sampling image of the training sample, wherein the sampling density of each pixel is proportional to its importance index.

[0026] The non-uniformly sampled image of the training sample is input into the segmentation module to perform image segmentation, thereby obtaining the initial segmented image of the training sample.

[0027] The segmentation marker map is sampled to the second resolution based on the pixel-by-pixel sampling density map of the training samples to obtain a sampling marker map, and a sampling density marker map is generated based on the sampling marker map;

[0028] The image segmentation network is trained according to a total loss function, which is determined by a first loss function and a second loss function. The first loss function is the loss function of the image segmentation task performed by the segmentation module, representing the error between the initial segmented image of the training samples and the sampling label map. The second loss function represents the error between the pixel-by-pixel sampling density map of the training samples and the sampling density label map.

[0029] In some embodiments, the expression for the second loss function is:

[0030] L2=MSE(d m ,d GT )

[0031] Where MSE() represents the mean squared error loss function, d m d represents the pixel-by-pixel sampling density map of the training samples. GT This represents the sampling density marker map.

[0032] In some embodiments, the sampling density marker map is determined by the following expression:

[0033] d GT =Edge(Gauss(l a )),

[0034] Among them l a The sampling marker map is represented by Gauss(), which represents the Gaussian smoothing function, and Edge(), which represents the edge detection function.

[0035] In some embodiments, the expression for the total loss function is:

[0036] L = αL1 + βL2,

[0037] Where L1 represents the first loss function, L2 represents the second loss function, and α and β are preset weight coefficients.

[0038] According to a second aspect, one embodiment provides a computer-readable storage medium storing a program that can be executed by a processor to implement the image segmentation method described in any of the above embodiments.

[0039] The image segmentation method based on the learnable sampling described in the above embodiments constructs an image segmentation network using deep learning technology. The image segmentation network includes a learnable non-uniform image sampling module and a segmentation module. The learnable non-uniform image sampling module predicts the importance index of each pixel after sampling the image to be segmented to a first resolution and generates a pixel-wise sampling density map. The image to be segmented is sampled according to the pixel-wise sampling density map, wherein the sampling density of each pixel is proportional to its importance index. The segmentation module performs image segmentation on the sampled image to obtain an initial segmented image. Then, the initial segmented image is restored to its original resolution to complete the entire image segmentation process. Because the learnable non-uniform image sampling module can perform deep learning, it can automatically increase information in regions beneficial to the segmentation task (usually regions with few pixels and difficult to segment) and reduce attention to general regions. It learns an adaptive sampling strategy, predicts the importance index of each pixel after sampling during segmentation, and performs non-uniform sampling on the image to be segmented. The sampling density is proportional to its importance index, which is beneficial for oversampling regions with few pixels and difficult to segment, and downsampling regions with many pixels and easy to segment. This fundamentally solves the imbalance between categories and improves the segmentation accuracy of the image segmentation network. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the overall process of a learnable sampling image segmentation method.

[0041] Figure 2 This is a flowchart of a learnable sampling image segmentation method in one embodiment;

[0042] Figure 3 This is a flowchart of one embodiment that uses a pre-trained image segmentation network to perform non-uniform sampling and image segmentation processing on the image to be segmented, to obtain an initial segmented image with a first resolution;

[0043] Figure 4 This is a schematic diagram of the structure of a learnable non-uniform image sampling module according to one embodiment.

[0044] Figure 5 This is a flowchart of an embodiment that uses a learnable non-uniform image sampling module to predict the importance index of each pixel after sampling the image to be segmented to a first resolution, and generates a pixel-by-pixel sampling density map, and then samples the image to be segmented to a first resolution based on the pixel-by-pixel sampling density map to obtain a non-uniform sampled image.

[0045] Figure 6 This is a schematic diagram of the structure of a positioning network according to one embodiment;

[0046] Figure 7 This is a flowchart illustrating the training process of an image segmentation network according to one embodiment.

[0047] Figure 8 This is a schematic diagram illustrating how the learnable sampling image segmentation method of the present invention is used to segment an image to be segmented to obtain a segmented image in one embodiment. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0049] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0050] The serial numbers used to designate components in this document, such as "first" and "second," are solely for distinguishing the described objects and have no sequential or technical meaning. Unless otherwise specified, the terms "connection" and "linkage" in this application include both direct and indirect connections (linkages). The "standard convolutional layer" referred to herein consists of a convolutional layer (Conv), a batch normalized layer (BN), and an activation layer connected sequentially, denoted as "convolutional layer + batch normalized layer + activation layer," where the activation layer can employ activation functions such as ReLU. For ease of description below, unless otherwise specified, "3×3 convolutional layer + batch normalized layer + activation layer" is referred to as a 3×3 standard convolutional layer, abbreviated as "C." 3x3 "; "1×1 convolutional layer + batch normalized layer + activation layer" is called a 1×1 standard convolutional layer, abbreviated as "C" 1x1 ", where 3×3 convolutional layer refers to a convolutional layer with a kernel size of 3×3, and 1×1 convolutional layer refers to a convolutional layer with a kernel size of 1×1.

[0051] Current industrial image segmentation faces two challenging problems: first, the class imbalance mentioned in the background; and second, the high resolution of input images, which hardware cannot handle. High-resolution industrial cameras offer numerous advantages, including high definition, high precision, and low noise. As production quality requirements continue to rise, manufacturers widely use high-resolution cameras to better image their products, obtaining more information about the target area and improving segmentation accuracy. However, high-resolution images are enormous, and hardware memory is insufficient to support their storage demands.

[0052] For high-resolution image segmentation, due to their huge size, hardware cannot process them directly. There are generally two solutions: image cropping into blocks or image downsampling.

[0053] Image cropping into blocks is rarely used in practical applications involving large images due to its inability to meet real-time inference requirements. Directly downsampling the original image using methods such as nearest neighbor interpolation or bilinear sampling to reduce resolution and train the model for inference at lower resolution typically yields poor results. This is because such downsampling relies on an unrealistic assumption that all pixels are equally important to the segmentation network. This leads to insufficient attention to critical regions of the input image, resulting in poor segmentation accuracy. Furthermore, large industrial images often exhibit severe foreground-background imbalance. Directly downsampling to reduce image resolution to meet hardware requirements exacerbates this imbalance. Regions that are already small in size become virtually nonexistent after downsampling, resulting in low segmentation accuracy that fails to meet requirements.

[0054] To alleviate or solve the above two problems, this invention creatively proposes a learnable sampling image segmentation method, which can greatly improve the class imbalance. When segmenting high-resolution images, it can reduce the image resolution and adaptively increase the sampling density of important regions, improving the performance of image segmentation tasks while being hardware-friendly. The learnable sampling image segmentation method provided by this invention is mainly based on a data-driven deformable adaptive image sampling method. It automatically adjusts the sampling density according to the importance of each pixel to the image segmentation task. Regions beneficial to the segmentation task are oversampled, while general regions are undersampled, greatly improving the class imbalance of the image to be segmented and, to a certain extent, realizing the discovery of important samples. That is, for regions that are difficult to segment, the image segmentation network will automatically upsample to increase attention and improve segmentation performance.

[0055] This invention primarily utilizes deep learning technology to build an image segmentation network, and then implements the image segmentation method of this invention based on this image segmentation network. Please refer to [link / reference]. Figure 1 The entire image segmentation network F consists of a learnable non-uniform image sampling module L and a segmentation module S. The learnable non-uniform image sampling module L is responsible for sampling the input image x to be segmented, obtaining a non-uniformly sampled image lx. The segmentation module S is responsible for performing segmentation on the non-uniformly sampled image lx, obtaining an initial segmented image ly. Finally, the initial segmented image ly output by the segmentation module S is backsampled non-uniformly to obtain a final segmented image y with the same resolution as the image x to be segmented, thus completing the segmentation task.

[0056] The core idea of ​​the learnable sampling image segmentation method of this invention is that the sampling strategy of the learnable non-uniform image sampling module L is learnable, and it learns adaptive sampling of the image to be segmented x, which is essentially learning a pixel-wise sampling density map d. m According to d m Adaptive sampling is performed on the image to be segmented, with high sampling density in regions that significantly improve segmentation performance and lower sampling density in other general regions, resulting in a non-uniformly sampled image lx (the non-uniformly sampled image lx produces a certain degree of distortion compared to the image to be segmented by the human eye, but it greatly improves segmentation performance). The segmentation module S is responsible for performing the segmentation task on the non-uniformly sampled image lx to obtain the initial segmented image ly. Then, non-uniform backsampling is performed on the initial segmented image ly to obtain the final segmented image y with the same resolution as the image to be segmented x. Essentially, non-uniform backsampling is the backsampling of the learnable non-uniform image sampling module L, that is, it adopts backsampling relative to the learnable non-uniform image sampling module L, which can eliminate the distortion caused by the adaptive sampling used earlier, so that the final segmented image corresponds to the input image to be segmented.

[0057] During training, the non-uniform image sampling module L and the segmentation module S are trained and optimized together under the control of the loss function until convergence. During inference, the image segmentation network F directly outputs the segmented image y in an end-to-end manner to complete the image segmentation task.

[0058] Please refer to Figure 2 The image segmentation method with learnable sampling in one embodiment of the present invention includes steps 100 to 300, which are described in detail below.

[0059] Step 100: Obtain the image to be segmented and its first resolution.

[0060] In this context, the image to be segmented is the image from which the target region needs to be segmented. For example, in the defect detection of industrial products, the image to be segmented is the image of the object to be detected, and the target region is the area on the object to be detected where there is a defect. The object to be detected can be a product on an industrial assembly line, a mechanical part in a box, a tool on an operating table, etc., without specific limitations.

[0061] The first resolution determines the resolution of the non-uniformly sampled image lx, which is set by the user. The first resolution can be greater than, less than, or equal to the resolution of the image to be segmented. Correspondingly, the learnable sampling strategy of the learnable non-uniform image sampling module L is divided into three types: learnable upsampling, learnable downsampling, and learnable leveling. Learnable upsampling means that the resolution of the image to be segmented is less than the resolution of the non-uniformly sampled image lx; learnable downsampling means that the resolution of the image to be segmented is greater than the resolution of the non-uniformly sampled image lx; and learnable leveling means that the resolution of the image to be segmented is equal to the resolution of the non-uniformly sampled image lx. Figure 1 This diagram illustrates the overall process of segmenting high-resolution images using learnable downsampling.

[0062] Generally speaking, these three strategies correspond to the following task scenarios: Learnable upsampling is mainly for images with low resolution (e.g., [128, 128]), which are difficult to segment and require further improvement in segmentation accuracy (after appropriate upsampling, the inference time is still within an acceptable range); Learnable downsampling is mainly for high-resolution images, generally images that current hardware cannot directly process and need to be downsized, such as images with a resolution of [10000, 8000]; Learnable flat sampling is mainly for images with average resolution (e.g., [512, 512]), but with low segmentation accuracy due to class imbalance or similar reasons, requiring performance improvement, but without the need to downsize the resolution.

[0063] The choice of the learnable sampling strategy and the size of the first resolution are determined by the user according to the specific situation. This invention does not limit them. In typical image segmentation tasks, the basic principles are generally that the hardware is acceptable, the inference time is feasible, and the segmentation accuracy meets the requirements.

[0064] Step 200: Use a pre-trained image segmentation network to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with a first resolution.

[0065] Those skilled in the art will understand that before using an image segmentation network to process the image to be segmented, it needs to be trained to learn sampling strategies and segmentation capabilities. Therefore, in this step, a pre-trained image segmentation network is used to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with a first resolution.

[0066] Please refer to Figure 3 In one embodiment, step 200 includes steps 210 and 220, which are described below.

[0067] Step 210: Use the learnable non-uniform image sampling module to predict the importance index of each pixel after sampling the image to be segmented to the first resolution, and generate a pixel-by-pixel sampling density map. Based on the pixel-by-pixel sampling density map, sample the image to be segmented to the first resolution to obtain a non-uniform sampling image, wherein the sampling density of each pixel is proportional to its importance index.

[0068] Importance metrics can be determined based on specific circumstances and tasks, depending on the regions or features (e.g., edges, colors) that require attention in the task. These metrics can be grayscale values, saliency, etc. The predictive ability of the learnable non-uniform image sampling module for importance metrics can be acquired through training. The pixel values ​​of the pixel-by-pixel sampling density map represent the sampling density of the corresponding pixel in the non-uniformly sampled image lx. Since sampling density is proportional to its importance metric, it helps to increase attention to regions beneficial for segmentation while reducing the sampling degree of general regions, correcting class imbalances at the pixel level of the data.

[0069] The learnable non-uniform image sampling module L adaptively samples the image to be segmented, x, to obtain a non-uniformly sampled image lx, which is then used by the subsequent segmentation module S. Please refer to [reference needed]. Figure 4 In one embodiment, the learnable non-uniform image sampling module L includes a localization network LN, a compression network CN, and a non-uniform consistency sampler G. Based on this, please refer to... Figure 5 Step 210 includes steps 211 to 215.

[0070] Step 211: Perform a scaling operation on the image to be segmented to obtain a scaled image (xlr) with the first resolution. Scaling methods can include bilinear interpolation, nearest neighbor interpolation, etc.

[0071] Step 212: Input the scaled image xlr into the localization network LN to determine the importance of each pixel in the scaled image xlr, and obtain an important region localization map with a first resolution.

[0072] The localization network LN is responsible for locating important regions of the scaled image xlr. In this embodiment, the importance of pixels is determined by saliency, and the pixel values ​​of each pixel in the obtained important region localization map represent the saliency information of the pixel.

[0073] Step 213: Input the important region localization map into the compression network CN to reduce the number of channels to 1 and obtain the region saliency map. The pixel value of each pixel in the region saliency map represents the saliency of that pixel.

[0074] The purpose of the compressed network (CN) is to compress and refine effective information, reduce the number of channels, and generate a regional saliency map.

[0075] Step 214: Generate a pixel-by-pixel sampling density map d based on the region saliency map.m The sampling density of each pixel is proportional to its saliency. For example... Figure 4 As shown, in one embodiment, the region saliency map can be normalized using the softmax function in the channel dimension to obtain a pixel-wise sampling density map d. m The obtained pixel-by-pixel sampling density map d m With first resolution, pixel-by-pixel sampling density map d m Each pixel value is between [0,1], representing the sampling density of that pixel.

[0076] Step 215: Combine the image to be segmented x and the pixel-wise sampling density map d m Input a non-uniform consistency sampler G to sample the pixel values ​​of the image to be segmented x to the corresponding positions in the non-uniformly sampled image lx according to the pixel-by-pixel sampling density map. The expression for this operation is:

[0077] lx = G(x, dm).

[0078] Among them, the non-uniform consistency sampler G essentially uses the grid sampling method, namely the GridSample sampling algorithm, based on the pixel-by-pixel sampling density map d. m The obtained density coordinate map is used to sample the image x to be segmented. The GridSample sampling algorithm is an existing technology, and is briefly described here. First, the GridSample sampling algorithm has two inputs: an input tensor and grid coordinates. Then, the grid coordinates are mapped to the corresponding positions in the input tensor. Finally, image interpolation methods such as bilinear interpolation or nearest neighbor interpolation are used for sampling (interpolation) to obtain the output result. In this embodiment, specifically, the pixel-wise sampled density map d obtained after normalization using the softmax function is used... m First, the pixel-by-pixel sampling density map d m Make a copy along the channel dimension, then multiply by 2 pixel by pixel and subtract, thus obtaining the pixel-by-pixel sampling density map d. m The pixel values ​​are mapped to the range [-1, 1] to obtain the final density coordinate map; finally, the GridSample algorithm is applied to sample the pixel values ​​of the image to be segmented x onto the non-uniformly sampled image lx based on the density coordinate map.

[0079] For example, the shape of the image x to be segmented is [3, 1, 256, 256] (where 3 is the batch dimension, 1 is the channel dimension, and the two 256s are the height and width of the image, respectively). Assuming that the resolution of the non-uniformly sampled image lx to be obtained is [64, 64], that is, the shape is [3, 1, 64, 64], then the obtained pixel-wise sampling density map d m The shape is [3,1,64,64], d mA copy is made in the channel dimension, the shape becomes [3,2,64,64] and then the shape is transformed to [3,64,64,2]. Then, each pixel is multiplied by 2 and subtracted by 1 to obtain the final density coordinate map. GridSample samples the pixel values ​​of the image to be segmented x onto the non-uniform sampling image lx according to the density coordinate map, that is, the shape of the non-uniform sampling image lx is [3,1,64,64].

[0080] In some embodiments, the learnable non-uniform image sampling module L is a plug-and-play lightweight network module. The role of the localization network LN is to encourage the network model to focus more on regions beneficial to the target object in the segmentation task. The network structure is simple and efficient, including multiple standard convolutional layers. The first standard convolutional layer has more output channels than input channels, and the number of output channels in each subsequent standard convolutional layer first increases or remains constant before decreasing. That is, from input to output, the number of channels generally increases first and then decreases. Increasing the number of channels first allows the network to learn to gradually abstract higher-order features from lower-order features, obtaining richer and more effective feature information, and further improving the network's fitting ability. Decreasing the number of channels is a process of information compression and refinement of the extracted feature information, prompting the network to extract effective information from numerous features and, to some extent, avoiding overfitting.

[0081] Please refer to Figure 6 Preferably, the localization network LN includes three standard convolutional layers, and the number of output channels of the three standard convolutional layers are μ×c, μ×c, and c, respectively, where c represents the number of channels of the scaled image xlr, and μ is a preset value and is an integer greater than 0. In one embodiment, the three standard convolutional layers are three 3×3 standard convolutional layers, and μ = 3.

[0082] The compressed network (CN) consists of a standard 1×1 convolutional layer. Its function is to realize the linear combination of information between channels, enabling the network to learn to extract effective feature information across channels, realize the interaction and integration of information across channels, and thus play the role of extracting key information.

[0083] The learnable non-uniform image sampling module in this embodiment has a simple structure, no complicated preprocessing or postprocessing, is easy to operate, and is plug-and-play, which greatly reduces the technical threshold for users and is easy to deploy.

[0084] Step 220: Input the non-uniformly sampled image into the segmentation module to perform image segmentation and obtain the initial segmented image.

[0085] The segmentation module S primarily performs segmentation operations on the non-uniformly sampled image lx output by the learnable non-uniform image sampling module L to obtain the initial segmented image ly. The segmentation module S can use classic UNet, HRNet networks, or custom segmentation networks, etc. The specific segmentation network used is determined by the user based on the actual situation, and this invention does not impose any limitations.

[0086] Step 300: Backsample the initial segmented image to restore it to the same resolution as the image to be segmented, and obtain the final segmented image.

[0087] Essentially, the backsampling here is the backsampling of the learnable non-uniform image sampling module L. It is a type of non-uniform backsampling, and its main purpose is to map the initial segmentation image ly output by the segmentation module S to a segmentation image y with the same resolution as the image to be segmented. Its expression is:

[0088] y = G -1 (ly,d m ),

[0089] Among them, G -1 This indicates backsampling relative to the learnable non-uniform image sampling module L. Those skilled in the art will understand that once the sampling method for the image to be segmented is known, backsampling is also known, and therefore will not be elaborated upon here.

[0090] This invention designs a dedicated loss function, Loss, for training and optimization of the constructed image segmentation network. The training method for the image segmentation network is described below. Please refer to [link / reference needed]. Figure 7 In one embodiment, the training process of the image segmentation network includes steps 10 to 50, which are described in detail below.

[0091] Step 10: Obtain the training sample images and the corresponding segmentation label maps, as well as the second resolution.

[0092] Referring to step 100 regarding the image to be segmented, those skilled in the art will understand how to collect training sample images. Those skilled in the art will know that the segmentation label map is obtained by labeling the segmentation target regions in the training sample images to represent the true result of performing the segmentation task on the training sample images. The second resolution may be the same as or different from the first resolution.

[0093] Step 20: Use the learnable non-uniform image sampling module to predict the importance index of each pixel after sampling the training sample image to the second resolution, and generate a pixel-by-pixel sampling density map of the training sample. Based on the pixel-by-pixel sampling density map of the training sample, sample the training sample image to the second resolution to obtain a non-uniform sampling image of the training sample, in which the sampling density of each pixel is proportional to its importance index.

[0094] This step can be referred to as step 210 above, and will not be repeated here.

[0095] Step 30: Input the non-uniformly sampled training sample image into the segmentation module to perform image segmentation and obtain the initial segmented training sample image.

[0096] This step can be referred to as step 220 above, and will not be repeated here. It should be noted that after obtaining the initial segmentation images of the training samples during the training phase, backsampling is not required; the backsampling operation is only performed during the inference phase.

[0097] Step 40: Sampling the segmentation marker map to the second resolution based on the pixel-by-pixel sampling density map of the training samples to obtain the sampling marker map, and generating the sampling density marker map based on the sampling marker map.

[0098] Sampling the segmentation marker map to a second resolution based on the pixel-by-pixel sampling density map of the training samples can be done by referring to step 215 above, which describes the sampling method of the image to be segmented based on the pixel-by-pixel sampling density map. The expression is as follows:

[0099] l a =G(label,d) m ),

[0100] Among them l a The label represents the sampling marker map, and the label represents the segmentation marker map.

[0101] The sampling density marker map has a second resolution and is used to identify the true value of the sampling density represented by each pixel in the pixel-by-pixel sampling density map of the training samples. Since the target region to be segmented is marked in the sampling marker map, the corresponding region or feature can be extracted from the sampling marker map based on the region of interest or the region that is beneficial to the segmentation task, thus obtaining the sampling density marker map.

[0102] Step 50: Train the image segmentation network based on the total loss function Loss, where the total loss function Loss is determined by the first loss function L1 and the second loss function L2. The first loss function L1 is the loss function of the image segmentation task performed by the segmentation module, representing the error between the initial segmented image of the training sample and the sampling label map. The second loss function L2 represents the error between the pixel-by-pixel sampling density map of the training sample and the sampling density label map.

[0103] The inputs to the first loss function L1 are the initial segmented image of the training samples and the corresponding segmentation labels, i.e., the sampling label map. The first loss function L1 is the loss function used for semantic segmentation, such as the cross-entropy loss function. During training, the user can decide which semantic segmentation loss function to use based on the specific situation.

[0104] The second loss function L2 is a loss function specifically designed for the learnable adaptive sampling strategy in this invention. If the first loss function L1 is directly used as the loss function for the entire image segmentation network during training and optimization, the network will oversample large, easily segmented regions, which is detrimental to the segmentation of small objects. Therefore, L2 is used to constrain this process. In one embodiment, the expression for L2 is:

[0105] L2=MSE(d m ,d GT ),

[0106] Here, MSE() represents the mean squared error loss function, d m d represents the pixel-wise sampling density map of the training samples. GT This represents a sampling density marker map.

[0107] In one implementation of this invention, it is desirable for the image segmentation network to sample more densely around the semantic boundaries of objects, meaning that the edge information of the objects is useful. This is beneficial for accommodating targets of all sizes that need to be segmented. Essentially, it is hoped that the network can adapt to the optimal sampling position and sampling degree according to the segmentation target. Therefore, the sampling density labeling map d GT By detecting the sampling markers in Figure 1 a The edge is obtained, and its expression is:

[0108] d GT =Edge(Gauss(l a )),

[0109] Where Gauss() represents the Gaussian smoothing function, such as a Gaussian smoothing function with a standard deviation of 1, and Edge() represents the edge detection function, such as edge detection operators like Canny. In one embodiment, the specific implementation of the Edge() edge detection function is to perform convolution extraction using a 3×3 convolution kernel K. Since convolution operations reduce resolution, the sampled labeled image l needs to be pre-processed. a Perform appropriate edge padding so that the sampling density marker map d GT With sampling markers Figure 1 a The resolution is consistent. The 3x3 convolution kernel K is defined as follows:

[0110]

[0111] Essentially, the role of the second loss function L2 is to encourage the network to pay attention to the edges of the target object. By combining it with the first loss function L1, better adaptive sampling can be achieved, ultimately improving the segmentation performance of the image segmentation network.

[0112] Figure 8An example of defect segmentation using the learnable sampling image segmentation method of the present invention is given. The left image is a high-resolution defect image to be segmented. After learning sampling image segmentation, the segmented image on the right is output. It can be seen that the defective area in the left image is segmented out.

[0113] Those skilled in the art will understand that, compared to existing technologies, the learnable sampling image segmentation method proposed in this invention helps solve two major industry pain points: class imbalance and the difficulty of handling high-resolution images with sufficient hardware memory. Its beneficial effects are mainly as follows:

[0114] (1) Solving class imbalance and realizing the mining of important samples: The sampling design of this invention is learnable. Through learning, the image segmentation network learns to adaptively oversample regions with few pixels and difficult to segment, while downsampling regions with many pixels and easy to segment. At the pixel level of data, it corrects the imbalance between classes, fundamentally solves the class imbalance problem, and to a certain extent realizes the mining of important samples, ultimately improving the segmentation accuracy of the image segmentation network.

[0115] (2) Learnable sampling improves segmentation accuracy: When dealing with high-resolution images, downsampling is required. Compared with directly performing downsampling on the original image, this invention uses a learnable method to perform adaptive sampling on the original image. It can automatically add information about the target region that is beneficial to the segmentation task and reduce attention to general regions, reduce redundant information, ensure inference speed and accuracy, and achieve a dual improvement in efficiency and performance. While reducing image resolution, it improves image segmentation performance, achieving a significant improvement in image segmentation performance and hardware-friendly real-time operation.

[0116] (3) Applicable to general-sized images: For general-sized images (hardware-capable), but with severe class imbalance or the required segmentation target is relatively "small and weak", making them difficult to segment, the image segmentation method of the present invention with learnable sampling can be used to segment and improve segmentation accuracy.

[0117] (4) Simple operation and easy deployment: The image segmentation method with learnable sampling proposed in this invention is implemented by a deep neural network. The training and inference run end-to-end without complicated preprocessing and postprocessing. It is simple to operate, greatly reduces the technical threshold for users, and is easy to deploy.

[0118] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0119] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A learnable sampling image segmentation method, characterized in that, include: Obtain the image to be segmented and its initial resolution; A pre-trained image segmentation network is used to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with the first resolution; The initial segmented image is backsampled to restore it to the same resolution as the image to be segmented, thus obtaining the final segmented image; The image segmentation network includes a learnable non-uniform image sampling module and a segmentation module. The step of using a pre-trained image segmentation network to perform non-uniform sampling and image segmentation processing on the image to be segmented to obtain an initial segmented image with the first resolution includes: The learnable non-uniform image sampling module is used to predict the importance index of each pixel after sampling the image to be segmented to the first resolution, and a pixel-wise sampling density map is generated. The image to be segmented is then sampled to the first resolution according to the pixel-wise sampling density map to obtain a non-uniform sampling image, wherein the sampling density of each pixel is proportional to its importance index. The non-uniformly sampled image is input into the segmentation module to perform image segmentation, thereby obtaining the initial segmented image.

2. The image segmentation method as described in claim 1, characterized in that, The learnable non-uniform image sampling module includes a localization network, a compression network, and a non-uniform consistency sampler. The step of using the learnable non-uniform image sampling module to predict the importance index of each pixel after sampling the image to be segmented to the first resolution, and generating a pixel-by-pixel sampling density map, and then sampling the image to be segmented to the first resolution based on the pixel-by-pixel sampling density map to obtain a non-uniform sampled image, includes: A scaling operation is performed on the image to be segmented to obtain a scaled image with the first resolution; The scaled image is input into the localization network to determine the importance of each pixel in the scaled image, thereby obtaining a localization map of important regions with the first resolution; The important region location map is input into the compression network to reduce the number of channels to 1, thereby obtaining a region saliency map. The pixel value of each pixel in the region saliency map represents the saliency of that pixel. A pixel-by-pixel sampling density map is generated based on the region saliency map, wherein the sampling density of each pixel is proportional to its saliency; The image to be segmented and the pixel-by-pixel sampling density map are input into the non-uniform consistency sampler to sample the pixel values ​​of the image to be segmented to the corresponding positions of the non-uniform sampling image according to the pixel-by-pixel sampling density map.

3. The image segmentation method as described in claim 2, characterized in that, The localization network includes multiple standard convolutional layers, and the number of output channels of the first standard convolutional layer is greater than the number of input channels. The number of output channels of each standard convolutional layer first increases or remains unchanged and then decreases. The standard convolutional layer includes a convolutional layer, a batch normalization layer and an activation layer connected in sequence.

4. The image segmentation method as described in claim 3, characterized in that, The localization network includes three standard convolutional layers, and the number of output channels of the three standard convolutional layers are μ×c, μ×c and c, respectively, where c represents the number of channels of the scaled image, and μ is a preset value and is an integer greater than 0.

5. The image segmentation method as described in claim 2, characterized in that, The compressed network includes a 1×1 standard convolutional layer, wherein the 1×1 standard convolutional layer comprises a 1×1 convolutional layer, a batch normalization layer and an activation layer connected in sequence.

6. The image segmentation method as described in claim 2, characterized in that, The step of generating a pixel-wise sampling density map based on the region saliency map includes: normalizing the region saliency map in the channel dimension using a softmax function to obtain the pixel-wise sampling density map.

7. The image segmentation method as described in claim 1, characterized in that, The image segmentation network is trained according to the following steps: Obtain the training sample images and their corresponding segmentation labels, as well as the second resolution; The learnable non-uniform image sampling module is used to predict the importance index of each pixel after the training sample image is sampled to the second resolution, and a pixel-by-pixel sampling density map of the training sample is generated. The training sample image is sampled to the second resolution according to the pixel-by-pixel sampling density map of the training sample to obtain a non-uniform sampling image of the training sample, wherein the sampling density of each pixel is proportional to its importance index. The non-uniformly sampled image of the training sample is input into the segmentation module to perform image segmentation, thereby obtaining the initial segmented image of the training sample. The segmentation marker map is sampled to the second resolution based on the pixel-by-pixel sampling density map of the training samples to obtain a sampling marker map, and a sampling density marker map is generated based on the sampling marker map; The image segmentation network is trained according to a total loss function, which is determined by a first loss function and a second loss function. The first loss function is the loss function of the image segmentation task performed by the segmentation module, representing the error between the initial segmented image of the training samples and the sampling label map. The second loss function represents the error between the pixel-by-pixel sampling density map of the training samples and the sampling density label map.

8. The image segmentation method as described in claim 7, characterized in that, The expression for the second loss function is: L2=MSE(d m ,d GT ) Where MSE() represents the mean squared error loss function, d m d represents the pixel-by-pixel sampling density map of the training samples. GT This represents the sampling density marker map.

9. The image segmentation method as described in claim 7 or 8, characterized in that, The sampling density marker map is determined by the following expression: d GT =Edge(Gauss(l a )), Among them l a The sampling marker map is represented by Gauss(), which represents the Gaussian smoothing function, and Edge(), which represents the edge detection function.

10. The image segmentation method as described in claim 7, characterized in that, The expression for the total loss function is: L = αL1 + βL2, Where L1 represents the first loss function, L2 represents the second loss function, and α and β are preset weight coefficients.

11. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the image segmentation method as described in any one of claims 1-10.

Images