A target image segmentation method and system

Abstract

The present application provides a method and system for target image segmentation, which relates to the technical field of image processing. The method includes: obtaining an image to be segmented containing several small targets, and performing preprocessing to obtain a processed segmented image, recorded as a first image; constructing a small target segmentation model; collecting historical images containing several small target images, and using the historical images to train the small target segmentation model; importing the first image into the trained small target segmentation model to obtain a binary mask of the first image, performing a bitwise AND operation on the binary mask of the first image and the first image, extracting the small target area, and completing the image segmentation task. The present application combines the U‑Net sub-model with the instance segmentation sub-model, which can effectively capture the features of small targets and perform independent segmentation processing on each small target, thereby greatly enhancing the segmentation accuracy and achieving accurate capture and independent segmentation of small target features.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a target image segmentation method and system. Background Art

[0002] Image segmentation is a core task in the field of computer vision, which aims to divide an image into several meaningful regions or objects to facilitate deeper analysis and processing of the image. This technology plays an important role in many application fields such as image analysis, computer vision, and medical image processing. Among them, how to accurately segment small targets is a research direction that has attracted much attention. Since small targets often have low resolution, blurred edges, and complex backgrounds, the accuracy of small target image segmentation faces great challenges.

[0003] The Chinese invention patent with application publication date of May 15, 2020 and application publication number CN 111612807A provides a small target image segmentation method based on scale and edge information. On the basis of the segmentation model of the encoder-decoder structure, an edge coding module is added. The module takes the last convolution layer of the encoder partial convolution block as input and is composed of an attention module and a gating module alternately connected. The patent also introduces global attention to increase the receptive field of the image, effectively avoiding the situation of ignoring the edge information of the small target area. At the same time, by weighting the gradient map of the input image label, the loss of edge information in the calculation process is reduced.

[0004] Regarding the above technical solution, there are the following deficiencies when processing images containing multiple overlapping small targets: on the one hand, weighted processing of the gradient map of the input image label will cause the gradient map to become complicated, making it difficult to accurately represent the edges of overlapping small targets, thereby affecting the segmentation accuracy of the small target image; on the other hand, the introduction of the global attention mechanism to increase the receptive field of the image will cause the model to be unable to effectively focus on each small target when processing images containing multiple overlapping small targets, thereby resulting in a decrease in the segmentation accuracy of the small target image. Summary of the invention

[0005] In order to improve the accuracy of small target segmentation, the present application provides a target image segmentation method and system.

[0006] In the first aspect, the present application provides a target image segmentation method, which adopts the following technical solution:

[0007] A target image segmentation method, the method comprising:

[0008] Data processing: obtaining an image to be segmented containing several small targets, and performing preprocessing to obtain a processed segmented image, which is recorded as the first image;

[0009] Model construction: Build a small target segmentation model, which includes an edge detection sub-model, a U-Net network sub-model, and an instance segmentation sub-model connected in sequence;

[0010] The edge detection sub-model is used to obtain edge information of each small target in the first image, and to splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image;

[0011] The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image;

[0012] The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of the first image;

[0013] Model training: Collect historical images containing several small target images to obtain a historical image dataset, use the historical image dataset to train the small target segmentation model, and obtain a trained small target segmentation model;

[0014] Image segmentation: Import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image to extract the small target area and complete the image segmentation task.

[0015] By adopting the above technical solution, the edge detection sub-model is introduced into the small target segmentation model, which helps to clearly distinguish the target from the background in the target overlapping area. At the same time, by combining the U-Net sub-model with the instance segmentation sub-model, the characteristics of the small target can be effectively captured, and the effect of independently segmenting each small target can be achieved, thereby greatly enhancing the image segmentation accuracy and effectively reducing the confusion caused by the overlap of small targets. In addition, based on the binary mask generated by the instance segmentation sub-model, the binary mask is bitwise ANDed with the first image, so that the final segmentation result can accurately focus on the target area, even in the face of complex scenes with multiple overlapping small targets, it can provide accurate and reliable segmentation results. The above technical solution solves the problem of decreased segmentation accuracy when processing multiple overlapping small target images, realizes the accurate capture and independent segmentation of small target features, and greatly improves the ability to accurately extract small target images in complex backgrounds.

[0016] Optionally, the model building step further includes:

[0017] Processing the first image using an image scaling technique to obtain a plurality of scaled images of different scales;

[0018] Annotate the edge regions of the small objects in the obtained zoomed images of different scales to obtain the embedded region of each zoomed image;

[0019] The extracted edge information is respectively embedded into the embedding area of ​​each scaled image to obtain a number of embedded images of different scales, which are then collected to obtain a second image.

[0020] By adopting the above technical solution, the edge detection sub-model performs multi-scale enhancement and annotation on the edges of small targets in the image, so that the small target segmentation model can better adapt to small targets of different sizes and shapes, and improve the robustness of segmentation. At the same time, the embedding of multi-scale edge information not only retains the information of the original image, but also enhances the edge information to make the outline and features of the small target more prominent, providing a more accurate and reliable basis for subsequent segmentation tasks, and also making the model pay more attention to the edge part of the small target during the segmentation process, thereby reducing the possibility of mis-segmentation.

[0021] Optionally, in the model building step,

[0022] The U-Net network sub-model includes encoder, decoder and skip connection;

[0023] After receiving the second image, the encoder uses a convolution layer to extract a feature map of the second image to obtain a first feature map, performs a pooling operation on the first feature map, and obtains a second feature map after multiple layers of convolution operations and pooling operations;

[0024] The encoder passes the second feature map to the decoder via a skip connection;

[0025] The decoder uses the second feature map to obtain a third feature map through a deconvolution operation, concatenates the second feature map with the third feature map, and converts the concatenated feature map into a segmentation mask of the second image through a thresholding operation.

[0026] By adopting the above technical solutions, it is possible to deeply extract image features, and use pooling operations to effectively compress information, eliminate background noise, and accurately capture key information. In addition, the decoder performs a deconvolution operation to restore the low-resolution feature map to a high resolution, which can completely retain the detailed information of the target area. At the same time, the jump connection directly transfers the high-resolution features in the encoder to the decoder, further strengthening the retention of the target area, especially the edge information of small targets. In addition, the introduction of the jump connection cleverly integrates the feature maps of the encoder and decoder, providing the decoder with richer contextual information, and significantly improving the model's ability to distinguish small target areas. Even in complex backgrounds, the U-Net network submodel can effectively reduce the mis-segmentation of overlapping areas of small targets and improve the accuracy of small target segmentation.

[0027] Optionally, in the step of model training, the training method of the small object segmentation model includes:

[0028] Define a comprehensive loss function, import the historical image dataset into the constructed small target segmentation model, use the historical images in the historical image dataset to iteratively train the edge detection sub-model, U-Net network sub-model and instance segmentation sub-model, use the defined comprehensive loss function to calculate the loss value of the small target segmentation model, use the back propagation algorithm to calculate the model parameter gradient of the small target segmentation model, and use the gradient descent method to update the model parameters of the small target segmentation model until the preset number of iterative training times is completed or the calculated loss value of the small target segmentation model no longer decreases.

[0029] By adopting the above technical solution and training on historical image data sets, the small target segmentation model can learn segmentation rules in various complex backgrounds and when multiple small targets overlap, thereby improving the segmentation accuracy of the small target segmentation model. In addition, the synergy of the edge detection submodel, U-Net network submodel, and instance segmentation submodel in the small target segmentation model improves the sensitivity of the small target segmentation model to the edges of small targets and its ability to handle overlapping targets. The trained small target segmentation model can adapt well to image segmentation tasks in different scenarios and conditions. Whether it is a complex background or multiple small targets overlap, the trained small target segmentation model can accurately segment small target images.

[0030] Optionally, in the training method of the small object segmentation model,

[0031] The instance segmentation sub-model includes an RPN network model, and the RPN network model includes a self-attention mechanism module;

[0032] The edge detection sub-model extracts edge information of each small target in the historical image, and splices the extracted edge information with the corresponding historical image to obtain a spliced ​​image, which is recorded as the second historical image. The U-Net network sub-model obtains a feature map of the second historical image, and generates a segmentation mask of the second historical image according to the feature map of the second historical image.

[0033] The RPN network model maps the feature map of the second historical image into Query, Key, and Value respectively;

[0034] Calculate the similarity between each query and each key, and integrate the calculation results into a similarity matrix between query and key;

[0035] The similarity matrix is ​​normalized using the Softmax function, and the similarity values ​​in the normalized similarity matrix are used as the attention weights of the self-attention mechanism module;

[0036] Use the attention weight to perform weighted summation on each Value to obtain a weighted feature map;

[0037] The RPN network model traverses the weighted feature map to obtain the local features of each pixel, and uses the local features of each pixel to obtain several candidate regions;

[0038] The candidate areas are classified, and the candidate areas classified as small targets are recorded as target candidate areas. The feature map corresponding to the target candidate areas is pooled to obtain the first historical target feature map. The segmentation mask of the second historical image is concatenated with the first historical target feature map to obtain the second historical target feature map. The second historical target feature map is convolved to obtain an output feature map. The probability value of each pixel value in the output feature map belonging to the target area is calculated by the sigmoid activation function. The output feature map is binarized according to the obtained probability value to obtain a binary mask corresponding to each target candidate area.

[0039] By adopting the above technical solution, adding a self-attention mechanism in the RPN network model can calculate the correlation between each pixel and other pixels, thereby capturing the global information and subtle features in the image, helping the small target segmentation model to better understand the relative position and relationship between targets, and helping the small target segmentation model to cut overlapping targets as a whole during the segmentation process, thereby improving image segmentation accuracy. Moreover, the added self-attention mechanism can help the small target segmentation model to consider more contextual information when generating candidate regions, thereby generating more accurate and tighter candidate regions, so that the small target segmentation model can better understand the contextual information in the image when dealing with complex backgrounds or multiple small target overlaps, improve the ability to express features, and also help the small target segmentation model to better identify targets in subsequent classification and regression tasks, improving the accuracy and robustness of target detection.

[0040] Optionally, after obtaining a number of candidate regions and before classifying the candidate regions, the method further includes:

[0041] The RPN network model also includes a bounding box regression network;

[0042] The obtained candidate region is recorded as the first candidate region, the predicted bounding box and the true bounding box of each first candidate region are obtained, the IOU threshold of each first candidate region is calculated, and according to the calculation result, it is determined whether there is a first candidate region whose IOU threshold is less than or equal to the first preset threshold:

[0043] If so, according to the formula , calculate the predicted IOU threshold, compare the IOU threshold of each first candidate area with the predicted IOU threshold, record the first candidate area whose IOU threshold is greater than the predicted IOU threshold as the second candidate area, record the candidate area whose IOU threshold is less than or equal to the predicted IOU threshold as the third candidate area, use the bounding box regression network to calculate the predicted bounding box adjustment amount of the second candidate area, and adjust the predicted bounding box of the second candidate area according to the calculation result, record the adjusted second candidate area and the third candidate area as the fourth candidate area, and use the fourth candidate area as the new candidate area;

[0044] If not, the first candidate region is used as a new candidate region;

[0045] in, Indicates the predicted IOU threshold, represents the adjustment factor, Represents the standard IOU threshold.

[0046] By adopting the above technical solution, an IOU threshold screening mechanism is introduced, and candidate areas are screened according to the IOU threshold, which can effectively remove low-quality candidate areas with low overlap with the true target bounding box and retain high-quality second candidate areas. At the same time, the predicted bounding box of each second candidate area is adjusted using a bounding box regression network, which can further improve the accuracy and tightness of the candidate area, making it closer to the bounding box of the true target, thereby improving the quality of the candidate area, reducing the false detection and missed detection of small target images, and improving the recall rate and precision of target detection. When the small target segmentation model processes images with complex backgrounds or multiple overlapping small targets, it can more accurately identify the position and range of small targets, thereby improving the accuracy and robustness of small target detection.

[0047] Optionally, the comprehensive loss function for:

[0048] ;

[0049] in, represents the segmentation loss function, represents the classification loss function, represents the bounding box regression loss function, represents the mask loss function, represents the marginal loss function, , , , is the weight coefficient.

[0050] By adopting the above technical scheme and combining multiple loss terms, different parts and performances of the small target segmentation model are optimized respectively. For example, segmentation loss and mask loss help to improve the small target segmentation model's ability to accurately capture the target area and shape; classification loss helps the small target segmentation model to correctly identify the target category; bounding box regression loss helps to improve the small target segmentation model's positioning accuracy of the target position; edge loss helps to improve the accuracy and precision of the small target segmentation model's edge detection. By considering these loss terms at the same time to train the small target segmentation model, the small target segmentation model can better adapt to target detection, segmentation and recognition tasks in complex scenes.

[0051] Optionally, in the step of image segmentation,

[0052] Importing the first image into the trained small object segmentation model to obtain an initial binary mask for each pixel in the first image;

[0053] The position of each pixel in the second image is traversed to obtain the value of the initial binary mask corresponding to each pixel in the first image, the initial binary mask of each pixel in the second image is merged, and the merged binary mask is recorded as a fused binary mask.

[0054] By adopting the above technical solution, the optimal segmentation result of each pixel at different scales can be retained, the segmentation error caused by a single scale can be reduced, and the binary masks of different scales can be fused to integrate the image information at different scales, which helps to more accurately identify the target area, enhance the robustness of small target segmentation, and improve the accuracy of small target segmentation.

[0055] Optionally, the image segmentation step further includes:

[0056] Construct a conditional random field model and define an energy function, which includes a data term and a smooth term;

[0057] Inputting the fused binary mask as a data item into the conditional random field model, calculating the similarity between each pair of adjacent pixel points, and calculating a smoothing item based on the similarity;

[0058] Each pixel is classified and labeled according to the classification result. The energy function is calculated according to the data term and the smoothness term. The gradient of the conditional random field model is calculated based on the label of each pixel. The energy function is minimized according to the gradient descent method. The label of each pixel is updated. The updated label of each pixel is converted through thresholding to obtain an optimized fused binary mask, which is recorded as the binary mask of the first image.

[0059] By adopting the above technical solution and inputting the fused binary mask as the data item into the conditional random field model, it is possible to make full use of multi-scale and multi-feature information, which helps to smooth the image segmentation boundaries, reduce noise and artifacts, reduce breaks and fragments in the segmentation results, improve the coherence and consistency of the segmentation results, further improve the segmentation quality, and improve the accuracy of small target image segmentation.

[0060] In a second aspect, the present application provides a target image segmentation system, the segmentation system is applicable to the segmentation method described in any one of the first aspects, and the segmentation system includes: adopting the following technical solution:

[0061] A target image segmentation system, the system comprising:

[0062] A data processing module is used to obtain an image to be segmented containing a plurality of small targets, and perform preprocessing to obtain a processed segmented image, which is recorded as a first image;

[0063] A model building module is used to build a small object segmentation model. The small object segmentation model includes an edge detection sub-model, a U-Net network sub-model and an instance segmentation sub-model connected in sequence.

[0064] The edge detection sub-model is used to obtain edge information of each small target in the first image, and to splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image;

[0065] The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image;

[0066] The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of the first image;

[0067] The model training module is used to collect historical images containing a number of small target images to obtain a historical image data set, and use the historical image data set to train a small target segmentation model to obtain a trained small target segmentation model;

[0068] The image segmentation module is used to import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image, extract the small target area, and complete the image segmentation task.

[0069] By adopting the above technical solution, the edge detection sub-model, U-Net sub-model and instance segmentation sub-model in the trained small target segmentation model in the model training module can be used to effectively capture the features of the small target after extracting the edge information of the small target image, and achieve the technical effect of performing independent segmentation processing on each small target, thereby greatly enhancing the segmentation accuracy and effectively reducing the confusion caused by the overlap of small targets. In addition, in the image segmentation module, the obtained binary mask is used to perform a bitwise AND operation with the first image, so that the final segmentation result can accurately focus on the target area, and even in the face of complex scenes with multiple overlapping small targets, accurate and reliable segmentation results can be provided.

[0070] In summary, the present application includes at least one of the following beneficial technical effects:

[0071] 1. Construct a small target segmentation model, introduce an edge detection sub-model, and combine the U-Net sub-model with the instance segmentation sub-model. In the overlapping area of ​​the image, the features of small targets can be effectively captured, and each small target can be independently segmented, thereby greatly enhancing the segmentation accuracy and effectively reducing the confusion caused by the overlap of small targets. Secondly, a binary mask is used to perform a bitwise AND operation with the first image, so that the final segmentation result can accurately focus on the target area, and even in the face of complex scenes with multiple overlapping small targets, it can provide accurate and reliable segmentation results. This technical solution solves the problem of decreased segmentation accuracy when processing multiple overlapping images of small targets, realizes the accurate capture and independent segmentation of small target features, and greatly improves the ability to accurately extract small target areas in complex backgrounds.

[0072] 2. Adding a self-attention mechanism to the instance segmentation sub-model can calculate the correlation between each pixel and other pixels, thereby capturing the global information and subtle features in the image, helping the small target segmentation model to better understand the relative position and relationship between targets, and helping the small target segmentation model to cut overlapping targets as a whole during the segmentation process, thereby improving the image segmentation accuracy. Moreover, the added self-attention mechanism can help the small target segmentation model to consider more contextual information when generating candidate regions, thereby generating more accurate and tighter candidate regions, so that the small target segmentation model can better understand the contextual information in the image when dealing with complex backgrounds or multiple overlapping small targets, improve the ability to express features, and also help the small target segmentation model to better identify targets in subsequent classification and regression tasks, thereby improving the accuracy and robustness of target detection.

[0073] 3. When the instance segmentation sub-model generates candidate regions, an IOU threshold screening mechanism is introduced. The candidate regions are screened according to the IOU threshold, which can effectively remove low-quality candidate regions with low overlap with the true target bounding box and retain high-quality second candidate regions. At the same time, the predicted bounding box of each second candidate region is adjusted using the bounding box regression network, which can further improve the accuracy and tightness of the candidate region, making it closer to the bounding box of the true target, improving the quality of the candidate region, reducing the false detection and missed detection of small target images, and improving the recall rate and precision of target detection. When the small target segmentation model processes images with complex backgrounds or multiple overlapping small targets, it can more accurately identify the position and range of small targets, thereby improving the accuracy and robustness of small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of Example 1 of the present application;

[0075] Figure 2 is a flow chart of S1 data processing of Example 1 of the present application;

[0076] Figure 3 This is a flow chart of the edge detection sub-model processing the first image in the S2 model construction of Example 1 of the present application;

[0077] Figure 4 This is a flowchart of the S3 model training of Example 1 of the present application;

[0078] Figure 5 This is a flowchart of S4 image segmentation of Example 1 of the present application. DETAILED DESCRIPTION

[0079] The following combination Figures 1 to 5 This application is described in further detail.

[0080] Embodiment 1: This embodiment discloses a target image segmentation method, such as Figure 1 As shown, the method includes:

[0081] S1 data processing: obtain an image to be segmented containing several small targets, and perform preprocessing to obtain a processed segmented image, which is recorded as the first image.

[0082] S2 model construction: Construct a small target segmentation model, which includes an edge detection sub-model, a U-Net network sub-model and an instance segmentation sub-model connected in sequence.

[0083] The edge detection sub-model is used to obtain edge information of each small target in the first image, and to splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image.

[0084] The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image.

[0085] The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of the first image.

[0086] S3 model training: Collect historical images containing several small target images to obtain a historical image dataset, use the historical image dataset to train the small target segmentation model, and obtain a trained small target segmentation model.

[0087] S4 image segmentation: Import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image to extract the small target area and complete the image segmentation task.

[0088] This embodiment introduces an edge detection sub-model, a U-Net sub-model and an instance segmentation sub-model into the small target segmentation model, which can effectively capture the characteristics of small targets and achieve the technical effect of independently segmenting each small target, thereby greatly enhancing the segmentation accuracy and effectively reducing the confusion caused by the overlap of small targets.

[0089] like Figure 2 As shown, S1 data processing includes:

[0090] S11 Image annotation: Use image annotation tools to annotate small objects in the image to be segmented, and obtain the annotated image to be segmented.

[0091] S12 Image enhancement: Randomly select any three operations among flipping, rotating, cropping, brightness adjustment and noise addition to perform on the annotated image to be segmented to obtain an enhanced image to be segmented.

[0092] S13 Image normalization: Normalize the pixel values ​​of the image to be segmented to adapt to the input of the small object segmentation model.

[0093] like Figure 3 As shown, in other embodiments, in constructing the S2 model, the method for the edge detection sub-model to process the first image includes:

[0094] S211 Image scaling: The first image is processed using an image scaling technology to obtain a plurality of scaled images of different scales.

[0095] S212 Edge region labeling: labeling edge regions of small objects in the obtained several scaled images at different scales to obtain an embedded region of each scaled image.

[0096] S213 Edge information embedding: embedding the extracted edge information into the embedding area of ​​each scaled image respectively, obtaining a plurality of embedded images of different scales, and assembling them to obtain a second image.

[0097] In other embodiments, in S2 model construction, the U-Net network submodel includes an encoder, a decoder, and a skip connection.

[0098] After receiving the second image, the encoder uses the convolution layer to extract the feature map of the second image to obtain the first feature map of the second image, performs a pooling operation on the first feature map to reduce the resolution of the first feature map, and obtains the second feature map of the second image after multiple layers of convolution and pooling operations. The encoder passes the second feature map to the decoder through a jump connection. The decoder restores the resolution of the second feature map to the resolution of the second image through a deconvolution operation, recorded as a third feature map, and splices the second feature map with the third feature map to obtain a probability map, i.e., a spliced ​​feature map, and converts the probability map into a segmentation mask of the second image through a thresholding operation.

[0099] In other embodiments, in S3 model training, the method for obtaining the historical image dataset includes:

[0100] Collect historical images containing several small target images, use image annotation tools to annotate labels on the small targets that need to be segmented in the collected historical images, and obtain annotated historical images.

[0101] The annotated historical image is randomly selected to perform any three operations among flipping, rotating, cropping, brightness adjustment and noise addition to obtain the historical image after image enhancement.

[0102] The pixel values ​​of the historical images after image enhancement are normalized to adapt to the input of the small target segmentation model to obtain the processed historical images, and the processed historical images are collected to obtain a historical image dataset.

[0103] like Figure 4 As shown, in other embodiments, in S3 model training, the training method of the small object segmentation model includes:

[0104] S31 defines loss function: defines a comprehensive loss function, which includes segmentation loss function, classification loss function, bounding box regression loss function, mask loss function and edge loss function.

[0105] S32 Small target segmentation model training: The historical image dataset is imported into the constructed small target segmentation model. The edge detection sub-model extracts the edge information of each small target in the historical image, and splices the extracted edge information with the corresponding historical image to obtain the spliced ​​image, which is recorded as the second historical image. The U-Net network sub-model obtains the feature map of the second historical image, and generates a segmentation mask of the second historical image based on the feature map of the second historical image. The feature map of the second historical image is input into the RPN network model in the instance segmentation sub-model, and the RPN network model generates several candidate regions based on the feature map of the second historical image. The candidate regions are classified, and the candidate regions with classification results as small targets are recorded as target candidate regions. The feature map corresponding to each target candidate region is mapped to a fixed-size feature map of the same size through a pooling operation, which is recorded as the first historical target feature map. The segmentation mask of the second historical image is used as the attention map, the segmentation mask is aligned with the first historical target feature map, and the segmentation mask of the second historical image is spliced ​​with the first historical target feature map to obtain the second historical target feature map, wherein the splicing method of the segmentation mask of the second historical image and the first historical target feature map is to multiply the aligned segmentation mask with each pixel value of the first historical target feature map. The second historical target feature map is convolved using a fully convolutional network to obtain an output feature map, and the probability value of each pixel value in the output feature map belonging to the target area is calculated by the sigmoid activation function, and the output feature map is binarized according to the obtained probability value to obtain a binary mask corresponding to each target candidate area.

[0106] S33 calculates loss: uses the defined comprehensive loss function to calculate the loss value of the small target segmentation model.

[0107] S34 Backpropagation: Use the backpropagation algorithm to calculate the model parameter gradients of the small object segmentation model.

[0108] S35 parameter update: Use the gradient descent method to update the model parameters of the small object segmentation model.

[0109] S36 iterative training: loop through the above S32 small target segmentation model training to S35 parameter update until the preset number of iterative training times is completed or the calculated loss value of the small target segmentation model no longer decreases, completing the model training and obtaining a trained small target segmentation model.

[0110] In this embodiment, both the first image and the historical image in the historical image data set can be directly input into the small object segmentation model for data processing or training.

[0111] In this embodiment, the RPN network model includes a self-attention mechanism module. In the small target segmentation model training in S32, the RPN network model generates a plurality of candidate regions according to the feature map of the second historical image, and also includes:

[0112] The RPN network model maps the feature map of the second historical image into Query, Key, and Value respectively through multiple convolutional layers.

[0113] Calculate the similarity between each Query and each Key, and integrate all the calculated similarities into a similarity matrix between Query and Key.

[0114] The similarity matrix is ​​normalized using the Softmax function, and the similarity values ​​in the normalized similarity matrix are used as the attention weights of the self-attention mechanism module.

[0115] Perform weighted summation on each Value according to the attention weight to obtain a weighted feature map.

[0116] The convolution kernel of the RPN network model is used to traverse the weighted feature map to obtain the local features of each pixel, and several candidate regions are obtained using the local features of each pixel.

[0117] The RPN network model also includes a bounding box regression network. After obtaining several candidate regions and before classifying the candidate regions, it also includes:

[0118] A number of candidate regions are obtained, recorded as first candidate regions, and the predicted bounding box and the true bounding box of each first candidate region are obtained. The IOU threshold of each first candidate region is calculated, that is, the IOU threshold of the predicted bounding box and the true bounding box of each first candidate region is calculated. According to the calculated IOU threshold of the first candidate region, it is determined whether there is a first candidate region whose IOU threshold is less than or equal to the first preset threshold:

[0119] If there is a first candidate region whose IOU threshold is less than or equal to the first preset threshold, then according to the formula , calculate the required IOU threshold, recorded as the predicted IOU threshold, compare the IOU threshold of each first candidate area with the predicted IOU threshold, screen out the first candidate area with an IOU threshold greater than the predicted IOU threshold, record it as the second candidate area, screen out the candidate area with an IOU threshold less than or equal to the predicted IOU threshold, record it as the third candidate area, use the bounding box regression network to calculate the predicted bounding box adjustment amount of each second candidate area, and adjust the predicted bounding box of the second candidate area according to the calculated predicted bounding box adjustment amount of the second candidate area, record the adjusted second candidate area and the third candidate area as the fourth candidate area, take the fourth candidate area as the new candidate area, and classify it in the subsequent process.

[0120] If there is no first candidate region whose IOU threshold is less than or equal to the first preset threshold, the first candidate region is taken as a new candidate region and is classified in a subsequent process.

[0121] in, Indicates the predicted IOU threshold, represents the adjustment factor, Represents the standard IOU threshold, taking a fixed value, .

[0122] In this embodiment, the instance segmentation sub-model may adopt the Mask R-CNN model and the Cascade Mask R-CNN model, the self-attention mechanism module may adopt the non-local block, and the first preset threshold is set to 0.6.

[0123] When the instance segmentation sub-model uses the Mask R-CNN model and the self-attention mechanism module uses the non-local module, the non-local module is integrated on the RPN network model of the Mask R-CNN model, the input of the RPN network model is used as the input of the non-local module, and the output of the non-local module is used as the input of the convolutional layer of the RPN network model. After the U-Net network sub-model generates the segmentation mask of the second historical image, the Mask R-CNN model performs the following operations:

[0124] T1 feature map mapping: The dimension of the feature map of the second historical image is ,in, is the batch size, is the number of channels, , are the height and width of the feature map respectively. The feature map of the second historical image is input to the non-local module, and three convolutional layers are used to map the feature map of the second historical image to the Query matrix (Q), Key matrix (K), and Value matrix (V). The Query matrix is ​​the query matrix, and its dimension is , the Key matrix is ​​the key matrix, and its dimension is , the Value matrix is ​​the value matrix, the dimension is .

[0125] T2 calculates the similarity matrix: before calculation, The feature map of is flattened into The dimension of the Query matrix becomes , the dimension of the Key matrix becomes , the dimension of the Value matrix becomes , through the formula , calculate the similarity between each Query value in the Query matrix and each Key value in the Key matrix, and integrate all the calculated similarities into the Query and Key similarity matrix.

[0126] Assumptions , ;

[0127] ;

[0128] in, represents the element of the cth row of the Query matrix, Represents the elements in the dth row of the Key matrix.

[0129] Then the similarity matrix between Query and Key is:

[0130] .

[0131] T3 calculates the attention weight: Use the Softmax function to normalize the similarity matrix, and use the normalized value as the attention weight of the self-attention mechanism module.

[0132] The calculation formula is: .

[0133] in, is the attention weight.

[0134] T4 obtains the weighted feature map: performs weighted summation on each Value according to the attention weight to obtain the weighted feature map.

[0135] The calculation formula is: .

[0136] in, Indicates the use The output value after weighted summation of Value.

[0137] T5 generates candidate areas: The weighted feature map is passed to the convolution kernel of the RPN network model as the output of the non-local module. The convolution kernel traverses the weighted feature map to obtain the local features of each pixel. A candidate area is generated for each pixel, which is recorded as the first candidate area.

[0138] T6 Candidate Region Screening and Classification: Get the predicted bounding box and true bounding box of each first candidate region, according to the formula , calculate the IOU threshold of the predicted bounding box and the true bounding box of each first candidate region, and determine whether there is a The first candidate region.

[0139] If exists The first candidate region of , calculate the required IOU threshold, recorded as the predicted IOU threshold, compare the IOU threshold of each first candidate area with the predicted IOU threshold, screen out the first candidate area whose IOU threshold is greater than the predicted IOU threshold, recorded as the second candidate area, screen out the candidate area whose IOU threshold is less than or equal to the predicted IOU threshold, recorded as the third candidate area, use the bounding box regression network to calculate the predicted bounding box adjustment amount of each second candidate area, and adjust the predicted bounding box of the second candidate area according to the calculated predicted bounding box adjustment amount of the second candidate area, record the adjusted second candidate area and the third candidate area as the fourth candidate area, take the fourth candidate area as the new candidate area, and classify it in the subsequent process.

[0140] If not present If the first candidate region is selected, the first candidate region is taken as a new candidate region and is classified in the subsequent process.

[0141] in, Represents the intersection area of ​​the predicted bounding box and the true bounding box, Represents the area of ​​the union of the predicted bounding box and the true bounding box, Indicates the predicted IOU threshold, represents the adjustment factor, Represents the standard IOU threshold, taking a fixed value, .

[0142] T7 obtains the output feature map: the candidate area classified as a small target is recorded as the target candidate area, and the feature map corresponding to each target candidate area is mapped to a fixed-size feature map of the same size through a pooling operation, which is recorded as the first historical target feature map.

[0143] The segmentation mask of the second historical image is used as the attention map, the segmentation mask is aligned with the first historical target feature map, the pixel value of the aligned segmentation mask is multiplied by each pixel value of the first historical target feature map to obtain the second historical target feature map, and a full convolutional network is used to perform a convolution operation on the second historical target feature map to obtain an output feature map.

[0144] T8 obtains the binary mask: the probability value of each pixel value in the output feature map belonging to the target area is calculated through the sigmoid activation function, and the output feature map is binarized according to the obtained probability value to obtain the binary mask corresponding to each target candidate area.

[0145] In this embodiment, we can try to adjust the factor by cross validation or grid search. Under different values, evaluate the performance of the small object segmentation model and select the best value.

[0146] In this embodiment, in the output feature map obtained by T7, by using the segmentation mask as an attention map combined with the target feature map, the interference of complex background can be reduced, which helps the instance segmentation sub-model to clearly focus on the small target image and its edges, thereby enhancing the precise positioning and segmentation of small targets. In addition, since the segmentation mask usually has a significant probability difference between the target area and the background, using the segmentation mask as an attention map helps the instance segmentation sub-model to focus on the boundaries and details of the small target based on the high probability areas of the segmentation mask, thereby improving the recognition accuracy of the boundaries of small targets.

[0147] In S33 loss calculation, the comprehensive loss function includes the segmentation loss function , classification loss function , Bounding box regression loss function , mask loss function And the marginal loss function .

[0148] The formula of the comprehensive loss function is:

[0149] ;

[0150] ;

[0151] ;

[0152] ;

[0153] ;

[0154] .

[0155] Where N is the number of historical images in the historical image dataset, is the first sub-model in the U-Net network The true labels of historical images, is the first sub-model in the U-Net network The probability of the predicted label of a historical image, M is the number of historical image categories in the historical image dataset, is the first The true category labels of historical images, is the first The probability of predicting the category label for a historical image, R is the number of true bounding boxes of historical images in the historical image dataset, is the first The coordinates of the ground-truth bounding boxes of the historical images, is the first The coordinates of the predicted bounding box of the historical image, H is the height of the binary mask, W is the width of the binary mask, is the pixel point in the Mask R-CNN model The real binary mask value of is the pixel point in the MaskR-CNN model The predicted binary mask of , P is the number of pixels of edge information of historical images in the Mask R-CNN model, is the pixel point in the Mask R-CNN model The pixel value of the real edge information, is the pixel point in the MaskR-CNN model The pixel value of the predicted edge information, , , , is the weight coefficient, .

[0156] In this embodiment, cross validation or leave-out method can be used to evaluate the small object segmentation model. , , , The performance of different weight settings and choose the best performing one , , , Weighted combination.

[0157] In this embodiment, the above steps combine multiple loss terms by performing model training on historical image datasets, so that the small target segmentation model can learn the segmentation rules under various complex backgrounds and multiple overlapping small targets, thereby improving the segmentation accuracy of the small target segmentation model.

[0158] like Figure 5 As shown, in other embodiments, S4 image segmentation includes:

[0159] S41 model reasoning: import the first image into the trained small object segmentation model to obtain an initial binary mask for each pixel in the first image.

[0160] S42 Binary mask merging: traverse the position of each pixel in the second image, obtain the value of the initial binary mask corresponding to each pixel in the first image, perform a maximum merging operation, merge the initial binary mask of each pixel in the second image, and record the merged binary mask as a fused binary mask.

[0161] The maximum merging operation is to compare the binary mask value of a pixel point in a scaled image in the second image with the binary mask values ​​of the pixel point in other scaled images, and select the maximum value as the binary mask value of the pixel point after merging.

[0162] For example, after performing S1 data processing on an image Z containing a small target, a processed Z image is obtained, and the processed Z image is input into the small target segmentation model. After the processed Z image is input into the edge detection sub-model, five embedded images of different scales are obtained after processing by the edge detection sub-model, which are recorded as Z1 image, Z2 image, Z3 image, Z4 image and Z5 image, and the Z scale image set is obtained. After that, the U-Net network sub-model processes each image in the Z scale image set to obtain the segmentation mask of the Z1 image and the feature map of the Z1 image, the segmentation mask of the Z2 image and the feature map of the Z2 image, the segmentation mask of the Z3 image and the feature map of the Z3 image, the segmentation mask of the Z4 image and the feature map of the Z4 image, and the segmentation mask of the Z5 image and the feature map of the Z5 image. The instance segmentation submodel performs instance segmentation processing on the segmentation mask and feature map of each image in the Z-scale image set obtained by the U-Net network submodel, and obtains the initial binary mask of the Z1 image, the initial binary mask of the Z2 image, the initial binary mask of the Z3 image, the initial binary mask of the Z4 image, and the initial binary mask of the Z5 image.

[0163] Use a double loop to traverse the position of each pixel in the Z1 image, the Z2 image, the Z3 image, the Z4 image, and the Z5 image, obtain the value of the initial binary mask corresponding to each pixel in the Z1 image, the Z2 image, the Z3 image, the Z4 image, and the Z5 image, and perform a maximum merging operation on each pixel in the Z image, which is to select the position of a certain pixel in the Z image, compare the initial binary mask values ​​of the pixel position in the Z1 image, the Z2 image, the Z3 image, the Z4 image, and the Z5 image, and select the maximum value as the binary mask value of the pixel in the Z image. The binary mask obtained by merging the initial binary mask values ​​of the Z1 image, the Z2 image, the Z3 image, the Z4 image, and the Z5 image is recorded as the fused binary mask of the Z image.

[0164] The double loop is an outer loop that traverses rows, and an inner loop that traverses columns.

[0165] S43 Conditional Random Field Model Construction: Construct a conditional random field model and define an energy function, which includes data terms and smooth terms.

[0166] S44 calculates the smoothness term: the fused binary mask is input into the conditional random field model as a data item, the color similarity between each pair of adjacent pixels is calculated through the Euclidean distance, and the smoothness term is calculated according to the color similarity between each pair of adjacent pixels.

[0167] S45 obtains pixel labels: classifies each pixel, and labels each pixel according to the classification result to obtain a label for each pixel.

[0168] S46 optimizes the fused binary mask: obtains pixel labels, calculates the energy function based on the data term and the smooth term, calculates the gradient of the conditional random field model based on the label of each pixel, minimizes the energy function according to the gradient descent method, updates each pixel label until the energy function no longer decreases, and transforms the updated label of each pixel through thresholding processing to obtain an optimized fused binary mask, which is recorded as the binary mask of the first image.

[0169] The threshold processing is that when the pixel label is 1, it means that the pixel belongs to the target area; when the pixel label is 0, it means that the pixel belongs to the background area.

[0170] S47 Small target segmentation: perform a bitwise AND operation on the binary mask of the first image and the first image to obtain a small target area in the first image, complete the small target image segmentation task, and output the small target area.

[0171] In the above steps, the bitwise AND operation is to compare each pixel point in the first image and the binary mask of the first image one by one. For the pixel point whose value in the binary mask of the first image is 1, the pixel point of the first image at the corresponding position is retained, and the pixel point of the first image at the corresponding position whose value in the binary mask of the first image is 0 is suppressed, that is, becomes zero, and the remaining area of ​​the first image is the small target area in the first image.

[0172] In this embodiment, the above steps use a conditional random field model to process the fused binary mask, which can smooth the small target segmentation boundaries, reduce noise and artifacts, reduce breaks and fragments in the segmentation results, improve the coherence and consistency of the segmentation results, further improve the segmentation quality, and improve the accuracy of small target image segmentation.

[0173] For example, a researcher needs to segment the vascular cell images in the bamboo stem slices. The specific steps are as follows:

[0174] 1. Obtain the image of vascular cells to be segmented:

[0175] A high-resolution image containing vascular cells in a bamboo stalk slice is obtained by a microscope or other equipment as the vascular cell image to be segmented, and an image annotation tool is used to annotate labels on the vascular cells to be segmented in the vascular cell image to be segmented.

[0176] The annotated vascular cell image is flipped, rotated, and noise added to obtain an image-enhanced vascular cell image.

[0177] The pixel values ​​of the enhanced vascular cell image are normalized, and the obtained image is recorded as the first cell image.

[0178] 2. Import the first cell image into the trained small target segmentation model to obtain the binary mask of the first cell image:

[0179] After the first cell image is imported into the trained small target segmentation model, the edge detection sub-model obtains the edge information of each small target in the first cell image, and splices the extracted edge information with the first cell image to obtain the spliced ​​image, which is recorded as the second cell image. The U-Net network sub-model obtains the feature map of the second cell image, and uses the feature map of the second cell image to generate the segmentation mask of the second cell image. The instance segmentation sub-model performs instance segmentation processing on the feature map of the second cell image and the segmentation mask of the second cell image to obtain the binary mask of each vascular cell in the first cell image.

[0180] 3. Segment the first cell image to obtain the vascular cell image:

[0181] The binary mask of the first cell image is compared with each pixel in the first cell image one by one. For the pixel whose median value in the binary mask of the first cell image is 1, the pixel of the first cell image at the corresponding position is retained; and the pixel of the first cell image at the corresponding position whose median value in the binary mask of the first cell image is 0 is suppressed, that is, becomes zero, and the remaining area of ​​the first cell image is the vascular cell image, thus completing the segmentation task.

[0182] In this embodiment, the bamboo stalk slice image is input into the constructed small target segmentation model. Through the processing of the edge detection sub-model, the U-Net sub-model and the instance segmentation sub-model, the area of ​​vascular cells is accurately focused, and the image features of vascular cells are accurately captured in the bamboo stalk slice image where multiple vascular cells overlap, thereby achieving accurate segmentation of vascular cells.

[0183] Embodiment 2: This embodiment discloses a target image segmentation system, the system comprising:

[0184] The data processing module obtains an image to be segmented containing several small targets, and uses an image annotation tool to annotate the small targets to be segmented in the image to be segmented, thereby obtaining an annotated image to be segmented. The annotated image to be segmented is randomly subjected to any three operations of flipping, rotating, cropping, brightness adjustment, and noise addition, thereby obtaining an image to be segmented after image enhancement. The pixel values ​​of the image to be segmented are normalized to obtain a processed segmented image, which is recorded as the first image.

[0185] A model building module is used to build a small target segmentation model, which includes an edge detection sub-model, a U-Net network sub-model and an instance segmentation sub-model connected in sequence; the edge detection sub-model is used to obtain edge information of each small target in the first image, and splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image.

[0186] The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image.

[0187] The U-Net network sub-model includes an encoder, a decoder, and skip connections.

[0188] Among them, after the encoder receives the second image, the convolution layer extracts the feature map of the second image to obtain the first feature map of the second image, performs a pooling operation on the first feature map to reduce the resolution of the first feature map, and obtains the second feature map of the second image after multiple layers of convolution and pooling operations. The encoder passes the second feature map to the decoder through a jump connection. The decoder restores the resolution of the second feature map to the resolution of the second image through a deconvolution operation, recorded as a third feature map, concatenates the second feature map with the third feature map to obtain a probability map, and converts the probability map into a segmentation mask of the second image through a thresholding operation.

[0189] The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of each small object in the first image.

[0190] The instance segmentation sub-model generates several candidate regions according to the feature map of the second image. The generated candidate regions are classified, and the candidate regions with classification results as small targets are recorded as target candidate regions. The feature map corresponding to each target candidate region is mapped to the first target feature map through pooling operation. The segmentation mask of the second image is used as the attention map, the segmentation mask is aligned with the first target feature map, and the segmentation mask of the second historical image is spliced ​​with the first historical target feature map to obtain the second historical target feature map, wherein the splicing method of the segmentation mask of the second historical image and the first historical target feature map is to multiply the aligned segmentation mask with each pixel value of the first historical target feature map. The second target feature map is convolved using a fully convolutional network to obtain an output feature map, and the probability value of each pixel value in the output feature map belonging to the target region is calculated by the sigmoid activation function. The output feature map is binarized according to the obtained probability value to obtain a binary mask corresponding to each target candidate region.

[0191] The model training module is used to collect historical images containing several small target images to obtain a historical image data set, and use the historical image data set to train the small target segmentation model to obtain a trained small target segmentation model.

[0192] Define a comprehensive loss function, import the historical image dataset into the constructed small target segmentation model, use the historical images in the historical image dataset to iteratively train the edge detection sub-model, U-Net network sub-model and instance segmentation sub-model, use the defined comprehensive loss function to calculate the loss value of the small target segmentation model, use the back propagation algorithm to calculate the model parameter gradient of the small target segmentation model, use the gradient descent method to update the model parameters of the small target segmentation model, until the preset number of iterative training times is completed or the calculated loss value of the small target segmentation model no longer decreases, thus obtaining a trained small target segmentation model.

[0193] The image segmentation module is used to import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image, extract the small target area, and complete the image segmentation task.

[0194] The first image is imported into the trained small object segmentation model to obtain an initial binary mask for each pixel in the first image.

[0195] The position of each pixel in the second image is traversed to obtain the value of the initial binary mask corresponding to each pixel in the first image, and a maximum value merging operation is performed to merge the initial binary masks of each pixel in the second image, and the merged binary mask is recorded as a fused binary mask.

[0196] Construct a conditional random field model and define an energy function, which includes data terms and smooth terms.

[0197] The fused binary mask is input as a data item into the conditional random field model, and the similarity between each pair of adjacent pixel points is calculated, and the smoothness item is calculated based on the similarity.

[0198] Each pixel is classified and labeled according to the classification result. The energy function is calculated according to the data term and the smoothness term. The gradient of the conditional random field model is calculated based on the label of each pixel. The energy function is minimized according to the gradient descent method. The label of each pixel is updated. The updated label of each pixel is converted through thresholding to obtain an optimized fused binary mask, which is recorded as the binary mask of the first image.

[0199] In this embodiment, the edge detection sub-model, U-Net sub-model and instance segmentation sub-model in the small target segmentation model trained in the model training module can effectively capture the characteristics of the small targets in the first image, and realize independent segmentation processing for each small target, thereby greatly enhancing the segmentation accuracy and effectively reducing the confusion caused by the overlap of small targets.

[0200] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A target image segmentation method, characterized in that: The method comprises: Data processing: obtaining an image to be segmented containing several small targets, and performing preprocessing to obtain a processed segmented image, which is recorded as the first image; Model construction: Build a small target segmentation model, which includes an edge detection sub-model, a U-Net network sub-model, and an instance segmentation sub-model connected in sequence; The edge detection sub-model is used to obtain edge information of each small target in the first image, and to splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image; The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image; The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of the first image; Model training: Collect historical images containing several small target images to obtain a historical image dataset, use the historical image dataset to train the small target segmentation model, and obtain a trained small target segmentation model; In the model training steps, the training method of the small object segmentation model includes: Define a comprehensive loss function, import the historical image dataset into the constructed small target segmentation model, use the historical images in the historical image dataset to iteratively train the edge detection sub-model, the U-Net network sub-model and the instance segmentation sub-model, use the defined comprehensive loss function to calculate the loss value of the small target segmentation model, use the back propagation algorithm to calculate the model parameter gradient of the small target segmentation model, and use the gradient descent method to update the model parameters of the small target segmentation model until the preset number of iterative training times is completed or the calculated loss value of the small target segmentation model no longer decreases; In the training method of the small target segmentation model, The instance segmentation sub-model includes an RPN network model, and the RPN network model includes a self-attention mechanism module; The edge detection sub-model extracts edge information of each small target in the historical image, and splices the extracted edge information with the corresponding historical image to obtain a spliced ​​image, which is recorded as the second historical image. The U-Net network sub-model obtains a feature map of the second historical image, and generates a segmentation mask of the second historical image according to the feature map of the second historical image. The RPN network model maps the feature map of the second historical image into Query, Key, and Value respectively; Calculate the similarity between each query and each key, and integrate the calculation results into a similarity matrix between query and key; The similarity matrix is ​​normalized using the Softmax function, and the similarity values ​​in the normalized similarity matrix are used as the attention weights of the self-attention mechanism module; Use the attention weight to perform weighted summation on each Value to obtain a weighted feature map; The RPN network model traverses the weighted feature map to obtain the local features of each pixel, and uses the local features of each pixel to obtain several candidate regions; Classify the candidate areas, record the candidate areas with classification results as small targets as target candidate areas, perform pooling operation on the feature map corresponding to the target candidate areas to obtain the first historical target feature map, concatenate the segmentation mask of the second historical image with the first historical target feature map to obtain the second historical target feature map, perform convolution operation on the second historical target feature map to obtain the output feature map, calculate the probability value of each pixel value in the output feature map belonging to the target area through the sigmoid activation function, binarize the output feature map according to the obtained probability value, and obtain the binary mask corresponding to each target candidate area; Image segmentation: Import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image to extract the small target area and complete the image segmentation task.

2. The target image segmentation method according to claim 1, characterized in that: The step of constructing the model also includes: Processing the first image using an image scaling technique to obtain a plurality of scaled images of different scales; Annotate the edge regions of the small objects in the obtained zoomed images of different scales to obtain the embedded region of each zoomed image; The extracted edge information is respectively embedded into the embedding area of ​​each scaled image to obtain a number of embedded images of different scales, which are then collected to obtain a second image.

3. The target image segmentation method according to claim 1, characterized in that: In the model building step, The U-Net network sub-model includes encoder, decoder and skip connection; After receiving the second image, the encoder uses a convolution layer to extract a feature map of the second image to obtain a first feature map, performs a pooling operation on the first feature map, and obtains a second feature map after multiple layers of convolution operations and pooling operations; The encoder passes the second feature map to the decoder via a skip connection; The decoder uses the second feature map to obtain a third feature map through a deconvolution operation, concatenates the second feature map with the third feature map, and converts the concatenated feature map into a segmentation mask of the second image through a thresholding operation.

4. The target image segmentation method according to claim 1, characterized in that: After obtaining several candidate regions and before classifying the candidate regions, the following steps are also included: The RPN network model also includes a bounding box regression network; The obtained candidate region is recorded as the first candidate region, the predicted bounding box and the true bounding box of each first candidate region are obtained, the IOU threshold of each first candidate region is calculated, and according to the calculation result, it is determined whether there is a first candidate region whose IOU threshold is less than or equal to the first preset threshold: If so, according to the formula , calculate the predicted IOU threshold, compare the IOU threshold of each first candidate area with the predicted IOU threshold, record the first candidate area whose IOU threshold is greater than the predicted IOU threshold as the second candidate area, record the candidate area whose IOU threshold is less than or equal to the predicted IOU threshold as the third candidate area, use the bounding box regression network to calculate the predicted bounding box adjustment amount of the second candidate area, and adjust the predicted bounding box of the second candidate area according to the calculation result, record the adjusted second candidate area and the third candidate area as the fourth candidate area, and use the fourth candidate area as the new candidate area; If not, the first candidate region is used as a new candidate region; in, Indicates the predicted IOU threshold, represents the adjustment factor, Represents the standard IOU threshold.

5. The target image segmentation method according to claim 1, characterized in that: The comprehensive loss function for: ； in, represents the segmentation loss function, represents the classification loss function, represents the bounding box regression loss function, represents the mask loss function, represents the marginal loss function, , , , is the weight coefficient.

6. The target image segmentation method according to claim 1, characterized in that: In the image segmentation step, Importing the first image into the trained small object segmentation model to obtain an initial binary mask for each pixel in the first image; The position of each pixel in the second image is traversed to obtain the value of the initial binary mask corresponding to each pixel in the first image, the initial binary mask of each pixel in the second image is merged, and the merged binary mask is recorded as a fused binary mask.

7. The target image segmentation method according to claim 6, characterized in that: The step of image segmentation also includes: Construct a conditional random field model and define an energy function, which includes a data term and a smooth term; Inputting the fused binary mask as a data item into the conditional random field model, calculating the similarity between each pair of adjacent pixel points, and calculating a smoothing item based on the similarity; Each pixel is classified and labeled according to the classification result. The energy function is calculated according to the data term and the smoothness term. The gradient of the conditional random field model is calculated based on the label of each pixel. The energy function is minimized according to the gradient descent method. The label of each pixel is updated. The updated label of each pixel is converted through thresholding to obtain an optimized fused binary mask, which is recorded as the binary mask of the first image.

8. A target image segmentation system, characterized in that: The segmentation system is applicable to the segmentation method according to any one of claims 1 to 7, and the segmentation system comprises: A data processing module is used to obtain an image to be segmented containing a plurality of small targets, and perform preprocessing to obtain a processed segmented image, which is recorded as a first image; A model building module is used to build a small object segmentation model. The small object segmentation model includes an edge detection sub-model, a U-Net network sub-model and an instance segmentation sub-model connected in sequence. The edge detection sub-model is used to obtain edge information of each small target in the first image, and to splice the extracted edge information with the first image to obtain a spliced ​​image, which is recorded as the second image; The U-Net network sub-model is used to obtain a feature map of the second image, and generate a segmentation mask of the second image using the feature map of the second image; The instance segmentation sub-model is used to perform instance segmentation processing using the feature map of the second image and the segmentation mask of the second image to obtain a binary mask of the first image; The model training module is used to collect historical images containing a number of small target images to obtain a historical image data set, and use the historical image data set to train a small target segmentation model to obtain a trained small target segmentation model; In the model training module, the training method of the small object segmentation model includes: Define a comprehensive loss function, import the historical image dataset into the constructed small target segmentation model, use the historical images in the historical image dataset to iteratively train the edge detection sub-model, the U-Net network sub-model and the instance segmentation sub-model, use the defined comprehensive loss function to calculate the loss value of the small target segmentation model, use the back propagation algorithm to calculate the model parameter gradient of the small target segmentation model, and use the gradient descent method to update the model parameters of the small target segmentation model until the preset number of iterative training times is completed or the calculated loss value of the small target segmentation model no longer decreases; In the training method of the small target segmentation model, The instance segmentation sub-model includes an RPN network model, and the RPN network model includes a self-attention mechanism module; The edge detection sub-model extracts edge information of each small target in the historical image, and splices the extracted edge information with the corresponding historical image to obtain a spliced ​​image, which is recorded as the second historical image. The U-Net network sub-model obtains a feature map of the second historical image, and generates a segmentation mask of the second historical image according to the feature map of the second historical image. The RPN network model maps the feature map of the second historical image into Query, Key, and Value respectively; Calculate the similarity between each query and each key, and integrate the calculation results into a similarity matrix between query and key; The similarity matrix is ​​normalized using the Softmax function, and the similarity values ​​in the normalized similarity matrix are used as the attention weights of the self-attention mechanism module; Use the attention weight to perform weighted summation on each Value to obtain a weighted feature map; The RPN network model traverses the weighted feature map to obtain the local features of each pixel, and uses the local features of each pixel to obtain several candidate regions; Classify the candidate areas, record the candidate areas with classification results as small targets as target candidate areas, perform pooling operation on the feature map corresponding to the target candidate areas to obtain the first historical target feature map, concatenate the segmentation mask of the second historical image with the first historical target feature map to obtain the second historical target feature map, perform convolution operation on the second historical target feature map to obtain the output feature map, calculate the probability value of each pixel value in the output feature map belonging to the target area through the sigmoid activation function, binarize the output feature map according to the obtained probability value, and obtain the binary mask corresponding to each target candidate area; The image segmentation module is used to import the first image into the trained small target segmentation model to obtain the binary mask of the first image, perform a bitwise AND operation on the binary mask of the first image and the first image, extract the small target area, and complete the image segmentation task.

Images