A method and apparatus for image instance segmentation

Abstract

This invention provides a method and apparatus for image instance segmentation. The method includes acquiring a visible light image of the object to be segmented and a corresponding binary edge image; extracting a second image feature from the binary edge image through convolution operations; fusing the first and second image features to obtain a third image feature; inputting the third image feature into a joint attention module, which adds spatial attention and coordinate attention to the third image feature to form a fourth image feature; and inputting the fourth image feature into a prediction module, which performs semantic prediction and mask prediction on the fourth image feature to output a segmentation scheme for the object in the visible light image. This invention effectively improves the network's ability to perceive boundary features and significantly enhances the network's instance segmentation performance.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more particularly to a method and apparatus for image instance segmentation. Background Technology

[0002] Different types of images contain different feature information. In recent years, computer vision tasks that use one image to guide another have become increasingly widespread. Binary edge images refer to the object edges in a binary image. They possess a certain degree of spatial continuity and serve as the boundary between the object and the background. Edges are a crucial source of information about objects, determining their characteristics. Therefore, using binary edge images to guide object segmentation in visible light images has become an effective method for improving performance. By utilizing binary edge images, more accurate edge features can be obtained, thereby enhancing object segmentation performance. In instance segmentation tasks, the edges of the target need to be accurately extracted. Using binary edge images to guide object segmentation can, to some extent, reduce the impact of noise in the image, improving segmentation accuracy. Simultaneously, edge structure and characteristics can help the model better understand the shape and structure of the object, thus improving segmentation robustness.

[0003] However, the drawback of existing technologies is that they are prone to incomplete segmentation or missed detection when dealing with targets with discontinuous or irregular edges. This limits the accurate acquisition of edge features in some practical applications, thus affecting the overall segmentation effect. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method and apparatus for image instance segmentation, which solves the limitations of existing technologies in processing targets with discontinuous or irregular edges.

[0005] According to a first aspect of the present invention, a method for image instance segmentation is provided, the method comprising the following steps:

[0006] Step S1: Obtain the visible light image to be segmented and the corresponding binary edge image; extract the first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; extract the second image feature of the binary edge image by convolution operation;

[0007] Step S2: Input the first image feature and the second image feature into the binary edge image guidance module. The binary edge image guidance module includes top-down and bottom-down fusion paths for fusing the first image feature and the second image feature to obtain the third image feature.

[0008] Step S3: Input the third image feature into the joint attention module, which is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature;

[0009] Step S4: Input the fourth image feature into the prediction module. The prediction module performs semantic prediction and mask prediction on the fourth image feature and outputs the segmentation scheme of the object in the visible light image.

[0010] Preferably, step S2 includes:

[0011] From the first image features, select hierarchical features from level 2 to level num in descending order of size. Input the hierarchical features from level 2 to level num and the second image features into the binary edge image guidance module, and process the level num hierarchical feature C. i Perform a 1×1 convolution to obtain P num Following the order of i decreasing from num-1 to 2, for P i+1 Upsampling, then applying the i-th level feature C i Perform a 1×1 convolution, and convert P i+1 The results of upsampling and C i The results of performing a 1×1 convolution are summed element-wise according to the corresponding pixel positions to obtain P. i ; Fuse P2 with the second image feature to obtain the fused image feature M2; Following the order of i increasing from 2 to num-1, process M... i Downsampling, for P i+1 Perform a 1×1 convolution, and convert M... i The results of downsampling and P i+1 The results of performing a 1×1 convolution are summed element-wise according to the corresponding pixel positions to obtain M. i+1 ; for M2 to M num Perform a 3×3 convolution to obtain N2 to N. num ; For N num Perform coordinate convolution to obtain N. num+1 ; where N2 to N num+1 is the output of the binary edge image guidance module, i.e., the third image feature; num is a natural number.

[0012] Preferably, in step S4, the prediction module includes a semantic category branch, a mask convolution kernel branch, and a mask feature branch. The semantic category branch is used to perform interpolation convolution operations on the input fourth image features to obtain the probability that the content corresponding to each grid cell belongs to the target category. The mask convolution kernel branch runs in parallel with the semantic category branch and is used to perform dynamic convolution calculation operations on the input fourth image features to obtain a prediction mask. The mask feature branch is used to perform convolution, batch normalized nonlinear activation, and adaptive interpolation operations on the input fourth image features to obtain a mask feature map. Based on the mask feature map, a segmentation scheme for objects in the visible light image is obtained.

[0013] Preferably, non-maximum suppression is applied to the output of the prediction module, including: obtaining all prediction scores from the prediction module output; performing non-maximum suppression on the top N prediction scores to obtain an IoU matrix of size N×N; performing a maximum value operation on the column directions of the IoU matrix; determining the mask to be retained based on the maximum value of each column of the IoU matrix; calculating the penalty of each prediction on other predictions using a linear function based on the prediction score of each prediction; fitting the prediction with the highest overlap to the suppression probability; calculating the attenuation factor using the penalty and suppression probability of each prediction; and obtaining the final prediction based on the attenuation factor.

[0014] Preferably, during the training process of the visible light image instance segmentation network model guided by binary edge images, a positive and negative sample label assignment strategy based on minimum loss matching is introduced. The positive and negative sample label assignment strategy based on minimum loss matching is as follows:

[0015] The hierarchical outputs N2 to N corresponding to the third image features output by the binary edge image guidance module are determined. num +1, determine the object size range corresponding to the output of each level. The object size ranges corresponding to each level are different and have no overlap. When an object has multiple possible sizes, each possible size corresponds to a level, and prediction is performed by the outputs of multiple levels corresponding to the object. In each corresponding level, by calculating the classification and segmentation loss between each grid in the real target box and the real target, the grid with the minimum comprehensive loss is found, and the grid with the minimum comprehensive loss is taken as a positive sample.

[0016] According to a second aspect of the present invention, an apparatus for image instance segmentation is provided, the apparatus comprising:

[0017] Feature extraction module: configured to acquire a visible light image to be segmented and a binary edge image corresponding to the visible light image; extract a first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; and extract a second image feature of the binary edge image by convolution operation.

[0018] Feature fusion module: configured to input the first image features and the second image features into the binary edge image guidance module, the binary edge image guidance module includes top-down and bottom-down fusion paths, used to fuse the first image features and the second image features to obtain a third image feature;

[0019] Attention Addition Module: Configured to input the third image feature into the joint attention module, the joint attention module is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature;

[0020] Segmentation module: configured to input the fourth image features into the prediction module, the prediction module performs semantic prediction and mask prediction on the fourth image features, and outputs a segmentation scheme for objects in the visible light image.

[0021] According to a third aspect of the present invention, a system for image instance segmentation is provided, comprising:

[0022] A processor is used to execute multiple instructions;

[0023] Memory, used to store multiple instructions;

[0024] The plurality of instructions are to be stored in the memory and loaded and executed by the processor as described above.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a plurality of instructions are stored therein; the plurality of instructions are configured to be loaded by a processor and executed as described above.

[0026] According to the above-described scheme of the present invention, the method of the present invention is implemented through a visible light image instance segmentation network model guided by binary edge images (BEIG-SOLO). This visible light image instance segmentation network model is guided by a binary edge image guidance module, which fuses binary edge image feature information and visible light image feature information. The binary edge image guidance module includes a binary edge image information extraction module and top-down and bottom-up fusion modules. Embedding the binary edge image guidance module into the output of the feature extraction network enhances the network's ability to perceive the boundary feature information of the visible light image. Furthermore, a positive and negative sample label assignment strategy based on minimum loss matching is set. By calculating the sum of the semantic classification and mask loss of all grids, the grid with the minimum comprehensive loss is labeled as a positive sample, thus mitigating the problem of poor segmentation performance caused by overlapping multiple objects to the greatest extent. Experimental results show that the visible light image instance segmentation network model based on binary edge images exhibits good instance segmentation performance on both the visible light image dataset and the COCO dataset.

[0027] The technical effects of this invention are as follows:

[0028] (1) This invention proposes a visible light image instance segmentation network model based on binary edge image guidance. The network adds a binary edge image guidance module and a positive and negative sample label allocation strategy based on minimum loss matching, which effectively improves the instance segmentation performance of the network.

[0029] (2) This invention proposes a binary edge image guidance module. This module integrates the feature information of the binary edge image into a multi-level feature fusion network, and adds a bottom-up fusion path to the multi-level feature fusion network, which can effectively improve the network's ability to perceive boundary features.

[0030] (3) This invention proposes a negative sample label allocation strategy based on minimum loss matching. This strategy can effectively alleviate the problem of poor segmentation results caused by multiple overlapping instance targets and improve the instance segmentation performance of the network.

[0031] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention, and the invention is illustrated by the following drawings. In the drawings:

[0033] Figure 1 This is a schematic flowchart of an image instance segmentation method according to one embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of a visible light image instance segmentation network model structure based on binary edge images, according to one embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the structure of a binary edge image guidance module according to one embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of an image instance segmentation apparatus according to one embodiment of the present invention. Detailed Implementation

[0037] First, combine Figure 1 This invention describes a method for image instance segmentation according to one embodiment of the present invention. For example... Figure 1 As shown, the method includes the following steps:

[0038] Step S1: Obtain the visible light image to be segmented and the corresponding binary edge image; extract the first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; extract the second image feature of the binary edge image by convolution operation;

[0039] Step S2: Input the first image feature and the second image feature into the binary edge image guidance module. The binary edge image guidance module includes top-down and bottom-down fusion paths for fusing the first image feature and the second image feature to obtain the third image feature.

[0040] Step S3: Input the third image feature into the joint attention module, which is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature;

[0041] Step S4: Input the fourth image feature into the prediction module. The prediction module performs semantic prediction and mask prediction on the fourth image feature and outputs the segmentation scheme of the object in the visible light image.

[0042] like Figure 2 As shown, the visible light image instance segmentation network model based on binary edge image guidance includes a feature extraction network, a binary edge image guidance module, a joint attention module, and a prediction module connected in sequence.

[0043] The feature extraction network uses a ResNet50 network to extract a first image feature from the visible light image. The first image feature is a multi-layer feature map with multiple levels of image features, and different levels focus on different objects and details.

[0044] In this embodiment, since it is impossible to obtain both visible light images and binary edge images of the same scene at the same time in practical applications, the BEIG-SOLO network is divided into a training phase and a testing phase. During the training phase, based on the visible light image dataset, an edge extraction algorithm is used to generate a corresponding binary edge image dataset. The guidance from these binary edge images improves the instance segmentation performance of the BEIG-SOLO network for visible light images. During the testing phase, the binary edge images are not required for guidance; the trained BEIG-SOLO network can be used directly to complete the instance segmentation task for visible light images. In the training phase, it is assumed that… (C, H, and W are the number of channels, height, and width of the input image, respectively) is the visible light image input to BEIG-SOLO. It is a binary edge image with BEIG-SOLO input, I Seg This is the output after instance splitting. First, I... Op The input feature extraction network extracts feature information to generate feature maps of different levels, resulting in C1 to C5. This process can be represented as follows:

[0045] C i =f fe (I Op ), i = 1~5 (Formula 1)

[0046] Among them, f fe (·) represents the implementation of the feature extraction network, C i The i represents the feature maps of different levels of the output of the feature extraction network, where i = 1 to 5 indicates that there are 5 levels of output.

[0047] In step S2, the first image feature and the second image feature are input into a binary edge image guidance module. The binary edge image guidance module includes top-down and bottom-down fusion paths for fusing the first image feature and the second image feature to obtain a third image feature, including:

[0048] From the first image features, select hierarchical features from level 2 to level num in descending order of size. Input the hierarchical features from level 2 to level num and the second image features into the binary edge image guidance module, and process the level num hierarchical feature C. i Perform a 1×1 convolution to obtain P num Following the order of i decreasing from num-1 to 2, for P i+1Upsampling, then applying the i-th level feature C i Perform a 1×1 convolution, and convert P i+1 The results of upsampling and C i The results of performing a 1×1 convolution are summed element-wise according to the corresponding pixel positions to obtain P. i ; Fuse P2 with the second image feature to obtain the fused image feature M2; Following the order of i increasing from 2 to num-1, process M... i Downsampling, for P i+1 Perform a 1×1 convolution, and convert M... i The results of downsampling and P i+1 The results of performing a 1×1 convolution are summed element-wise according to the corresponding pixel positions to obtain M. i+1 ; for M2 to M num Perform a 3×3 convolution to obtain N2 to N. num ; For N num Perform coordinate convolution to obtain N. num+1 ; where N2 to N num+1 It is the output of the binary edge image guidance module, i.e., the third image feature.

[0049] To use binary edge images to guide object segmentation in visible light images, reduce the impact of noise in the image, and improve segmentation accuracy, a method was designed as follows: Figure 3 The demonstration showcases a binary edge image guidance module. A binary edge image refers to the edge portion of a target within a binary image. It possesses a degree of spatial continuity and serves as the boundary between the target and the background. Edges are a crucial source of information about objects, determining their characteristics. By utilizing binary edge images, more precise edge features can be obtained, thereby improving object segmentation performance. In object segmentation tasks, accurately extracting the target's edges is essential. Using binary edge images to guide object segmentation can, to some extent, reduce the impact of noise in the image, improving segmentation accuracy. Furthermore, the edge structure and characteristics help the model better understand the object's shape and structure, thus enhancing segmentation robustness.

[0050] In this embodiment, to obtain better feature representation and incorporate feature information from the binary edge image, the generated multi-layer feature maps C1-C5 and the features of the binary edge image are input into the binary edge image guidance module for feature fusion, resulting in N2-N5. Through a feature fusion method combining top-down and bottom-up approaches, high-level and low-level feature maps containing different features are fused step-by-step, thereby combining semantic information and shallow information to obtain a more accurate feature representation. The binary edge image guidance module enables multi-scale prediction of targets of different sizes and can also alleviate the overlap problem of different instance targets to some extent. This process can be represented as:

[0051] N j =f BEIG (C i ,I Be ), i = 1~5, j = 2~5 (Formula 2)

[0052] Among them, f BEIG (·) indicates the binary edge image guidance module, C i N represents the feature maps at different levels input to the binary edge image guidance module. j The binary edge image guidance module outputs feature maps at different levels, where i = 1 to 5 indicates that there are 5 levels in the input and j = 2 to 5 indicates that there are 4 levels in the output.

[0053] In this embodiment, the highest-level feature map N output by the binary edge image guidance module is... num The purpose of coordinate convolution is to add x and y channel information to enhance the processing of positional information. This process can be represented as:

[0054] N6=f CoordConv (N5) (Formula 3)

[0055] Among them, f CoordConv (·) represents coordinate convolution.

[0056] In this embodiment, after the visible light image passes through the feature extraction network, the output can be divided into 5 layers based on the feature map size, ranging from C1 to C5 in descending order of size, which also corresponds to the order of layers from low to high. According to the principles of image processing, lower-layer feature maps typically focus more on small targets and detailed features in the image, as well as their positional information within the image. Higher-layer feature maps focus more on large targets in the image and the semantic information they contain. To reduce network performance overhead, {C2, C3, C4, C5} are selected as the input to the binary edge image guidance module, with their sizes reduced to {1 / 4, 1 / 8, 1 / 16, 1 / 32} compared to the input image size. First, a 1×1 convolution is performed on C5 to obtain P5. This process can be represented as:

[0057] P5 = Conv 1×1 (C5)

[0058] Among them, Conv 1×1 (·) represents a 1×1 convolution.

[0059] Next, P5 is upsampled, and C4 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain P4. P4 is then upsampled, and C3 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain P3. P3 is then upsampled, and C2 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain P2. This top-down process fuses feature maps from different levels to better extract and identify target features. This process can be represented as:

[0060]

[0061] in, This means that the two feature maps are added element-wise according to their corresponding pixel positions. Up 2× (·) indicates a 2x upsampling, Conv 1×1 (·) represents a 1×1 convolution.

[0062] Then, 3×3 convolutions are performed on P2 to P5 to obtain M2 to M5. This process can be represented as follows:

[0063] M i =Conv 3×3 (P i ), i = 2 ~ 5

[0064] Among them, Conv 3×3 (·) represents a 3×3 convolution.

[0065] Due to differences in spatial structure, the feature distributions of visible light images and binary edge images differ, making it difficult to directly fuse the two images for better instance segmentation of the target. Therefore, considering this problem, we first use convolution to extract image features from the binary edge image. Through convolution, edge features can be extracted from the image, making subsequent image processing and analysis more accurate and precise. This process can be represented as:

[0066] F Be =σ(BN(Conv) 3×3 (I Be )))

[0067] Among them, F Be I represents the binary edge image features extracted through convolution. Be The input is a binary edge image, σ(·) represents the nonlinear activation of the activation function, BN(·) represents batch normalization, and Conv 3×3 (·) represents a 3×3 convolution.

[0068] Next, the visible light image feature P2 and the binary boundary image feature F are compared. Be A fusion operation is performed, and then the number of channels is aligned using a 1×1 convolution to obtain the fused image features M2. This process can be represented as:

[0069]

[0070] Among them, Conv 1×1 (·) represents a 1×1 convolution. This indicates splicing along the channel dimension.

[0071] Then, M2 is downsampled, and P3 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain M3. M3 is downsampled, and P4 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain M4. M4 is downsampled, and P5 is convolved with a 1×1 convolution. The resulting feature maps are summed element-wise according to pixel positions to obtain M5. This bottom-up fusion process fuses feature maps from different levels again to better extract and identify target features. This process can be represented as:

[0072]

[0073] in, This indicates that the two feature maps are added element-wise according to their corresponding pixel positions. Down 2× (·) indicates a 2x downsampling, Conv 1×1 (·) represents a 1×1 convolution.

[0074] Next, perform a 3×3 convolution on M2 to M5 to obtain N2 to N5. This process can be represented as:

[0075] N i =Conv 3×3 (M i ), i = 2 ~ 5

[0076] Among them, Conv 3×3 (·) represents a 3×3 convolution.

[0077] Then, to obtain more accurate location feature information, coordinate convolution is performed on N5 to obtain N6. This process can be represented as:

[0078] N6 = CoordConv(N5)

[0079] Here, CoordConv(·) represents coordinate convolution. Thus, N2 to N6 are the complete outputs of the binary edge image guidance module.

[0080] Step S3: The third image feature is input into the joint attention module. The joint attention module adds spatial attention and coordinate attention to the third image feature to form a fourth image feature. The output feature maps N2 to N6 of the binary edge image guidance module are used as input to the joint attention module to obtain I'. After the joint attention module applies spatial attention and coordinate attention to the input feature map, the network's sensitivity to positional features is enhanced. This process can be represented as:

[0081] I'=f JAM (M i ), i = 2 ~ 6

[0082] Among them, f JAM (·) represents the joint attention module, and i = 2 to 6 indicates that the input has 5 levels.

[0083] In this embodiment, the joint attention module can simultaneously apply spatial attention and coordinate attention. Embedding the joint attention module into the output of each layer of the binary edge image module can improve the network's ability to perceive location features.

[0084] Step S4: The fourth image feature is input into the prediction module. The prediction module performs semantic prediction and mask prediction on the fourth image feature and outputs the object segmentation scheme in the visible light image. The prediction module includes a semantic category branch, a mask convolution kernel branch, and a mask feature branch. The semantic category branch is used to perform interpolation convolution operation on the input fourth image feature to obtain the probability that the content corresponding to each grid cell belongs to the target category. The mask convolution kernel branch runs in parallel with the semantic category branch and is used to perform dynamic convolution calculation operation on the input fourth image feature to obtain the prediction mask. The mask feature branch is used to perform convolution, batch normalized nonlinear activation, and adaptive interpolation operation on the input fourth image feature to obtain the mask feature map. Based on the mask feature map, the object segmentation scheme in the visible light image is obtained.

[0085] In this embodiment, the multi-level feature map I' after attention is applied is input into the prediction module for mask prediction. The semantic category branch and the mask convolution kernel branch in the prediction module are parallel. The mask feature branch is directly generated from the multi-level feature map after attention is applied, dynamically generating the prediction mask. Instance segmentation of the image is completed through matrix non-maximum suppression, resulting in I. Seg This process can be represented as:

[0086] I Seg =f M-NMS (f PM (I'))

[0087] Among them, f M-NMS(·) denotes the implementation of matrix nonmaximum suppression, f PM (·) indicates the implementation of the prediction module.

[0088] To reduce the computational cost of traditional non-maximum suppression (NMS) in the mask output process, a matrix non-maximum suppression (Matrix NMS) mechanism is adopted, which has the characteristics of hard deletion and sequential operation.

[0089] Furthermore, non-maximum suppression is applied to the output of the prediction module, including: obtaining all prediction scores from the prediction module output; performing non-maximum suppression on the top N prediction scores to obtain an IoU matrix of size N×N; performing a maximum value operation on the column directions of the IoU matrix; determining the mask to be retained based on the maximum value of each column of the IoU matrix; calculating the penalty of each prediction on other predictions using a linear function based on the prediction score of each prediction; fitting the most overlapping predictions to the suppression probability; finally calculating the attenuation factor using the penalty and suppression probability of each prediction; and obtaining the final prediction based on the attenuation factor.

[0090] Furthermore, predict m j The attenuation factor is affected by the following factors: (a) for each prediction m i For m j (s i >s j The punishment of ) where s i and s j It is the confidence score; (b)m i The probability of being suppressed.

[0091] For (a), each prediction m i For m j (s i >s j The penalty can be determined by f(iou). i,j The result is calculated using m. For (b), the inhibition probability is usually positively correlated with IoU. To simplify the calculation, m is used instead. i The probability is approximated by the most overlapping predictions.

[0092]

[0093] Therefore, the final attenuation factor can be expressed as:

[0094]

[0095] The updated score is from s j =s j·decay j Calculate. Where f(iou) i,j ) = 1 - iou i,j .

[0096] The prediction module is the core of BEIG-SOLO. It takes the attention-enhanced feature map as input to complete the instance segmentation task. Meanwhile, to address the issue of poor segmentation results caused by overlapping multiple targets, a positive and negative sample label assignment strategy based on minimum loss matching is proposed.

[0097] Furthermore, during the training process of the visible light image instance segmentation network model guided by binary edge images, a positive and negative sample label assignment strategy based on minimum loss matching is introduced. This strategy is as follows:

[0098] The hierarchical outputs N2 to N corresponding to the third image features output by the binary edge image guidance module are determined. num +1, determine the object size range corresponding to the output of each level. The object size ranges corresponding to each level are different and have no overlap. When an object has multiple possible sizes, each possible size corresponds to a level, and prediction is performed by the outputs of multiple levels corresponding to the object. In each corresponding level, by calculating the classification and segmentation loss between each grid in the real target box and the real target, the grid with the minimum comprehensive loss is found, and the grid with the minimum comprehensive loss is taken as a positive sample.

[0099] In this embodiment, label assignment is a crucial step during network training, significantly impacting the upper limit of model accuracy. The purpose of label assignment is to distinguish between positive and negative samples during training, thus providing the network with appropriate learning targets. The SOLO-based instance segmentation network divides the feature map into equally sized S×S grids in the prediction module, determining which grids to label as positive samples and which as negative samples based on the centroid of the ground truth box (GT box).

[0100] Assume the center of the ground truth box of the given target is (c x ,c yA grid with width w and height h is used. A scaling factor ε is set to shrink the ground truth (GT) box of the target according to ε to obtain the positive sample box (PS box). When the grid at position (i,j) falls within the PS box, that grid is marked as a positive sample. When there are two or more instance targets in the input image with a large degree of overlap, or when one instance target is completely surrounded by another instance target, the positive sample boxes of these two or more instance targets may fall in the same grid. This causes the problem of different instance targets being predicted by the same grid, resulting in severe confusion of the outlines of different instance targets and greatly interfering with the prediction results.

[0101] To address the aforementioned issues, this invention proposes a positive and negative sample label allocation strategy based on Minimum Cost Assignment (MCA), which is divided into positive and negative sample label allocation strategies at different levels and at the same level.

[0102] First, a positive and negative sample label assignment strategy with different levels is proposed. The role of this strategy is to assign each level to a target within a specific size range. The output of the binary edge image guidance module contains outputs of 5 levels, from low to high, namely N2 to N6, with corresponding size ranges of ((384,2048),(192,768),(96,384),(48,192),(1,96)).

[0103] There are two possibilities for the size of a single instance target: it belongs to one size range or it belongs to multiple size ranges. Let's assume that instance target A's size falls only within the range (384, 2048). In this case, the corresponding N2 layer will be responsible for predicting instance target A. If instance target A's size falls simultaneously within both (384, 2048) and (192, 768), then both the corresponding N2 and N3 layers will be responsible for predicting instance target A. This approach allows the network to more effectively predict targets of various sizes. When an instance target falls within multiple ranges simultaneously, it can be predicted within different ranges, thereby increasing the number of positive samples and mitigating the problem of excessively large discrepancies between the number of positive and negative samples.

[0104] In special cases, when the size range of levels N2 to N6 does not contain any instance target size, all mesh samples at that level will be ignored and not counted as positive or negative samples. For example, if the range (384, 2048) of level N2 does not contain any instance target size, then all meshes at level N2 will be ignored. This strategy can effectively improve the network's performance and robustness, enabling it to perform well on various target sizes and shapes.

[0105] Secondly, a positive and negative sample labeling strategy within the same level is proposed. When a level is assigned the task of predicting instance targets, positive and negative sample labels are assigned to the grids at that level. Assuming level N2 is responsible for predicting instance target A, positive and negative sample labels are assigned to the grids at level N2. MCA treats the assignment of positive and negative sample labels to the grids as an adaptive matching problem. By calculating the classification and segmentation losses between each grid within the ground truth bounding box and the ground truth target, the grid with the minimum comprehensive loss is found and labeled as a positive sample. This method can better handle the overlap problem between instance targets, improving the accuracy and stability of the network. This process can be represented as:

[0106] M = min(focalloss(c i,j ,c gt )+diceloss(m i,j ,m gt ))

[0107] Where M represents the grid with the minimum overall loss within the true target bounding box, it is labeled as a positive sample. i,j Let c be the classification prediction value of the grid at position i,j. gt Let m be the true class value of the sample, and focalloss be the loss function for classification. i,j Let m be the predicted segmentation value of the grid at position (i,j). gt Let be the ground truth mask for the sample, and diceloss be the segmentation loss function. Let i,j be the range of the ground truth bounding box mapped onto the S×S grid.

[0108] During training, instance targets are arranged in ascending order of their true bounding box area. Positive sample grids for each instance target are calculated sequentially. Grids that have already been marked as positive samples are ignored in the next calculation to prevent overlapping instance targets from being marked on the same grid.

[0109] The overall loss function of the visible light image instance segmentation network model guided by binary edge images is:

[0110] L = L cate +λL mask

[0111] Among them, L cate It is the loss function for semantic branches, L mask This is the loss function for the masked branch, where λ is a hyperparameter set to 3; L cate Focus loss is used. mask The expression is:

[0112]

[0113] Where k = i × S + j, N pos p is the number of positive samples. * m* represents the class truth value and the mask truth value, respectively. It is an indicator function, if The value is 1 if it is m, otherwise it is 0. k To predict the mask, The mask's true value;

[0114] d mask The implementation is as follows, using dice loss and boundary loss:

[0115] d mask =L Dice +L Boundary

[0116] Among them, L Dice The definition is as follows:

[0117]

[0118] Where, p x,y ,q x,y These are the pixel values ​​of the predicted mask and the real mask at position (x,y), respectively.

[0119] L Boundary This is the boundary loss, and its calculation method is as follows:

[0120] Based on the target's true mask m and predicted mask m * The formula for constructing the true boundary and predicted boundary of the target is as follows:

[0121] b = pool(1-m, θ) - (1-m)

[0122] b * =pool(1-m) * ,θ)-(1-m * )

[0123] Among them, b,b * Representing the true boundary and the predicted boundary respectively, pool(·,·) denotes pixel-level max pooling operation, m,m * These represent the true mask and the predicted mask, respectively. θ is a hyperparameter, set to 3.

[0124] Based on the obtained true boundary b and predicted boundary b * The boundary loss is constructed and calculated using the following formula:

[0125]

[0126] Where i is the boundary index, b i For the i-th real boundary, Let be the i-th prediction boundary.

[0127] The present invention further provides an apparatus for image instance segmentation, such as... Figure 4 As shown, the device includes:

[0128] Feature extraction module: configured to acquire a visible light image to be segmented and a binary edge image corresponding to the visible light image; extract a first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; and extract a second image feature of the binary edge image by convolution operation.

[0129] Feature fusion module: configured to input the first image features and the second image features into the binary edge image guidance module, the binary edge image guidance module includes top-down and bottom-down fusion paths, used to fuse the first image features and the second image features to obtain a third image feature;

[0130] Attention Addition Module: Configured to input the third image feature into the joint attention module, the joint attention module is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature;

[0131] Segmentation module: configured to input the fourth image features into the prediction module, the prediction module performs semantic prediction and mask prediction on the fourth image features, and outputs a segmentation scheme for objects in the visible light image.

[0132] This invention further provides an image instance segmentation system, comprising:

[0133] A processor is used to execute multiple instructions;

[0134] Memory, used to store multiple instructions;

[0135] The plurality of instructions are to be stored in the memory and loaded and executed by the processor as described above.

[0136] The present invention further provides a computer-readable storage medium storing a plurality of instructions; the plurality of instructions are used by a processor to load and execute the method as described above.

[0137] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0141] The integrated unit, implemented as a software functional unit, can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a physical server, or a cloud server, etc., requiring the Ubuntu operating system to be installed) to execute some steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0142] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for image instance segmentation, characterized in that, The method includes the following steps: Step S1: Obtain the visible light image to be segmented and the corresponding binary edge image; extract the first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; extract the second image feature of the binary edge image by convolution operation; Step S2: Input the first image feature and the second image feature into the binary edge image guidance module. The binary edge image guidance module includes top-down and bottom-down fusion paths for fusing the first image feature and the second image feature to obtain the third image feature. Step S3: Input the third image feature into the joint attention module, which is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature; Step S4: Input the fourth image feature into the prediction module. The prediction module performs semantic prediction and mask prediction on the fourth image feature and outputs the segmentation scheme of the object in the visible light image.

2. The method as described in claim 1, characterized in that, Step S2 includes: From the first image features, select hierarchical features from level 2 to level num in descending order of size. Input the hierarchical features from level 2 to level num and the second image features into the binary edge image guidance module, and process the level num hierarchical feature C. i conduct Convolution, to obtain P num Following the order of i decreasing from num-1 to 2, for P i+1 Upsampling, then applying the i-th level feature C i conduct Convolution, P i+1 The results of upsampling and C i conduct The result of the convolution is obtained by adding corresponding elements at pixel positions, resulting in P. i ; Fuse P2 with the second image feature to obtain the fused image feature M2; Following the order of i increasing from 2 to num-1, process M... i Downsampling, for P i+1 conduct Convolution, M i The results of downsampling and P i+1 conduct The result of the convolution is obtained by adding corresponding elements at pixel positions, resulting in M. i+1 ; for M2 to M num Do Convolution yields N² to N. num ; For N num Perform coordinate convolution to obtain N. num+1 ; where N2 to N num+1 is the output of the binary edge image guidance module, i.e., the third image feature; num is a natural number.

3. The method as described in claim 2, characterized in that, In step S4, the prediction module includes a semantic category branch, a mask convolution kernel branch, and a mask feature branch. The semantic category branch performs interpolation convolution on the input fourth image features to obtain the probability that the content corresponding to each grid cell belongs to the target category. The mask convolution kernel branch runs in parallel with the semantic category branch and performs dynamic convolution calculation on the input fourth image features to obtain a prediction mask. The mask feature branch performs convolution, batch normalization nonlinear activation, and adaptive interpolation on the input fourth image features to obtain a mask feature map. Based on the mask feature map, a segmentation scheme for objects in the visible light image is obtained.

4. The method as described in claim 3, characterized in that, The non-maximum suppression (NMS) of the prediction module output includes: obtaining all prediction scores from the prediction module output; performing N×N IoU matrix on the top N prediction scores with the highest scores after NMS; performing a maximum value operation on the column directions of the IoU matrix; determining the mask to be retained based on the maximum value of each column of the IoU matrix; calculating the penalty of each prediction on other predictions using a linear function based on the prediction score of each prediction; fitting the prediction with the highest overlap to the suppression probability; calculating the attenuation factor using the penalty and suppression probability of each prediction; and obtaining the final prediction based on the attenuation factor.

5. The method as described in claim 4, characterized in that, In the training process of the visible light image instance segmentation network model guided by binary edge images, a positive and negative sample label assignment strategy based on minimum loss matching is introduced. The positive and negative sample label assignment strategy based on minimum loss matching is as follows: The hierarchical outputs N2 to N corresponding to the third image features output by the binary edge image guidance module are determined. num +1, determine the object size range corresponding to the output of each level. The object size ranges corresponding to each level are different and have no overlap. When an object has multiple possible sizes, each possible size corresponds to a level, and prediction is performed by the outputs of multiple levels corresponding to the object. In each corresponding level, by calculating the classification and segmentation loss between each grid in the real target box and the real target, the grid with the minimum comprehensive loss is found, and the grid with the minimum comprehensive loss is taken as a positive sample.

6. An apparatus for image instance segmentation, characterized in that, The device includes: Feature extraction module: configured to acquire a visible light image to be segmented and a binary edge image corresponding to the visible light image; extract a first image feature of the visible light image by a feature extraction network, wherein the first image feature is a multi-layer feature map with num layers of image features; and extract a second image feature of the binary edge image by convolution operation. Feature fusion module: configured to input the first image features and the second image features into the binary edge image guidance module, the binary edge image guidance module includes top-down and bottom-down fusion paths, used to fuse the first image features and the second image features to obtain a third image feature; Attention Addition Module: Configured to input the third image feature into the joint attention module, the joint attention module is used to add spatial attention and coordinate attention to the third image feature to form a fourth image feature; Segmentation module: configured to input the fourth image features into the prediction module, the prediction module performs semantic prediction and mask prediction on the fourth image features, and outputs a segmentation scheme for objects in the visible light image.

7. A system for image instance segmentation, comprising: A processor is used to execute multiple instructions; Memory, used to store multiple instructions; The plurality of instructions are to be stored in the memory and loaded by the processor and executed as described in any one of claims 1-5.

8. A computer-readable storage medium storing a plurality of instructions; the plurality of instructions being loaded by a processor and executing the method as claimed in any one of claims 1-5.

Images