A boundary-guided image instance segmentation method and device and related medium

Abstract

The application discloses a boundary guidance-based image instance segmentation method and device and related media, which comprises the following steps: performing multi-stage feature extraction on an input image to be segmented to obtain an encoded feature set object; constructing feature groups according to adjacent levels for the encoded feature set object, and inputting each feature group into a global feature fusion module to perform context aggregation and attention weighting, thereby obtaining a global fusion feature set object; respectively performing decoding reconstruction processing on the encoded feature set object and the global fusion feature set object to obtain a decoded feature set object and an instance segmentation prediction map; performing edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generating an edge label map according to a preset label map, and calculating a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map and the decoded feature set object, updating network parameters and outputting an instance segmentation result, thereby improving the boundary positioning accuracy of the instance segmentation result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a boundary-guided image instance segmentation method, apparatus, and related medium. Background Technology

[0002] While existing image instance segmentation techniques have proposed various implementation methods based on edge operators, thresholds, and deep learning, mainstream solutions, represented by convolutional neural networks, typically focus on semantic feature learning and mask reconstruction of the main body region. During encoding and decoding, they mainly rely on continuous convolution, pooling, and upsampling to complete segmentation prediction. Due to the lack of a dedicated learning and constraint mechanism that uses the segmentation target boundary information as a key guiding signal, targeted attention weighting and edge supervision of the boundary region are not implemented during training and inference. This results in insufficient localization accuracy of the output instance segmentation prediction map at the target boundary, manifesting as unclear edge details, discontinuous contours, or local boundary offsets, leading to inaccurate boundary conditions. Summary of the Invention

[0003] This invention provides a boundary-guided image instance segmentation method, apparatus, and related medium, aiming to solve the technical problem in the prior art where the boundary positioning accuracy of instance segmentation results is insufficient due to the lack of targeted guided learning and supervision constraints on the boundary information of the segmentation target.

[0004] In a first aspect, embodiments of the present invention provide a boundary-guided image instance segmentation method, comprising: Multi-stage feature extraction is performed on the input image to be segmented to generate at least five levels of coded feature maps and aggregate them into a coded feature set object; The encoded feature set object is constructed into feature groups according to adjacent levels, and each feature group is input into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature map corresponding to each feature group and collecting them into a global fused feature set object; The encoded feature set object and the global fusion feature set object are respectively decoded and reconstructed to obtain the decoded feature set object and the instance segmentation prediction map; Edge prediction maps are obtained by performing edge operator processing on the instance segmentation prediction map, and edge label maps are generated based on preset annotation maps. Then, a weighted loss object is calculated based on the instance segmentation prediction map, edge prediction map, edge label map, and decoded feature set object. The network parameters are updated based on the weighted loss object, and the instance segmentation result is output.

[0005] Secondly, embodiments of the present invention provide a boundary-guided image instance segmentation apparatus, comprising: The feature extraction unit is used to perform multi-stage feature extraction on the input image to be segmented, so as to generate at least five levels of coded feature maps and collect them into a coded feature set object; The feature fusion unit is used to construct feature groups according to adjacent levels of the encoded feature set object, input each feature group into the global feature fusion module to perform context aggregation and attention weighting, output the fused feature map corresponding to each feature group and collect them into a global fused feature set object; The decoding and reconstruction unit is used to perform decoding and reconstruction processing on the encoded feature set object and the global fusion feature set object respectively, to obtain the decoded feature set object and the instance segmentation prediction map. The image output unit is used to perform edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generate an edge label map according to a preset annotation map, calculate a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map and the decoded feature set object, update the network parameters according to the weighted loss object and output the instance segmentation result.

[0006] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the boundary-guided image instance segmentation method of the first aspect.

[0007] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the boundary-guided image instance segmentation method of the first aspect.

[0008] This invention provides a boundary-guided image instance segmentation method, comprising: performing multi-stage feature extraction on the input image to be segmented to generate at least five levels of encoded feature maps and assembling them into an encoded feature set object; constructing feature groups according to adjacent levels on the encoded feature set object, and inputting each feature group into a global feature fusion module to perform context aggregation and attention weighting, outputting fused feature maps corresponding to each feature group and assembling them into a global fused feature set object; performing decoding and reconstruction processing on the encoded feature set object and the global fused feature set object respectively to obtain a decoded feature set object and an instance segmentation prediction map; performing edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, and generating an edge label map according to a preset annotation map; calculating a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map, and the decoded feature set object; updating network parameters according to the weighted loss object and outputting the instance segmentation result, thereby improving the boundary localization accuracy of the instance segmentation result.

[0009] This invention also provides a boundary-guided image instance segmentation apparatus, a computer device, and a storage medium, which have the same beneficial effects as described above. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a boundary-guided image instance segmentation method provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of an image instance segmentation device based on boundary guidance, provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see below. Figure 1 , Figure 1 The flowchart of an image instance segmentation method based on boundary guidance provided in an embodiment of the present invention specifically includes steps S101 to S104.

[0017] S101. Perform multi-stage feature extraction on the input image to be segmented to generate at least five levels of coded feature maps and collect them into a coded feature set object. S102. Construct feature groups for the encoded feature set object according to adjacent levels, and input each feature group into the global feature fusion module to perform context aggregation and attention weighting, output the fused feature map corresponding to each feature group and collect them into a global fused feature set object; S103. Decode and reconstruct the encoded feature set object and the global fusion feature set object respectively to obtain the decoded feature set object and the instance segmentation prediction map. S104. The instance segmentation prediction map is processed by edge operators to obtain an edge prediction map, and an edge label map is generated according to the preset annotation map. Then, a weighted loss object is calculated based on the instance segmentation prediction map, the edge prediction map, the edge label map and the decoded feature set object. The network parameters are updated according to the weighted loss object and the instance segmentation result is output.

[0018] In step S101, this embodiment uses Res2Net-50 as the backbone network to construct a five-layer network structure. The feature images corresponding to the first-layer decoder and encoder are E1 and D1, respectively (the second layer uses E2 and D2, and so on). There are skip links between the upsampling processes of E1 and D1 / D2. The upsampling process from D2 to D1 includes convolution for channel dimensionality reduction and upsampling of the feature image using an upsampling operator. Simultaneously, the feature image of E1 is concatenated to D1 using the concat operator to supplement the feature information of image instance segmentation. For example, the feature image size of E1 is H×W×C. E1 The feature image size of D1 is H×W×C. D1 Since E1 and D1 have the same feature size H and W, they can be concatenated by channel C using the concat operator to obtain H×W×C. cat The fused feature image, where C cat =C E1 +C D1 This operation is called channel fusion. Additionally, when E2, E3, and E4 are concatenated to D2, D3, and D4 respectively using the concat operator, the feature image outputs of the additional modules proposed in this application are also concatenated, thereby forming at least five levels of encoded feature maps during the multi-stage feature extraction process, and the encoded feature maps are then aggregated into the encoded feature set object.

[0019] In step S102, based on the hierarchical order of the encoded feature set object, the encoded feature maps of adjacent levels are grouped to obtain multiple feature groups, and each feature group is input into the global feature fusion module; the global feature fusion module performs context information aggregation on each feature group, and performs weighted fusion of the feature expressions within the feature group based on attention weights, outputting a fused feature map corresponding to each feature group, and then gathers the fused feature maps into the global fused feature set object.

[0020] In one embodiment, step S102 includes: Based on the encoded feature set object, the encoding stage hierarchical sequence is determined, and the encoded feature maps of each layer are numbered to obtain the hierarchical identifier object; Based on the hierarchical identifier object, the previous feature layer, the current feature layer, and the next feature layer are determined to obtain a combined index object; Based on the combined index object, the corresponding three-layer encoded feature map is extracted from the encoded feature set object and encapsulated in hierarchical order to obtain a feature group; wherein, the feature group includes three-layer feature group objects for each of the first feature layer to the third feature layer, the second feature layer to the fourth feature layer, and the third feature layer to the fifth feature layer.

[0021] In this embodiment, the encoding stage hierarchy sequence is determined based on the encoded feature set object, and each layer of encoded feature map is numbered according to the hierarchy order to generate a hierarchy identifier object. The hierarchy identifier object is used to indicate the hierarchy position of each layer of encoded feature map in the encoding stage and its corresponding relationship. For example, the output features of the encoding stage are sequentially identified as (E1, E2, E3, E4, E5), and a searchable hierarchy identifier object is formed with this numbering result to ensure that stable positioning and retrieval can be performed according to the adjacent hierarchy relationship of the previous feature layer, the current feature layer, and the next feature layer.

[0022] After generating the hierarchical identifier object, the previous feature layer, the current feature layer, and the next feature layer are determined based on the hierarchical identifier object to obtain a combined index object. The combined index object is used to limit the range of current levels that can participate in constructing feature groups and explicitly provides the three-level hierarchical index relationship corresponding to each feature group. Specifically, in this embodiment, the intermediate layer of the encoding stage is used as the current layer that can construct three adjacent combinations. The current level i is set to 2, 3, or 4, and the adjacent previous level i-1 and next level i+1 are determined accordingly, thereby generating three sets of combined indices: (1,2,3), (2,3,4), and (3,4,5). Furthermore, based on the combined index object, the corresponding three-layer encoded feature map is extracted from the encoded feature set object and encapsulated in hierarchical order to obtain a feature group. Thus, the feature group includes three-layer feature group objects [E1, E2, E3] from the first feature layer to the third feature layer, three-layer feature group objects [E2, E3, E4] from the second feature layer to the fourth feature layer, and three-layer feature group objects [E3, E4, E5] from the third feature layer to the fifth feature layer. The above three-layer feature group objects are used as the input feature group of the subsequent adjacent three-way global feature fusion module for performing context aggregation and attention weighting.

[0023] Furthermore, to maintain consistency with the computation interface of the adjacent-level three-way global feature fusion module, this embodiment encapsulates each three-layer feature group object in the hierarchical order of the previous feature layer, the current feature layer, and the next feature layer, and constructs the fusion input for global attention learning accordingly; wherein, the output feature of the adjacent-level three-way global feature fusion module is denoted as HGA. i And HGA i These can be respectively referred to as HGA1, HGA2, and HGA3, HGA i The expression satisfies the following formula: Among them, E l This represents the output of the l-th layer features in the encoding stage. This indicates that the output features of the previous feature layer, the current feature layer, and the next feature layer are fused together. H l This represents a neighboring global attention module, which includes convolution, adaptive average pooling layers, L2 regularization, etc.

[0024] In one embodiment, step S102 further includes: The input feature map of the previous feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the upper-layer processed feature map. The upper-layer processed feature map is multiplied element-wise with the input feature map of the previous feature layer to obtain the upper-layer fused feature map; The upper-layer fused feature map is subjected to downsampling convolution operation to obtain the upper-layer aggregated feature map.

[0025] In this embodiment, taking HGA1 as an example, the input feature images of the previous feature layer, the current feature layer, and the next feature layer are E1, E2, and E3, respectively, and correspond to the input feature images l, x, and h, respectively. The dimensions of these three input features are respectively , , ,in, = = , = = In this upper-level path, the input feature image l from the previous feature layer is first sequentially input into an adaptive pooling layer, L2 regularization, and the Upsample bilinear upsampling operator to obtain an upper-level processed feature map aligned with the spatial scale of l. This smooths and aggregates feature information without changing the consistency of channel representation, enabling more effective aggregation of contextual information in subsequent steps. Then, the upper-level processed feature map is multiplied element-wise with the original input feature image l to obtain an upper-level fused feature map. This allows the global information obtained through adaptive pooling, L2 regularization, and Upsample bilinear upsampling to perform gated weighting on the original feature image.

[0026] After element-wise multiplication, a downsampling convolution operation is performed on the upper-layer fused feature map. The downsampling convolution operation can use a downsampling convolution layer with a convolution size of 2×2 and a stride of 2 to obtain the upper-layer aggregated feature map l'. The combination of adaptive pooling layer, L2 regularization, and upsample bilinear upsampling operator is used to perform more effective smoothing of feature information before aggregating context information, thereby providing more comprehensive global feature information than ordinary convolution for subsequent feature fusion. This global information is then solidified into an upper-layer aggregated feature map that can be used for subsequent fusion calculations through element-wise multiplication and downsampling convolution.

[0027] In one embodiment, step S102 further includes: The input feature map of the next feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the lower layer processed feature map; The lower-layer processed feature map is multiplied element-wise with the input feature map of the next feature layer to obtain the lower-layer fused feature map; The lower-layer fused feature map is subjected to convolution operation and channel dimensionality reduction processing to obtain the intermediate feature map; The intermediate feature map is subjected to adaptive pooling, L2 regularization, and bilinear interpolation upsampling operations in sequence to obtain the lower-level aggregated feature map.

[0028] In this embodiment, in the three inputs of HGA1 as an example, the input feature map of the next feature layer is denoted as h, and h is a feature image that is deeper than the input feature map x of the current feature layer and the input feature map of the previous feature layer. Specifically, the first adaptive pooling operation, L2 regularization operation and Upsample bilinear interpolation upsampling operation are sequentially performed on the input feature map h of the next feature layer to obtain a lower-level processed feature map aligned with the spatial scale of the original h; then, the lower-level processed feature map and the original input feature map h are multiplied element-wise to obtain a lower-level fused feature map, so that the global context information obtained by adaptive pooling, L2 regularization and Upsample bilinear interpolation upsampling is used to perform gated weighted fusion on the deep feature map h, thereby obtaining a feature image with more comprehensive global feature information after feature fusion. Furthermore, convolution operations are performed on the lower-layer fused feature map, and channel dimensionality reduction is performed. The convolution operation adopts a convolution size of 1×1 and a stride of 1 to achieve channel dimensionality reduction and obtain an intermediate feature map. Then, a second adaptive pooling operation, an L2 regularization operation, and an Upsample bilinear interpolation upsampling operation are sequentially performed on the intermediate feature map to obtain the lower-layer aggregated feature map h'.

[0029] Furthermore, since the input feature map h of the next feature layer is a feature image that is deeper than feature maps x and l, in order to obtain more comprehensive global feature information from the deep feature information, this embodiment uses a combination of three operations ("first adaptive pooling layer, L2 regularization, and upsample bilinear upsampling") and "second adaptive pooling layer, L2 regularization, and upsample bilinear upsampling") to form a two-level global information aggregation path for the input feature map h of the next feature layer. Channel dimensionality reduction is completed between the two aggregations using a 1×1 convolution with a stride of 1, thereby stably aggregating the deep global context information into the lower-level aggregated feature map for subsequent attention weighting and cross-layer fusion calculations.

[0030] In one embodiment, step S102 further includes: The input feature map of the current feature layer is multiplied element-wise with the aggregated feature map of the lower layer to obtain a global feature information fusion feature map; The global feature information fusion feature map is subjected to linear rectified activation, channel dimensionality reduction convolution, and logical activation operations in sequence to obtain the attention matrix; The attention matrix is ​​multiplied element-wise with the upper-layer aggregated feature map to obtain an attention-weighted feature map; The attention-weighted feature map is fused with the input feature map of the current feature layer using a feature matrix weighting operation to obtain a global fused feature set object.

[0031] In this embodiment, the input feature map x of the current feature layer is first multiplied element-wise with the lower-layer aggregated feature map h' to obtain the global feature information fusion feature map x'. This means that by mapping the global context information aggregated in the deeper layers onto the spatial location and channel response of the current feature layer, the current feature layer features carry the enhancement of global feature information after fusion. Then, the global feature information fusion feature map x' is sequentially subjected to ReLU activation, a 1×1 channel-dimension-reduced convolution, and a Sigmoid logical activation to generate an attention matrix for instance segmentation; wherein the size of the attention matrix is... The attention matrix is ​​configured to provide precise instance segmentation attention weights for subsequent fusion processes. Then, the attention matrix is ​​multiplied element-wise with the upper-layer aggregated feature map l' to obtain an attention-weighted feature map. This allows the attention weights to act on the spatial location and channel response of the upper-layer aggregated feature map, thereby achieving adaptive weighting of the upper-layer aggregated feature map.

[0032] Finally, the attention-weighted feature map and the input feature map x of the current feature layer are subjected to a weighted fusion operation on the feature matrix to obtain the final output feature map of the adjacent three-way global feature fusion module, and the final output feature map is encapsulated and aggregated into the global fusion feature set object; wherein, the final output feature map simultaneously fuses the global feature information of the adjacent upper layer and the next layer, and performs global feature information enhancement processing on each channel to meet the call requirements of global context enhancement features in the subsequent decoding and reconstruction stage.

[0033] In step S103, during the decoding stage, the encoded feature set object and the global fusion feature set object are used as decoding and reconstruction inputs. Feature reconstruction and scale recovery are performed step by step according to the preset decoding path to generate decoding feature maps at each level and collect them into the decoding feature set object. After the decoding and reconstruction is completed, an instance segmentation prediction map corresponding to the image to be segmented is generated based on the decoding feature set object.

[0034] In one embodiment, step S103 includes: Obtain the output prediction map and align its scale to obtain the decoding input object; The foreground boundary is calculated on the output prediction map to obtain the foreground boundary object; Based on the foreground boundary object, the boundary information of the output prediction map is decoupled and extracted, and a boundary mapping is generated to obtain the boundary information object; The boundary information object is subjected to boundary attention learning at a predetermined feature layer to obtain a multi-scale attention object; Associating the multi-scale attention object with the decoded input object yields a convolutional feature object; Boundary decoupling operations are performed on the convolutional feature objects, and the multi-scale attention objects are fused to obtain a decoupling enhancement object; The decoupling enhancement object is multiplied element-wise with the foreground boundary object and then added together to obtain the boundary blending object; The boundary fusion object is connected to a preset output channel to obtain the current output feature map; The current output feature map is encapsulated with the corresponding output prediction map to obtain a decoded feature set and output an instance segmentation prediction map.

[0035] In this embodiment, the output prediction map of the next feature layer in the decoding stage is first obtained as Prediction, and the output feature image corresponding to the current decoding layer is obtained as the input of the edge feature enhancement module for boundary decoupling and guidance. ; and then the Prediction and Perform scale alignment processing to make Prediction's spatial resolution consistent with... The spatial resolution is consistent, thus obtaining a decoded input object that can be uniformly referenced. Based on the decoded input object, foreground boundary calculation is performed on the output prediction map Prediction to obtain a foreground boundary object. ,in This represents the boundary region of the foreground target to be segmented, and Prediction represents the output prediction map of the next feature layer in the decoding stage.

[0036] In obtaining the foreground boundary object Then, based on the foreground boundary object, boundary information is decoupled and extracted from the output prediction map Prediction, and a boundary mapping is generated to form a boundary information object; specifically, the output feature image of the decoder is... As input to the boundary decoupling part, the parallel convolutional module CIE is first input to extract boundary-related representations, where CIE consists of parallel convolutional modules with convolutional sizes of 3×3 and 5×5. Then, the output of CIE is input to the boundary decoupling operation Decouple to decompose the features into boundary part output and main part output, and the boundary part output is denoted as... Thus by This forms the basis for the boundary mapping in the boundary information object.

[0037] Furthermore, the boundary information object undergoes boundary attention learning at a predetermined feature layer to obtain a multi-scale attention object. The next layer output prediction map (Prediction) in the decoding stage guides the boundary information learning of this module. To avoid excessive computational overhead, multi-scale boundary attention learning is performed at predetermined levels, including the first, third, and fifth feature layers. Boundary attention learning (MEA1, MEA2, and MEA3) is performed at these corresponding levels to obtain the multi-scale attention object. The multi-scale attention object is... and As a joint input, attention weights are applied to the boundary portion, and the attention result is used for boundary enhancement association of subsequent decoded features.

[0038] After obtaining the multi-scale attention object, the multi-scale attention object is associated with the decoded input object to obtain the convolutional feature object; specifically, and The result of element-wise multiplication and The features are superimposed to form a boundary feature representation with enhanced boundary attention, and this boundary feature representation is then combined with the output of the boundary decoupling and guidance module from the previous stage. The channels are connected using the Concat operation to obtain the fused input; a Conv3×3 convolution kernel operation is then performed on this fused input to obtain the output feature image of the boundary decoupling and guidance module in this stage. The output is used as the core component of the convolutional feature object to support subsequent boundary decoupling operations and multi-scale attention fusion.

[0039] Furthermore, boundary decoupling operations are performed on the convolutional feature objects, and the multi-scale attention objects are fused to obtain decoupling enhancement objects; in this process, the decoupling enhancement objects and the foreground boundary objects... Element-wise multiplication and addition are performed to obtain a boundary fusion object. The boundary fusion object is then connected to a preset output channel to obtain the current output feature map. The current output feature map and the output prediction map Prediction are encapsulated according to the hierarchical correspondence to obtain a decoded feature set object and output an instance segmentation prediction map. In the process of transitioning from the encoding stage to the decoding stage, boundary decoupling and guidance module outputs can be introduced in the decoding upsampling paths from D5 to D4, D4 to D3, and D3 to D2, respectively, to ensure that the decoded output feature maps at each level have consistent boundary enhancement representations and can be encapsulated into a decoded feature set object.

[0040] The formula for calculating the foreground boundary based on prediction map boundary decoupling and guidance can be referenced as follows: The calculation formula for the multi-scale boundary attention module can be found below: .

[0041] In step S104, an edge operator (e.g., Sobel edge operator) is applied to the instance segmentation prediction map to extract the boundary response and generate an edge prediction map; simultaneously, an edge label map corresponding to the edge prediction map is constructed based on a preset annotation map; then, the instance segmentation prediction map, the edge prediction map, the edge label map, and the decoded feature set object are input into the loss calculation process to obtain a weighted loss object reflecting the deviation between the segmentation prediction and the boundary prediction; finally, the network parameters are updated according to the weighted loss object, and the instance segmentation result is output under the updated network parameters.

[0042] In one embodiment, step S104 includes: The instance segmentation prediction map is used as the output object, and edge operator processing is performed on the output object to obtain the edge prediction map; The output edge result of the edge feature enhancement module is obtained as the edge output object, and its size is aligned with the edge prediction map object to obtain the aligned object; Obtain a preset annotation map as the real annotation object, and generate an edge label map based on the real annotation object; Based on the edge label map and alignment object, the edge consistency loss term is calculated to obtain a multi-layer loss set object; The real labeled object is downsampled multiple times to obtain a multi-scale labeled set, and the multi-scale labeled set is combined with the decoded feature set object to calculate a deep supervision loss set object; The multi-layer loss set object and the deep supervised loss set object are weighted to obtain a weighted loss object, and the network parameters are updated to output the instance segmentation result.

[0043] In this embodiment, the instance segmentation prediction map output in step S103 is first used as the output object Output, and edge operator processing is performed on the Output. In this embodiment, the edge operator is the Sobel edge operator, so that the Output is processed by the Sobel edge operator to obtain the method output result Output through Sobel operator, and the Output through Sobel operator is used as the edge prediction map to characterize the boundary response of the output object in the sense of pixel gradient.

[0044] Subsequently, the output edge result of the edge feature enhancement module is obtained as the edge output object. In this embodiment, the edge feature enhancement module is the edge feature enhancement module for boundary decoupling and guidance in step S103, and its output boundary result in the decoding stage is denoted as Edge Output (e.g., the boundary result output by MEA1). To ensure the consistency of edge supervision, the Edge Output is spatially scale aligned with the Output through Sobel operator (including size resampling and alignment clipping) to obtain an aligned object, so that the edge output object and the edge prediction map can correspond element-wise at the same resolution.

[0045] After aligning edge prediction and edge output, a preset annotation map is obtained as the ground truth, and an edge label map is generated based on the ground truth. Specifically, edge masks are generated from the ground truth, and the edge masks correspond to the output through the Sobel operator at the same size, thus forming an edge supervision relationship where output through the Sobel operator = edge masks. Simultaneously, a correspondence is established between edge masks and edge output to form an edge consistency supervision relationship between edge masks and edge output. Based on the edge masks and the aligned object, an edge consistency loss term is calculated to obtain a multi-level loss set object. This multi-level loss set object includes at least one set of edge losses constrained by the output through the Sobel operator and edge masks, and another set of edge losses constrained by edge masks and edge output, thus incorporating both edge labels and edge output into the loss calculation.

[0046] Furthermore, to achieve deep supervision, in this embodiment, Ground Truth will perform four downsampling operations to obtain a multi-scale annotation set. This multi-scale annotation set will then be matched with the corresponding layer's decoding feature set object in the decoding stage, ensuring that all feature images from the highest to the lowest layer are constrained by the loss function; thus, a deep supervision loss set object is obtained. The aforementioned deep supervision loss set object and the aforementioned multi-layer loss set object participate in weighted fusion to obtain a weighted loss object for parameter updates. The composition of the network loss function L, and the calculation methods for deep supervision segmentation loss and boundary loss, are determined in this embodiment according to the following formula. After the loss calculation is completed, the network parameter update step continues, gradually converging the network error to a predetermined range, thereby outputting the instance segmentation result and iteratively obtaining the optimal network parameters. The network loss function... Perform the following formula: Among them, the deep supervised segmentation loss function in the encoding stage The calculation is performed according to the following formula: Among them, the boundary loss function The calculation is performed according to the following formula: in, Let i represent the i-th sub-loss function, and j represent the j-th pixel. Indicates total 1 pixel, Indicates the first The predicted value of each pixel. Indicates the first The actual value of each pixel. and for The hyperparameters were set to 0.25 and 2 respectively.

[0047] In summary, this application constructs adjacent-level feature groups and introduces a three-way global feature fusion module in the encoding stage to achieve cross-layer context information aggregation and attention weighting, thus mitigating the attenuation of detail information caused by continuous convolution and pooling. Simultaneously, in the decoding stage, a boundary decoupling and guided edge feature enhancement module is introduced, using the foreground boundary of the output prediction map as a guiding signal for boundary information decoupling extraction and multi-scale boundary attention learning, enabling targeted enhancement of boundary regions. Furthermore, in the training stage, edge consistency constraints are constructed based on the edge operator output of the instance segmentation prediction map, the edge output of the edge feature enhancement module, and the edge label map generated from real annotations. Combined with multiple downsamplings of real annotations to form a deep supervised loss, joint optimization and weighted update of segmentation loss and boundary loss are achieved. This makes the output instance segmentation results more stable in terms of target contour continuity, edge localization accuracy, and fine-grained structure preservation, improving the overall accuracy and robustness of instance segmentation in complex scenes.

[0048] Combination Figure 2 As shown, Figure 2 A schematic block diagram of a boundary-guided image instance segmentation device 200 provided in an embodiment of the present invention includes: The feature extraction unit 201 is used to perform multi-stage feature extraction on the input image to be segmented, so as to generate at least five levels of coded feature maps and collect them into a coded feature set object; The feature fusion unit 202 is used to construct feature groups according to adjacent levels of the encoded feature set object, input each feature group into the global feature fusion module to perform context aggregation and attention weighting, output the fused feature map corresponding to each feature group and collect them into a global fused feature set object; The decoding and reconstruction unit 203 is used to perform decoding and reconstruction processing on the encoded feature set object and the global fusion feature set object respectively, to obtain the decoded feature set object and the instance segmentation prediction map. The image output unit 204 is used to perform edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generate an edge label map according to a preset annotation map, calculate a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map and the decoded feature set object, update the network parameters according to the weighted loss object and output the instance segmentation result.

[0049] In this embodiment, the feature extraction unit 201 performs multi-stage feature extraction on the input image to be segmented to generate at least five levels of encoded feature maps and aggregates them into an encoded feature set object; the feature fusion unit 202 constructs feature groups according to adjacent levels of the encoded feature set object, and inputs each feature group into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature maps corresponding to each feature group and aggregating them into a global fused feature set object; the decoding and reconstruction unit 203 performs decoding and reconstruction processing on the encoded feature set object and the global fused feature set object respectively to obtain a decoded feature set object and an instance segmentation prediction map; the image output unit 204 performs edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generates an edge label map according to a preset annotation map, calculates a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map, and the decoded feature set object, updates the network parameters according to the weighted loss object, and outputs the instance segmentation result.

[0050] In one embodiment, the feature fusion unit 202 is specifically used for: Based on the encoded feature set object, the encoding stage hierarchical sequence is determined, and the encoded feature maps of each layer are numbered to obtain the hierarchical identifier object; Based on the hierarchical identifier object, the previous feature layer, the current feature layer, and the next feature layer are determined to obtain a combined index object; Based on the combined index object, the corresponding three-layer encoded feature map is extracted from the encoded feature set object and encapsulated in hierarchical order to obtain a feature group; wherein, the feature group includes three-layer feature group objects for each of the first feature layer to the third feature layer, the second feature layer to the fourth feature layer, and the third feature layer to the fifth feature layer.

[0051] In one embodiment, the feature fusion unit 202 is further specifically used for: The input feature map of the previous feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the upper-layer processed feature map. The upper-layer processed feature map is multiplied element-wise with the input feature map of the previous feature layer to obtain the upper-layer fused feature map; The upper-layer fused feature map is subjected to downsampling convolution operation to obtain the upper-layer aggregated feature map.

[0052] In one embodiment, the feature fusion unit 202 is further specifically used for: The input feature map of the next feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the lower layer processed feature map; The lower-layer processed feature map is multiplied element-wise with the input feature map of the next feature layer to obtain the lower-layer fused feature map; The lower-layer fused feature map is subjected to convolution operation and channel dimensionality reduction processing to obtain the intermediate feature map; The intermediate feature map is subjected to adaptive pooling, L2 regularization, and bilinear interpolation upsampling operations in sequence to obtain the lower-level aggregated feature map.

[0053] In one embodiment, the feature fusion unit 202 is further specifically used for: The input feature map of the current feature layer is multiplied element-wise with the aggregated feature map of the lower layer to obtain a global feature information fusion feature map; The global feature information fusion feature map is subjected to linear rectified activation, channel dimensionality reduction convolution, and logical activation operations in sequence to obtain the attention matrix; The attention matrix is ​​multiplied element-wise with the upper-layer aggregated feature map to obtain an attention-weighted feature map; The attention-weighted feature map is fused with the input feature map of the current feature layer using a feature matrix weighting operation to obtain a global fused feature set object.

[0054] In one embodiment, the decoding and reconstruction unit 203 is specifically used for: Obtain the output prediction map and align its scale to obtain the decoding input object; The foreground boundary is calculated on the output prediction map to obtain the foreground boundary object; Based on the foreground boundary object, the boundary information of the output prediction map is decoupled and extracted, and a boundary mapping is generated to obtain the boundary information object; The boundary information object is subjected to boundary attention learning at a predetermined feature layer to obtain a multi-scale attention object; Associating the multi-scale attention object with the decoded input object yields a convolutional feature object; Boundary decoupling operations are performed on the convolutional feature objects, and the multi-scale attention objects are fused to obtain a decoupling enhancement object; The decoupling enhancement object is multiplied element-wise with the foreground boundary object and then added together to obtain the boundary blending object; The boundary fusion object is connected to a preset output channel to obtain the current output feature map; The current output feature map is encapsulated with the corresponding output prediction map to obtain a decoded feature set and output an instance segmentation prediction map.

[0055] In one embodiment, the image output unit 204 is specifically used for: The instance segmentation prediction map is used as the output object, and edge operator processing is performed on the output object to obtain the edge prediction map; The output edge result of the edge feature enhancement module is obtained as the edge output object, and its size is aligned with the edge prediction map object to obtain the aligned object; Obtain a preset annotation map as the real annotation object, and generate an edge label map based on the real annotation object; Based on the edge label map and alignment object, the edge consistency loss term is calculated to obtain a multi-layer loss set object; The real labeled object is downsampled multiple times to obtain a multi-scale labeled set, and the multi-scale labeled set is combined with the decoded feature set object to calculate a deep supervision loss set object; The multi-layer loss set object and the deep supervised loss set object are weighted to obtain a weighted loss object, and the network parameters are updated to output the instance segmentation result.

[0056] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0057] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0058] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, a power supply, a graphics card, etc., to utilize the graphics card's performance to operate the model, such as for inference and training.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0060] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A boundary-guided image instance segmentation method, characterized in that, include: Multi-stage feature extraction is performed on the input image to be segmented to generate at least five levels of coded feature maps and aggregate them into a coded feature set object; The encoded feature set object is constructed into feature groups according to adjacent levels, and each feature group is input into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature map corresponding to each feature group and collecting them into a global fused feature set object; The encoded feature set object and the global fusion feature set object are respectively decoded and reconstructed to obtain the decoded feature set object and the instance segmentation prediction map; Edge prediction maps are obtained by performing edge operator processing on the instance segmentation prediction map, and edge label maps are generated according to the preset annotation map. Then, a weighted loss object is calculated based on the instance segmentation prediction map, edge prediction map, edge label map and decoded feature set object. The network parameters are updated according to the weighted loss object and the instance segmentation result is output. The step of performing decoding and reconstruction processing on the encoded feature set object and the global fusion feature set object respectively to obtain the decoded feature set object and the instance segmentation prediction map includes: The output prediction map of the next feature layer in the decoding stage is obtained, and the output feature image corresponding to the current decoding layer is obtained as the input of the edge feature enhancement module for boundary decoupling and guidance. The scale alignment processing is performed on the output prediction map and the input of the edge feature enhancement module to make the spatial resolution of the output prediction map consistent with the spatial resolution of the input of the edge feature enhancement module, so as to obtain the decoding input object. Based on the decoded input object, foreground boundary calculation is performed on the output prediction map to obtain a foreground boundary object, which represents the boundary region of the foreground target to be segmented; Based on the foreground boundary object, the boundary information of the output prediction map is decoupled and extracted, and a boundary mapping is generated to obtain the boundary information object; The boundary information object is subjected to boundary attention learning at a predetermined feature layer to obtain a multi-scale attention object; Associating the multi-scale attention object with the decoded input object yields a convolutional feature object; Boundary decoupling operations are performed on the convolutional feature objects, and the multi-scale attention objects are fused to obtain a decoupling enhancement object; The decoupling enhancement object is multiplied element-wise with the foreground boundary object and then added together to obtain the boundary blending object; The boundary fusion object is connected to a preset output channel to obtain the current output feature map; The current output feature map is encapsulated with the corresponding output prediction map to obtain a decoded feature set and output an instance segmentation prediction map.

2. The image instance segmentation method based on boundary guidance according to claim 1, characterized in that, The construction of feature groups for the encoded feature set object according to adjacent levels includes: Based on the encoded feature set object, the encoding stage hierarchical sequence is determined, and the encoded feature maps of each layer are numbered to obtain the hierarchical identifier object; Based on the hierarchical identifier object, the previous feature layer, the current feature layer, and the next feature layer are determined to obtain a combined index object; Based on the combined index object, the corresponding three-layer encoded feature map is extracted from the encoded feature set object and encapsulated in hierarchical order to obtain a feature group; wherein, the feature group includes three-layer feature group objects for each of the first feature layer to the third feature layer, the second feature layer to the fourth feature layer, and the third feature layer to the fifth feature layer.

3. The image instance segmentation method based on boundary guidance according to claim 2, characterized in that, The step of inputting each of the feature groups into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature maps corresponding to each feature group and aggregating them into a global fused feature set object includes: The input feature map of the previous feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the upper-layer processed feature map. The upper-layer processed feature map is multiplied element-wise with the input feature map of the previous feature layer to obtain the upper-layer fused feature map; The upper-layer fused feature map is subjected to downsampling convolution operation to obtain the upper-layer aggregated feature map.

4. The image instance segmentation method based on boundary guidance according to claim 3, characterized in that, The step of inputting each of the feature groups into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature maps corresponding to each feature group and aggregating them into a global fused feature set object, further includes: The input feature map of the next feature layer is subjected to adaptive pooling, L2 regularization and bilinear interpolation upsampling operations in sequence to obtain the lower layer processed feature map; The lower-layer processed feature map is multiplied element-wise with the input feature map of the next feature layer to obtain the lower-layer fused feature map; The lower-layer fused feature map is subjected to convolution operation and channel dimensionality reduction processing to obtain the intermediate feature map; The intermediate feature map is subjected to adaptive pooling, L2 regularization, and bilinear interpolation upsampling operations in sequence to obtain the lower-level aggregated feature map.

5. The image instance segmentation method based on boundary guidance according to claim 4, characterized in that, The step of inputting each of the feature groups into the global feature fusion module to perform context aggregation and attention weighting, outputting the fused feature maps corresponding to each feature group and aggregating them into a global fused feature set object, further includes: The input feature map of the current feature layer is multiplied element-wise with the aggregated feature map of the lower layer to obtain a global feature information fusion feature map; The global feature information fusion feature map is subjected to linear rectified activation, channel dimensionality reduction convolution, and logical activation operations in sequence to obtain the attention matrix; The attention matrix is ​​multiplied element-wise with the upper-layer aggregated feature map to obtain the attention-weighted feature map; The attention-weighted feature map is fused with the input feature map of the current feature layer using a feature matrix weighting operation to obtain a global fused feature set object.

6. The image instance segmentation method based on boundary guidance according to claim 1, characterized in that, The process involves performing edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generating an edge label map based on a preset annotation map, calculating a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map, and the decoded feature set object, updating network parameters based on the weighted loss object, and outputting the instance segmentation result, including: The instance segmentation prediction map is used as the output object, and edge operator processing is performed on the output object to obtain the edge prediction map; The output edge result of the edge feature enhancement module is obtained as the edge output object, and its size is aligned with the edge prediction map object to obtain the aligned object; Obtain a preset annotation map as the real annotation object, and generate an edge label map based on the real annotation object; Based on the edge label map and alignment object, the edge consistency loss term is calculated to obtain a multi-layer loss set object; The real labeled object is downsampled multiple times to obtain a multi-scale labeled set, and the multi-scale labeled set is combined with the decoded feature set object to calculate a deep supervision loss set object; The multi-layer loss set object and the deep supervised loss set object are weighted to obtain a weighted loss object, and the network parameters are updated to output the instance segmentation result.

7. A boundary-guided image instance segmentation device, characterized in that, include: The feature extraction unit is used to perform multi-stage feature extraction on the input image to be segmented, so as to generate at least five levels of coded feature maps and collect them into a coded feature set object; The feature fusion unit is used to construct feature groups according to adjacent levels of the encoded feature set object, input each feature group into the global feature fusion module to perform context aggregation and attention weighting, output the fused feature map corresponding to each feature group and collect them into a global fused feature set object; The decoding and reconstruction unit is used to perform decoding and reconstruction processing on the encoded feature set object and the global fusion feature set object respectively, to obtain the decoded feature set object and the instance segmentation prediction map. The image output unit is used to perform edge operator processing on the instance segmentation prediction map to obtain an edge prediction map, generate an edge label map according to a preset annotation map, calculate a weighted loss object based on the instance segmentation prediction map, the edge prediction map, the edge label map and the decoded feature set object, update the network parameters according to the weighted loss object and output the instance segmentation result. Specifically, the decoding and reconstruction unit is used for: The output prediction map of the next feature layer in the decoding stage is obtained, and the output feature image corresponding to the current decoding layer is obtained as the input of the edge feature enhancement module for boundary decoupling and guidance. The scale alignment processing is performed on the output prediction map and the input of the edge feature enhancement module to make the spatial resolution of the output prediction map consistent with the spatial resolution of the input of the edge feature enhancement module, so as to obtain the decoding input object. Based on the decoded input object, foreground boundary calculation is performed on the output prediction map to obtain a foreground boundary object, which represents the boundary region of the foreground target to be segmented; Based on the foreground boundary object, the boundary information of the output prediction map is decoupled and extracted, and a boundary mapping is generated to obtain the boundary information object; The boundary information object is subjected to boundary attention learning at a predetermined feature layer to obtain a multi-scale attention object; Associating the multi-scale attention object with the decoded input object yields a convolutional feature object; Boundary decoupling operations are performed on the convolutional feature objects, and the multi-scale attention objects are fused to obtain a decoupling enhancement object; The decoupling enhancement object is multiplied element-wise with the foreground boundary object and then added together to obtain the boundary blending object; The boundary fusion object is connected to a preset output channel to obtain the current output feature map; The current output feature map is encapsulated with the corresponding output prediction map to obtain a decoded feature set and output an instance segmentation prediction map.

8. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the boundary-guided image instance segmentation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the boundary-guided image instance segmentation method as described in any one of claims 1 to 6.

Images