A multi-stage instance segmentation method and system based on parallel feature completion

Abstract

The application discloses a kind of multi-stage instance segmentation method and system based on parallel feature completion, after data pre-processing obtains the data format suitable for system input, in the introduction GC Block of backbone network ResNet50 obtains instance association information.Then parallel semantic branch and parallel global branch are proposed, the semantic and global information of feature layer are extracted, and the completion of RoI feature is realized.Again after boundary box dynamic interaction module and detection branch, the regression of boundary box is obtained.In addition, feature transfer structure is also proposed, by explicitly increasing the connection between detection and segmentation branch, changing the path of gradient back propagation and then indirectly completing RoI feature, after mask dynamic interaction module and mask branch, instance mask is obtained.It is worth emphasizing that the above structure is parallel, that is, there is no interference between stages, and it is compatible with the parallel supervision mechanism of QueryInst.In addition, it is compared with non-parallel structure, and the superior performance of the proposed parallel structure is further highlighted.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent automatic driving car environment perception, and particularly relates to a multi-stage instance segmentation method and system based on parallel feature completion. BACKGROUND

[0002] Instance segmentation is a basic yet challenging computer vision task that requires identifying objects in an image and segmenting each object of interest at the pixel level. Therefore, applying instance segmentation algorithms to autonomous driving scenarios can simultaneously distinguish between objects of interest such as pedestrians, cars, and cyclists at both the instance and pixel levels, which has engineering significance.

[0003] Instance segmentation methods can be broadly classified into single-stage, two-stage, and multi-stage methods based on the number of stages for generating masks. Single-stage methods vary in their mask generation processes, with the key being whether the mask modeling and supervision are coupled. The advantage of single-stage methods is fast inference speed, but the accuracy is not high enough. Two-stage methods are generally based on Mask R-CNN, which first obtains the region of interest and then introduces a mask branch to binarize the local region. Two-stage methods generally have higher accuracy than one-stage methods, but are slightly slower. Multi-stage methods are an extension of two-stage methods, aiming to achieve higher accuracy. Currently, mainstream instance segmentation methods are based on attention mechanisms, which encode-decode object queries by combining mask information and object queries to obtain segmentation results. Therefore, considering the accuracy requirements of autonomous driving cars, algorithms that combine multi-stage and attention mechanisms are increasingly attracting attention from academia and industry.

[0004] QueryInst is the first multi-stage method that integrates attention mechanisms, but the single nature of RoI (region of interest) feature completion, the lack of connection with the backbone network ResNet50, and the locality and passivity of RoIAlign (region of interest align) all contribute to the inadequacy of RoI features. The inadequacy of RoI features leads to the inability to distinguish between background features, resulting in the failure to distinguish objects of interest at the instance level and to segment them at the pixel level. Therefore, it is worth studying how to complete the features of QueryInst to achieve more comprehensive instance segmentation. SUMMARY

[0005] To solve the problem, the application provides a multi-stage instance segmentation method and system based on parallel feature completion, which is used for parallel feature completion of each stage of the QueryInst method, so that the instance information on the feature layer and the surrounding background information are more easily distinguished, and the attention object of segmentation is more comprehensive, so as to improve the precision of the QueryInst method. The specific technical scheme is as follows:

[0006] A multi-stage instance segmentation method based on parallel feature completion, comprising the following steps:

[0007] S1 data preprocessing;

[0008] S2 backbone network ResNet50 feature extraction with global context;

[0009] S3 input the feature into the QueryInst-Parallel Completion network head to obtain the completed mask result;

[0010] S4 parallel semantic branch feature completion;

[0011] S5 parallel global branch feature completion;

[0012] S6 parallel feature transmission, output segmentation result.

[0013] Further, the data preprocessing of S1 is as follows:

[0014] S1.1 load the data set from the local file,

[0015] S1.2 load the label file from the local file;

[0016] S1.3 resize the input image and label;

[0017] S1.4 random flip processing;

[0018] S1.5 normalization and boundary zero padding processing to obtain an image in digital form with three channels.

[0019] Further, the S2 is as follows:

[0020] Insert the GC Block into the backbone network ResNet50, capture long-distance dependency according to each query position, extract global context information, and then aggregate the information into the feature of each query position;

[0021] The logic framework of the GC Block comprises:

[0022] (a) modeling the context relationship of the input feature map to obtain global context features;

[0023] (b) converting the global context features to capture channel dependency relationships;

[0024] (c) aggregating the features with global context information and the features of each corresponding position to make the features more complete;

[0025] The feature extraction steps are as follows:

[0026] First, the feature map is globally context modeled. For a query position j in the feature map, first, the j position relationship weight is obtained through 1x1 convolution, then SoftMax normalization processing is performed, and finally the corresponding multiplication with the j position feature is performed and the sum of all query positions is obtained to obtain the corresponding global context feature j a j x j ;

[0027] Then, the feature conversion δ is performed. In order to reduce the calculation amount caused by the increase in the number of channels in the deeper layer, the 1x1 convolution is set as a bottleneck transformation, so that the number of channels after convolution is C / r, where r is the bottleneck ratio. The obtained global context feature has channel dependency relationship;

[0028] Finally, the element positions are added by using the broadcast mechanism to obtain the complete feature.

[0029] Further, the S3 is specifically as follows:

[0030] The feature obtained by S2 is input into the QueryInst-Parallel Completion network head, and a random generated bounding box is initialized to perform RoIAlign processing, parallel semantic branch processing and parallel global branch processing on the feature, to obtain RoI features with complete semantic information and global information. A random initialized object query is generated, which is processed by MSA to obtain an enhanced object query. The enhanced object query and the RoI feature are input into the bounding box dynamic interaction module to obtain an enhanced bounding box RoI feature and an object query of the next stage. Then, the enhanced bounding box RoI feature is input into the detection branch to obtain the bounding box information. After that, the mask generation link is entered. According to the generated bounding box, the positive and negative samples are divided, and according to the loss cost, the positive sample with smaller loss is selected. The positive sample is processed by RoIAlign, parallel semantic branch processing and parallel global branch processing to obtain RoI features with complete semantic information and global information. At the same time, the enhanced bounding box RoI feature after the bounding box dynamic interaction in the detection part is indexed according to the positive sample, and the corresponding enhanced bounding box RoI feature is filtered out and input into the feature transmission part. The enhanced mask RoI feature obtained by the dynamic interaction of the mask is summed to realize completion. Finally, the completion result is input into the mask branch to obtain the mask result.

[0031] Further, the implementation of S4 includes:

[0032] The feature layers of the first two stages of the backbone network ResNet50 are taken as inputs, which are stages 0 and 1 respectively. First, 1x1 convolution is performed for feature conversion to make it further close to semantic features. Then, the output of stage 1 is up-sampled and aligned with the results of stage 0. The fused features are obtained by element-wise addition. After that, the fused features are input into the pyramid pooling module. Specifically, the fused feature layer is divided into 6x6, 3x3, 2x2, and 1x1 grids. Each grid performs average pooling respectively. The pooling results are up-sampled and aggregated. Then, 4 3x3 convolutions and 1x1 convolution are performed to adjust the channel to adapt to the dimension of the RoI feature. Finally, the obtained semantic features are fused with the RoI features to make them have semantic information.

[0033] Further, the implementation of S5 includes:

[0034] The features of stages 2 and 3 of the backbone network ResNet50 and the down-sampled stage 4 feature are taken as inputs. First, 1x1 convolution is performed for feature channel number adjustment. Then, it is aligned to stage 4. After that, the fused features are obtained by element-wise addition. Then, 4 3x3 convolutions are performed to extract global information and flatten the fused features. A fully connected layer is used to assemble the local information to obtain a feature vector with high global information. The number of sub-branches and parallel semantic branches is the same. One of them supervises the feature vector by predicting the class of each object of interest through image classification. The other branch outputs the global feature to complete the RoI feature.

[0035] Further, the implementation of S6 includes:

[0036] The enhanced bounding box feature after dynamic interaction of the detection branch is taken as the input of parallel feature transmission. Then, the enhanced bounding box feature is filtered according to the positive sample index, corresponding to the enhanced RoI feature of the mask branch. Then, it is sent into the fully connected layer. After that, shape remodeling processing is performed to align the mask feature. Finally, the fusion with the mask feature is realized through up-sampling and element-wise summation.

[0037] Further, it also includes S7 non-parallel feature transmission, which includes the following specific implementation: four kinds of mask information flow horizontal interaction are designed. The first stage interaction: four convolutions at the same position of the previous stage are used. The second interaction is to fully utilize the dynamic mask interaction module of the previous stage. The third interaction is to fuse the enhanced mask features between stages. The fourth interaction is to unify the previous three non-parallel structures and completely interact the mask branch horizontally.

[0038] Further, it also includes S8 design of loss function:

[0039] The overall loss function is composed of the following parts:

[0040]

[0041] For the detection branch, λ cls , λ reg , λ giou are 2, 5, 2 respectively; for the segmentation branch, β is 8, the loss function is DiceLoss, and for the parallel semantic branch, a cross-entropy loss function is used for supervision, and the specific formula is as follows:

[0042]

[0043] For the parallel global branch, the application adopts a binary cross-entropy loss function for supervision of multiple target classes:

[0044]

[0045] The application also proposes a multi-stage instance segmentation system based on parallel feature completion, which includes a data preprocessing module, a backbone network ResNet50 feature extraction, a parallel semantic branch module, a parallel global branch module, an MSA (multi-head attention mechanism) module, a mask dynamic interaction module, a bounding box dynamic interaction module, a feature transmission module, a detection branch module and a mask branch module.

[0046] The relationship and function of each module in the system are as follows:

[0047] The data preprocessing module is the starting stage of the entire system. Then the parallel processing of the backbone network ResNet50 feature extraction and the MSA module is carried out. After ResNet50 feature extraction, the parallel semantic branch module and the parallel global branch module are processed. Then the bounding box dynamic interaction module combines the outputs of the parallel semantic branch module, the parallel global branch module and the MSA module to obtain the interaction result. The interaction result is input to the detection branch to obtain the detection result. The detection result is processed again by the parallel semantic branch module and the parallel global branch module to obtain the fused features. At the same time, the feature transmission module outputs the transmission features. The mask dynamic interaction module processes the outputs of the fused features, the transmission features and the MSA module to obtain the mask features. Finally, the mask branch module converts the mask features into instance mask results. The above process is iterated 6 times, and the last iteration is taken as the output of the system.

[0048] The data preprocessing module: adjusts the size, randomly flips and normalizes the input image and label to make the obtained image data more standardized. The image data is then used as the input of the system.

[0049] The backbone network feature extraction extracts features of image data, uses shallow features to distinguish simple target areas, uses deep features to distinguish complex target areas, and enables the system to have certain robustness.

[0050] The parallel semantic branch module strengthens the connection between the ResNet50 features of the backbone network and the RoI features, and better distinguishes foreground points and background points.

[0051] The parallel global branch module actively extracts features, and the extracted features are independent of the quality of the bounding box and irrelevant to the quality of the bounding box. Overall, it compensates for the passivity and locality of the RoI alignment operation, and also strengthens the connection with the backbone network.

[0052] The MSA module processes all queries through attention mechanism, and strengthens the connection between the queries.

[0053] The bounding box dynamic interaction module decodes the RoI features of the query bounding box, and the instance information such as the center position and the category information of the query itself is given to the RoI features of the bounding box. This stage is a transition to facilitate the processing of the subsequent stage.

[0054] The mask dynamic interaction module decodes the RoI features of the query mask, and the instance information such as the shape information of the query itself is given to the RoI features of the mask. This stage is a transition to facilitate the processing of the subsequent stage.

[0055] The feature transmission module establishes a channel between the connected and interactive mask features and the bounding box features. The bounding box features are used to complete the mask features, and the mask features are used to correct the bounding box features through gradient backpropagation, so as to improve the accuracy of each subtask.

[0056] The detection branch module performs regression on the RoI features of the bounding box to obtain the detection result, which is used as the input of the next stage and the input of the mask branch.

[0057] The mask branch module performs regression on the RoI features of the mask to obtain the mask result.

[0058] The system of the application is specifically implemented as follows: first, data preprocessing is performed on the input two-dimensional image, specifically, including loading a data set from a local file, then loading a label file from a local file, adjusting the size of the input image and label, then performing random flip processing to enhance the robustness of the model, and finally performing normalization and boundary zero padding processing to obtain an image in digital form with three channels.

[0059] The three-channel digital form image is input to a backbone network ResNet50 for feature extraction.

[0060] The feature extraction of the backbone network ResNet50 is the input of each structure module, therefore, in order to make the feature extracted by the backbone network ResNet50 more sufficient, the application introduces a GC Block (global context relationship module) in the backbone network ResNet50, which can capture long-distance dependency relationship, so that the feature layer output by the module has more complete information.

[0061] The dynamic interaction module can be considered as the completion of the RoI feature by the object query, but the way is too single, the diversity of the connection with the backbone network ResNet50 and the locality and passivity of the RoIAlign lead to that the instance information of the backbone network ResNet50 is not fully extracted. Therefore, the application proposes a parallel semantic branch to extract the semantic features of the backbone network ResNet50 and complete the RoI feature, so that the RoI feature has semantic information.

[0062] The application also proposes a parallel global branch to complete the global information of the RoI feature. The difference between the parallel global branch and the parallel semantic branch is not only the different types of extracted information, but more importantly, the parallel global branch does not involve the RoIAlign operation, so it is not passive and has nothing to do with the advantages and disadvantages of the bounding box.

[0063] The introduction of the two parallel branches fully utilizes all feature layers of the backbone network ResNet50 and strengthens the connection with the backbone network, so the completion way is more comprehensive. The introduction of the parallel branch is a direct feature completion, and the application also theoretically analyzes the gradient backpropagation path of QueryInst and proposes a feature transfer method for indirect feature completion. Specifically, the structure establishes an explicit connection between the detection branch and the segmentation branch, fully utilizes the mutual relationship between the detection and segmentation subtasks, enhances the detection box feature to provide prior knowledge for the enhanced mask feature, and the mask branch refines the bounding box feature through gradient backpropagation, the close coupling relationship finally feeds back to the RoI feature, and then realizes the completion.

[0064] The application has the following beneficial effects:

[0065] 1. In view of the single nature of QueryInst for RoI feature completion, the insufficient connection with backbone network ResNet50, and the locality and passivity of RoIAlign, which cause the problem of insufficient RoI features, the present application proposes a novel parallel feature completion network framework system QueryInst-Parallel Completion around feature completion.

[0066] 2. GC Block is introduced into the backbone network ResNet50 to obtain more sufficient instance correlation information.

[0067] 3. A parallel semantic branch module is proposed to complete the semantic information of RoI features.

[0068] 4. A parallel global branch module is proposed to complete the global information of RoI features.

[0069] 5. A feature transmission module is proposed to increase the explicit connection between detection and segmentation branches and the explicit interaction of RoI features.

[0070] 6. The above structures are all parallel structures, and the present application also proposes a non-parallel structure for comparison, further highlighting the advantages of the proposed parallel structure and the parallel supervision mechanism of QueryInst. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 The flowchart of the present application

[0072] Figure 2 The overall framework diagram of the network head of QueryInst-Parallel Completion

[0073] Figure 3 The structure diagram of GC Block (Global Context Module)

[0074] Figure 4 The structure diagram of parallel semantic branch

[0075] Figure 5 The structure diagram of parallel global branch

[0076] Figure 6 The structure diagram of parallel feature transmission

[0077] Figure 7 The structure diagram of non-parallel structure DETAILED DESCRIPTION

[0078] QueryInst is the first multi-stage method to integrate an attention mechanism, but its limited feature completion for RoIs, insufficient connection with the ResNet50 backbone network, and the locality and passivity of RoIAlign all contribute to inadequate RoI features. To address this issue, this invention improves upon the method by proposing a novel parallel feature completion network framework, QueryInst-Parallel Completion. First, a GC Block is introduced into the ResNet50 backbone network to obtain instance association information. Then, parallel semantic branches and parallel global branches are proposed to extract semantic and global information from the feature layer, achieving RoI feature completion. Furthermore, a feature transfer structure is proposed, which indirectly completes RoI features by explicitly increasing the connection between the detection and segmentation branches and altering the gradient backpropagation path. It is worth emphasizing that all of the above structures are parallel, meaning there is no interference between stages, and it aligns with the parallel supervision mechanism of QueryInst. A comparison with non-parallel structures further highlights the superior performance of the proposed parallel structure.

[0079] The invention will now be further described with reference to the accompanying drawings.

[0080] like Figure 1 , 2 As shown, the implementation of the present invention includes the following steps:

[0081] Step 1: Data Preprocessing

[0082] First, prepare the labeled datasets for model training: the COCO dataset and the Cityscapes dataset. During model training, the image data is preprocessed using the following methods: (1) Load the dataset from a local file. (2) Load the label file from a local file. (3) Resize the input images and labels. (4) Perform random flipping to enhance the robustness of the model. (5) Normalize and pad the edges with zeros to obtain images in digital form with three channels.

[0083] Step 2: Feature Extraction from the ResNet50 Backbone Network with Global Context

[0084] The parallel branch structure and the region of interest alignment operation need to take the feature layer output by the backbone network ResNet50 as input, so it is necessary to optimize the structure of the backbone network ResNet50, so that the feature layer output by the backbone network ResNet50 has more complete context information. The traditional neural network extracts local features through convolution, in order to obtain global context information, it is necessary to stack the depth direction of convolution to expand the receptive field to model the global relationship. In order to pursue a more effective way, the application inserts the GC Block into the backbone network ResNet50, according to each query position, captures long-distance dependency relationship, extracts global context information, and then aggregates this information into the features of each query position.

[0085] As shown in Figure 3 The logical framework of the GC Block module mainly consists of the following three parts: (a) modeling the context relationship of the input feature map to obtain global context features (b) converting the global context features to capture channel dependency relationship (c) aggregating the features with global context information with the features of each corresponding position to make the features more complete.

[0086] The steps of feature extraction are specifically implemented as shown in Figure 3 (b): first, the feature map is globally context modeled, for the query position j in the feature map, first get the j position relationship weight through 1x1 convolution, then perform SoftMax normalization processing, finally multiply the corresponding features of j position and sum (matrix multiplication) all query positions to obtain the corresponding global context features j a j x j Then, the feature conversion δ is performed, in order to reduce the calculation amount caused by the increase of the channel number in the deeper layer, the application sets the 1x1 convolution as a bottleneck transformation module, so that the channel number after convolution is C / r, wherein r is a bottleneck ratio. The obtained global context features have channel dependency relationship. Finally, the element positions are added by using the broadcast mechanism to obtain complete features.

[0087] Step 3 uses the MSA module, the boundary box dynamic interaction module, the mask dynamic interaction module, the detection branch and the mask branch of the system to process the features

[0088] The application inputs the obtained features into the QueryInst-Parallel Completion network head, and the specific implementation is as shown in Figure 2The RoI features are obtained by simultaneously initializing the random generated bounding box, RoIAlign processing of the features, parallel semantic branch processing and parallel global branch processing, and the RoI features are supplemented with semantic information and global information. The object query is randomly initialized, and the enhanced object query is obtained through MSA processing. The enhanced object query and the RoI features are input into the bounding box dynamic interaction module to obtain enhanced bounding box RoI features and the object query of the next stage. Then, the enhanced bounding box RoI features are input into the detection branch to obtain the bounding box information such as the bounding box position coordinates and the confidence score. Then, the mask generation link is entered. The positive and negative samples are divided according to the generated bounding box, and the positive sample with smaller loss is selected according to the loss cost. The positive sample is also subjected to RoIAlign processing, parallel semantic branch processing and parallel global branch processing to obtain RoI features supplemented with semantic information and global information. Meanwhile, the enhanced bounding box RoI features after the bounding box dynamic interaction module in the detection part are indexed according to the positive sample, and the corresponding enhanced bounding box RoI features are screened and input into the feature transmission module. The enhanced mask RoI features are obtained through the mask dynamic interaction module, and the elements are summed to realize the completion. Finally, the completion result is input into the mask branch to obtain the mask result. The above part is expressed by the formula as follows:

[0089] The detection process is expressed by the formula as follows:

[0090]

[0091]

[0092]

[0093]

[0094] The segmentation process is expressed by the formula as follows:

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] wherein b t-1 is the bounding box of the t-1 stage; x gc is the feature map after GC Block processing; G is the parallel global branch module; S is the parallel semantic branch module; P box and P maskare 7x7 and 14x14 size alignment operations on feature maps, respectively; and are 7x7 and 14x14 size RoI features, respectively; MSA t is a multi-head attention mechanism module for t stage; q t-1 is a query input to t stage; is the converted query of t stage; and are the bounding box and mask RoI features enhanced by the dynamic interaction module, respectively; and are the bounding box and mask dynamic interaction modules, respectively; B t and M t are the detection branch and the mask branch, respectively; FT is a feature transfer module; b t and m t are the detection result bounding box and the instance segmentation result instance mask, respectively.

[0101] The above is only the result of one stage, therefore, the present application inputs the bounding box information and the object query obtained by the bounding box dynamic interaction module to the next stage. The overall cycle ends after 6 stages. The detection branch and the mask branch in each stage and the parallel semantic branch and the parallel global branch are supervised at the same time. The specific parallel branch and the feature transfer structure will be introduced in the following steps.

[0102] Step 4 Parallel semantic branch feature completion

[0103] In order to make full use of the complete features provided by the backbone network ResNet50, inspired by the HTC semantic branch, the present application builds a parallel semantic branch to strengthen the connection between the backbone network ResNet50 features and the RoI features. Compared with the original RoIAlign operation, the parallel semantic branch of the present application has obvious purpose and pertinence in extracting RoI features, because it is supervised by a sub-branch which is cross-entropy loss between each pixel of the image and the label through a traditional semantic segmentation loss function, so as to ensure that another sub-branch outputs semantic features. Moreover, the semantic features output by the parallel semantic branch are equivalent to the results of the traditional semantic segmentation network, which have better distinction between foreground points and background points.

[0104] Further, the newly added parallel semantic branch realizes the interaction between the segmentation task and the original task, and the close combination between each task helps to improve the feature representation and brings performance gain to the original task. The specific implementation is expressed by the formula as follows:

[0105]

[0106]

[0107] Figure 4 (a) shows the specific structure of the parallel semantic segmentation branch of the present application. Unlike the semantic branch mentioned in HTC, the present application only takes the feature layers of the first two stages of the backbone network ResNet50 as input, which are stage 0 and stage 1 respectively. The reduction of the number of input feature layers reduces the upsampling operation and the corresponding convolution operation, and reduces the computational complexity. While deeper semantic information is obtained through the top-down path of the feature pyramid, the GC Block of the backbone network ResNet50 and the feature pyramid pooling module. The PPM module structure is shown in Figure 4 (b) shows that first, the feature is converted through 1x1 convolution to further approach the semantic feature, and then the stage1 output result is aligned with the stage0 result through upsampling, and the fused feature is obtained by element-wise addition. After that, the fused feature is input into the PPM module, and specifically the fused feature layer is divided into 6x6, 3x3, 2x2, and 1x1 grids, and each grid is averaged pooled respectively, and the pooling results are aggregated through upsampling, and then 4 3x3 convolutions and 1x1 convolution are used to adjust the channel to adapt to the dimension of the RoI feature. Finally, the obtained semantic feature is fused with the RoI feature to complete it and make it have semantic information.

[0108] Step 5 parallel global branch feature completion

[0109] In order to make full use of the features of stages 2 and 3, and inspired by the SCNet global branch, the present application also constructs a parallel global branch. Compared with the parallel semantic branch, it focuses more on the extraction of global information. Therefore, more global instance correlation information is completed to the RoI feature. It is worth mentioning that compared with the original RoIAlign operation, the parallel global branch of the present application is more active in feature extraction, which is specifically manifested in that the extracted features are independent of the quality of the bounding box and are irrelevant to the quality of the bounding box. Overall, it makes up for the passivity and locality of RoIAlign, and also strengthens the connection with the backbone network. The specific implementation is represented by the formula as follows:

[0110]

[0111]

[0112] The specific parallel global branch structure is shown in Figure 5The application takes the stage 2 and 3 and the down-sampled stage 4 features of the backbone network ResNet50 as input, first adjusts the feature channel number through 1x1 convolution, then aligns to stage 4, and then adds according to the element correspondence to obtain the fusion features, and then extracts and flattens the global information of the fusion features through four 3x3 convolutions, and assembles the local information through a fully connected layer to obtain a feature vector with high global information. The number of sub-branches is the same as the number of parallel semantic branches, wherein the branch outputting the multi-label class prediction realizes the supervision of the feature vector by predicting the class of each object of interest through image classification, and the branch outputting the global vector is the output of the global feature, which is completed to the RoI feature.

[0113] Step 6 Parallel feature transmission

[0114] Whether explicit or implicit, the interaction between sub-tasks is reciprocal. QueryInst realizes the implicit interaction between the detection and segmentation sub-tasks through shared object queries and MSA. Inspired by the SCNet feature relay structure, the application proposes a parallel feature transmission structure for explicit interaction. In general, the channels of the mask feature and the bounding box feature after connection interaction are established. The mask feature is completed by the bounding box feature, and the mask feature is used to correct the bounding box feature through gradient backpropagation, so as to improve the accuracy of each sub-task. The parallel feature transmission structure of the application is an indirect feature completion compared with the parallel branch. Specifically, when the gradient is backpropagated, the calculated loss is backpropagated to the RoI feature before interaction and the object query. Parallel feature transmission improves the head gradient information and optimizes the gradient update of the RoI feature.

[0115] Figure 6 Details of the parallel feature transmission structure of the application are shown. Specifically, the application takes the enhanced bounding box feature after dynamic interaction of the detection branch as the input of the parallel feature transmission, then filters the enhanced bounding box feature according to the positive sample index, and corresponds to the enhanced RoI feature of the mask branch one by one, then sends it into the fully connected layer, then performs reshape processing, aligns the mask feature, and finally realizes the fusion with the mask feature through upsampling and element-wise summation.

[0116] Step 7 Non-parallel structure

[0117] The branch structure and feature transmission of the present application have the characteristic of parallelism. In general, their feature complementation all occurs within each stage, and follows the parallel supervision mechanism of the QueryInst algorithm, complementing each other and achieving good results. Specifically, the semantic branch and the global branch insert the extracted semantic and global features into each stage in parallel to complement the RoI features. The feature transmission structure uses the enhanced features of the detection branch in this stage to complement the enhanced features of the mask branch. There is no interference between the previous stage and the next stage of the above structure. In order to prove the superiority and effectiveness of the above three parallel feature complementations, the present application proposes a non-parallel structure compared with the parallel structure, that is, the different stages interact with each other. Referring to the mask information flow interaction of HTC, the present application makes four improvements on the QueryInst algorithm for horizontal interaction of mask information flow, specifically as shown in Figure 7 .

[0118] The overall logical framework is shown in Figure 7 . The essence of the four structures is the horizontal interaction between stages. The first structure represents the use of four convolutions at the same position in the previous stage before performing convolution in the current stage. The second structure makes full use of the dynamic mask interaction module of the previous stage. The third structure performs fusion of enhanced mask features between stages. The fourth structure unifies the first three non-parallel structures and completely interacts the mask branch horizontally. The formula corresponding to the occurrence of the above four structures is:

[0119]

[0120]

[0121]

[0122]

[0123]

[0124] Step 8 Model training and loss function setting

[0125] The overall loss function is composed of the following parts:

[0126]

[0127] For the detection branch, λ cls , λ reg , λ giou2, 5, 2. For the segmentation branch, the application follows the hyperparameter settings of QueryInst, and β is 8, and γ and δ represent the weight coefficients of the loss functions of the parallel semantic branch module and the parallel global branch module, respectively, and the values are 0.01 and 0.1, and the loss function is Dice Loss, represents the class loss function used by the t-stage detection branch, which is used to supervise the output class results. represents the bounding box loss function used by the t-stage detection branch, which is used to supervise the output bounding box position. represents the auxiliary bounding box loss function used by the t-stage detection branch, which is used to supervise the output bounding box shape. represents the mask loss function used by the t-stage segmentation branch, which is used to supervise the output instance mask. sema represents the segmentation loss function used by the parallel semantic branch module, which is used to supervise the output semantic segmentation result. global represents the global loss function used by the parallel global branch module, which is used to supervise the output multi-class result. For the parallel semantic branch, the application adopts the cross-entropy loss function for supervision, and the specific formula is as follows:

[0128]

[0129] For the parallel global branch, the application adopts the binary cross-entropy loss function for supervision of multiple target classes:

[0130]

[0131] CE stands for Cross Entropy, which represents the cross-entropy loss function. The semantic segmentation result s and the label are calculated by cross-entropy loss.

[0132] BCE stands for Binary Cross Entropy, which represents the binary cross-entropy loss function. The multi-class label result c and the classification label are calculated by binary cross-entropy loss.

[0133] In summary, the application proposes a multi-stage instance segmentation method based on parallel feature completion.

[0134] The above series of detailed descriptions are only specific descriptions of the feasible embodiments of the application, and are not intended to limit the protection scope of the application. Any equivalent means or changes that do not deviate from the technology of the application shall be included in the protection scope of the application.

Claims

1. A multi-stage instance segmentation method based on parallel feature completion, characterized in that, Comprising the following steps: S1 data preprocessing; S2 backbone network feature extraction with global context; The S2 is specifically as follows: Inserting the GC Block into the backbone network, capturing long-distance dependency relationships according to each query position, extracting global context information, and then aggregating this information into the features of each query position; The logical framework of the GC Block includes: (a) modeling the context relationship of the input feature map to obtain global context features; (b) converting the global context features to capture channel dependency relationships; (c) aggregating the features with global context information with the features of each corresponding position to make the features more complete; The feature extraction step is as follows: First, the feature map is globally context modeled. For the query position j in the feature map, first, the j position relationship weight is obtained through 1x1 convolution, then SoftMax normalization processing is performed, finally the corresponding multiplication with the j position feature is performed, and the sum of all query positions is obtained to obtain the corresponding global context feature ; Then feature conversion is performed In order to reduce the calculation amount caused by the increase of the number of channels in the deeper layer, the 1x1 convolution is set as the bottleneck conversion, so that the number of channels after the convolution is C / r, wherein r is a bottleneck ratio, and the obtained global context feature has a channel dependency relationship. Finally, the complete features are obtained by adding the element positions using the broadcast mechanism; S3 input the features into the QueryInst-Parallel Completion network head to obtain the completed mask result; The S3 is specifically as follows: The features obtained in S2 are input into the QueryInst-Parallel Completion network head, and the random generated bounding box is initialized to perform RoIAlign processing, parallel semantic branch processing and parallel global branch processing, to obtain RoI features with completed semantic information and global information, and a random initialized object query is generated, which is processed by MSA to obtain an enhanced object query. The enhanced object query and the RoI features are input into the bounding box dynamic interaction module to obtain enhanced bounding box RoI features and the next stage object query, and then the enhanced bounding box RoI features are input into the detection branch to obtain the bounding box information. Then, enter the mask generation link, divide the positive and negative samples according to the generated bounding box, select the positive samples according to the loss cost, perform RoIAlign processing, parallel semantic branch processing and parallel global branch processing on the positive samples to obtain RoI features with completed semantic information and global information, and index the enhanced bounding box RoI features after the bounding box dynamic interaction in the detection part according to the positive samples, filter out the corresponding enhanced bounding box RoI features, input them into the feature transmission part, and perform dynamic interaction with the mask to obtain enhanced mask RoI features. Perform element summation to realize completion, and finally input the completed result into the mask branch to obtain the mask result; S4 parallel semantic branch feature completion; The implementation of the S4 includes: The feature layers of the first two stages of the backbone network are taken as inputs, that is, stages 0 and 1, and first, 1x1 convolution is performed for feature conversion, so that the features are further close to semantic features, then the output result of stage 1 is up-sampled and aligned with the result of stage 0, and the fused features are obtained by element-wise addition; then the fused features are input into a pyramid pooling module, specifically, the fused feature layer is divided into 6x6, 3x3, 2x2 and 1x1 grids, each grid is averaged-pooled respectively, the pooled results are up-sampled and aggregated, and then four 3x3 convolutions and 1x1 convolution are performed to adjust the channels to adapt to the dimensions of the RoI features; finally, the obtained semantic features are fused with the RoI features to complete the RoI features, so that the RoI features have semantic information; S5 parallel global branch feature completion; S6 parallel feature transmission, output segmentation result.

2. The method of claim 1, wherein, The data preprocessing of S1 is specifically as follows: S1.1 loading a data set from a local file, S1.2 loading a label file from a local file; S1.3 adjusting the size of the input image and label; S1.4 random flip processing; S1.5 normalization and boundary zero padding processing to obtain an image in digital form with three channels.

3. The method of claim 1, wherein, The implementation of S5 includes: The features of stages 2 and 3 of the backbone network and the down-sampled stage 4 features are taken as inputs, first, 1x1 convolution is performed for feature channel number adjustment, then alignment is performed to stage 4, then element-wise addition is performed, to obtain fused features, and then four 3x3 convolutions are performed on the fused features for global information extraction and flattening, a full connection layer is used to assemble local information as a whole, to obtain a feature vector with high global information, the number of sub-branches is the same as the number of parallel semantic branches, wherein the branch outputting multi-label class prediction realizes supervision on the feature vector by predicting the class of each object of interest through image classification, and the branch outputting a global vector outputs a global feature to complete the RoI feature.

4. The method of claim 1, wherein, The implementation of S6 includes: The enhanced bounding box feature after dynamic interaction of the detection branch is taken as the input of parallel feature transmission, then the enhanced bounding box feature is filtered according to the positive sample index, and is one-to-one corresponding with the enhanced RoI feature of the mask branch, then is input into a full connection layer, then is subjected to shape remodeling processing, is aligned with the mask feature, and finally is fused with the mask feature through up-sampling and element-wise summation.

5. The method of claim 1, wherein, Further including S7 non-parallel feature transmission, and the specific implementation includes: designing four kinds of mask information flow horizontal interaction, the first stage interaction: using four convolutions at the same position of the previous stage; the second interaction is to fully utilize the dynamic mask interaction module of the previous stage; the third interaction is to fuse the enhanced mask features between stages; the fourth interaction is to unify the first three non-parallel structures, and completely interact the mask branches horizontally.

6. The method of claim 1, wherein, Further including S8 designing a loss function: The overall loss function is composed of the following parts: For the detection branch, , , For the segmentation branch, is 8, the loss function is DiceLoss, and respectively represent the weight coefficients of the loss functions of the parallel semantic branch module and the parallel global branch module, and for the parallel semantic branch, a cross-entropy loss function is used for supervision. For the parallel global branch, the present application adopts a binary cross-entropy loss function to supervise multiple target classes.

7. A system based on the multi-stage instance segmentation method based on parallel feature completion of claim 1, characterized in that, The system comprises a data preprocessing module, a backbone network feature extraction module, a parallel semantic branch module, a parallel global branch module, an MSA module, a mask dynamic interaction module, a bounding box dynamic interaction module, a feature transmission module, a detection branch module and a mask branch module. The data preprocessing is a starting stage of the whole system, and then the backbone network feature extraction and the MSA module are processed in parallel. After the backbone network feature extraction, the parallel semantic branch module and the parallel global branch module are processed. Then the bounding box dynamic interaction module processes the outputs of the parallel semantic branch module, the parallel global branch module and the MSA module to obtain an interaction result. The interaction result is input to the detection branch to obtain a detection result. The detection result is processed by the parallel semantic branch module and the parallel global branch module again to obtain fused features. Meanwhile, the feature transmission module outputs transmission features. The mask dynamic interaction module processes the outputs of the fused features, the transmission features and the MSA module to obtain mask features. Finally, the mask branch module converts the mask features into instance mask results. The above process is iterated for several times, and the last iteration is taken as the output of the system. The data preprocessing module adjusts the size, randomly flips and normalizes the input image and label, so that the obtained image data is more standardized, and the image data is taken as the input of the system. The backbone network feature extraction module extracts features from the image data, uses shallow features to distinguish simple target regions, and uses deep features to distinguish complex target regions, so that the system has robustness. The parallel semantic branch module strengthens the connection between the backbone network features and the RoI features, and better distinguishes foreground points and background points. The parallel global branch module actively extracts features, which are independent of the quality of the bounding box and irrelevant to the quality of the bounding box. Overall, it compensates for the passivity and locality of the RoI alignment operation, and strengthens the connection with the backbone network. The MSA module processes all queries through an attention mechanism to strengthen the connection between the queries. The bounding box dynamic interaction module decodes the RoI features of the query bounding box, and the instance information such as the center position and the category information of the query is assigned to the RoI features. This stage is a transition to facilitate the processing of the subsequent stage. The mask dynamic interaction module decodes the mask RoI features of the query, and the instance information such as the shape information of the query is assigned to the mask RoI features. This stage is a transition to facilitate the processing of the subsequent stage. The feature transmission module establishes a channel between the connected mask features and the bounding box features, uses the bounding box features to complete the mask features, and uses the mask features to correct the bounding box features through gradient backpropagation, so as to improve the accuracy of each subtask. The detection branch module regresses the RoI features of the bounding box to obtain a detection result, which is used as the input of the next stage and the input of the mask branch. The mask branch module regresses the mask RoI features to obtain a mask result.

Images