Training method for Transform instance segmentation model

Abstract

The invention discloses a training method based on a Transform instance segmentation model, and the method comprises the steps: carrying out the graphic transformation of a truth value mask, obtaining an enhanced truth value mask, calculating the mask attention based on the enhanced truth value mask in a training stage, and calculating the mask attention based on a prediction mask in an inference stage. Multi-path query feature propagation is executed in the training stage, that is, query features output by the (s-1) th decoding stage and the (s-2) th decoding stage are received in the s-th decoding stage to serve as input query features of the decoding stage. In the inference stage, the Transform instance segmentation model still keeps executing single-path query feature propagation, that is, the query feature output by the (s-1) th decoding stage is accepted as the input query feature of the decoding stage in the s decoding stage; the enhanced mask module (EMM) and the query multi-path propagation strategy (MPQ) provided by the invention are applied to different Transform instance segmentation models, and the instance segmentation precision of the models can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and is specifically applied to instance segmentation tasks based on a Transformer architecture. Background Art

[0002] Instance segmentation is a core task in the field of computer vision, which aims to simultaneously detect each independent object in an image and generate an accurate pixel-by-pixel mask for each object. The output of the instance segmentation model contains a category label and a binary mask, where the binary mask covers the contour information of the object, thereby achieving accurate segmentation of each instance in the image. Instance segmentation has a wide range of application value in the fields of autonomous driving, medical image analysis, and intelligent security.

[0003] With the rapid development of Transformer architecture, Transformer-based instance segmentation models have attracted widespread attention from researchers. Figure 1 The FastInst model shown is a typical Transformer-based instance segmentation model. Its core architecture includes five key components: backbone network, pixel decoder, query generator, Transformer decoder, and decoding head. The backbone network is responsible for extracting multi-scale (multi-resolution) semantic feature maps from the input image. These feature maps are denoted as where F i represents the i-th feature map, L represents the total number of feature maps, and the larger the i value, the smaller the corresponding feature map F i The lower the resolution, the richer the semantic information. The pixel decoder first decodes the feature map F L Perform the convolution operation and get the same as F L Feature maps F′ of the same scale L ; Then, for F′ L Upsample and get the same value as F L-1 Feature maps F″ of the same scale L , and F″ L With F L-1 Fusion, get F′ L-1 ; Then F′ L-1 Upsample to F L-2 size, and with F L-2 Fusion to get F′ L-2 , and so on, until F1′ is finally obtained through layer-by-layer upsampling and feature fusion. In this way, a multi-scale fusion feature containing rich semantic information is obtained

[0004] From multi-scale fusion features Select a fusion feature map F of a specific scale i 'Use the query generator to generate initial query features Query by N objects The N value is usually set to 100, 300 or 900. Figure 1 The FastInst network shown will be used to generate The fusion feature map F i 'Select F4'.

[0005] The initial query features and multi-scale fusion features or The fusion feature map F′ of a certain scale in j (where 1≤j≤L) are input to the decoding head H 0 , get the prediction mask Where H and W represent the height and width of the feature map respectively. is the object query The prediction mask of . Decoding header H 0 At the same time, the predicted category corresponding to each predicted mask is output. Next, the multi-scale fusion feature or The fusion feature map F′ of a certain scale in j With the initial query features Input into the Transformer decoder together. Figure 1 The FastInst network shown will be used to decode the "multi-scale fusion features or The fusion feature map F′ of a certain scale in j ” is selected as the fusion feature map F3′.

[0006] The Transformer decoder consists of S decoding stages T s (where 1≤s≤S) is composed of a cascade, and each decoding stage is equipped with a decoding head H s , usually S is set to 6. Define query features For those who have experienced 1 to T s-1 The query features updated after s-1 Transformer decoding stages, where s ≥ 2. Initial query features is a special case, which is generated by the query generator and is not updated by other Transformer decoding stages. The current methods for generating initial query features include zero initialization, normal distribution initialization, learnable parameter initialization, dynamic initialization based on regional features, and hybrid query selection. Figure 1In the FastInst model shown, the initial query features are generated by a dynamic initialization method based on regional features.

[0007] Decoding stage T s It consists of a sequentially connected masked-attention module, a self-attention module, and a feed-forward network (FFN). In the masked-attention module, each object query q n The self-attention module is used to infer the relationship between all object queries; the feedforward network further maps these object queries to a higher dimensional space, providing a rich query feature representation for the output of the current decoding stage. The process of generating initial query features is defined as a special decoding stage in this invention, called T 0 , T 0 is a special case of the decoding stage, which does not belong to the Transformer decoder and only contains a query generator. 0 Equipped with a decoding head H 0 .

[0008] Figure 1 The query features generated in the decoding process of the sth decoding stage are shown in is the input of the s+1th decoding stage. The query features generated by the sth decoding stage Together with the fused feature map F3′, it is sent to the decoding head H s , generate the prediction mask For example, the initial query features After T 1 Generate query features after update Then Together with F3′, it is input to the decoding head H 1 , get the predicted mask P 0→1 ; Query features Input to T 2 , get the query features Then, and F3′ are input to the decoding head H 2 , get the predicted mask P 0→1→2 And so on, until all predefined Transformer decoding stages and their corresponding decoding heads are executed. For the sake of simplicity, Figure 1 The data flow from the feature map F3' to the decoding head is not shown.

[0009] like Figure 1As shown in Figure 1, when training the Transformer instance segmentation model, the query features are propagated stage by stage along a fixed single path and are iteratively optimized and updated at each decoding stage. Figure 1 The iterative update process of the query features in can be expressed as

[0010] In the training phase of the model, the Hungarian matching algorithm is used to s Output of the nth prediction mask Assign true values, assign results using matching matrix Record, where N is the number of prediction masks, B is the number of true value masks, B ≤ N. Indicates that the Hungarian matching algorithm assigns the bth true value mask to the nth predicted mask, It means that the Hungarian matching algorithm does not assign a true value mask to the nth predicted mask. Then, the loss is calculated according to formula (1) to supervise the training of the model.

[0011] The total loss of S decoding stages is:

[0012]

[0013] Figure 1 Loss function L at each decoding stage s (0≤s≤S) as shown in formula (2):

[0014]

[0015] Among them, L ce and L dice They represent the binary cross entropy loss and dice loss of the predicted mask, respectively, and L cls is the cross entropy loss for predicting the masked object classification. ce ,λ dice and λ cls is a hyperparameter used to balance the three loss functions. ce =5.0;λ dice =5.0; when the true value is the “background” category cls The value is 0.1, otherwise it is 1. s (·,·) represents the decoding header H s , the first parameter is the query feature The second parameter is the fused feature map F3′.

[0016] Next, we discuss two issues in the Transformer-based instance segmentation method:

[0017] Question 1: The accuracy of the predicted mask output by the s-1th decoding stage has a direct impact on the mask attention calculation result of the sth decoding stage. To simplify the description, the predicted mask output by the sth decoding stage is abbreviated as P s In the s-th decoding stage, the model uses the predicted mask P from the s-1-th decoding stage s-1 Compute the attention mask The specific calculation method is shown in formula (3):

[0018]

[0019] Among them, P s-1 (:,x,y)=1 means that the decoding header H s-1 The pixel at position (x, y) is predicted as an instance. The mask attention calculation at the s-th decoding stage is shown in formula (4):

[0020]

[0021] in, is the query feature input to the s-th decoding stage; Yes Rearrange by row; They represent the query, key and value required for attention calculation respectively; is the query feature after being updated by the mask attention module in the s-th decoding stage; K s =f K (F3′), V s =f V (F3′), f Q 、f K 、f V They are all linear maps.

[0022] Formula (4) describes how the model uses the attention mask in the s-th decoding stage to focus on the foreground area predicted by the s-1 decoding stage. However, there is a key problem in this process: the prediction mask P generated by the s-1 decoding stage s-1 is not always completely accurate. This inaccuracy may cause the masked attention mechanism in the s-th decoding stage to fail to accurately focus on the true instance region. If the masked attention mechanism cannot accurately identify the instance region, it may mistakenly focus on the non-instance region, which will negatively affect the performance of the model, including reducing the accuracy of the prediction and slowing down the convergence of the model.

[0023] Problem 2: There are feature degradation and error cascading issues in the decoding stage of the cascade. As the model decodes stage by stage, the quality of the predicted mask and the classification confidence score are continuously corrected. In most cases, as the decoding stage progresses backward, the quality of the predicted mask will improve and its classification confidence score will increase accordingly. However, the model shows the opposite trend for some samples.

[0024] against Figure 1 We tested the FastInst model shown in (https: / / github.com / junjiehe96 / FastInst.git) on the COCO val2017 dataset. The test results show that the average segmentation accuracy (AP) of the FastInst model is 38.8%. Figure 2 and Figure 3 Visualization of the prediction results output by different decoding stages of the FastInst model for two sample images in the dataset, 000000000036.jpg and 000000579635.jpg.

[0025] When predicting the sample 000000000036.jpg, the mask prediction results of each decoding stage are as follows Figure 2 As shown. In the first decoding stage, both the umbrella and the person obtained high-confidence predictions (umbrela: 0.97, person: 0.98); in the second decoding stage, part of the umbrella was predicted to be a cellphone (cellphone: 0.06), and in the third decoding stage, part of the umbrella was predicted to be a cellphone (cellphone: 0.08). Usually, the model decides whether to use the prediction result as the final output based on a certain confidence threshold. In this case, the low-confidence prediction mask such as 0.08 will be discarded and not used as the model output result. Therefore, in the third decoding stage, the final prediction mask of the umbrella output by the model is not as complete as the prediction mask output in the first decoding stage.

[0026] When predicting the sample 000000579635.jpg, the mask prediction results at each decoding stage are as follows Figure 3 As shown. In the first decoding stage, the surfboard, person, and boat all obtained high confidence predictions (surtboard: 0.95, person: 0.98, boat: 0.87); in the second decoding stage, the prediction confidence of the boat increased (boat: 0.93), and the prediction confidence of the surfboard and person remained unchanged; in the third decoding stage, the prediction confidence of the surfboard and person decreased (surtboard: 0.65, boat: 0.91). Therefore, the confidence of the prediction mask about the surfboard output by the model in the third decoding stage is lower than the confidence of the prediction mask about the surfboard output by the model in the first decoding stage.

[0027] From the above analysis of the prediction results of sample 000000000036.jpg and sample 000000579635.jpg, we can observe that for some samples, the final prediction results of the FastInst model did not reach the expected optimal state, but were not as accurate as the prediction results of the early decoding stage. This phenomenon is contrary to the expectation that "the prediction results produced by the model after all decoding stages are the most accurate." Summary of the invention

[0028] The technical purpose of the present invention is to improve the aforementioned Problems 1 and 2, and to propose a new training method for a Transformer-based instance segmentation model to improve the segmentation accuracy and overall performance of the Transformer instance segmentation model.

[0029] In response to problem 1, the present invention proposes an Enhanced Mask Training Strategy (EMT). In the training stage, the core operation of the strategy is: calculating the attention mask based on the enhanced mask, and using the attention mask to calculate the attention score. Specific implementation steps: First, for the prediction mask output by the decoding head equipped in each decoding stage of the model, the Hungarian matching algorithm is used to assign the corresponding truth mask. Then, the truth mask is morphologically transformed to generate an enhanced truth mask, and the enhanced truth mask is used to replace the prediction mask assigned with the truth mask. The prediction mask that is not replaced and the enhanced truth mask together constitute an enhanced mask. Finally, the attention mask is calculated based on the enhanced mask, and the attention score is calculated using the attention mask. In the inference stage, the model calculates the attention mask based on the prediction mask, and then uses the attention mask to calculate the attention score.

[0030] In response to problem 2, the present invention proposes a query feature multi-path propagation training strategy (Multi-path Propagation of Queries Features, MPQ). In the training phase, multi-path propagation of query features is performed. To facilitate the implementation of multi-path propagation of query features, the present invention converts query features into Expand to The new dimension is named groupsize. G query features are stacked in the groupsize dimension. Form a query feature group, denoted as During the training phase, the input to the s-th decoding stage (s ≥ 2) of the Transformer decoder is the query feature set where Y s-1 and Y s-2 T s-1 and T s-2The number of query features output. In particular, the decoding stage T 1 The input is the query feature set In the inference phase, the Transformer instance segmentation model performs a single-path propagation of query features, that is, the s-th decoding stage only accepts the query features output by the s-1-th decoding stage. as input.

[0031] When the model is trained using the enhanced mask training strategy (EMT) and the query feature multi-path propagation training strategy (MPQ), the network structure of the Transformer-based instance segmentation model remains unchanged. Figure 5 As shown in Figure 1, we first use the backbone network to extract features from the input image and generate image feature maps of multiple scales. Multi-scale image feature map It is then sent to the pixel decoder to obtain multi-scale fusion features through layer-by-layer fusion.

[0032] From multi-scale fusion features Select a feature map F of a specific scale from i ′, input to the query generator to obtain the initial query features When the present invention is implemented, F i ' is selected as F4'. Then, the initial query feature and multi-scale fusion features or The fusion feature map F′ of a certain scale in j (where 1≤j≤L) are input to the decoding head H 0 , generate the prediction mask P 0→0 Next, the multi-scale fusion features or The fusion feature map F′ of a certain scale in j With the initial query features The multi-scale fusion feature or The fusion feature map F′ of a certain scale in j " is selected as F3'. In order to keep the diagram simple, Figure 5 The data flow from the feature map F3′ to the decoding head is not shown. The above settings are consistent with the existing work. The improvement of the present invention is reflected in the calculation of the Transformer decoder. The specific process is as follows:

[0033] The input of the 0th decoding stage is the fused feature map F4′, and the query generator is used to obtain the initial query feature Afterwards, And the fused feature map F3′ is input to the decoding head H 0 , get the predicted mask P 0→0 Next, predict the mask P 0→0 Together with the ground truth GT, it is input into the Mask Enhancement Module (MEM). In MEM, the Hungarian matching algorithm is used to achieve one-to-one matching between N predicted masks and B ground truth masks, where B ≤ N. For each ground truth mask, a random morphological transformation is applied to generate an enhanced ground truth mask. Then, the enhanced mask is calculated based on the enhanced ground truth mask and the predicted mask. Finally, the attention mask is calculated based on the enhanced mask

[0034] The first decoding stage T 1 The input is the query feature set and attention mask The masked attention module utilizes attention masks Calculate the attention score and use the attention score to Update to get This improvement helps the mask attention module to more accurately identify and focus on foreground features. Obtained sequentially through the self-attention module and the feed-forward network Next, query features And the fused feature map F3′ is input to the decoding head H 1 , generating the decoding stage T 1 The prediction mask P 0→1 Finally, the predicted mask P 0→1 Input into MEM together with the true value GT to obtain the attention mask

[0035] Second decoding stage T 2 The input is given by and The query feature group A obtained by stacking in the groupsize dimension 2 And the attention mask and The masked attention module utilizes attention masks right Update to get Using Attention Mask right Update to get Two sets of updated query features and By sequentially passing through the self-attention module and the feedforward network, we get and The query feature And the fused feature map F3′ is input to the decoding head H 2 , generate the prediction mask P for the second decoding stage 0→2 and P 0 →1→2 Then, the prediction mask P 0→2 , P 0→1→2 Input into MEM together with the true value GT to obtain the attention mask and

[0036] Similarly, starting from the second decoding stage, each decoding stage expands the query features output by the groupsize dimension of the two adjacent previous stages. Then, they are stacked in the groupsize dimension to form a query feature group as the input of the current decoding stage. Specifically, the s-th decoding stage accepts the query features output by the s-1-th decoding stage and the s-2-th decoding stage, expands these features by the groupsize dimension, and then stacks them in the groupsize dimension to form a query feature group A. s , A s Used as the input of the sth decoding stage. Let G s A s The size of the groupsize dimension is G 0 =G 1 =1; when s≥2, G s =G s-1 +G s-2 .

[0037] Furthermore, the training strategy formula of the present invention is described in two parts:

[0038] Part I: Enhanced Mask Training Strategy (EMT);

[0039] The present invention designs a mask enhancement module (MEM). The module receives the true value mask and the predicted mask as input and outputs the attention mask. Taking the s-th decoding stage as an example, the attention mask input to the mask attention module is The specific calculation process is as follows:

[0040] First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix Indicates that the Hungarian matching algorithm assigns the bth true value mask to the nth predicted mask, Indicates that the Hungarian matching algorithm did not assign a true value mask to the nth predicted mask.

[0041] Then, the mask enhancement module randomly selects one of the three sizes of 3×3, 5×5 or 7×7 as the size of the mask enhancement convolution kernel; after the size is selected, the convolution operation is performed on the true value mask. After the convolution is completed, one of the three operations, Dilate(·), Erode(·) and Comp(·), is selected and applied to the result of the convolution to obtain the enhanced true value mask. After replacing the matching predicted mask with the enhanced true value mask, the enhanced mask is obtained.

[0042] Finally, the attention mask is calculated based on the enhanced mask

[0043] The above attention mask The calculation process can be divided into two main steps, convolution operation and morphological transformation operation:

[0044] 1) Convolution operation

[0045] First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix

[0046] Then, calculate the truth mask Ground truth mask GT b Enhanced truth mask for (1≤b≤B) That is, according to formula (5), GT b Perform convolution operation to obtain a temporary matrix

[0047]

[0048] in, Represents the convolution operation, mask enhancement convolution kernel ω k×k It is a two-dimensional matrix with all elements being 1 and size k×k. The possible sizes are 3×3, 5×5, and 7×7.

[0049] 2) Morphological transformation operation

[0050] Next, the temporary matrix Randomly perform a morphological transformation operation to obtain the true value mask GT b The corresponding enhanced truth mask Specifically, the morphological transformation operations include three types: dilation operation (Formula 6), corrosion operation (Formula 7) and composite operation (Formula 8):

[0051] right Perform dilation operation Dilate(·) to obtain the dilation enhanced truth mask

[0052] right Perform the erosion operation Erode(·) to obtain the erosion enhanced truth mask

[0053]

[0054] right Perform the composite operation Comp(·) to obtain the edge enhancement truth mask

[0055]

[0056] The above calculations are performed during the network training process, and MEM is each truth mask GT b Randomly select mask-enhanced convolution kernel ω k×k size, randomly select a mask enhancement operation. If the dilation operation Dilate(·) is selected, then If the erosion operation Erode(·) is chosen, then If the composite operation Comp(·) is selected, then

[0057] Finally, according to formula (9), the enhanced mask is obtained

[0058]

[0059] in

[0060] Finally, according to formula (10), the required attention mask is calculated

[0061]

[0062] Input to the masked attention module of the s+1th decoding stage for masked attention calculation.

[0063] Take the mask attention calculation in the s+1th decoding stage as an example. Definition Represents the query feature set A input at the s+1th decoding stage s+1 The calculation process of updating a query feature in based on the attention mask is shown in formula (11):

[0064]

[0065] in, represents the mask attention module pair at the s+1th decoding stage The query features obtained after updating; Yes The attention mask obtained by rearranging rows; They represent the query, key and value required for attention calculation respectively. K s+1 =f K (F3′), V s+1 =f V (F3′), where f Q 、f K 、f V They are all linear transformations.

[0066] Part II: Multi-path Propagation of Queries Features (MPQ) training strategy;

[0067] Decoding stage T 0 Generate initial query features Then, Expanding on the groupsize dimension based on Get query feature group A 1 , Then, A 1 is sent to the first decoding stage T 1 , after processing, output query features Starting from the second decoding stage, the sth decoding stage accepts the query feature group A formed by stacking multiple query features output by the s-1th decoding stage and the s-2th decoding stage in the groupsize dimension s , A s As the input of the sth decoding stage. The specific form of the query feature group input in each decoding stage is:

[0068]

[0069] Among them, ·||· indicates that the query features are stacked in the groupsize dimension. When the query feature group passes through the mask attention module, the self-attention module and the feedforward network in sequence, each module performs independent calculations on the query features stacked in the groupsize dimension in the query feature group, and each query feature is independently assigned a true value mask and independently calculates the loss. Therefore, under the premise that the number of object queries contained in each query feature is fixed, the present invention uses the MPQ strategy in the training stage and does not use the MPQ strategy in the reasoning stage. As a result, the number of query features input to the network in the training stage is allowed to be greater than the number of query features input in the reasoning stage, thereby allowing the network to be more fully trained based on a larger number of object queries.

[0070] When training the model based on the query feature multi-path propagation training strategy, the total loss function of the S decoding stages is shown in formula (13):

[0071]

[0072] Figure 5 Loss function L at each decoding stage s (0≤s≤S) as shown in formula (14):

[0073]

[0074] Among them, L ce and L dice They represent the binary cross entropy loss and dice loss of the predicted mask, respectively, and L cls is the cross entropy loss for predicting the masked object classification. ce ,λ dice and λ cls is a hyperparameter used to balance the three loss functions. ce =5.0;λ dice =5.0; when the true value is the “background” category cls The value is 0.1, otherwise it is 1. As can be seen from formula (13), as the decoding stage deepens, the supervision information of each decoding stage gradually increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a Transformer-based instance segmentation network framework diagram (taking the FastInst instance segmentation model as an example).

[0076] Figure 2 This is a diagram showing the cascading errors in the prediction results of the FastInst instance segmentation model at each decoding stage.

[0077] Figure 3 This is a diagram showing the feature degradation phenomenon in the prediction results of the FastInst instance segmentation model at each decoding stage.

[0078] Figure 4 It is a visualization of the enhanced truth mask obtained by calculating the truth mask based on different enhancements

[0079] Figure 5 This is the training strategy framework diagram for the Transformer instance segmentation model proposed in this invention DETAILED DESCRIPTION

[0080] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0081] The technical solution adopted by the present invention is a training method for a Transformer instance segmentation model, which is divided into two training strategies:

[0082] Strategy 1 is the enhanced mask training strategy (EMT). In the training phase, the core operation of this strategy is to calculate the attention mask based on the enhanced mask and use the attention mask to calculate the attention score. Specific implementation steps: First, for the prediction mask output by the decoding head equipped in each decoding stage of the model, the Hungarian matching algorithm is used to assign the corresponding truth mask to it. Then, the truth mask is morphologically transformed to generate an enhanced truth mask, and the enhanced truth mask is used to replace the prediction mask assigned with the truth mask. The prediction mask that is not replaced and the enhanced truth mask together constitute the enhanced mask. Finally, the attention mask is calculated based on the enhanced mask, and the attention score is calculated using the attention mask. In the inference phase, the model calculates the attention mask based on the prediction mask, and then uses the attention mask to calculate the attention score.

[0083] The second strategy is the query feature multi-path propagation training strategy (MPQ). In the training phase, the query feature multi-path propagation is performed. To facilitate the implementation of the query feature multi-path propagation, the present invention converts the query feature Expand to The new dimension is named groupsize. G query features are stacked in the groupsize dimension. Form a query feature group, denoted as During the training phase, the input to the s-th decoding stage (s ≥ 2) of the Transformer decoder is the query feature set where Y s-1 and Y s-2 T s-1 and T s-2 The number of query features output. In particular, the decoding stage T 1 The input is the query feature set In the inference phase, the Transformer instance segmentation model performs a single-path propagation of query features, that is, the s-th decoding stage only accepts the query features output by the s-1-th decoding stage. as input.

[0084] When the model is trained using the enhanced mask training strategy (EMT) and the query feature multi-path propagation training strategy (MPQ), the network structure of the Transformer-based instance segmentation model remains unchanged. Figure 5 As shown in Figure 1, we first use the backbone network to extract features from the input image and generate image feature maps of multiple scales. Multi-scale image feature map It is then sent to the pixel decoder to obtain multi-scale fusion features through layer-by-layer fusion.

[0085] From multi-scale fusion features Select a feature map F of a specific scale from i ′, input to the query generator to generate initial query features When the present invention is implemented, F i ' is selected as F4'. Then, the initial query feature and multi-scale fusion features or The fusion feature map F′ of a certain scale in j (where 1≤j≤L) are input to the decoding head H 0 , generate the prediction mask P 0→0 Next, the multi-scale fusion features or The fusion feature map F′ of a certain scale in j With the initial query features The multi-scale fusion feature or The fusion feature map F′ of a certain scale in j " is selected as F3'. In order to keep the diagram simple, Figure 5 The data flow from the feature map F3′ to the decoding head is not shown. The above settings are consistent with the existing work. The improvement of the present invention is reflected in the calculation of the Transformer decoder. The specific process is as follows:

[0086] The input of the 0th decoding stage is the fused feature map F4′, and the query generator is used to obtain the initial query feature Afterwards, And the fused feature map F3′ is input to the decoding head H 0 , get the predicted mask P 0→0 Next, predict the mask P 0→0 Together with the true value GT, it is input into the Mask Enhancement Module (MEM). In MEM, the Hungarian matching algorithm is used to achieve a one-to-one match between N predicted masks and B true masks, where B ≤ N. For each true mask, a random morphological transformation is applied to generate an enhanced true mask. Then, the enhancement mask is calculated based on the enhanced true mask and the prediction mask. Finally, the attention mask is calculated based on the enhancement mask

[0087] The first decoding stage T 1 The input is the query feature set and attention mask The masked attention module utilizes attention masks Calculate the attention score and use the attention score to Update to get This improvement helps the mask attention module to more accurately identify and focus on foreground features. Obtained sequentially through the self-attention module and the feed-forward network Next, query features And the fused feature map F3′ is input to the decoding head H 1 , generating the decoding stage T 1 The prediction mask P 0→1 Finally, the predicted mask P 0→1 Input into MEM together with the true value GT to obtain the attention mask

[0088] Second decoding stage T 2 The input is given by and The query feature group A obtained by stacking in the groupsize dimension 2 And the attention mask and The masked attention module utilizes attention masks right Update to get Using Attention Mask right Update to get Two sets of updated query features and By sequentially passing through the self-attention module and the feedforward network, we get and The query feature And the fused feature map F3′ is input to the decoding head H 2 , generate the prediction mask P for the second decoding stage 0→2 and P 0 →1→2 Then, the prediction mask P 0→2 , P 0→1→2 Input into MEM together with the true value GT to obtain the attention mask and

[0089] Similarly, starting from the second decoding stage, each decoding stage expands the query features output by the groupsize dimension of the two adjacent previous stages. Then, they are stacked in the groupsize dimension to form a query feature group as the input of the current decoding stage. Specifically, the s-th decoding stage accepts the query features output by the s-1-th decoding stage and the s-2-th decoding stage, expands these features by the groupsize dimension, and then stacks them in the groupsize dimension to form a query feature group A. s , A s Used as the input of the sth decoding stage. Let G s A s The size of the groupsize dimension is G 0 =G 1 =1; when s≥2, G s =G s-1 +G s-2 .

[0090] Furthermore, the training strategy formula of the present invention is described in two parts:

[0091] 1. Enhanced Mask Training Strategy (EMT)

[0092] The present invention designs a mask enhancement module (MEM). The module receives the true value mask and the predicted mask as input and outputs the attention mask. Taking the s-th decoding stage as an example, the attention mask input to the mask attention module is The specific calculation process is as follows:

[0093] First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix Indicates that the Hungarian matching algorithm assigns the bth true value mask to the nth predicted mask, Indicates that the Hungarian matching algorithm did not assign a true value mask to the nth predicted mask.

[0094] Then, the mask enhancement module randomly selects one of the three sizes of 3×3, 5×5 or 7×7 as the size of the mask enhancement convolution kernel; after the size is selected, the convolution operation is performed on the true value mask. After the convolution is completed, one of the three operations, Dilate(·), Erode(·) and Comp(·), is selected and applied to the result of the convolution to obtain the enhanced true value mask. After replacing the matching predicted mask with the enhanced true value mask, the enhanced mask is obtained.

[0095] Finally, the attention mask is calculated based on the enhanced mask

[0096] The above attention mask The calculation process can be divided into two main steps, convolution operation and morphological transformation operation:

[0097] 1) Convolution operation

[0098] First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix

[0099] Then, calculate the truth mask Ground truth mask GT b Enhanced truth mask for (1≤b≤B) That is, according to formula (15), GT b Perform convolution operation to obtain a temporary matrix

[0100]

[0101] in, Represents the convolution operation, mask enhancement convolution kernel ω k×k It is a two-dimensional matrix with all elements being 1 and size k×k. The possible sizes are 3×3, 5×5, and 7×7.

[0102] 2) Morphological transformation operation

[0103] Next, the temporary matrix Randomly perform a morphological transformation operation to obtain the true value mask GT b The corresponding enhanced truth mask Specifically, the morphological transformation operations include three types: dilation operation (Formula 16), corrosion operation (Formula 17) and composite operation (Formula 18):

[0104] right Perform dilation operation Dilate(·) to obtain the dilation enhanced truth mask

[0105]

[0106] right Perform the erosion operation Erode(·) to obtain the erosion enhanced truth mask

[0107]

[0108] right Perform the composite operation Comp(·) to obtain the edge enhancement truth mask

[0109]

[0110] The above calculations are performed during the network training process, and MEM is each truth mask GT b Randomly select mask-enhanced convolution kernel ω k×k The size of the image is randomly selected as a mask enhancement operation. If the dilation operation Dilate(·) is selected, then If the erosion operation Erode(·) is chosen, then If the composite operation Comp(·) is selected, then

[0111] Finally, according to formula (19), the enhanced mask is obtained

[0112]

[0113] in

[0114] Finally, according to formula (20), when the query feature is input into the masked attention module of the s+1th decoding stage, the attention mask required for attention calculation is obtained.

[0115]

[0116] Input to the masked attention module of the s+1th decoding stage for masked attention calculation.

[0117] Take the mask attention calculation in the s+1th decoding stage as an example. Definition Represents the query feature set A input at the s+1th decoding stage s+1 The calculation process of updating a query feature in based on the attention mask is shown in formula (21):

[0118]

[0119] in, represents the mask attention module pair at the s+1th decoding stage The query features obtained after updating; Yes The attention mask obtained by rearranging rows; They represent the query, key and value required for attention calculation respectively. V s+1 =f V (F3′), where f Q 、f K 、fV They are all linear transformations.

[0120] Next, we will introduce the specific implementation of the query feature multi-path propagation training strategy (MPQ) in detail. 2. Multi-path Propagation of Queries Features (MPQ)

[0121] When query features are propagated through multiple paths, the data flow of query features at each decoding stage is as follows: Figure 5 This process starts from the initialization stage, that is, the decoding stage T 0 Start, decoding phase T 0 Generate initial query features Then, Expanding on the groupsize dimension based on Get query feature group A 1 , Then, A 1 is sent to the first decoding stage T 1 , after processing, output query features Starting from the second decoding stage, the sth decoding stage accepts the query feature group A formed by stacking multiple query features output by the s-1th decoding stage and the s-2th decoding stage in the groupsize dimension s , A s As the input of the sth decoding stage. The specific form of the query feature group input in each decoding stage is:

[0122]

[0123] Among them, ·||· indicates that the query features are stacked in the groupsize dimension. When the query feature group passes through the mask attention module, the self-attention module and the feedforward network in sequence, each module performs independent calculations on the query features stacked in the groupsize dimension in the query feature group, and each query feature is independently assigned a true value mask and independently calculates the loss. Therefore, under the premise that the number of object queries contained in each query feature is fixed, the present invention uses the MPQ strategy in the training stage and does not use the MPQ strategy in the reasoning stage. As a result, the number of query features input to the network in the training stage is allowed to be greater than the number of query features input in the reasoning stage, thereby allowing the network to be more fully trained based on a larger number of object queries.

[0124] When training the model based on the query feature multi-path propagation training strategy, the total loss function is shown in formula (23):

[0125]

[0126] Figure 5 In the decoding stage, the loss function L s (0≤s≤S) as shown in formula (24):

[0127]

[0128] Among them, L ce and L dice They represent the binary cross entropy loss and dice loss of the predicted mask, respectively, and L cls is the cross entropy loss for predicting the masked object classification. ce ,λ dice and λ cls is a hyperparameter used to balance the three loss functions. ce =5.0;λ dice =5.0; when the true value is the “background” category cls The value is 0.1, otherwise it is 1. As can be seen from formula (22), as the decoding stage deepens, the supervision information of each decoding stage gradually increases.

[0129] The COCO 2017 dataset provides instance-level segmentation annotations, and the instance category annotations cover 80 target categories. The dataset is divided into three subsets: train2017, validation set val2017, and test set test-dev. Train2017 contains 115,000 images, val2017 contains 5,000 images, and test-dev contains 20,000 images. Following standard practice, we train the model on train2017 and evaluate the performance of the model on val2017.

[0130] The evaluation index is AP μ and AP.

[0131] AP μ is the instance segmentation accuracy under a specified IoU (Intersection over Union) threshold, where μ is the IoU threshold. For example, AP 50 is the instance segmentation accuracy calculated under the condition of μ = 0.50, that is, (M P ∩M G ) / (M P ∪M G )≥50% of the instance segmentation accuracy. P Represents the set of pixels contained in the predicted mask area, M G Represents the set of pixels contained in the true value mask area. The smaller the μ value, the smaller the overlap ratio between the detection result and the true annotation is required, so AP μ The looser. For example, AP 75 is better than AP 50More rigorous evaluation metrics to measure the performance of the model in high-precision detection tasks.

[0132] AP is the average of instance segmentation accuracy under multiple IoU thresholds. According to the usual practice, the AP is calculated by taking 10 different thresholds with IoU thresholds between 0.50 and 0.95 and a step size of 0.05 to calculate the average instance segmentation accuracy. AP integrates the performance of the model under different accuracy standards, so it is a general indicator to measure the overall quality of the model.

[0133] Experimental setup

[0134] The transformer-based instance segmentation model with the highest real-time performance (FastInst) and the model with the highest segmentation accuracy (Mask2Former) are selected as the baseline of the present invention. The experiment is implemented based on the Detectron2 code library (https: / / github.com / facebookresearch / detectron2.git). The model performs data enhancement on the training samples during the training phase. The data enhancement methods include scaling, jittering, and horizontal flipping. The scaling factor is set to a range of 0.5 to 2.0. The short side of the image is adjusted to between 416 and 640 pixels, and the long side does not exceed 640 pixels.

[0135] The backbone network uses ResNet101, ResNet50 variant R50-d-DCN and Swin-transformer pre-trained on ImageNet. The network training uses the AdamW optimizer, with an initial learning rate of 0.0001, a weight decay of 0.05, and a momentum of 0.1; a cosine annealing learning rate scheduler; the model is trained for 50 epochs with a batch size of 16. The number of object queries is set to 100, and the weight coefficient λ in the loss function is ce , dice and λ cls The values ​​of are 2.0, 5.0, and 1.0 respectively.

[0136] Experimental Results

[0137] The results in Table 1 show that the “enhanced mask training strategy (EMT)” and “query feature multi-path propagation training strategy (MPQ)” proposed in the present invention are applied to different Transformer instance segmentation models, and both can effectively improve the instance segmentation accuracy of the models.

[0138] Table 1 Comparison of instance segmentation accuracy on COCO val2017 dataset (%)

[0139]

Claims

1. A training method for a Transformer instance segmentation model, characterized in that: The method consists of two training strategies; Strategy 1 is the enhanced mask training strategy EMT. In the training phase, the core operation of this strategy is to calculate the attention mask based on the enhanced mask and calculate the attention score using the attention mask. Specific implementation steps: First, for the prediction mask output by the decoding head equipped in each decoding stage of the model, the Hungarian matching algorithm is used to assign the corresponding true value mask; then, the true value mask is morphologically transformed to generate an enhanced true value mask; next, the enhanced true value mask is used to replace the prediction mask assigned with the true value mask, and the prediction mask that is not replaced and the enhanced true value mask together constitute the enhanced mask; Finally, the attention mask is calculated based on the enhanced mask, and the attention score is calculated using the attention mask; In the inference phase, the model calculates the attention mask based on the predicted mask, and then uses the attention mask to calculate the attention score; Strategy 2 is the query feature multi-path propagation training strategy MPQ; In the training phase, multi-path propagation of query features is performed; to facilitate the multi-path propagation of query features, the query features are Expand to Where N is the number of object queries in the query feature, C is the dimension of the object query; the newly added dimension is named groupsize dimension; G query features are stacked in the groupsize dimension Form a query feature group, denoted as During the training phase, the input to the s-th decoding stage (s ≥ 2) of the Transformer decoder is the query feature set where Y s-1 and Y s-2 T s-1 and T s-2 The number of query features output; decoding stage T 1 The input is the query feature set In the inference phase, the Transformer instance segmentation model performs a single-path propagation of query features, that is, the s-th decoding stage only accepts the query features output by the s-1-th decoding stage. As input; When training the model using the enhanced mask training strategy EMT and the query feature multi-path propagation training strategy MPQ, the network structure of the Transformer-based instance segmentation model remains unchanged; first, the backbone network is used to extract features from the input image and generate image feature maps at multiple scales. where F i Represents the i-th feature map, L represents the total number of feature maps; image feature maps of multiple scales It is then sent to the pixel decoder to obtain multi-scale fusion features through layer-by-layer fusion. where F i ′ represents the i-th fusion feature map; From multi-scale fusion features Select a feature map F of a specific scale from i ′, input to the query generator to generate initial query features F i ' is selected as F4'; then, the initial query feature and multi-scale fusion features or The fusion feature map F of a certain scale in j ′(1≤j≤L) are input to the decoding head H 0 , generate the prediction mask P 0→0 ; The multi-scale fusion feature or The fusion feature map F of a certain scale in j ′” is selected as F3′; F3′ is compared with the initial query feature They are then input into the Transformer decoder for feature update and optimization.

2. The training method based on the Transformer instance segmentation model according to claim 1, characterized in that: The specific process of feature optimization is as follows: The input of the 0th decoding stage is the fused feature map F4′, and the query generator is used to obtain the initial query feature Afterwards, And the fused feature map F3′ is input to the decoding head H 0 , get the predicted mask P 0→0 ; Next, predict the mask P 0→0 Together with the true value GT, it is input into the mask enhancement module MEM; in MEM, the Hungarian matching algorithm is used to achieve one-to-one matching between N predicted masks and B true value masks, where B ≤ N; a randomly selected morphological transformation operation is applied to each true value mask to generate an enhanced true value mask; Then, an enhanced mask is calculated based on the enhanced ground truth mask and the predicted mask; Finally, the attention mask is calculated based on the enhanced mask The first decoding stage T 1 The input is the query feature set and attention mask The masked attention module utilizes attention masks Calculate the attention score and use the attention score to Update to get This improvement helps the mask attention module to more accurately identify and focus on foreground image features; Then, Obtained sequentially through the self-attention module and the feed-forward network Next, query features And the fused feature map F3′ is input to the decoding head H 1 , generating the decoding stage T 1 The prediction mask P 0→1 ; Finally, the predicted mask P 0→1 Input into MEM together with the true value GT to obtain the attention mask Second decoding stage T 2 The input is given by and The query feature group A obtained by stacking in the groupsize dimension 2 And the attention mask and The masked attention module utilizes attention masks right Update to get Using Attention Mask right Update to get Two sets of updated query features and By sequentially passing through the self-attention module and the feed-forward network, we get and The query feature And the fused feature map F3′ is input to the decoding head H 2 , generate the prediction mask P for the second decoding stage 0→2 and P 0 →1→2 ; Then, the prediction mask P 0→2 , P 0→1→2 Input into MEM together with the true value GT to obtain the attention mask and Similarly, starting from the second decoding stage, each decoding stage expands the query features output by the groupsize dimension of the two adjacent previous stages; then, they are stacked in the groupsize dimension to form a query feature group as the input of the current decoding stage; the s-th decoding stage accepts the query features output by the s-1-th decoding stage and the s-2-th decoding stage, expands these features by the groupsize dimension, and then stacks them in the groupsize dimension to form a query feature group A s , A s Used as the input of the s-th decoding stage; let G s A s The size of the groupsize dimension is G 0 =G 1 =1; when s≥2, G s =G s-1 +G s -2 .

3. The training method based on the Transformer instance segmentation model according to claim 1, characterized in that: The specific formula for the training strategy for the Transformer instance segmentation model is described in two parts: Part 1: Enhanced Mask Training Strategy EMT; Design a mask enhancement module MEM; this module receives the true value mask and the predicted mask as input and outputs the attention mask; In the s-th decoding stage, the attention mask input to the masked attention module The specific calculation process is as follows: First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix It means that the Hungarian matching algorithm assigns the bth true value mask to the nth predicted mask. Indicates that the Hungarian matching algorithm does not assign a true value mask to the nth predicted mask; Then, the mask enhancement module randomly selects one of the three sizes of 3×3, 5×5 or 7×7 as the size of the mask enhancement convolution kernel; after the size is selected, the convolution operation is performed on the true value mask; after the convolution is completed, one of the three operations of Dilate(·), Erode(·) and Comp(·) is selected and applied to the result after convolution to obtain the enhanced true value mask; After replacing the predicted mask that matches it with the enhanced truth mask, the enhanced mask is obtained; Finally, the attention mask is calculated based on the enhanced mask Attention Mask The calculation process is divided into two steps, convolution operation and morphological transformation operation: 1) Convolution operation; First, the Hungarian matching algorithm is used to predict the mask P s Perform one-to-one matching with the true value mask GT to obtain the matching matrix I; Then, calculate the truth mask Ground truth mask GT b Enhanced truth mask for (1≤b≤B) That is, according to formula (1), GT b Perform convolution operation to obtain a temporary matrix in, Represents the convolution operation, mask enhancement convolution kernel ω k×k is a two-dimensional matrix with all elements being 1 and size k×k. The possible sizes are 3×3, 5×5, and 7×7. 2) Morphological transformation operations; Next, the temporary matrix Randomly perform a morphological transformation operation to obtain the true value mask GT b The corresponding enhanced truth mask Specifically, the morphological transformation operations include three types: dilation operation (Formula 2), corrosion operation (Formula 3) and composite operation (Formula 4): right Perform dilation operation Dilate(·) to obtain the dilation enhanced truth mask right Perform the erosion operation Erode(·) to obtain the erosion enhanced truth mask right Perform the composite operation Comp(·) to obtain the edge enhancement truth mask The above calculations are performed during the network training process, and MEM is each truth mask GT b Randomly select mask-enhanced convolution kernel ω k×k size, randomly select a mask enhancement operation; if the dilation operation Dilate(·) is selected, then If the erosion operation Erode(·) is chosen, then If the composite operation Comp(·) is selected, then Finally, according to formula (5), the enhanced mask is obtained in Finally, according to formula (6), when the query feature is input into the mask attention module of the s+1th decoding stage, the attention mask required for attention calculation is obtained. Input to the masked attention module of the s+1th decoding stage for masked attention calculation; Part II: Query feature multi-path propagation training strategy MPQ; Decoding stage T 0 Generate initial query features Then, Expanding on the groupsize dimension based on Get query feature group A 1 , Then, A 1 is sent to the first decoding stage T 1 , after processing, output query features Starting from the second decoding stage, the sth decoding stage accepts the query feature group A formed by stacking multiple query features output by the s-1th decoding stage and the s-2th decoding stage in the groupsize dimension s , A s As the input of the sth decoding stage; the specific form of the query feature group input in each decoding stage is: Among them, ·||· indicates that the query features are stacked in the groupsize dimension; when the query feature group passes through the mask attention module, the self-attention module and the feedforward network in sequence, each module performs independent calculations on the query features stacked in the groupsize dimension in the query feature group, and each query feature is independently assigned a true value mask and independently calculates the loss; under the premise that the number of object queries contained in each query feature is fixed, the MPQ strategy is used in the training phase and the MPQ strategy is not used in the inference phase, which results in allowing the network to input more query features in the training phase than the number of query features input in the inference phase, thereby allowing the network to be more fully trained based on a larger number of object queries; When training the model based on the query feature multi-path propagation training strategy, the total loss function of the S decoding stages is shown in formula (9): Loss function L at each decoding stage s (0≤s≤S) as shown in formula (10): Among them, L ce and L dice They represent the binary cross entropy loss and dice loss of the predicted mask, respectively, and L cls is the cross entropy loss for predicting the classification of masked targets; ce ,λ dice and λ cls is a hyperparameter used to balance the three loss functions; ce =5.0;λ dice =5.0; when the true value is the "background" category cls The value is 0.1, otherwise it is 1; it can be seen from formula (10) that as the decoding stage deepens, the supervision information of each decoding stage gradually increases.

Images