A method, system, and readable storage medium for segmenting aircraft target instances.

Abstract

This invention discloses an aircraft target instance segmentation method, system, and readable storage medium. Based on SOLOv2, this invention proposes a multi-view visible light aircraft target instance segmentation algorithm, named CABL-SOLO, based on coordinate attention and boundary loss. The CABL-SOLO algorithm enhances the model's sensitivity to positional features by adding a coordinate attention module after the output of each layer of the feature pyramid network; it optimizes the network structure and reduces computational parameters by using second-generation deformable convolutions to replace traditional convolutions; and it optimizes the loss function of the mask branch and increases the calculation of boundary loss, resulting in more accurate and smoother segmentation contours. Experimental results show that the CABL-SOLO algorithm can effectively improve the segmentation accuracy of aircraft targets in multi-view visible light aircraft images.

Description

Technical Field

[0001] This application relates to the fields of data processing and data transmission, and more specifically, to a method, system, and readable storage medium for segmenting aircraft target instances. Background Technology

[0002] Identifying aircraft target models in visible light images by contour recognition has broad application prospects in civil aviation and military fields. Currently, obtaining the aircraft target contour through instance segmentation and post-processing is a relatively effective method. Accurately segmenting aircraft targets is both a key research focus and a challenge. Summary of the Invention

[0003] In view of the above problems, the object of the present invention is to provide an aircraft target instance segmentation method, system and readable storage medium.

[0004] The first aspect of this invention provides a method for segmenting aircraft target instances, comprising:

[0005] Get the current image data;

[0006] Send the current image data to the preset first model;

[0007] Obtain the segmentation result information;

[0008] The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch.

[0009] In this solution, sending the current image data to a preset first model includes:

[0010] The current image data is sent to the ResNet backbone network for feature extraction.

[0011] Obtain the feature data of the current image;

[0012] The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data.

[0013] The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches.

[0014] In this scheme, after obtaining the multi-scale prediction data of the current image, the following steps are also included:

[0015] The multi-scale prediction data of the current image is sent to the semantic branch to obtain the target category number data;

[0016] The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data;

[0017] Based on the target category number data, out-of-space data, and final mask feature map data, matrix nonmaximum suppression processing is performed to obtain the segmentation result information;

[0018] The mask branch includes a convolution kernel branch and a feature branch.

[0019] In this scheme, the coordinate attention module includes an information embedding module and an attention generation module.

[0020] In this solution, before sending the current image data to the preset first model, the following steps are also included:

[0021] Obtain dataset information;

[0022] The dataset information is used for training according to preset rules;

[0023] The first preset model is obtained;

[0024] The preset first model uses DCNv2 to replace the traditional convolutional network.

[0025] In this plan,

[0026] The dataset information is incorporated into the boundary loss function for fusion training;

[0027] The dataset information includes the training dataset and the training validation set.

[0028] A second aspect of the present invention provides an aircraft target instance segmentation system, including a memory and a processor. The memory includes an aircraft target instance segmentation method program, which, when executed by the processor, performs the following steps:

[0029] Get the current image data;

[0030] Send the current image data to the preset first model;

[0031] Obtain the segmentation result information;

[0032] The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch.

[0033] In this solution, sending the current image data to a preset first model includes:

[0034] The current image data is sent to the ResNet backbone network for feature extraction.

[0035] Obtain the feature data of the current image;

[0036] The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data.

[0037] The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches.

[0038] In this scheme, after obtaining the multi-scale prediction data of the current image, the following steps are also included:

[0039] The multi-scale prediction data of the current image is sent to the semantic branch to obtain the target category number data;

[0040] The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data;

[0041] Based on the target category number data, out-of-space data, and final mask feature map data, matrix nonmaximum suppression processing is performed to obtain the segmentation result information;

[0042] The mask branch includes a convolution kernel branch and a feature branch.

[0043] In this scheme, the coordinate attention module includes an information embedding module and an attention generation module.

[0044] In this solution, before sending the current image data to the preset first model, the following steps are also included:

[0045] Obtain dataset information;

[0046] The dataset information is used for training according to preset rules;

[0047] The first preset model is obtained;

[0048] The preset first model uses DCNv2 to replace the traditional convolutional network.

[0049] In this plan,

[0050] The dataset information is incorporated into the boundary loss function for fusion training;

[0051] The dataset information includes the training dataset and the training validation set.

[0052] A third aspect of the present invention provides a computer-readable storage medium comprising an aircraft target instance segmentation method program, wherein when the aircraft target instance segmentation method program is executed by a processor, it implements the steps of the aircraft target instance segmentation method as described in any of the preceding claims.

[0053] This invention discloses an aircraft target instance segmentation method, system, and readable storage medium. Based on SOLOv2, this invention proposes a multi-view visible light aircraft target instance segmentation algorithm, named CABL-SOLO, based on coordinate attention and boundary loss. The CABL-SOLO algorithm enhances the model's sensitivity to positional features by adding a coordinate attention module after the output of each layer of the feature pyramid network; it optimizes the network structure and reduces computational parameters by using second-generation deformable convolutions instead of traditional convolutions; and it optimizes the loss function of the mask branch and increases the calculation of boundary loss, resulting in more accurate and smoother segmentation contours. Experimental results show that the CABL-SOLO algorithm can effectively improve the segmentation accuracy of aircraft targets in multi-view visible light aircraft images. Attached Figure Description

[0054] Figure 1 A flowchart of an aircraft target instance segmentation method according to the present invention is shown;

[0055] Figure 2 A flowchart of a multi-scale image prediction method according to the present invention is shown;

[0056] Figure 3 A flowchart of a multi-scale image prediction data segmentation method according to the present invention is shown;

[0057] Figure 4 A block diagram of an aircraft target instance segmentation system according to the present invention is shown;

[0058] Figure 5 A schematic diagram of a CABL-SOLO structure according to the present invention is shown;

[0059] Figure 6 This diagram illustrates a method of the present invention where the left side represents a normal convolution and the right side represents a convolution with two additional channels.

[0060] Figure 7 A schematic diagram of the structure of a coordinate attention module according to the present invention is shown;

[0061] Figure 8 A schematic diagram showing some of the experimental results of this invention is provided. Detailed Implementation

[0062] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0063] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0064] Figure 1 A flowchart of an aircraft target instance segmentation method according to the present invention is shown.

[0065] like Figure 1 As shown, this invention discloses a method for segmenting aircraft target instances, comprising:

[0066] S102, Get the current image data;

[0067] S104, The current image data is sent to the preset first model;

[0068] S106, obtain the segmentation result information;

[0069] The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch.

[0070] According to an embodiment of the present invention, the preset first model is a multi-view visible light aircraft target instance segmentation algorithm based on coordinate attention and boundary loss, named CABL-SOLO, which is based on SOLOv2. The network structure is as follows: Figure 5 As shown, the CABL-SOLO algorithm model framework is mainly divided into five parts: ResNet backbone network, FPN feature pyramid, CA coordinate attention module, semantic category branch, and mask branch.

[0071] According to an embodiment of the present invention, sending the current image data to a preset first model includes:

[0072] S202, The current image data is sent to the ResNet backbone network for feature extraction;

[0073] S204, obtain the current image feature data;

[0074] S206, The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data;

[0075] The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches.

[0076] It's important to note that the core idea of ​​CABL-SOLO is to redefine instance segmentation as a problem of classifying objects based on the position and size of each pixel within an instance. The CABL-SOLO network first extracts features from the input image through a backbone network and then feeds them into a feature pyramid. Different levels of the pyramid are used to perform multi-scale predictions for targets of different sizes. A coordinate attention module is connected after each feature layer of the FPN to enhance the network's sensitivity to positional features. Finally, the feature maps of each layer are connected to two branches: a semantic branch for predicting the semantic category and a mask branch for generating the mask.

[0077] According to an embodiment of the present invention, after obtaining the multi-scale prediction data of the current image, the method further includes:

[0078] S302, The current image multi-scale prediction data is sent to the semantic branch to obtain the target category number data;

[0079] S304, The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data;

[0080] S306, perform matrix nonmaximum suppression processing based on the target category number data, outspace data and final mask feature map data to obtain the segmentation result information;

[0081] The mask branch includes a convolution kernel branch and a feature branch.

[0082] It should be noted that the semantic category branch is responsible for predicting the target category. After the input image passes through the feature fusion network, the H×W×E feature map is input to the semantic category branch, which divides the input feature map into an S×S network. The branch needs to predict the probability of each category for each grid cell. The aligned feature map has a size of S×S×E, and finally, after semantic category processing, the output is S×S×C, where C is the number of target categories.

[0083] The mask branch is responsible for generating the target mask. CABL-SOLO uses dynamic mask prediction. The mask branch is further divided into a kernel branch and a feature branch, which learn the mask kernel G∈R respectively. S×S×D and mask features F∈R H×W×E Then, the results from the two branches are dynamically convolved to obtain the mask prediction result. The calculation process is as follows:

[0084] In the formula, (i,j) represents the coordinates in the original graph, and M i,j G is the single mask prediction result at position (i,j).i,j Predict the convolution kernel on (i,j).

[0085] M i,j =G i,j ×F

[0086] In the CABL-SOLO prediction head, the convolutional kernel branch is parallel to the semantic category branch. Feature maps from different layers are input to the kernel branch, aligned to an S×S×E size, and after feature extraction through four convolutional layers, predicted by the last left convolutional kernel, resulting in G∈S×S×D. For each grid cell, the kernel branch predicts a D-dimensional output (D being the number of parameters) to represent the kernel weights. To generate weights for a 1×1 convolution with E input channels, D = E; for a 3×3 convolution, D = 9E. The generated weights are determined by the grid cells. When the input image is divided into S×S grid cells, the output is an S×S×D output space.

[0087] For the feature branch, the method employed is to fuse feature maps from different levels. This fusion is achieved through convolution and upsampling to ensure all layers have the same size, resulting in a feature map that is 1 / 4 the size of the input features. At the smallest input layer, CoordConv is used to enhance positional encoding information. Specifically, the P2-P5 feature layers in the FPN undergo 3x3 convolution, group normalization, ReLU activation, and bilinear interpolation to merge them into a single 1 / 4 size (the original image). Then, these layers are summed point-by-point, and finally, a 1x1 convolution, group normalization, and ReLU activation are applied to obtain the final mask feature map.

[0088] Finally, the instance segmentation results for each grid are collected and processed by matrix nonmaximum suppression (Matrix NMS) to obtain the final segmentation result.

[0089] According to an embodiment of the present invention, it further includes:

[0090] The coordinate attention module includes an information embedding module and an attention generation module.

[0091] It should be noted that the core idea of ​​SOLO is to segment instances based on location, using a fully convolutional neural network to predict instance masks. However, traditional convolutional operations possess a certain degree of spatial invariance, which improves the network's robustness. But since SOLO's segmentation masks are highly dependent on grid cells and must be separated by different feature channels, this invention constructs a location-sensitive model—the coordinate attention module.

[0092] To address this issue, a tensor with the same spatial size as the input is created, containing pixel coordinate information and normalized to [-1, 1]. This tensor is then concatenated with the input features and passed to the next layer. Feeding the input coordinate information to the convolution operation adds spatial location features to the traditional FCN model, specifically as follows: Figure 6 As shown.

[0093] Figure 6 The left side shows a regular convolution, and the right side shows a convolution with two additional channels.

[0094] A model with higher sensitivity means more accurate segmentation. By adding a coordinate attention module after the output of each layer of FPN, the position sensitivity of the model can be effectively improved.

[0095] like Figure 7 As shown, the Coordinate Attention (CA) module comprises two parts: information embedding and attention generation. In the information embedding stage, features are pooled. For all channels of the input feature map, average pooling is performed along the horizontal and vertical coordinates to obtain feature maps of size C×H×1 and C×1×W, respectively. In the attention generation stage, the two feature maps are concatenated into a C×1×(H+W) feature map. Then, a 1×1 convolution is used to compress its channel dimension from C to C / r dimensions with a shrinkage rate r, followed by non-linear activation using the ReLU function. The result is then decomposed along the spatial dimension into a C / r×H×1 horizontal attention tensor and a C / r×1×W vertical attention tensor. Next, two sets of 1×1 convolutions are used to increase the channel dimension from C / r to C dimensions, followed by non-linear activation using the Sigmoid function. Finally, the two attention maps, C×H×1 and C×1×W, are multiplied by the input feature map to complete the application of coordinate attention.

[0096] According to an embodiment of the present invention, before sending the current image data to a preset first model, the method further includes:

[0097] Obtain dataset information;

[0098] The dataset information is used for training according to preset rules;

[0099] The first preset model is obtained;

[0100] The preset first model uses DCNv2 to replace the traditional convolutional network.

[0101] It should be noted that visible light images of aircraft from multiple perspectives contain various flight attitudes and sizes of aircraft. There are small inter-class differences between different aircraft models, but large intra-class differences between aircraft of the same model. Since conventional convolutional structures are fixed, their receptive field for target shape is not flexible enough and their generalization ability is not strong. Therefore, DCNv2 is introduced to replace the traditional convolutional network.

[0102] DCNv2 builds upon traditional convolution by adjusting the direction vector of the convolution kernel. It uses offsets to learn the geometric deformation of instances, allowing sampling to more closely approximate the shape and size of the target, significantly improving subsequent segmentation accuracy. However, while deformable convolution introduces random offsets, it also generates more irrelevant interference information outside the target region. DCNv2 addresses this by adding weights to each sampling point, increasing the learning degrees of freedom and allowing the weights of some irrelevant sampling points to be learned to be 0, ensuring the convolution region is within the target object region.

[0103] For the feature at point p in the feature map, the formula for calculating y(p) is:

[0104]

[0105] In the formula: K is the total number of sampling positions within the range of the convolution kernel at p; w k The weight at position k; x(p+p) k +Δp k ) represents the eigenvalue after offset at position p; p k It is the pre-set offset at point p; Δp k Δm is the learnable offset at position k. k This is the adjustable weight at the k-th position.

[0106] According to an embodiment of the present invention, it further includes:

[0107] The dataset information is incorporated into the boundary loss function for fusion training;

[0108] The dataset information includes the training dataset and the training validation set.

[0109] It's important to note that most widely used loss functions today are based on segmented regions. During training, they are not sensitive enough to boundaries; even if the predicted boundary deviates from the true value by 5-10 pixels, it won't significantly affect their score. To obtain more accurate boundaries, a boundary loss function is introduced and fused with the original loss function of the mask branch during training.

[0110] Boundary Loss: First, the boundary loss is constructed using a custom boundary and an extended boundary. The boundary is defined as follows:

[0111] b = pool(1-m) k,θ0)-(1-m k (2)

[0112]

[0113] Among them, b,b * Let m and m represent the predicted boundary and the true boundary, respectively. * θ represents the predicted and true values ​​of the mask, respectively. 0 It is a hyperparameter, set to 3 in the experiment.

[0114] The extended boundary is defined as follows:

[0115] b exp =pool(b,θ1) (4)

[0116]

[0117] Among them, b exp , Let θ represent the predicted extension boundary and the true extension boundary, respectively. 1 It is a hyperparameter, set to 5 in the experiment.

[0118] Precision P and recall R are defined as follows:

[0119]

[0120]

[0121] Where “o” means multiplying pixel by pixel, and sum(·) means summing pixel by pixel.

[0122] Finally, the boundary loss is defined as:

[0123]

[0124] The overall loss function is defined as follows:

[0125] L = L cate +λL mask (9)

[0126] Among them, L cate It is the loss function for semantic branches, L mask λ is the loss function for the mask branch, and λ is a hyperparameter, set to 3 in the experiment. cate The conventional focus loss method is used. mask The expression is:

[0127]

[0128] Where, k = i × S + j, Npos p is the number of positive samples. * m* represents the class truth value and the mask truth value, respectively. It is an indicator function, if The value is 1 if it is d, otherwise it is 0. mask The implementation is as follows, using Dice Loss and Boundary Loss:

[0129] d mask =L Dice +L Boundary (11)

[0130] Among them, L Dice The definition is as follows:

[0131]

[0132] p x,y ,q x,y These are the pixel values ​​of the predicted mask and the real mask at position (x,y), respectively.

[0133] It should be noted that the dataset used in the experiment comprises 2643 visible light images of aircraft from multiple perspectives. Of these, 393 aircraft images were selected from open-source aircraft photography websites, and the remaining 2250 images were selected from aircraft category images from COCO and other public datasets. Furthermore, all aircraft images in the dataset contain at least one aircraft target. The specific construction process of the dataset is as follows:

[0134] First, visible light images of aircraft from multiple perspectives were selected and collected from open-source websites and public datasets. The collected visible light images were not limited to specific aircraft models; images of any aircraft model could be collected. Furthermore, during the selection process, efforts were made to ensure the dataset covered a sufficient number of different aerial attitudes of the aircraft target to guarantee its comprehensiveness. Then, all visible light images were re-labeled for image instance segmentation according to the instance segmentation annotation format of the COCO dataset. Specifically, the dataset only possesses class-agnostic binary classification labels and lacks detailed classification labels for specific aircraft models. Finally, the dataset was divided into a regular training set and a training / validation set. 90% of the visible light images in the training dataset were used for network training, while the remaining 10% served as the training / validation set.

[0135] According to an embodiment of the present invention, it further includes:

[0136] Ablation experiment.

[0137] It should be noted that the ablation experiments were conducted on a custom multi-view aircraft visible light image dataset. All trained models were trained for 12 epochs with an initial learning rate of 0.01. The segmentation improvement effects of the coordinate attention module, second-generation deformable convolution, and boundary loss were compared with those of CABL-SOLO.

[0138]

[0139]

[0140] These algorithms were quantitatively analyzed, and their performance was assessed using AP and AP. 50 AP 75 AP S AP M AP L The effectiveness of different models was evaluated. Table 1 records the experimental results under different conditions. The experimental results show that the improvements made to the base model are indeed effective. By adding a coordinate attention module, introducing second-generation deformable convolution, and adding boundary loss, the AP was improved by 5.5%, 4.3%, and 3.1%, respectively. The final CABL-SOLO algorithm showed a significant improvement, with an AP increase of 9% compared to the base model. The experimental results fully demonstrate the superiority of the CABL-SOLO algorithm. Some experimental results are shown in the table below. Figure 8 As shown.

[0141] According to an embodiment of the present invention, it further includes:

[0142] Comparative experiment.

[0143] It should be noted that, to verify the performance of the proposed algorithm, CABL-SOLO was compared with other state-of-the-art instance segmentation algorithms on the MS COCO test-dev dataset. Table 2 records the experimental results under different conditions. The experimental results show that CABL-SOLO achieves better results than both bounding box-based and non-bounding box-based instance segmentation algorithms.

[0144] Table 2. Instance segmentation mask AP (%) on COCO test-dev

[0145] Table 2 Comparison of Instance Segmentation Algorithm Accuracy on COCO Test Set

[0146]

[0147]

[0148] Figure 4 A block diagram of an aircraft target instance segmentation system according to the present invention is shown.

[0149] like Figure 4 As shown, a second aspect of the present invention provides an aircraft target instance segmentation system 6, including a memory 41 and a processor 42. The memory includes an aircraft target instance segmentation method program, which, when executed by the processor, performs the following steps:

[0150] Get the current image data;

[0151] Send the current image data to the preset first model;

[0152] Obtain the segmentation result information;

[0153] The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch.

[0154] According to an embodiment of the present invention, the preset first model is a multi-view visible light aircraft target instance segmentation algorithm based on coordinate attention and boundary loss, named CABL-SOLO, which is based on SOLOv2. The network structure is as follows: Figure 5 As shown, the CABL-SOLO algorithm model framework is mainly divided into five parts: ResNet backbone network, FPN feature pyramid, CA coordinate attention module, semantic category branch, and mask branch.

[0155] According to an embodiment of the present invention, sending the current image data to a preset first model includes:

[0156] The current image data is sent to the ResNet backbone network for feature extraction.

[0157] Obtain the feature data of the current image;

[0158] The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data.

[0159] The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches.

[0160] It's important to note that the core idea of ​​CABL-SOLO is to redefine instance segmentation as a problem of classifying objects based on the position and size of each pixel within an instance. The CABL-SOLO network first extracts features from the input image through a backbone network and then feeds them into a feature pyramid. Different levels of the pyramid are used to perform multi-scale predictions for targets of different sizes. A coordinate attention module is connected after each feature layer of the FPN to enhance the network's sensitivity to positional features. Finally, the feature maps of each layer are connected to two branches: a semantic branch for predicting the semantic category and a mask branch for generating the mask.

[0161] According to an embodiment of the present invention, after obtaining the multi-scale prediction data of the current image, the method further includes:

[0162] The multi-scale prediction data of the current image is sent to the semantic branch to obtain the target category number data;

[0163] The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data;

[0164] Based on the target category number data, out-of-space data, and final mask feature map data, matrix nonmaximum suppression processing is performed to obtain the segmentation result information;

[0165] The mask branch includes a convolution kernel branch and a feature branch.

[0166] It should be noted that the semantic category branch is responsible for predicting the target category. After the input image passes through the feature fusion network, the H×W×E feature map is input to the semantic category branch, which divides the input feature map into an S×S network. The branch needs to predict the probability of each category for each grid cell. The aligned feature map has a size of S×S×E, and finally, after semantic category processing, the output is S×S×C, where C is the number of target categories.

[0167] The mask branch is responsible for generating the target mask. CABL-SOLO uses dynamic mask prediction. The mask branch is further divided into a kernel branch and a feature branch, which learn the mask kernel G∈R respectively. S×S×D and mask features F∈R H×W×E Then, the results from the two branches are dynamically convolved to obtain the mask prediction result. The calculation process is as follows:

[0168] In the formula, (i,j) represents the coordinates in the original graph, and M i,j G is the single mask prediction result at position (i,j). i,j Predict the convolution kernel on (i,j).

[0169] M i,j =G i,j ×F

[0170] In the CABL-SOLO prediction head, the convolutional kernel branch is parallel to the semantic category branch. Feature maps from different layers are input to the kernel branch, aligned to an S×S×E size, and after feature extraction through four convolutional layers, predicted by the last left convolutional kernel, resulting in G∈S×S×D. For each grid cell, the kernel branch predicts a D-dimensional output (D being the number of parameters) to represent the kernel weights. To generate weights for a 1×1 convolution with E input channels, D = E; for a 3×3 convolution, D = 9E. The generated weights are determined by the grid cells. When the input image is divided into S×S grid cells, the output is an S×S×D output space.

[0171] For the feature branch, the method employed is to fuse feature maps from different levels. This fusion is achieved through convolution and upsampling to ensure all layers have the same size, resulting in a feature map that is 1 / 4 the size of the input features. At the smallest input layer, CoordConv is used to enhance positional encoding information. Specifically, the P2-P5 feature layers in the FPN undergo 3x3 convolution, group normalization, ReLU activation, and bilinear interpolation to merge them into a single 1 / 4 size (the original image). Then, these layers are summed point-by-point, and finally, a 1x1 convolution, group normalization, and ReLU activation are applied to obtain the final mask feature map.

[0172] Finally, the instance segmentation results for each grid are collected and processed by matrix nonmaximum suppression (Matrix NMS) to obtain the final segmentation result.

[0173] According to an embodiment of the present invention, it further includes:

[0174] The coordinate attention module includes an information embedding module and an attention generation module.

[0175] It should be noted that the core idea of ​​SOLO is to segment instances based on location, using a fully convolutional neural network to predict instance masks. However, traditional convolutional operations possess a certain degree of spatial invariance, which improves the network's robustness. But since SOLO's segmentation masks are highly dependent on grid cells and must be separated by different feature channels, this invention constructs a location-sensitive model—the coordinate attention module.

[0176] To address this issue, a tensor with the same spatial size as the input is created, containing pixel coordinate information and normalized to [-1, 1]. This tensor is then concatenated with the input features and passed to the next layer. Feeding the input coordinate information to the convolution operation adds spatial location features to the traditional FCN model, specifically as follows: Figure 6As shown.

[0177] Figure 6 The left side shows a regular convolution, and the right side shows a convolution with two additional channels.

[0178] A model with higher sensitivity means more accurate segmentation. By adding a coordinate attention module after the output of each layer of FPN, the position sensitivity of the model can be effectively improved.

[0179] like Figure 7 As shown, the Coordinate Attention (CA) module comprises two parts: information embedding and attention generation. In the information embedding stage, features are pooled. For all channels of the input feature map, average pooling is performed along the horizontal and vertical coordinates to obtain feature maps of size C×H×1 and C×1×W, respectively. In the attention generation stage, the two feature maps are concatenated into a C×1×(H+W) feature map. Then, a 1×1 convolution is used to compress its channel dimension from C to C / r dimensions with a shrinkage rate r, followed by non-linear activation using the ReLU function. The result is then decomposed along the spatial dimension into a C / r×H×1 horizontal attention tensor and a C / r×1×W vertical attention tensor. Next, two sets of 1×1 convolutions are used to increase the channel dimension from C / r to C dimensions, followed by non-linear activation using the Sigmoid function. Finally, the two attention maps, C×H×1 and C×1×W, are multiplied by the input feature map to complete the application of coordinate attention.

[0180] According to an embodiment of the present invention, before sending the current image data to a preset first model, the method further includes:

[0181] Obtain dataset information;

[0182] The dataset information is used for training according to preset rules;

[0183] The first preset model is obtained;

[0184] The preset first model uses DCNv2 to replace the traditional convolutional network.

[0185] It should be noted that visible light images of aircraft from multiple perspectives contain various flight attitudes and sizes of aircraft. There are small inter-class differences between different aircraft models, but large intra-class differences between aircraft of the same model. Since conventional convolutional structures are fixed, their receptive field for target shape is not flexible enough and their generalization ability is not strong. Therefore, DCNv2 is introduced to replace the traditional convolutional network.

[0186] DCNv2 builds upon traditional convolution by adjusting the direction vector of the convolution kernel. It uses offsets to learn the geometric deformation of instances, allowing sampling to more closely approximate the shape and size of the target, significantly improving subsequent segmentation accuracy. However, while deformable convolution introduces random offsets, it also generates more irrelevant interference information outside the target region. DCNv2 addresses this by adding weights to each sampling point, increasing the learning degrees of freedom and allowing the weights of some irrelevant sampling points to be learned to be 0, ensuring the convolution region is within the target object region.

[0187] For the feature at point p in the feature map, the formula for calculating y(p) is:

[0188]

[0189] In the formula: K is the total number of sampling positions within the range of the convolution kernel at p; w k The weight at position k; x(p+p) k +Δp k ) represents the eigenvalue after offset at position p; p k It is the pre-set offset at point p; Δp k Δm is the learnable offset at position k. k This is the adjustable weight at the k-th position.

[0190] According to an embodiment of the present invention, it further includes:

[0191] The dataset information is incorporated into the boundary loss function for fusion training;

[0192] The dataset information includes the training dataset and the training validation set.

[0193] It's important to note that most widely used loss functions today are based on segmented regions. During training, they are not sensitive enough to boundaries; even if the predicted boundary deviates from the true value by 5-10 pixels, it won't significantly affect their score. To obtain more accurate boundaries, a boundary loss function is introduced and fused with the original loss function of the mask branch during training.

[0194] Boundary Loss: First, the boundary loss is constructed using a custom boundary and an extended boundary. The boundary is defined as follows:

[0195] b = pool(1-m) k ,θ0)-(1-m k (14)

[0196]

[0197] Among them, b,b * Let m and m represent the predicted boundary and the true boundary, respectively. *θ represents the predicted and true values ​​of the mask, respectively. 0 It is a hyperparameter, set to 3 in the experiment.

[0198] The extended boundary is defined as follows:

[0199] b exp =pool(b,θ1) (16)

[0200]

[0201] Among them, b exp , Let θ represent the predicted extension boundary and the true extension boundary, respectively. 1 It is a hyperparameter, set to 5 in the experiment.

[0202] Precision P and recall R are defined as follows:

[0203]

[0204]

[0205] Where “o” means multiplying pixel by pixel, and sum(·) means summing pixel by pixel.

[0206] Finally, the boundary loss is defined as:

[0207]

[0208] The overall loss function is defined as follows:

[0209] L = L cate +λL mask (twenty one)

[0210] Among them, L cate It is the loss function for semantic branches, L mask λ is the loss function for the mask branch, and λ is a hyperparameter, set to 3 in the experiment. cate The conventional focus loss method is used. mask The expression is:

[0211]

[0212] Where, k = i × S + j, N pos p is the number of positive samples. * m* represents the class truth value and the mask truth value, respectively. It is an indicator function, if The value is 1 if it is d, otherwise it is 0. maskThe implementation is as follows, using Dice Loss and Boundary Loss:

[0213] d mask =L Dice +L Boundary (twenty three)

[0214] Among them, L Dice The definition is as follows:

[0215]

[0216] p x,y ,q x,y These are the pixel values ​​of the predicted mask and the real mask at position (x,y), respectively.

[0217] It should be noted that the dataset used in the experiment comprises 2643 visible light images of aircraft from multiple perspectives. Of these, 393 aircraft images were selected from open-source aircraft photography websites, and the remaining 2250 images were selected from aircraft category images from COCO and other public datasets. Furthermore, all aircraft images in the dataset contain at least one aircraft target. The specific construction process of the dataset is as follows:

[0218] First, visible light images of aircraft from multiple perspectives were selected and collected from open-source websites and public datasets. The collected visible light images were not limited to specific aircraft models; images of any aircraft model could be collected. Furthermore, during the selection process, efforts were made to ensure the dataset covered a sufficient number of different aerial attitudes of the aircraft target to guarantee its comprehensiveness. Then, all visible light images were re-labeled for image instance segmentation according to the instance segmentation annotation format of the COCO dataset. Specifically, the dataset only possesses class-agnostic binary classification labels and lacks detailed classification labels for specific aircraft models. Finally, the dataset was divided into a regular training set and a training / validation set. 90% of the visible light images in the training dataset were used for network training, while the remaining 10% served as the training / validation set.

[0219] According to an embodiment of the present invention, it further includes:

[0220] Ablation experiment.

[0221] It should be noted that the ablation experiments were conducted on a custom multi-view aircraft visible light image dataset. All trained models were trained for 12 epochs with an initial learning rate of 0.01. The segmentation improvement effects of the coordinate attention module, second-generation deformable convolution, and boundary loss were compared with those of CABL-SOLO.

[0222]

[0223] These algorithms were quantitatively analyzed, and their performance was assessed using AP and AP. 50 AP 75 AP S AP M AP L The effectiveness of different models was evaluated. Table 1 records the experimental results under different conditions. The experimental results show that the improvements made to the base model are indeed effective. By adding a coordinate attention module, introducing second-generation deformable convolution, and adding boundary loss, the AP was improved by 5.5%, 4.3%, and 3.1%, respectively. The final CABL-SOLO algorithm showed a significant improvement, with an AP increase of 9% compared to the base model. The experimental results fully demonstrate the superiority of the CABL-SOLO algorithm. Some experimental results are shown in the table below. Figure 8 As shown.

[0224] According to an embodiment of the present invention, it further includes:

[0225] Comparative experiment.

[0226] It should be noted that, to verify the performance of the proposed algorithm, CABL-SOLO was compared with other state-of-the-art instance segmentation algorithms on the MS COCO test-dev dataset. Table 2 records the experimental results under different conditions. The experimental results show that CABL-SOLO achieves better results than both bounding box-based and non-bounding box-based instance segmentation algorithms.

[0227] Table 2. Instance segmentation mask AP (%) on COCO test-dev

[0228] Table 2 Comparison of Instance Segmentation Algorithm Accuracy on COCO Test Set

[0229]

[0230] A third aspect of the present invention provides a computer-readable storage medium comprising an aircraft target instance segmentation method program, wherein when the aircraft target instance segmentation method program is executed by a processor, it implements the steps of the aircraft target instance segmentation method as described in any of the preceding claims.

[0231] This invention discloses an aircraft target instance segmentation method, system, and readable storage medium. Based on SOLOv2, this invention proposes a multi-view visible light aircraft target instance segmentation algorithm, named CABL-SOLO, based on coordinate attention and boundary loss. The CABL-SOLO algorithm enhances the model's sensitivity to positional features by adding a coordinate attention module after the output of each layer of the feature pyramid network; it optimizes the network structure and reduces computational parameters by using second-generation deformable convolutions instead of traditional convolutions; and it optimizes the loss function of the mask branch and increases the calculation of boundary loss, resulting in more accurate and smoother segmentation contours. Experimental results show that the CABL-SOLO algorithm can effectively improve the segmentation accuracy of aircraft targets in multi-view visible light aircraft images.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0233] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0235] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for segmenting aircraft target instances, characterized in that, include: Get the current image data; Send the current image data to the preset first model; Obtain the segmentation result information; The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch. Sending the current image data to a preset first model includes: The current image data is sent to the ResNet backbone network for feature extraction. Obtain the feature data of the current image; The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data. The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches. After obtaining the multi-scale prediction data of the current image, the process also includes: The multi-scale prediction data of the current image is sent to the semantic branch to obtain the target category number data; The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data; Based on the target category number data, outspace data, and final mask feature map data, matrix nonmaximum suppression processing is performed to obtain the segmentation result information; The mask branch includes a convolution kernel branch and a feature branch.

2. The aircraft target instance segmentation method according to claim 1, characterized in that, The coordinate attention module includes an information embedding module and an attention generation module.

3. The aircraft target instance segmentation method according to claim 1, characterized in that, Before sending the current image data to the preset first model, the method further includes: Obtain dataset information; The dataset information is used for training according to preset rules; The first preset model is obtained; The preset first model uses DCNv2 to replace the traditional convolutional network.

4. The aircraft target instance segmentation method according to claim 3, characterized in that, The dataset information is incorporated into the boundary loss function for fusion training; The dataset information includes the training dataset and the training validation set.

5. An aircraft target instance segmentation system, characterized in that, The system includes a memory and a processor. The memory contains a program for segmenting aircraft target instances. When the program for segmenting aircraft target instances is executed by the processor, it performs the following steps: Get the current image data; Send the current image data to the preset first model; Obtain the segmentation result information; The preset first model includes a ResNet backbone network, an FPN feature pyramid, a CA coordinate attention module, a semantic category branch, and a mask branch. Sending the current image data to a preset first model includes: The current image data is sent to the ResNet backbone network for feature extraction. Obtain the feature data of the current image; The current image feature data is sent to the feature pyramid for multi-scale prediction processing to obtain the current image multi-scale prediction data. The feature pyramid has multiple feature layers, each of which is connected to a coordinate attention block. The feature maps all have semantic branches and mask branches. After obtaining the multi-scale prediction data of the current image, the process also includes: The multi-scale prediction data of the current image is sent to the semantic branch to obtain the target category number data; The current image multi-scale prediction data is sent to the mask branch to obtain output spatial data and final mask feature map data; Based on the target category number data, outspace data, and final mask feature map data, matrix nonmaximum suppression processing is performed to obtain the segmentation result information; The mask branch includes a convolution kernel branch and a feature branch.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program for an aircraft target instance segmentation method, which, when executed by a processor, implements the steps of an aircraft target instance segmentation method as described in any one of claims 1 to 4.

Images