SAR image aircraft detection and classification method based on knowledge assistance

Abstract

The application provides a kind of SAR image aircraft detection classification method based on knowledge assistance, comprising: obtaining the SAR image of the scene to be measured, extracting the region of interest of the airport, obtaining a plurality of slices according to the region of interest, and inputting the target feature extraction network constructed based on ConvNeXt-T network, obtaining the image feature of the region of interest;A multi-modal domain knowledge feature extraction model is constructed through an MLP network, and multi-modal domain knowledge features are obtained by combining domain knowledge;The multi-modal domain knowledge features are dynamically updated into the image features by adaptive projection, and self-attention enhancement is performed to obtain the target features;An anchor-free detection head is designed, the aircraft target in the target features is subjected to position regression and target classification through the anchor-free detection head, and the target detection classification result of the SAR image is obtained.The application can effectively reduce the difficulty of similar aircraft target classification and improve the accuracy of aircraft target detection classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of SAR image target detection, and particularly relates to a SAR image airplane detection and classification method based on knowledge assistance. BACKGROUND

[0002] Synthetic Aperture Radar (SAR) is based on distance measurement and mainly observes the physical and chemical properties of the target scene, has the imaging ability of all-weather and all-day, and plays an important role in military and civil airplane target detection and classification. With the development of SAR imaging technology, high-resolution large-size SAR images can be obtained. However, complex ground environment background, different sizes of targets, dense arrangement and other factors will affect the performance of the airplane detection and classification network. High-resolution large-scene SAR image airplane target detection and classification has become an important and challenging task.

[0003] At present, the target detection method based on deep learning has become the mainstream in the field of SAR image target detection and classification because it does not need a complex modeling process and has strong generalization ability. This method mainly includes two categories: 1) two-stage target detection method, including R-CNN, Faster R-CNN, etc.; 2) single-stage target detection method, including SSD, YOLO, etc. The above methods are anchor-based detection methods, and there are still some problems for airplane target detection in large-scene SAR images, for example, the design of anchor frame is very sensitive to size, aspect ratio and number, and it is difficult to adjust and not easy to use; due to the fixed size of anchor frame, the universality of the network is not strong, and it needs to be adjusted artificially according to experience in different tasks, otherwise it is difficult to achieve good application effect; the IOU (Intersection over Union) size needs to be compared between target frames, which wastes computing resources.

[0004] For SAR image airplane target detection and classification, most of the large-scene SAR images are background areas, and the area of the airport area is not large, so pixel-by-pixel detection is not only computationally complex, but also interfered by other ground information, which will produce many false alarms; due to the imaging mechanism of SAR, the airplane target in the SAR image presents as a discrete scattering point distribution, and the overall is weak; the size of the airplane target is different, and it faces the difficulty of multi-scale target detection; when encountering target dense parking, a part of the target is missed under the operation of non-maximum suppression, and it is difficult to achieve good detection and classification accuracy. In addition, there is a high degree of nonlinearity between remote sensing observation results and spatio-temporal geographic information, which is difficult to explain directly by physical model, which hinders people from obtaining prior information and making related applications, and the level of remote sensing application lags far behind the development of spatial remote sensing technology.

[0005] Therefore, there is an urgent need for an aircraft target detection and classification method that can reduce the difficulty of target classification of large scene SAR images and improve the accuracy of target classification. SUMMARY

[0006] Therefore, it is necessary to provide a SAR image aircraft detection and classification method based on knowledge assistance in view of the above technical problems.

[0007] A SAR image aircraft detection and classification method based on knowledge assistance, comprising the following steps: acquiring a SAR image of a large scene to be detected; when it is detected that there is an airport in the SAR image, extracting a region of interest of the airport; slicing the region of interest of the airport to obtain a plurality of slices; constructing a target feature extraction network based on a ConvNeXt-T network, inputting the plurality of slices into the target feature extraction network, and obtaining image features of the region of interest; constructing a multi-modal domain knowledge feature extraction model through an MLP network, obtaining domain knowledge, inputting the domain knowledge into the multi-modal domain knowledge feature extraction model, and obtaining multi-modal domain knowledge features; dynamically updating the multi-modal domain knowledge features to the image features through an adaptive projection, and performing a self-attention enhancement operation on the updated image features to obtain target features with domain knowledge weights; designing an anchor-free detection head, performing position regression on aircraft targets in the target features through the anchor-free detection head, classifying the aircraft targets after the position regression, and obtaining target detection and classification results of the SAR image.

[0008] In one embodiment, the acquiring a SAR image of a large scene to be detected, and when it is detected that there is an airport in the SAR image, extracting a region of interest of the airport, specifically comprises: acquiring a SAR image of a large scene to be detected, and detecting whether there is an airport in the SAR image; if there is no airport in the SAR image, ending the process; if there is an airport in the SAR image, comparing geographic location information carried by the SAR image with a latitude and longitude information library of actual airports; when the latitude and longitude information of the actual airport covers the geographic location information, it is determined that the SAR image contains an airport region, and the name and geographic location of the airport in the SAR image are obtained.

[0009] In one of the embodiments, the slicing the region of interest of the airport to obtain a plurality of slices, constructing a target feature extraction network according to the ConvNeXt-T network, inputting the plurality of slices into the target feature extraction network, and obtaining image features of the region of interest, specifically includes: dividing the region of interest of the airport into a plurality of slices with a size of 1024*1024; constructing a target feature extraction network according to the ConvNeXt-T network, inputting the plurality of slices into the target feature extraction network, and obtaining image features of the region of interest, wherein the target feature extraction network includes convolution layers, a first LN layer, a first ConvNeXt Block module, a first down-sampling layer, a second ConvNeXt Block module, a second down-sampling layer, a third ConvNeXt Block module, a third down-sampling layer, a fourth ConvNeXt Block module, a global pooling layer, a fourth LN layer, and a full connection layer.

[0010] In one of the embodiments, the constructing a multi-modal domain knowledge feature extraction model through an MLP network, obtaining domain knowledge, and inputting the domain knowledge into the multi-modal domain knowledge feature extraction model to obtain multi-modal domain knowledge features, specifically includes: constructing a multi-modal domain knowledge feature extraction model through an MLP network; obtaining domain knowledge information, wherein the domain knowledge information includes longitude, latitude, date, target size, wing area, wing sweep angle, and number of aircraft engines, the domain knowledge information is standardized to an interval of [-1, 1], and the domain knowledge information is connected according to channels:

[0011]

[0012] In the formula, lon, lat, date, length, width, wing area , sweep angle , and num engine respectively represent latitude, longitude, date information, length, width, wing area, wing sweep angle, and number of aircraft engines related to the image, and Concat(·) represents channel connection. represents an intermediate encoding result of additional information, and the domain knowledge information is mapped to R 16 , that is:

[0013]

[0014] According to the mapping result, the MLP network is used to obtain multi-modal domain knowledge features z e , that is:

[0015] z e = ReLU(LN(f(x e ))

[0016] where ReLU(·) represents a ReLU activation function, LN(·) represents layer normalization, and f(·) represents a fully connected layer.

[0017] In one of the embodiments, the multi-modal domain knowledge features are dynamically updated into the image features by adaptive projection, and the updated image features are subjected to self-attention enhancement operation to obtain target features with domain knowledge weights, specifically including: according to the multi-modal domain knowledge features z e The weight W of the dynamic projection is generated:

[0018] W = Reshape(f(z e ))

[0019] where Reshape(·) represents reconstructing one-dimensional features into two-dimensional features, and f(·) represents a fully connected layer; the multi-modal domain knowledge features are updated into the image features by adaptive projection to obtain target features, and the target features are subjected to self-attention enhancement;

[0020]

[0021]

[0022] where dim represents dimension information, and for the image features z i extracted by ConvNeXt-T feature extraction and the multi-modal features z e extracted by MLP, the dimensions d i and d e are expanded by 0 padding:

[0023]

[0024] The final dimension information is represented as:

[0025] dim = Concat({z i ,z e}) = z′ i +z′ e

[0026] The target features subjected to self-attention enhancement and the image features are connected by a skip connection.

[0027] In one of the embodiments, the anchor-free detection head is designed, and the position of the aircraft target in the target features is regressed by the anchor-free detection head, specifically including: a set of adaptive feature points is established:

[0028]

[0029] In the formula, n is the size of the convolution kernel; an initial convolution layer using an anchor-free detection head obtains a feature map, and a first convolution layer generates a first set of offsets; the feature point set is updated according to the first offsets, to obtain a first set of point sets corresponding to each feature point, wherein the updating of the point set is driven by the distance loss between the left top and right bottom points of the prediction box and the ground truth bounding box and the target classification loss in the subsequent stage; the feature map and the first offsets are passed through a second convolution layer to obtain a second set of offsets, and the second set of offsets is used to fine-tune the first set of point sets to obtain a second set of point sets; the feature map and the second offsets are sequentially passed through a third convolution layer to obtain a third set of offsets The third set of point sets is fine-tuned based on the second set of point sets through the third set of offsets to obtain a third set of point sets corrected by offsets, which is:

[0030]

[0031] The third set of point sets is converted into a bounding box, and a conversion function is defined as:

[0032] T:R p →B p

[0033] In the formula, R p represents the key points of the target P, B p represents a pseudo bounding box; the mean value of all points in the third set of point sets is calculated as the center point coordinates of the bounding box through the conversion function, and the second moment is used as the width and height of the bounding box, that is, the mean value and variance of all points are calculated, and the mean value and variance are restored to the bounding box through two globally learned coefficients; the distance between the left top corner and the right bottom corner of the bounding box is position-regressed through smooth L1, and the formula is:

[0034]

[0035] In the formula, x represents the positional difference between the prediction box and the real box.

[0036] In one embodiment, the initial convolution layer includes three 256-d 3x3 convolution layers; the first convolution layer includes a 256-d 3x3 convolution layer and an 18-d 1x1 convolution layer; the second convolution layer includes a 256-d 3x3 deformable convolution layer and an 18-d 1x1 convolution layer; and the third convolution layer includes two 256-d 3x3 deformable convolution layers and an 18-d 1x1 convolution layer.

[0037] In one of the embodiments, the aircraft target after position regression is classified to obtain the target detection classification result of the SAR image, specifically comprising: after the feature map is obtained, the feature map and the first offset are input into a classification convolution layer to obtain a target feature map; regions of each position in the target feature map are identified as positive samples if the intersection over union with the true value is greater than 0.5, and are identified as background if the intersection over union with the true value is less than 0.4; a focal loss loss function is used to classify the target feature map to obtain the target detection classification result of the SAR image.

[0038] FL(p t )=-α t (1-p t ) γ log(p t )

[0039] In the formula, y is a true label category, p is a probability value of y = 1, and p ∈ [0, 1]; the parameter α ∈ [0, 1] controls the contribution weight of positive and negative samples, and the default value is 0.25; and the parameter γ is a focusing parameter, and the default value is 2.

[0040] In one of the embodiments, the classification convolution layer comprises a 256-d 3×3 deformable convolution layer and a num_class-d 1×1 convolution layer, and num_class represents the number of categories of the classification task.

[0041] Compared with the prior art, the application has the advantages and beneficial effects that: by obtaining the SAR image of the to-be-measured large scene, when it is detected that there is an airport in the SAR image, the region of interest of the airport is extracted, the region of interest of the airport is sliced to obtain a plurality of slices, a target feature extraction network is constructed according to a ConvNeXt-T network, the plurality of slices are input into the target feature extraction network to obtain image features of the region of interest; a multi-modal domain knowledge feature extraction model is constructed through an MLP network, domain knowledge is obtained and input into the model to obtain multi-modal domain knowledge features; the multi-modal domain knowledge features are dynamically updated into the image features through a self-adaptive projection mode, and the updated image features are subjected to a self-attention enhancement operation to obtain target features with domain knowledge weights; an anchor-free detection head is designed, position regression and target classification of the aircraft target in the target features are performed according to the anchor-free detection head, and the target detection classification result of the SAR image is obtained; by using higher-dimensional domain knowledge, the difficulty of similar aircraft target classification is effectively reduced, the SAR image is easy to understand, the network interpretability is enhanced, and more fine-grained position representation and target classification can be performed through the anchor-free detection head, and the accuracy of the aircraft target detection classification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1A flowchart of a knowledge-assisted SAR image aircraft detection and classification method in an embodiment is shown in FIG. 1.

[0043] Figure 2 A flowchart of a knowledge-assisted SAR image aircraft detection and classification method in an embodiment is shown in FIG. 1.

[0044] Figure 3 A schematic diagram of the overall network structure of a knowledge-assisted SAR image aircraft detection and classification method in an embodiment is shown in FIG. 2.

[0045] Figure 4 A schematic diagram of the structure of an anchor-free detection head in an embodiment is shown in FIG. 3. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] In an embodiment, as shown in FIG. 1, a knowledge-assisted SAR image aircraft detection and classification method is provided, including the following steps: Figures 1 to 4

[0048] Step S101, acquiring a SAR image of a large scene to be detected, and extracting a region of interest of an airport when the airport is detected in the SAR image.

[0049] Specifically, a SAR image of a large scene to be detected is acquired by a synthetic aperture radar, and it is detected whether an airport exists in the SAR image. When the airport does not exist in the SAR image, the process is ended. When the airport exists in the SAR image, the airport region is identified, and the airport region is extracted as a region of interest.

[0050] Step S102, slicing the region of interest of the airport to acquire a plurality of slices, constructing a target feature extraction network according to a ConvNeXt-T network, inputting the plurality of slices into the target feature extraction network, and acquiring image features of the region of interest.

[0051] Specifically, the region of interest of the airport is sliced to acquire a plurality of slices, a target feature extraction network is constructed by a ConvNeXt-T network, the plurality of slices are input into the ConvNeXt-T network, target feature extraction is performed, and image features of the region of interest are acquired.

[0052] ​The ConvNeXt-T network is an improved version of the classic ResNet network based on some advanced ideas of the Transformer network. By combining the advantages of both networks, the performance of the CNN network is improved, enabling faster inference and higher accuracy.

[0053] In step S103, a multi-modal domain knowledge feature extraction model is constructed through an MLP network. Domain knowledge is obtained and input into the multi-modal domain knowledge feature extraction model to obtain multi-modal domain knowledge features.

[0054] Specifically, an MLP (Multi-layer Perceptrons) network is used to construct a multi-modal domain knowledge feature extraction model, and domain knowledge is input into the multi-modal domain knowledge feature extraction model to obtain multi-modal domain knowledge features with domain knowledge. The multi-modal domain knowledge features use higher-dimensional information than traditional target detection methods, making the SAR image easier to understand, improving the accuracy of the data, and enhancing the network's interpretability.

[0055] The domain knowledge can be information related to the image, such as longitude, latitude, and date, and information related to the target category, such as target size, wing area, wing sweep angle, and number of aircraft engines.

[0056] In step S104, the multi-modal domain knowledge features are dynamically updated into the image features through adaptive projection, and the updated image features are subjected to self-attention enhancement operations to obtain target features with domain knowledge weights.

[0057] Specifically, the multi-modal domain knowledge features with domain knowledge information are dynamically updated into the image features through adaptive projection, and the updated image features are subjected to self-attention enhancement operations through a self-attention mechanism to obtain target features with domain knowledge weights.

[0058] The key idea of the self-attention mechanism is to calculate the weighted average of the values calculated by the hidden units, and the weights used in the weighted average operation are dynamically obtained through a similarity function between the hidden units.

[0059] Specifically, the image features and the multi-modal domain knowledge features are input through adaptive projection, and N iterations are performed to dynamically update the multi-modal domain knowledge features with domain knowledge information into the image features, and the updated image features are subjected to self-attention enhancement through a self-attention mechanism to obtain target features with domain knowledge weights.

[0060] Step S105, an anchor-free bounding box detection head is designed, the position regression of the aircraft target in the target feature is performed through the anchor-free bounding box detection head, and the aircraft target after the position regression is classified to obtain the target detection result of the SAR image.

[0061] Specifically, the anchor-free bounding box detection head is designed as shown in the figure. Figure 4 It is noted that the structure of the anchor-free bounding box detection head is not limited in the present application, as long as the functions of position regression and target classification can be realized.

[0062] The position regression of the target feature with the domain knowledge weight is performed through the anchor-free bounding box detection head, so that the detection frame finally output can frame the aircraft target in the best shape, and the classification of the aircraft target after the position regression is performed to obtain the target detection result of the SAR image, thereby improving the detection and classification precision of the aircraft target.

[0063] In the present embodiment, the SAR image of the to-be-detected large scene is acquired, when it is detected that the airport exists in the SAR image, the region of interest of the airport is extracted, the region of interest of the airport is sliced to obtain a plurality of slices, the target feature extraction network is constructed according to the ConvNeXt-T network, the image features of the region of interest are acquired by inputting the plurality of slices into the target feature extraction network; the multi-modal domain knowledge feature extraction model is constructed through the MLP network, the domain knowledge is acquired and input into the model to obtain the multi-modal domain knowledge feature; the multi-modal domain knowledge feature is dynamically updated into the image feature through the self-adaptive projection, and the self-attention enhancement operation is performed on the updated image feature to obtain the target feature with the domain knowledge weight, the anchor-free bounding box detection head is designed, the position regression and target classification of the aircraft target in the target feature are performed according to the anchor-free bounding box detection head, and the target detection and classification result of the SAR image is acquired. By using higher-dimensional domain knowledge, the difficulty of similar aircraft target classification is effectively reduced, the SAR image is easy to understand, the network interpretability is enhanced, and the accuracy of the aircraft target detection and classification is improved through the anchor-free bounding box detection head for more fine-grained position representation and target classification.

[0064] The step S101 specifically includes: acquiring the SAR image of the to-be-detected large scene, detecting whether the airport exists in the SAR image; if the airport does not exist in the SAR image, the process is ended; if the airport exists in the SAR image, the geographic position information carried by the SAR image is compared with the latitude and longitude information library of the actual airport; when the latitude and longitude information of the actual airport covers the geographic position information, it is determined that the SAR image contains the airport region, and the name and geographic position of the airport in the SAR image are acquired.

[0065] Specifically, a SAR image of a large scene to be detected is acquired by a synthetic aperture radar, the SAR image is detected to determine whether it contains an airport, and when the SAR image does not contain an airport, the process is ended; when the SAR image contains an airport, geographic position information carried by the SAR image is acquired, and the geographic position information is compared with a latitude and longitude information library of actual airports, when the geographic position information in the SAR image is covered by certain latitude and longitude information in the latitude and longitude information library, it is determined that the image contains an airport area, and the airport name and geographic position of the airport on the SAR image are acquired according to the latitude and longitude information library.

[0066] The step S102 specifically comprises: dividing the region of interest of the airport into a plurality of slices with a size of 1024*1024; constructing a target feature extraction network according to a ConvNeXt-T network, inputting the plurality of slices into the target feature extraction network, and acquiring image features of the region of interest, the target feature extraction network comprising a convolution layer, a first LN layer, a first ConvNeXt Block module, a first down-sampling layer, a second ConvNeXt Block module, a second down-sampling layer, a third ConvNeXt Block module, a third down-sampling layer, a fourth ConvNeXt Block module, a global pooling layer, a fourth LN layer, and a full connection layer.

[0067] Specifically, after the region of interest of the airport is acquired, the region of interest is divided into a plurality of slices with a size of 1024*1024, the plurality of slices are input into a target feature extraction network according to a ConvNeXt-T network, and image features of the region of interest are acquired. The structure of the target feature extraction network comprises an input layer, a convolution layer, a first LN layer, a first ConvNeXt Block module, a first down-sampling layer, a second ConvNeXt Block module, a second down-sampling layer, a third ConvNeXt Block module, a third down-sampling layer, a fourth ConvNeXt Block module, a global pooling layer, a fourth LN layer, and a full connection layer.

[0068] The first LN layer and the fourth LN layer are both normalization layers, which are operations of normalizing the image layer. The global pooling layer is used for performing a mean pooling of the entire image of the feature map of the last layer to form a feature point, and the feature points are combined into a final feature vector for calculation in a softmax. The full connection layer is connected to all nodes of the previous layer by each node to realize the comprehensive of all extracted features.

[0069] Specifically, step S103 includes: constructing a multimodal domain knowledge feature extraction model through an MLP network; acquiring domain knowledge information, including longitude, latitude, date, target size, wing area, wing sweep angle, and number of aircraft engines; standardizing the domain knowledge information to the interval [-1, 1] and connecting them according to channels:

[0070]

[0071] In the formula, lon, lat, date, length, width, wing area ,sweep angle ,num engine These represent the latitude, longitude, date, length, width, wing area, wing sweep angle, and number of aircraft engines associated with the image, respectively. Concat(·) indicates channel connection. The intermediate encoding result represents the additional information, including domain knowledge information. Mapping to R 16 ,for:

[0072]

[0073] Based on the mapping results, an MLP network is used to obtain multimodal domain knowledge features z. e ,for:

[0074] z e =ReLU(LN(f(x) e )))

[0075] In the formula, ReLU(·) represents the ReLU activation function, LN(·) represents layer normalization, and f(·) represents a fully connected layer.

[0076] Specifically, a multimodal domain knowledge feature extraction model is constructed using an MLP network. Domain knowledge such as longitude, latitude, date, length, width, wing area, wing sweep angle, and number of aircraft engines is received through multimodal paths. The domain knowledge is standardized to the interval [-1,1] and connected according to channels to map the domain knowledge. The mapped multimodal features are then extracted based on the multimodal domain knowledge feature extraction model. The MLP network is a simple sequence of fully connected layers, layer normalization, and ReLU activation functions.

[0077] Specifically, step S104 includes: based on the multimodal domain knowledge features z e The weights W for generating the dynamic projection:

[0078] W = Reshape(f(z) e ))

[0079] In the formula, Reshape(·) represents reconstructing one-dimensional features into two-dimensional features, and f(·) represents a fully connected layer; through adaptive projection, the multi-modal field knowledge feature is updated into the image feature to obtain the target feature, and the target feature is self-attention enhanced;

[0080]

[0081]

[0082] In the formula, dim represents dimension information, and for the image feature z i extracted through the ConvNeXt-T feature extraction e , the multi-modal feature z i extracted through the MLP is expanded in the form of 0 supplement: e

[0083]

[0084] The final dimension information is represented as:

[0085] dim=Concat({z i ,z e})=z′ i +z′ e

[0086] Through the way of jump connection, the target feature after self-attention enhancement and the image feature are connected.

[0087] Specifically, the weight of dynamic projection is generated according to the multi-modal field knowledge feature, the multi-modal field knowledge feature is updated into the image feature in the form of adaptive projection to obtain the target feature, and the target feature is self-attention enhanced, in addition, the jump connection is used between the target feature after attention enhancement and the image feature, so as to avoid the model degradation.

[0088] When position regression is performed through the anchor-free detection head, specifically including: establishing a set of adaptive feature point set:

[0089]

[0090] ​In the formula, n is the size of the convolution kernel; a feature map is obtained by an initial convolution layer of an anchor-free detection head, and a first group of offsets is generated by a first convolution layer; a first group of point sets corresponding to each feature point is obtained by updating the feature point set according to the first offsets, wherein the updating of the point set is driven by the distance loss between the left top and right bottom points of the prediction box and the ground truth bounding box and the target classification loss in the subsequent stage; the feature map and the first offsets are input into a second convolution layer to obtain a second group of offsets, and a second group of point sets is obtained by fine-tuning the first group of point sets based on the second group of offsets; the feature map and the second offsets are input into a third convolution layer in sequence to obtain a third group of offsets The third group of point sets is obtained by fine-tuning the second group of point sets based on the third group of offsets, and is as follows:

[0091]

[0092] The third group of point sets is converted into a bounding box, and a conversion function is defined as follows:

[0093] T:R p →B p

[0094] In the formula, R p represents the key points of the target P, and B p represents a pseudo bounding box; the mean value of all points in the third group of point sets is calculated as the center point coordinates of the bounding box by the conversion function, and the second moment is taken as the width and height of the bounding box, that is, the mean value and variance of all points are calculated, and the mean value and variance are restored to the bounding box by two global learning coefficients; the distance between the left top corner and the right bottom corner of the bounding box is position-regressed by smooth L1, and the formula is as follows:

[0095]

[0096] In the formula, x represents the positional difference between the prediction box and the real box.

[0097] The initial convolution layer includes three layers of 256-d 3x3 convolution layers; the first convolution layer includes a 256-d 3x3 convolution layer and an 18-d 1x1 convolution layer; the second convolution layer includes a 256-d 3x3 deformable convolution layer and an 18-d 1x1 convolution layer; and the third convolution layer includes two layers of 256-d 3x3 deformable convolution layers and one layer of 18-d 1x1 convolution layers.

[0098] Specifically, as Figure 4As shown, it is a structural schematic diagram of the anchor-free bounding box detection head; when designing the anchor-free bounding box detection head, a set of point sets distributed in the target space range and having important semantic information positions are used to supervise the training of the detection classification network. The feature map is obtained through 3 layers of 256-d 3×3 convolution layers, and then sequentially passes through 256-d 3×3 convolution layers and 18-d 1×1 convolution layers to generate the first set of offsets. Thus, 9 offsets for the horizontal and vertical coordinates of each point in the point set are generated, a total of 18, and the first set of point sets corresponding to each position is obtained on the basis of the feature point set. The learning of the point set is driven by two targets, i.e. the distance loss between the top-left and bottom-right points of the predicted box and the true value bounding box and the target classification loss in the subsequent stage.

[0099] The feature map and the first set of offsets are sequentially passed through 256-d 3×3 deformable convolution layers and 18-d 1×1 convolution layers to obtain the second set of offsets. The second set of offsets is used to fine-tune the first set of point sets to obtain the second set of point sets.

[0100] The feature map and the second set of offsets are sequentially passed through two layers of 256-d 3×3 deformable convolution layers and one layer of 18-d 1×1 convolution layers to obtain the third set of offsets. The third set of offsets is used to fine-tune the second set of point sets to obtain the third set of point sets corrected by the offsets.

[0101] The points in the third set of point sets are converted into bounding boxes. A conversion function is defined. The mean value of all points in the point set is calculated as the center coordinates of the bounding box, and the second moment is calculated as the width and height of the bounding box. That is, the mean value and variance of all points are calculated. The mean value and variance are restored to the bounding box through two globally learned coefficients. The distance between the top-left corner and the bottom-right corner of the bounding box is position-regressed through the smooth L1 loss algorithm to obtain the position-regressed feature map. Through the above offset correction, the detection accuracy of the aircraft target in the SAR image can be improved, and the difficulty of classifying similar aircraft targets is reduced.

[0102] When the anchor-free bounding box detection head is used for target detection classification, the method specifically includes: after obtaining the feature map, inputting the feature map and the first offset into the classification convolution layer to obtain a target feature map; regarding the region of each position in the target feature map as a positive sample if the intersection over union with the true value is greater than 0.5, and as a background if it is less than 0.4; using a focal loss loss function to classify the aircraft target of the target feature map to obtain the target detection classification result of the SAR image:

[0103] FL(p t )=-α t (1-p t ) γ log(p t )

[0104] In the formula, y is a real label class, p is a probability value of y=1, p∈[0,1]; the parameter α∈[0,1] controls the contribution weight of positive and negative samples, and the default value is 0.25; the parameter γ is a focus parameter, and the default value is 2.

[0105] The classification convolution layer includes a 256-d 3*3 deformable convolution layer and a num_class-d 1*1 convolution layer, and num_class represents the number of categories of the classification task.

[0106] Specifically, when target detection classification is performed by the anchor-free detection head, after regression through three convolution layers, classification is directly performed, and a 256-d 3*3 deformable convolution and a num_class-d 1*1 convolution are performed on the feature map and the first offset, and num_class represents the number of categories of the classification task. After the num_class-d 1*1 convolution layer, the region of each position in the target feature map is greater than 0.5 in terms of the intersection over union with the true value, and is less than 0.4 in terms of the intersection over union with the true value, and the rest is ignored. The target detection classification result of the SAR image is obtained through the focal loss loss function, the plane target detection classification of the large scene SAR image is realized, and good detection classification precision is achieved.

[0107] The anchor-free detection head of the network is constructed by using the deformable convolution, a set of feature points distributed in the target space range and having important semantic information positions are provided, finer-grained position representation and classification information are provided, the training of these points is driven by target positioning and classification, the points are closely combined with the true value boundary box, correct classification targets are guided, the limitation of the boundary box is got rid of, the universality of the network is enhanced, and the calculation complexity is reduced.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM).

[0109] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in computer storage media (ROM / RAM, magnetic disc, optical disc) and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module. Therefore, the present application is not limited to any specific hardware and software combination.

[0110] The above is a further detailed description of the present application in combination with specific embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those skilled in the art, some simple deductions or substitutions can be made without departing from the concept of the present application, which should be considered as falling within the protection scope of the present application.

Claims

1. A knowledge-assisted SAR image aircraft detection and classification method, characterized in that, Includes the following steps: Acquire SAR images of the large scene to be tested, and when an airport is detected in the SAR image, extract the region of interest of the airport; The region of interest of the airport is sliced ​​to obtain multiple slices. A target feature extraction network is constructed based on the ConvNeXt-T network. The multiple slices are input into the target feature extraction network to obtain the image features of the region of interest. A multimodal domain knowledge feature extraction model is constructed using an MLP network. Domain knowledge is acquired and input into the multimodal domain knowledge feature extraction model to obtain multimodal domain knowledge features, including: A multimodal domain knowledge feature extraction model is constructed using an MLP network. Domain knowledge information is acquired, including longitude, latitude, date, target size, wing area, wing sweep angle, and number of aircraft engines. This domain knowledge information is standardized to the interval [-1, 1] and connected according to channels. In the formula, lon, lat, date, length, width, wing area ,sweep angle ,num engine These represent the latitude, longitude, date information, length, width, wing area, wing sweep angle, and number of aircraft engines related to the image, respectively. Concat(·) indicates channel connection. The intermediate encoding result represents the additional information, including domain knowledge information. Mapping to R 16 ,for: Based on the mapping results, an MLP network is used to obtain multimodal domain knowledge features z. e ,for: z e =ReLU(LN(f(x e ))) In the formula, ReLU(·) represents the ReLU activation function, LN(·) represents layer normalization, and f(·) represents a fully connected layer; By using adaptive projection, multimodal domain knowledge features are dynamically updated into image features, and self-attention enhancement is performed on the updated image features to obtain target features with domain knowledge weights, including: based on multimodal domain knowledge features z e The weights W for generating the dynamic projection: W=Reshape(f(z e )) In the formula, Reshape(·) represents reconstructing a one-dimensional feature into a two-dimensional feature, and f(·) represents a fully connected layer; By using adaptive projection, the multimodal domain knowledge features are updated into the image features to obtain the target features, and self-attention enhancement is performed on the target features. In the formula, dim represents dimensional information, which is the image feature z obtained after ConvNeXt-T feature extraction. i and the multimodal features z extracted by MLP e Use zero-padding to reduce the dimension d i and d e To expand: The final dimensional information is represented as follows: dim=Concat({z i ,from e })=z′ i +z′ e By using skip connections, the self-attention-enhanced target features and image features are connected; Design a frameless detection head, perform position regression on aircraft targets in target features using the frameless detection head, classify the regressed aircraft targets, and obtain the target detection and classification results of SAR images.

2. The knowledge-assisted SAR image aircraft detection and classification method according to claim 1, characterized in that, The process of acquiring a SAR image of the large scene to be tested, and extracting the region of interest of the airport when an airport is detected in the SAR image, specifically includes: Acquire SAR images of the large scene to be tested, and detect whether an airport exists in the SAR images; If no airport is found in the SAR image, the process ends. If an airport exists in the SAR image, the geographical location information carried by the SAR image is compared with the latitude and longitude information database of the actual airport. When the latitude and longitude information of an actual airport covers the geographical location information, the SAR image is identified as containing an airport area, and the name and geographical location of the airport in the SAR image are obtained.

3. The knowledge-assisted SAR image aircraft detection and classification method according to claim 1, characterized in that, The process of slicing the region of interest (ROI) of the airport into multiple slices, constructing a target feature extraction network based on the ConvNeXt-T network, and inputting the multiple slices into the target feature extraction network to obtain image features of the ROI specifically includes: The region of interest of the airport is divided into multiple slices of size 1024*1024; A target feature extraction network is constructed based on the ConvNeXt-T network. The multiple slices are input into the target feature extraction network to obtain image features of the region of interest. The target feature extraction network includes a concatenated convolutional layer, a first LN layer, a first ConvNeXt Block module, a first downsampling layer, a second ConvNeXt Block module, a second downsampling layer, a third ConvNeXt Block module, a third downsampling layer, a fourth ConvNeXt Block module, a global pooling layer, a fourth LN layer, and a fully connected layer.

4. The knowledge-assisted SAR image aircraft detection and classification method according to claim 1, characterized in that, The frameless detection head is designed to perform position regression on aircraft targets within target features. Specifically, this includes: Establish an adaptive set of feature points: In the formula, n is the size of the convolution kernel; The feature map is obtained by using the initial convolutional layer of the anchorless detection head, and the first set of offsets is generated by the first convolutional layer; The feature point set is updated based on the first set of offsets to obtain the first set of points corresponding to each feature point. The update of the point set is driven by the distance loss between the upper left and lower right points between the predicted box and the ground truth bounding box and the target classification loss in the subsequent stage. The feature map and the first set of offsets are passed through the second convolutional layer to obtain the second set of offsets. The second set of offsets is then fine-tuned based on the first set of points to obtain the second set of points. The feature map and the second set of offsets are passed sequentially through the third convolutional layer to obtain the third set of offsets. By fine-tuning the second set of points using the third set of offsets, a third set of points is obtained after offset correction, as follows: Transform the third set of points into a bounding box by defining a transformation function: T:R p →B p In the formula, R p B represents the key point of target P. p This represents a pseudo-boundary box; The mean of all points in the third set of points is calculated using a transformation function, which is used as the coordinates of the center point of the bounding box. The second moment is used as the width and height of the bounding box, that is, the mean and variance of all points are calculated. The mean and variance are then converted back into the bounding box using two globally learned coefficients. Position regression is performed using the distance between the top-left and bottom-right corners of the bounding box using smooth L1, with the following formula: In the formula, x represents the positional difference between the predicted bounding box and the ground truth bounding box.

5. The knowledge-assisted SAR image aircraft detection and classification method according to claim 4, characterized in that, The initial convolutional layer comprises three 256-d 3×3 convolutional layers; the first convolutional layer comprises a 256-d 3×3 convolutional layer and an 18-d 1×1 convolutional layer; the second convolutional layer comprises a 256-d 3×3 deformable convolutional layer and an 18-d 1×1 convolutional layer; and the third convolutional layer comprises two 256-d 3×3 deformable convolutional layers and one 18-d 1×1 convolutional layer.

6. The knowledge-assisted SAR image aircraft detection and classification method according to claim 4, characterized in that, The process of classifying the aircraft targets after location regression and obtaining the target detection and classification results of the SAR image specifically includes: After obtaining the feature map, the feature map and the first offset are input into the classification convolutional layer to obtain the target feature map; The region in the target feature map with an intersection-union ratio (IoU) greater than 0.5 with the ground truth is considered a positive sample, and the region with an IoU ratio less than 0.4 is considered background. The target feature map is classified into aircraft targets using the focal loss function to obtain the target detection and classification results of the SAR image. FL(p t )=-a t (1-p t ) γ log(p t ) In the formula, y is the true label category, p∈[0,1] is the probability value of y=1; the parameter α∈[0,1] controls the contribution weight of positive and negative samples, with a default value of 0.25; the parameter γ is the focusing parameter, with a default value of 2.

7. The knowledge-assisted SAR image aircraft detection and classification method according to claim 6, characterized in that, The classification convolutional layer includes a 256-d 3×3 deformable convolutional layer and a num_class-d 1×1 convolutional layer, where num_class represents the number of categories in the classification task.

Images