Aircraft target fine granularity identification method and system driven by configuration knowledge

Through the configuration knowledge-driven method, combined with the technologies such as image slice generation, random disruption of parts and confrontation learning, the problems of local details and overall discrimination capabilities in aircraft target recognition are solved, and the accuracy of aircraft target fine-grained recognition and efficiency.

CN118918501AActive Publication Date: 2024-11-08WUHAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411047559.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-08
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The prior art is difficult to effectively amplify the local details of the image in aircraft target fine-grained recognition and take into account the overall discrimination ability, especially when the aircraft types are diverse and the target scale and direction change greatly, resulting in low recognition accuracy.

Method used

Using configuration knowledge-driven methods, a high-spectral optical aircraft target fine-grained recognition model is constructed through technologies such as aircraft image slice generation, affine transformation, random component disruption and backbone classification network, and combined with head direction correction, confrontation learning and Regional reorganization module, optimize model training and prediction process.

Benefits of technology

The aircraft target attitude mutually poses is reduced, the model's ability to pay attention to local detailed characteristics is improved, and the accuracy and efficiency of aircraft target fine-grained recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918501A_ABST
    Figure CN118918501A_ABST
Patent Text Reader

Abstract

The invention provides a configuration knowledge-driven aircraft target fine-grained identification method and a configuration knowledge-driven aircraft target fine-grained identification system. The method comprises the following steps: acquiring four-way alignment images of a remote sensing aircraft, and marking the aircraft category and the aircraft direction of each four-way alignment image of the remote sensing aircraft; constructing a high-resolution optical aircraft target fine-grained recognition model, obtaining a predicted aircraft category and a predicted aircraft direction of each remote sensing aircraft four-way alignment image through prediction, constructing a loss function model, and carrying out optimization training through an Adam algorithm to obtain an optimized high-resolution optical aircraft target fine-grained recognition model; and performing prediction through the optimized high-resolution optical aircraft target fine-grained recognition model to obtain a predicted aircraft category and a predicted aircraft direction of a real-time remote sensing aircraft four-way alignment image. According to the method, the attitude heterogeneity of the aircraft target in the image is reduced, the cost of focusing on the detail features of local parts of the aircraft by the model is reduced, and the efficiency of extracting the detail features of the target by the model and the fine-grained recognition capability of the model are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a configuration knowledge-driven fine-grained recognition method and system for aircraft targets. Background Art

[0002] High-resolution optical remote sensing images have the characteristics of high spatial resolution, obvious geometric structure of objects, clear texture information, and large data volume. Such images can provide rich object information and spatial information, which helps to more accurately identify and extract ground features and improve the accuracy and efficiency of remote sensing applications.

[0003] In the military field, fine-grained identification of aircraft targets is a key task. By distinguishing the types and categories of aircraft targets in remote sensing images, the military can better understand the deployment and movement of enemy military forces, obtain accurate combat power assessment, battlefield situation and movement intentions. This is of great significance for making strategic decisions and implementing military operations.

[0004] However, due to the wide variety of aircraft types and models, various appearance features, and the varying scales and directions of targets in images, there is no fixed paradigm, which makes it very difficult to use deep learning classification models to infer fine-grained categories of aircraft.

[0005] In addition, different fine-grained categories usually share a relatively similar global structure in the network model, and only have certain minor differences in some local details. Therefore, compared with the overall structure of the target image, the local details of the image are more important. How to amplify the differences in local details in the target image while maintaining the ability to distinguish and control the target image as a whole is a research direction worth exploring in depth. Summary of the invention

[0006] In view of the problems mentioned in the above background technology, the present invention proposes a configuration knowledge-driven fine-grained aircraft target recognition method and system to achieve high-precision optical aircraft target fine-grained recognition.

[0007] The technical solution of the method of the present invention is a configuration knowledge-driven fine-grained aircraft target recognition method, comprising the following steps:

[0008] Step 1: Each remote sensing image is introduced into the aircraft image slice generation module. According to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked;

[0009] Step 2: Construct a high-resolution optical aircraft target fine-grained recognition model, input each remote sensing aircraft four-way aligned image into the high-resolution optical aircraft target fine-grained recognition model in turn to predict the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image, and build a loss function model based on the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training;

[0010] Step 3: Acquire real-time remote sensing images. The real-time remote sensing aircraft four-way aligned images are obtained through step 1. The predicted aircraft category and predicted aircraft direction of the real-time remote sensing aircraft four-way aligned images are obtained through prediction using the optimized high-resolution aircraft target fine-grained recognition model.

[0011] Preferably, the fine-grained category of the marked aircraft and the direction of the marked aircraft in each remote sensing aircraft four-way aligned image in step 1 are defined as:

[0012]

[0013] in, Indicates that the aircraft target in the k-th remote sensing aircraft four-way aligned image is the t-th aircraft category; Indicates that the aircraft direction of the kth remote sensing aircraft four-way aligned image is the i-th direction. If the aircraft target nose is facing right, it corresponds to "0", if the aircraft target nose is facing upward, it corresponds to "1", if the aircraft target nose is facing left, it corresponds to "2", if the aircraft target nose is facing downward, it corresponds to "3", TY=20 indicates the number of aircraft categories;

[0014] The certain size in step 1 is: P*P

[0015] P is a pixel unit of a certain size;

[0016] Preferably, the high-resolution aircraft target fine-grained recognition model described in step 2 includes:

[0017] Aircraft image slice generation module, nose direction recognition and correction module, parts random shuffling mechanism module, backbone classification network, regional reorganization module, adversarial learning module;

[0018] Among them, the nose direction correction module, the parts random scrambling mechanism module, and the backbone classification network module are cascaded in sequence;

[0019] The regional reorganization module and the adversarial learning module are connected in parallel through a backbone classification network module.

[0020] The aircraft head direction recognition and correction module includes: an aircraft head direction recognition module and an aircraft head direction correction module;

[0021] The nose direction recognition module is used to recognize one of the directions in the four-direction aligned image of the remote sensing aircraft, and realizes the recognition task of one of the four directions of the nose in the image by extracting the direction features of the aircraft target in the image and training through feature measurement and feature matching;

[0022] If the recognition result is that the remote sensing aircraft four-way alignment image is facing right, it is marked as 0; if the remote sensing aircraft four-way alignment image is facing upward, it is marked as 1; if the remote sensing aircraft four-way alignment image is facing left, it is marked as 2; if the remote sensing aircraft four-way alignment image is facing downward, it is marked as 3; output each remote sensing aircraft four-way alignment image and the recognition result of the remote sensing aircraft four-way alignment image to the nose direction correction module;

[0023] The nose direction correction module corrects the four-way alignment images of the remote sensing aircraft marked as 0, 1, and 2 to face upward;

[0024] The details are as follows:

[0025] The four-way aligned image of the remote sensing aircraft marked as 0 is rotated 90° counterclockwise to obtain each corresponding corrected aircraft image;

[0026] The four-way aligned images of the remote sensing aircraft marked as 1 remain unchanged, and each corresponding corrected aircraft image is obtained;

[0027] The four-way aligned image of the remote sensing aircraft marked as 2 is rotated 90° clockwise to obtain each corresponding corrected aircraft image;

[0028] The four-way aligned image of the remote sensing aircraft marked as 3 is rotated 180° counterclockwise to obtain each corresponding corrected aircraft image;

[0029] The component random scrambling mechanism module is composed of a component partitioning module and a random scrambling module;

[0030] The component partitioning module performs 3×3 partitioning processing on each corrected P×P pixel aircraft image, that is, partitioning the image into 9 P / 3×P / 3 pixel image blocks, which can be expressed as:

[0031]

[0032] in, represents the qth sub-region in the kth corrected remote sensing aircraft image;

[0033] Then, according to the special configuration of the aircraft, the small partitions in the same parts area are merged into a large partition, and the final partition result of the large and small partitions is obtained, which is as follows:

[0034] and The sub-area is the nose area A, and Area is left wing B; and The area is fuselage area C; and Area D is the right wing;

[0035] The random shuffling module performs random shuffling of parts on the calibrated aircraft image according to the final partition output by the aforementioned parts partitioning module, so as to obtain each aircraft image after the parts are shuffled;

[0036] The random scrambling module workflow is as follows:

[0037] Each component area is randomly rotated 0° or 180° to adapt the model to aircraft targets in different directions; the BCD area is randomly swapped; two of the BCD areas are randomly rotated 90°. Finally, the shuffled sample slice p(I k ).

[0038] The backbone classification network module is composed of a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a feature matching module;

[0039] This module is mainly used to extract the image features of aircraft targets and support the final fine-grained aircraft recognition task.

[0040] The details are as follows:

[0041] When training the model, each image of the target nose direction correction result of the aircraft, its shuffled image and category label are taken as input, each image of the target nose direction correction result of the aircraft and its shuffled image are defined as a double image, and each image of the target nose direction correction result of the aircraft and its shuffled image are sent to the network in sequence;

[0042] When the model is inferring, it only needs to take each remote sensing aircraft four-way aligned image as input;

[0043] The image is processed by four convolution modules in turn to obtain four output feature layers. The fourth output feature layer, i.e., the feature vector, is used as the input of the feature matching module. By matching the feature vectors, vectors with similar feature vectors are classified into the same category, and finally the fine-grained recognition result of the aircraft image is output.

[0044] The backbone classification network module consists of four convolution blocks connected in series, which output four convolutional feature layers in sequence; each convolution block is used for feature mining processing of the feature layer, and each feature layer serves as the input and output of the convolution block to cooperate with the convolution block to mine the hidden detail features in the target image.

[0045] The kth pair of dual images is processed by the first convolution module of the backbone classification network and outputs the first output feature map, which is defined as:

[0046] The kth pair of dual images The feature map is processed by the second convolution module of the backbone classification network to output the second output feature map, which is defined as:

[0047] The kth pair of dual images The feature map is processed by the third convolution module of the backbone classification network to output the third output feature map, which is defined as:

[0048] The kth pair of dual images The feature vector output by the fourth convolution module of the backbone classification network after the feature map is processed is defined as: That is, V(I k ,α cls );

[0049] The kth pair of dual images After the feature vector is calculated by the feature matching module of the backbone classification network, its category is finally output, which is expressed as:

[0050] The regional reorganization network module is composed of a single convolution block and a regional matrix reconstruction result generation module;

[0051] This module is used to input the kth pair of dual images output by the second convolution module of the backbone classification network. Feature map, k-th dual image After being processed by a single convolution block of this module, the output The feature map is then passed into the regional matrix reconstruction result generation module.

[0052] The regional matrix reconstruction result generation module inputs the kth scrambled image The feature maps are rearranged to The position of the components is restored to the characteristic graph of the kth unshuffled graph The localization accuracy of different regions in the dual-image feature map is then measured, inducing the backbone network to simulate the semantic correlation between regions through end-to-end training.

[0053] The adversarial learning module is used to determine the feature map output by the third convolution module of the backbone classification network of the kth pair of dual images. Derived from the original image of the k-th aircraft head direction correction result or its scrambled image;

[0054] The specific steps are as follows: the kth pair of dual images is processed by the third convolution module of the backbone classification network module, and the feature map of the kth pair of dual images is output. Enter the adversarial learning network module (ALN) for adversarial learning, and obtain the feature map distinction result of the kth original image or its scrambled image, denoted as v k ∈{0,1} 2 ;

[0055] The method by which this module determines whether the dual image is destroyed is expressed as:

[0056]

[0057] In the formula, The kth pair of dual images is classified by the backbone network k The feature vector output by the layer, For the classification network from layer 1 to layer m k The learnable parameters of the layer, is a linear mapping, which is adversarial to the learning loss L Adv It can be expressed as:

[0058]

[0059] Where p(I k ) represents the kth shuffled image.

[0060] The aforementioned direction recognition loss function L O Defined as:

[0061]

[0062] in The characteristic vector indicating that the direction of the aircraft target in the k-th remote sensing aircraft four-way aligned image is the i-th true direction; The feature vector representing the direction of the aircraft target in the k-th remote sensing aircraft four-way alignment image is the i-th predicted direction;

[0063] The loss function L of the aforementioned backbone classification network cls It can be expressed as:

[0064]

[0065] where α cls are all learnable parameters in the classification network; Indicates that the aircraft target in the k-th pair of dual images is of the t-th category; I k represents the kth image.

[0066] The aforementioned area reorganization loss L loc The definitions are as follows:

[0067]

[0068] Among them, M σ(q) (p(I k )) indicates p(I k ) in I k The predicted position number in M (q) (I k ) indicates I k The components in I k The predicted position numbers in , their true values ​​are all q.

[0069] In summary, the overall loss calculation formula is as follows:

[0070] L=αL0+βL cls +γL Adv +δL loc

[0071] Among them, α, β, γ, and δ represent adjustable setting parameters; L0 is the loss of the head direction recognition module; L cls is the backbone classification network loss; L Adv To combat learning loss; L loc Losses due to regional restructuring.

[0072] The present invention also provides a configuration knowledge-driven aircraft target fine-grained recognition system, comprising:

[0073] The sample label acquisition module is used to introduce each remote sensing image into the aircraft image slice generation module, and according to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked;

[0074] The network training module is used to build a high-resolution optical aircraft target fine-grained recognition model. Each remote sensing aircraft four-way aligned image is sequentially input into the high-resolution optical aircraft target fine-grained recognition model for prediction to obtain the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image. The loss function model is constructed by combining the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training.

[0075] The real-time remote sensing image prediction module is used to obtain real-time remote sensing images. The real-time remote sensing aircraft four-way aligned image is obtained through step 1. The predicted aircraft category and predicted aircraft direction of the real-time remote sensing aircraft four-way aligned image are obtained through the optimized high-resolution aircraft target fine-grained recognition model.

[0076] The beneficial effects produced by the present invention are:

[0077] Reduce the posture differences of aircraft targets in the image, reduce the cost of the model focusing on the detailed features of local aircraft components, and improve the efficiency of the model in extracting target detailed features.

[0078] The aircraft target feature map shuffle reconstruction method driven by configuration knowledge can force the model to strengthen its ability to pay attention to the local detail features of the target, thereby improving the model's fine-grained recognition capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 : A flow chart of a method according to an embodiment of the present invention;

[0080] Figure 2 : Flowchart of aircraft target nose direction correction according to an embodiment of the present invention;

[0081] Figure 3 : Schematic diagram of randomly disrupting four types of machine target configurations in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0084] Combine the following Figure 1-3 The specific implementation mode of the present invention is a configuration knowledge-driven fine-grained aircraft target recognition method and system.

[0085] The so-called configuration knowledge is: the minimum circumscribed bounding box of most aircraft is approximately a square, and the distribution of the nose part of the aircraft in the square is basically the same as above. The distribution of the wings on both sides in the square is basically left and right Next Area, the distribution of the fuselage in the square is basically in the middle Next area.

[0086] like Figure 1 The figure shows a method flow chart of an embodiment of the present invention. The method embodiment of the present invention is a configuration knowledge-driven fine-grained aircraft target recognition method, comprising the following steps:

[0087] Step 1: Each remote sensing image is introduced into the aircraft image slice generation module. According to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked;

[0088] The fine-grained category and direction of the marked aircraft in each remote sensing aircraft four-way aligned image described in step 1 are defined as:

[0089]

[0090] in, Indicates that the aircraft target in the k-th remote sensing aircraft four-way aligned image is the t-th aircraft category; Indicates that the aircraft direction of the kth remote sensing aircraft four-way aligned image is the i-th direction. If the aircraft target nose is facing right, it corresponds to "0", if the aircraft target nose is facing upward, it corresponds to "1", if the aircraft target nose is facing left, it corresponds to "2", if the aircraft target nose is facing downward, it corresponds to "3", TY=20 indicates the number of aircraft categories;

[0091] The certain size in step 1 is: P*P

[0092] P is a pixel unit of a certain size;

[0093] P=129

[0094] 129×129

[0095] Step 2: Construct a high-resolution optical aircraft target fine-grained recognition model, input each remote sensing aircraft four-way aligned image into the high-resolution optical aircraft target fine-grained recognition model in turn to predict the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image, and build a loss function model based on the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training;

[0096] The high-resolution optical aircraft target fine-grained recognition model described in step 2 includes:

[0097] Aircraft image slice generation module, nose direction recognition and correction module, parts random shuffling mechanism module, backbone classification network, regional reorganization module, adversarial learning module;

[0098] Among them, the nose direction correction module, the parts random scrambling mechanism module, and the backbone classification network module are cascaded in sequence;

[0099] The regional reorganization module and the adversarial learning module are connected in parallel through a backbone classification network module.

[0100] The aircraft head direction recognition and correction module includes: an aircraft head direction recognition module and an aircraft head direction correction module;

[0101] The nose direction recognition module is used to identify one of the four directions of the remote sensing aircraft in the four-way alignment image. By extracting the directional features of the aircraft target in the image and training through feature measurement and feature matching, the recognition task of one of the four directions of the nose in the image is realized. Figure 2 As shown;

[0102] If the recognition result is that the remote sensing aircraft four-way alignment image is facing right, it is marked as 0; if the remote sensing aircraft four-way alignment image is facing upward, it is marked as 1; if the remote sensing aircraft four-way alignment image is facing left, it is marked as 2; if the remote sensing aircraft four-way alignment image is facing downward, it is marked as 3; output each remote sensing aircraft four-way alignment image and the recognition result of the remote sensing aircraft four-way alignment image to the nose direction correction module;

[0103] The nose direction correction module corrects the four-way alignment images of the remote sensing aircraft marked as 0, 1, and 2 to face upward;

[0104] The details are as follows:

[0105] The four-way aligned image of the remote sensing aircraft marked as 0 is rotated 90° counterclockwise to obtain each corresponding corrected aircraft image;

[0106] The four-way aligned images of the remote sensing aircraft marked as 1 remain unchanged, and each corresponding corrected aircraft image is obtained;

[0107] The four-way aligned image of the remote sensing aircraft marked as 2 is rotated 90° clockwise to obtain each corresponding corrected aircraft image;

[0108] The four-way aligned image of the remote sensing aircraft marked as 3 is rotated 180° counterclockwise to obtain each corresponding corrected aircraft image;

[0109] The component random scrambling mechanism module is composed of a component partitioning module and a random scrambling module;

[0110] The component partitioning module performs 3×3 partitioning processing on each corrected P×P pixel aircraft image, that is, partitioning the image into 9 P / 3×P / 3 pixel image blocks, which can be expressed as:

[0111]

[0112] in, represents the qth sub-region in the kth corrected remote sensing aircraft image;

[0113] Then, according to the special configuration of the aircraft, the small partitions in the same parts area are merged into a large partition, and the final partition result of the large and small partitions is obtained, which is as follows:

[0114] and The sub-area is the nose area A, and Area is left wing B; and The area is fuselage area C; and Area D is the right wing;

[0115] The random shuffling module performs random shuffling of the components of the corrected aircraft image according to the final partition output by the aforementioned component partitioning module, and obtains each aircraft image after the components are shuffled, such as Figure 3 shown.

[0116] The random scrambling module workflow is as follows:

[0117] Each component area is randomly rotated 0° or 180° to adapt the model to aircraft targets in different directions; the BCD area is randomly swapped; two of the BCD areas are randomly rotated 90°. Finally, the shuffled sample slice p(I k ).

[0118] The backbone classification network module is composed of a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a feature matching module;

[0119] This module is mainly used to extract the image features of aircraft targets and support the final fine-grained aircraft recognition task.

[0120] The details are as follows:

[0121] When training the model, each image of the target nose direction correction result of the aircraft, its shuffled image and category label are taken as input, each image of the target nose direction correction result of the aircraft and its shuffled image are defined as a double image, and each image of the target nose direction correction result of the aircraft and its shuffled image are sent to the network in sequence;

[0122] When the model is inferring, it only needs to take each remote sensing aircraft four-way aligned image as input;

[0123] The image is processed by four convolution modules in turn to obtain four output feature layers. The fourth output feature layer, i.e., the feature vector, is used as the input of the feature matching module. By matching the feature vectors, vectors with similar feature vectors are classified into the same category, and finally the fine-grained recognition result of the aircraft image is output.

[0124] The backbone classification network module consists of four convolution blocks connected in series, which output four convolutional feature layers in sequence; each convolution block is used for feature mining processing of the feature layer, and each feature layer serves as the input and output of the convolution block to cooperate with the convolution block to mine the hidden detail features in the target image.

[0125] The kth pair of dual images is processed by the first convolution module of the backbone classification network and outputs the first output feature map, which is defined as:

[0126] The kth pair of dual images The feature map is processed by the second convolution module of the backbone classification network to output the second output feature map, which is defined as:

[0127] The kth pair of dual images The feature map is processed by the third convolution module of the backbone classification network to output the third output feature map, which is defined as:

[0128] The kth pair of dual images The feature vector output by the fourth convolution module of the backbone classification network after the feature map is processed is defined as: That is, V(I k ,α cls );

[0129] The kth pair of dual images After the feature vector is calculated by the feature matching module of the backbone classification network, its category is finally output, which is expressed as:

[0130] The regional reorganization network module is composed of a single convolution block and a regional matrix reconstruction result generation module;

[0131] This module is used to input the k-th dual image output by the second convolution module of the backbone classification network. Feature map, k-th dual image After being processed by a single convolution block of this module, the output The feature map is then passed into the regional matrix reconstruction result generation module.

[0132] The regional matrix reconstruction result generation module inputs the kth scrambled image The feature maps are rearranged to The position of the components is restored to the characteristic graph of the kth unshuffled graph The localization accuracy of different regions in the dual-image feature map is then measured, inducing the backbone network to simulate the semantic correlation between regions through end-to-end training.

[0133] The adversarial learning module is used to determine the feature map output by the third convolution module of the backbone classification network of the kth pair of dual images. Derived from the original image of the k-th aircraft head direction correction result or its scrambled image;

[0134] The specific steps are as follows: the kth pair of dual images is processed by the third convolution module of the backbone classification network module, and the feature map of the kth pair of dual images is output. Enter the adversarial learning network module (ALN) for adversarial learning, and obtain the feature map distinction result of the kth original image or its scrambled image, denoted as v k,n ∈{0,1} 2 ;

[0135] The method by which this module determines whether the dual image is destroyed is expressed as:

[0136]

[0137] In the formula, The kth pair of dual images is classified by the backbone network mth k The feature vector output by the layer, For the classification network from layer 1 to layer m k The learnable parameters of the layer, is a linear mapping, which is adversarial to the learning loss L Adv It can be expressed as:

[0138]

[0139] Where p(I k ) represents the kth shuffled image.

[0140] The aforementioned direction recognition loss function L O Defined as:

[0141]

[0142] in The characteristic vector indicating that the direction of the aircraft target in the k-th remote sensing aircraft four-way aligned image is the i-th true direction; The feature vector representing the direction of the aircraft target in the k-th remote sensing aircraft four-way alignment image is the i-th predicted direction;

[0143] The loss function L of the aforementioned backbone classification network cls It can be expressed as:

[0144]

[0145] where α cls are all learnable parameters in the classification network; Indicates that the aircraft target in the k-th dual image is of the t-th category; I k represents the kth image.

[0146] The aforementioned area reorganization loss L loc The definitions are as follows:

[0147]

[0148] Among them, M σ(q) (p(I k )) indicates p(I k ) in I k The predicted position number in M (q) (I k ) indicates I k The components in I k The predicted position numbers in , their true values ​​are all q.

[0149] In summary, the overall loss calculation formula is as follows:

[0150] L=αL0+βL cls +γL Adv +δL loc

[0151] Among them, α, β, γ, and δ represent adjustable setting parameters; L0 is the loss of the head direction recognition module; L cls is the backbone classification network loss; L Adv To combat learning loss; L loc Losses due to regional restructuring.

[0152] The beneficial effects produced by the present invention are:

[0153] By performing posture alignment on the aircraft targets in optical remote sensing images, the interference in the image background can be effectively eliminated, the cost of the model focusing on the detailed features of local aircraft components can be reduced, and the efficiency of the model in extracting target detailed features can be improved; the processing of disrupting and reconstructing aircraft parts can force the network to pay attention to the detailed features of aircraft components, strengthen the network's ability to pay attention to the detailed features of aircraft targets of different models and levels, and improve the accuracy and interpretability of network model recognition.

[0154] The system embodiment of the present invention is a high-spectral aircraft target fine-grained recognition system driven by configuration knowledge, comprising:

[0155] The sample label acquisition module is used to introduce each remote sensing image into the aircraft image slice generation module, and according to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked;

[0156] The network training module is used to build a high-resolution optical aircraft target fine-grained recognition model. Each remote sensing aircraft four-way aligned image is sequentially input into the high-resolution optical aircraft target fine-grained recognition model for prediction to obtain the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image. The loss function model is constructed by combining the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training.

[0157] The real-time remote sensing image prediction module is used to obtain the real-time remote sensing image. The real-time remote sensing aircraft four-way aligned image is obtained through step 1. The predicted aircraft category and predicted aircraft direction of the real-time remote sensing aircraft four-way aligned image are obtained through the optimized high-resolution aircraft target fine-grained recognition model.

[0158] The sample label acquisition module, network training module, and real-time remote sensing image prediction module are all deployed on the server.

[0159] As can be seen from Table 1, the attitude alignment processing of the samples by this method can effectively improve the accuracy of different methods in remote sensing aircraft target recognition. At the same time, this method has improved accuracy compared with other methods.

[0160] Table 1: Comparison of experimental results

[0161]

[0162] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0163] It should be understood that the above description of the embodiments is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A configuration knowledge-driven fine-grained aircraft target recognition method. It is characterized in that The following steps are involved: Step 1: Each remote sensing image is introduced into the aircraft image slice generation module. According to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked; Step 2: Construct a high-resolution optical aircraft target fine-grained recognition model, input each remote sensing aircraft four-way aligned image into the high-resolution optical aircraft target fine-grained recognition model in turn to predict the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image, and build a loss function model based on the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training; Step 3: Obtain real-time remote sensing images. The real-time remote sensing aircraft four-way aligned images are obtained through step 1. The predicted aircraft category and predicted aircraft direction of the real-time remote sensing aircraft four-way aligned images are obtained through prediction using the optimized high-resolution aircraft target fine-grained recognition model.

2. According to claim 1, the configuration knowledge-driven fine-grained aircraft target recognition method, Features: The fine-grained category and direction of the marked aircraft in each remote sensing aircraft four-way aligned image described in step 1 are defined as: in, Indicates that the aircraft target in the k-th remote sensing aircraft four-way aligned image is the t-th aircraft category; Indicates that the aircraft direction of the kth remote sensing aircraft four-way aligned image is the i-th direction. If the aircraft target nose is facing right, it corresponds to "0", if the aircraft target nose is facing upward, it corresponds to "1", if the aircraft target nose is facing left, it corresponds to "2", if the aircraft target nose is facing downward, it corresponds to "3", TY=20 indicates the number of aircraft categories; The certain size in step 1 is: P*P P is a pixel unit of a certain size.

3. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 2, Features: The high-resolution optical aircraft target fine-grained recognition model described in step 2 includes: Aircraft image slice generation module, nose direction recognition and correction module, parts random shuffling mechanism module, backbone classification network, regional reorganization module, adversarial learning module; Among them, the nose direction correction module, the parts random shuffling mechanism module, and the backbone classification network module are cascaded in sequence; The regional reorganization module and the adversarial learning module are connected in parallel via a backbone classification network module; The aircraft head direction recognition and correction module comprises: an aircraft head direction recognition module and an aircraft head direction correction module.

4. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 3, Features: The nose direction recognition module is used to recognize one of the directions in the four-direction aligned image of the remote sensing aircraft, and realizes the recognition task of one of the four directions of the nose in the image by extracting the direction features of the aircraft target in the image and training through feature measurement and feature matching; If the recognition result is that the remote sensing aircraft four-way alignment image is facing right, it is marked as 0; if the remote sensing aircraft four-way alignment image is facing upward, it is marked as 1; if the remote sensing aircraft four-way alignment image is facing left, it is marked as 2; if the remote sensing aircraft four-way alignment image is facing downward, it is marked as 3; output each remote sensing aircraft four-way alignment image and the recognition result of the remote sensing aircraft four-way alignment image to the nose direction correction module; The nose direction correction module corrects the four-way alignment images of the remote sensing aircraft marked as 0, 1, and 2 to face upward; The details are as follows: The four-way aligned image of the remote sensing aircraft marked as 0 is rotated 90° counterclockwise to obtain each corresponding corrected aircraft image; The four-way aligned images of the remote sensing aircraft marked as 1 remain unchanged, and each corresponding corrected aircraft image is obtained; The four-way aligned image of the remote sensing aircraft marked as 2 is rotated 90° clockwise to obtain each corresponding corrected aircraft image; The four-way aligned image of the remote sensing aircraft marked as 3 is rotated 180° counterclockwise to obtain each corresponding corrected aircraft image.

5. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 4, Features: The component random scrambling mechanism module is composed of a component partitioning module and a random scrambling module; The component partitioning module performs 3×3 partitioning processing on each corrected P×P pixel aircraft image, that is, partitioning the image into 9 image blocks of P / 3×P / 3 pixels, which can be expressed as: in, represents the qth sub-region in the kth corrected remote sensing aircraft image; Then, according to the special configuration of the aircraft, the small partitions in the same parts area are merged into a large partition, and the final partition result of the large and small partitions is obtained, which is as follows: and The sub-area is the nose area A, and Area is left wing B; and The area is fuselage area C; and Area D is the right wing.

6. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 5, Features: The random shuffling module performs random shuffling of parts on the calibrated aircraft image according to the final partition output by the aforementioned parts partitioning module, so as to obtain each aircraft image after the parts are shuffled; The random scrambling module workflow is as follows: Each component area is randomly rotated 0° or 180° to adapt the model to aircraft targets in different directions; the BCD area is randomly swapped; two of the BCD areas are randomly rotated 90°; finally, the shuffled sample slice p(I k ).

7. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 6, Features: The backbone classification network module is composed of a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a feature matching module; This module is mainly used to extract the image features of aircraft targets and support the final aircraft fine-grained type recognition task; The details are as follows: When training the model, each image of the target nose direction correction result of the aircraft, its shuffled image and category label are taken as input, each image of the target nose direction correction result of the aircraft and its shuffled image are defined as a double image, and each image of the target nose direction correction result of the aircraft and its shuffled image are sent to the network in sequence; When the model is inferring, it only needs to take each remote sensing aircraft four-way aligned image as input; The image is processed by four convolution modules in turn to obtain four output feature layers. The fourth output feature layer, i.e., the feature vector, is used as the input of the feature matching module. By matching the feature vectors, vectors with similar feature vectors are classified into the same category, and finally the fine-grained recognition result of the aircraft image is output. The backbone classification network module consists of four convolution blocks connected in series, which output four layers of convolution feature layers in sequence. Each convolution block is used for feature mining processing of the feature layer. Each feature layer serves as the input and output of the convolution block to cooperate with the convolution block to mine the hidden detail features in the target image. The kth pair of dual images is processed by the first convolution module of the backbone classification network and outputs the first output feature map, which is defined as: The kth pair of dual images The feature map is processed by the second convolution module of the backbone classification network to output the second output feature map, which is defined as: The kth pair of dual images The feature map is processed by the third convolution module of the backbone classification network to output the third output feature map, which is defined as: The kth pair of dual images The feature vector output by the fourth convolution module of the backbone classification network after the feature map is processed is defined as: That is, V(I k ,α cls ); The kth pair of dual images After the feature vector is calculated by the feature matching module of the backbone classification network, its category is finally output, which is expressed as:

8. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 7, Features: The regional reorganization network module is composed of a single convolution block and a regional matrix reconstruction result generation module; This module is used to input the kth pair of dual images output by the second convolution module of the backbone classification network. Feature map, k-th dual image After being processed by a single convolution block of this module, the output The feature map is then passed into the regional matrix reconstruction result generation module; The regional matrix reconstruction result generation module inputs the kth scrambled image The feature maps are rearranged to The position of the components is restored to the characteristic graph of the kth unshuffled graph The localization accuracy of different regions in the dual-image feature map is then measured, inducing the backbone network to simulate the semantic correlation between regions through end-to-end training.

9. The configuration knowledge-driven fine-grained aircraft target recognition method according to claim 8, Features: The adversarial learning module is used to determine the feature map output by the third convolution module of the backbone classification network of the kth pair of dual images. Derived from the original image of the k-th aircraft head direction correction result or its scrambled image; The specific steps are as follows: the kth pair of dual images is processed by the third convolution module of the backbone classification network module, and the feature map of the kth pair of dual images is output. Enter the adversarial learning network module (ALN) for adversarial learning, and obtain the feature map distinction result of the kth original image or its scrambled image, represented as v k ∈{0,1} 2 ; The method by which this module determines whether the dual image is destroyed is expressed as: In the formula, The kth pair of dual images is classified by the backbone network mth k The feature vector output by the layer, For the classification network from layer 1 to layer m k The learnable parameters of the layer, is a linear mapping, which is adversarial to the learning loss L Adv It can be expressed as: Where p(I k ) represents the kth shuffled image; The aforementioned direction recognition loss function L O Defined as: in The characteristic vector indicating that the direction of the aircraft target in the k-th remote sensing aircraft four-way aligned image is the i-th true direction; The feature vector representing the direction of the aircraft target in the k-th remote sensing aircraft four-way alignment image is the i-th predicted direction; The loss function L of the aforementioned backbone classification network cls It can be expressed as: where α cls are all learnable parameters in the classification network; Indicates that the aircraft target in the k-th pair of dual images is of the t-th category; I k represents the kth image; The aforementioned area reorganization loss L loc The definitions are as follows: Among them, M σ(q) (p(I k )) indicates p(I k ) in I k The predicted position number in M (q) (I k ) indicates I k The components in I k The predicted position numbers in , their true values ​​are all q; To sum up, the overall loss calculation formula is as follows: L=αL 0 +βL cls +γL Adv +δL loc Among them, α, β, γ, and δ represent adjustable setting parameters; L 0 L is the loss of the nose direction recognition module; cls is the backbone classification network loss; L Adv To combat learning loss; L loc Losses due to regional restructuring.

10. A configuration knowledge-driven fine-grained aircraft target recognition system. Features: The sample label acquisition module is used to introduce each remote sensing image into the aircraft image slice generation module, and according to the bounding box where the aircraft target is located, the target is cropped to generate the initial aircraft image slice, and then the slice is affine transformed and scaled to a certain pixel size to obtain the remote sensing aircraft four-way aligned image, and the aircraft category and aircraft direction of each remote sensing aircraft four-way aligned image are marked; The network training module is used to build a high-resolution optical aircraft target fine-grained recognition model. Each remote sensing aircraft four-way aligned image is sequentially input into the high-resolution optical aircraft target fine-grained recognition model for prediction to obtain the predicted aircraft category and predicted aircraft direction of each remote sensing aircraft four-way aligned image. The loss function model is constructed by combining the marked aircraft category and marked aircraft direction of each remote sensing aircraft four-way aligned image. The optimized high-resolution optical aircraft target fine-grained recognition model is obtained through Adam algorithm optimization training. The real-time remote sensing image prediction module is used to obtain real-time remote sensing images. The real-time remote sensing aircraft four-way aligned image is obtained through step 1. The predicted aircraft category and predicted aircraft direction of the real-time remote sensing aircraft four-way aligned image are obtained through the optimized high-resolution aircraft target fine-grained recognition model.

Citation Information

Patent Citations

  • Training method of image recognition model and image recognition method

    CN112016591A

  • Remote sensing image aircraft identification method based on deep learning and part prior

    CN112966555A

  • Remote sensing image target fine granularity identification method, system and device and storage medium

    CN115019182A

  • Boundary-optimized remote sensing image semantic segmentation method and apparatus, and device and medium

    WO2023077816A1