Aircraft target detection and identification method and device based on optical and SAR remote sensing data

Abstract

The invention relates to an aircraft target detection and identification method and device based on optical and SAR remote sensing data. The method comprises the following steps: S1, selecting completely labeled SAR images to form a data set A for training a first target detection model; s2, selecting the optical images in the same region and the same moment with the data set A to form a data set B for training a second target detection model; s3, acquiring SAR and optical images in the same area and at the same moment, respectively inputting the SAR and optical images into a first target detection model and a second target detection model for detection, and respectively performing rotation transformation on the detection results of the first target detection model and the second target detection model according to the attitude angle of the detected target to obtain a first detection result and a second detection result; and S4, performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm. According to the method, a deep learning image processing method is adopted, and the accuracy of target identification is effectively improved through an optical and SAR multi-sensor data fusion detection technology.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing satellite data processing, and in particular to an aircraft target detection and recognition method and device based on optical and SAR remote sensing data. Background Art

[0002] Intelligent processing of remote sensing satellite data on orbit is one of the current research hotspots. Optical data and synthetic aperture radar (SAR) data are the two most common data types in the field of satellite remote sensing. Due to their different imaging principles, they have their own advantages in earth observation. SAR sensors have all-day and all-weather detection capabilities, can penetrate clouds and fog, and are not affected by shadows and lighting time, but they have problems such as insufficient texture and ground radiation information and greater difficulty in interpretation. Optical remote sensing images can intuitively reflect information such as texture, color and shape to users, but due to lighting and weather restrictions, the ability to obtain data is limited.

[0003] Therefore, how to improve the efficiency of satellite remote sensing target detection and recognition, combine the respective advantages of SAR and optics, use the advantages of SAR remote sensing images in geometric characteristics, and the advantages of optical remote sensing images in radiation characteristics that can extract rich spectral information for classification and interpretation, and construct an on-orbit intelligent target detection method and device based on optical and SAR remote sensing data, has become one of the hot issues in the field of on-orbit intelligent processing of remote sensing satellite data. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and device for aircraft target detection and recognition based on optical and SAR remote sensing data, which can realize aircraft target detection and recognition based on optical and SAR satellite remote sensing data.

[0005] To achieve the above-mentioned object of the invention, the present invention provides an aircraft target detection and recognition method based on optical and SAR remote sensing data, comprising the following steps:

[0006] Step S1, selecting fully annotated SAR images to form a data set A, and using the data set A to train a SAR image target detection model to obtain a first target detection model;

[0007] Step S2, selecting an optical image in the same area and at the same time as the SAR image in the data set A to form a data set B, and using the data set B to train an optical image target test model to obtain a second target detection model;

[0008] Step S3, acquiring a SAR image and an optical image of the same area at the same time, and inputting the images into the first target detection model and the second target detection model for detection respectively, and performing rotation transformation on the detection results of the first target detection model and the second target detection model respectively according to the attitude angle of the detection target, to obtain a first detection result and a second detection result;

[0009] Step S4: performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm.

[0010] According to a technical solution of the present invention, in step S3, the first target detection model outputs a first detection result, specifically including:

[0011] Step S311, using the Canny edge detection operator to extract edge information from the input SAR image, extracting the strong scattering points of the aircraft, using the positional relationship of the scattering points to build a model, obtaining the structural relationship characteristics of the aircraft, and forming an edge detection map;

[0012] Step S312: using a pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map;

[0013] Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center );

[0014] Step S314, extracting the corresponding SAR suspected target slice according to the center point of the suspected target;

[0015] Step S315: extracting geometric features, invariant moment features, and electromagnetic scattering features of the SAR suspected target slices, and classifying the suspected targets using a single-classification support vector machine to obtain a first aircraft target;

[0016] Step S316: Physically modeling the first aircraft target according to the scattering parameters and in combination with the attitude angle of the first aircraft target, performing a rotation transformation on the first aircraft target according to the wing deployment direction information of the first aircraft target, and obtaining a SAR slice of the first aircraft target as the first detection result and outputting it.

[0017] According to a technical solution of the present invention, in step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equation (1) and equation (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtaining the first detection result:

[0018]

[0019] Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

[0020] According to a technical solution of the present invention, in step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum point of the suspected target in the edge saliency map to obtain the SAR suspected target slice.

[0021] According to a technical solution of the present invention, in step S3, the second target detection model outputs the second detection result, specifically including:

[0022] Step S321, using ResNet-50 as the backbone network, performing feature extraction on the input optical image, and obtaining features of the optical image downsampled by 4, 8, 16, and 32 times, which are recorded as P1, P2, P3, and P4 respectively;

[0023] Step S322, divide the features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After the Transformer structure, the feature maps T1, T2, T3, and T4 of the same size are output respectively;

[0024] Step S323, performing feature fusion on the feature map obtained in step S22 through a bidirectional fusion feature pyramid network, and using a skip residual connection method at the initial position and final position of the feature of each layer to fuse the shallow features with the deep features;

[0025] Step S324, performing aircraft target detection in the feature map obtained by the position sub-network after the feature fusion in step S323, obtaining the center point of the second aircraft target, regressing the width w and height h of the bounding box from the center point of the second aircraft target, and obtaining the attitude angle θ2 of the second aircraft target;

[0026] Step S325: rotating the optical image of the second aircraft target according to the unfolding direction information of the wing of the aircraft target so that the angle of rotation is uniform; the calculation formula of the rotation change is as follows:

[0027]

[0028] in, represents the transformed coordinates of the center point of the second aircraft target, represents the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the origin of the coordinate axis before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°);

[0029] Step S326, pass each channel in the optical image of the second aircraft target after the rotation change through a Sigmoid function, and the heat map value of each point in the obtained result is the confidence of the second aircraft target with the point as the center point, obtain the target category through the classification subnetwork, obtain the optical image slice of the second aircraft target, and output it as the second target detection result.

[0030] According to a technical solution of the present invention, the attention-based decision fusion algorithm includes:

[0031] Step S41: According to the first target detection result, obtain the scattering key features, divide the scattering key points into cluster center points through the clustering algorithm, strengthen the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is N SAR ;

[0032] Step S42: according to the second target detection result, ResNet is used to extract the target features in the optical image slice, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ;

[0033] Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain a position feature set F of the SAR target and the optical target. SAR and P optics , the sizes are F SAR and N optics ;

[0034] Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij :

[0035]

[0036] in, And there is

[0037] γ and σ represent the aggregation functions of the image target features and position features respectively, and are calculated as follows:

[0038]

[0039] Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector,

[0040] Step S45: The obtained attention value a ij Normalized using the Softmax algorithm

[0041] Step S46: construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the category, bounding box and angle information of the target; Y is the optical image target detection set, where any detection result y j Contains the category, bounding box and angle information of the target; according to any result pair in the X,Y set (x i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible match result, and adds an (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E;

[0042] Step S47, using the maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection;

[0043] Step S48: Based on the matching result, decision fusion is performed on the matched first target detection result and the second target detection result, and a final target detection and recognition result is output.

[0044] According to a technical solution of the present invention, in step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including:

[0045] Remember C plane Class aircraft targets The detection result of the first target and the second target involved in the fusion are Z = [zSAR , z optics ], where z SAR represents the first target detection result, z optics represents the second target detection result, z k The posterior probability of belonging to class k is:

[0046]

[0047] Among them are different types of posterior probabilities of the two target detection results. The decisions of the first target detection model and the second target detection model are independent of each other. The joint probability distribution based on Bayesian theory is:

[0048] P(T c ||Y)=P(T c |y SAR )P(T c |y optics )

[0049] The target category is determined according to the maximum a posteriori probability criterion to obtain the final aircraft target category:

[0050] Identity(Y)=argmax c P(T c ||Y).

[0051] According to one aspect of the present invention, an aircraft target recognition device based on optical and SAR remote sensing data is provided, which is used to implement the above-mentioned aircraft target recognition method. The whole device adopts VPX architecture design, and includes:

[0052] The main control module adopts the CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing;

[0053] High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing and data distribution of the device;

[0054] SAR imaging module, which adopts FPGA+DSP architecture design, is used for imaging processing of SAR raw data;

[0055] The SAR target detection module adopts the AI ​​chip + FPGA architecture design and runs the first target detection model to perform target detection and recognition on the SAR image;

[0056] The optical target detection module adopts the AI ​​chip + FPGA architecture design and runs the second target detection model to perform target detection and recognition on the optical image;

[0057] The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data;

[0058] The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module;

[0059] A backplane is used to connect the main control module, the high-speed interface module, the SAR target detection module, the SAR target detection module, the storage module and the power module.

[0060] The present invention proposes a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data, which has the following advantages:

[0061] (1) Target detection and recognition are performed on optical remote sensing data and SAR remote sensing data respectively, and decision analysis is performed on the optical and SAR target detection and recognition results, making full use of the advantages of optical and SAR image data, expanding the data acquisition capability of optical remote sensing image recognition, and eliminating the problems of insufficient texture and geographic radiation information and difficulty in interpretation in SAR recognition, thereby improving the accuracy of target detection and recognition;

[0062] (2) An attention-based fusion module is designed to calculate the attention values ​​between the target detection results from two different sources, construct a target matching bipartite graph, and use the maximum matching algorithm to obtain the matching results of target detection. On this basis, Bayesian decision fusion is used to make decision fusion on the category recognition results output by the first target detection model and the second target detection model, effectively eliminating the shortcomings of different recognition methods when applied to aircraft target recognition, correcting the accuracy of the recognition results, and effectively improving the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A schematic diagram schematically showing an aircraft target detection device for optical and SAR remote sensing data according to an embodiment of the present invention;

[0065] Figure 2 A flow chart schematically showing a method for detecting a target using SAR data according to an embodiment of the present invention;

[0066] Figure 3 A flow chart schematically showing an optical data target detection method according to an embodiment of the present invention;

[0067] Figure 4 A flowchart schematically showing a method for detecting aircraft targets using optical and SAR remote sensing data according to an embodiment of the present invention;

[0068] Figure 5 The flowchart of the aircraft target detection and recognition method based on optical and SAR remote sensing data of the present invention is schematically shown. DETAILED DESCRIPTION

[0069] The description of the embodiments of this specification should be combined with the corresponding drawings, which should be considered as part of the complete specification. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated for simplification or convenience. Furthermore, the parts of each structure in the drawings will be described separately. It is worth noting that the elements not shown in the drawings or not described in words are in a form known to ordinary technicians in the relevant technical field.

[0070] The description of the embodiments herein and any reference to directions and orientations are only for the convenience of description and are not to be construed as any limitation on the scope of protection of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0071] Based on the solution of the present invention, this embodiment designs a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data, such as Figure 1-5 shown.

[0072] The present embodiment is a method for detecting and identifying aircraft targets based on optical and SAR remote sensing data, such as Figure 5 As shown, the following steps are included:

[0073] Step S1, selecting fully annotated SAR images to form a data set A, and using the data set A to train a SAR image target detection model to obtain a first target detection model;

[0074] Step S2, selecting optical images in the same area and at the same time as the SAR images in the data set A to form a data set B, and using the data set B to train an optical image target test model to obtain a second target detection model;

[0075] Step S3, obtaining a SAR image and an optical image of the same area at the same time, and inputting them into a first target detection model and a second target detection model for detection respectively, and performing rotation transformation on the detection results of the first target detection model and the second target detection model respectively according to the attitude angle of the detection target to obtain a first detection result and a second detection result;

[0076] Figure 2This is a flow chart of the SAR data target detection method of this embodiment;

[0077] Figure 3 This is a flow chart of the optical data target detection method of this embodiment;

[0078] In step S3, the first target detection model outputs a first detection result, which specifically includes:

[0079] Step S311, using the Canny edge detection operator to extract edge information from the input SAR image, extracting the strong scattering points of the aircraft, using the positional relationship of the scattering points to build a model, obtaining the structural relationship characteristics of the aircraft, and forming an edge detection map;

[0080] Step S312: using the pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map;

[0081] Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center );

[0082] Step S314, extracting the corresponding SAR suspected target slice according to the center point of the suspected target;

[0083] In step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum points of the suspected target in the edge saliency map to obtain a SAR suspected target slice.

[0084] Step S315: extracting geometric features, invariant moment features, and electromagnetic scattering features of the SAR suspected target slices, and classifying the suspected targets by using a single-classification support vector machine or a trained single-classification convolutional neural network, and dividing the suspected targets into aircraft targets and non-aircraft targets, and taking the classified aircraft targets as the first aircraft targets;

[0085] Step S316: Physically modeling the aircraft target according to the scattering parameters and the attitude angle of the first aircraft target, performing a rotation transformation on the first aircraft target according to the wing deployment direction information of the first aircraft target, obtaining a SAR slice of the first aircraft target as the first detection result and outputting it.

[0086] In step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equation (1) and equation (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtaining the first detection result:

[0087]

[0088] Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

[0089] In step S3, the second target detection model outputs a second detection result, which specifically includes:

[0090] Step S321, using ResNet-50 as the backbone network, extracting features of the input optical image, and obtaining features of the optical image downsampled by 4, 8, 16, and 32 times, which are recorded as P1, P2, P3, and P4 respectively;

[0091] Step S322, divide the features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After the Transformer structure, the feature maps T1, T2, T3, and T4 of the same size are output respectively;

[0092] Step S323, feature fusion is performed on the feature map obtained in step S322 through a bidirectional fusion feature pyramid network, and a skip residual connection method is used at the initial position and final position of the feature of each layer to fuse the shallow features with the deep features;

[0093] Step S324, performing aircraft target detection in the feature map obtained by the position sub-network after the feature fusion in step S323, obtaining the center point of the second aircraft target, regressing the width w and height h of the bounding box from the center point of the second aircraft target, and obtaining the attitude angle θ2 of the second aircraft target;

[0094] Step S325: according to the wing deployment direction information of the aircraft target, the optical image of the second aircraft target is rotated to make the angle of rotation uniform; the calculation formula of the rotation change is as follows:

[0095]

[0096] in, Indicates the transformed coordinates of the center point of the second aircraft target, Indicates the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the origin of the coordinate axis before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°);

[0097] Step S326, pass each channel in the optical image of the second aircraft target after the rotation change through a Sigmoid function, and the heat map value of each point in the obtained result is the confidence of the second aircraft target with the point as the center point, obtain the target category through the classification subnetwork, obtain the optical image slice of the second aircraft target, and output it as the second target detection result.

[0098] Step S4: performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm.

[0099] The target slices of the same-direction detection results output by the first target detection model and the second target detection model are matched to output the final target detection matching results. Then, Bayesian decision fusion is used to make decision fusion on the category recognition results between the results to output the final target detection recognition results.

[0100] Attention-based decision fusion algorithms include:

[0101] Step S41: According to the first target detection result, obtain the scattering key features, divide the scattering key points into cluster center points through the clustering algorithm, enhance the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is F SAR ;

[0102] Step S42: According to the second target detection result, ResNet is used to extract the target features in the optical image slice, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ;

[0103] Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain a position feature set P of the SAR target and the optical target. SAR and P optics , the sizes are N SAR and N optics ;

[0104] Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij :

[0105]

[0106] in, And there is

[0107] γ and σ represent the aggregation functions of the image target features and position features respectively, and are calculated as follows:

[0108]

[0109] Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector,

[0110] Step S45: The obtained attention value a ij Normalized using the Softmax algorithm

[0111] Step S46: construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the category, bounding box and angle information of the target; Y is the optical image target detection set, where any detection result y j Contains the target category, bounding box, and angle information. i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible match result, and adds an (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E;

[0112] Step S47, using the maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection;

[0113] Step S48: Based on the matching result, a decision fusion is performed on the matched first target detection result and the second target detection result, and a final target detection and recognition result is output.

[0114] In step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including:

[0115] Remember C plane Class aircraft targets The detection result of the first target and the second target involved in the fusion are Z = [z SAR , z optics ], where zSAR represents the first target detection result, z optics represents the second target detection result, z k The posterior probability of belonging to class k is:

[0116]

[0117] Among them are different types of posterior probabilities of the two target detection results. The decisions of the first target detection model and the second target detection model are independent of each other. The joint probability distribution based on Bayesian theory is:

[0118] P(T c |Y)=P(T c |y SAR )P(T c |y optics )

[0119] The target category is determined according to the maximum a posteriori probability criterion to obtain the final aircraft target category:

[0120] Identity(Y)=argmax c P(T c |Y).

[0121] This embodiment also provides an aircraft target recognition device based on optical and SAR remote sensing data, which is used to implement the above-mentioned aircraft target recognition method. The whole device adopts VPX architecture design. Figure 1 As shown in the figure, it includes a main control module, a high-speed interface module, a target detection module, a storage module, a power module, and a backplane. Each module is dynamically interconnected based on the topological architecture of the Space VPX standard, and the high-speed data transmission of each module is interacted through the PCIE high-speed interface.

[0122] The main control module adopts the CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing;

[0123] The main control module is composed of CPU (such as FT D2000), FPGA (such as JFM7K325T), DDR (such as VD3D16G72RB), Flash, power supply circuit, interface circuit, high-speed data transmission circuit and main control software, which realizes the operation, monitoring, control, partial information processing, communication between modules and external communication of the device. The main control module exchanges data with SAR imaging module, target detection module (SAR), target detection module (optical), high-speed interface module and storage module through a group of PCIE×4 interfaces; and synchronizes and controls with high-speed interface module, SAR imaging module, target detection module (SAR), target detection module (optical) and storage module through 16-channel LVDS. The main control module uses CAN bus as management bus to monitor the working voltage, current, temperature, working status and other parameters of each module, and control the reset and restart of each module.

[0124] High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing and data distribution of the device;

[0125] SAR imaging module, which adopts FPGA+DSP architecture design, is used for imaging processing of SAR raw data;

[0126] The SAR imaging module consists of FPGA (such as JFM7VX690T-RT), DSP (such as FT-M6678H), DDR (such as VD3D16G72RB), Flash, power supply circuit, interface circuit, high-speed data transmission circuit and SAR imaging processing algorithm. It is used to receive SAR raw data in real time and perform imaging processing through FPGA+DSP array. The imaging results can be transmitted to the target detection module, high-speed interface module or main control module for subsequent processing. The SAR imaging module exchanges data with the main control module, high-speed interface module, target detection module (SAR) and storage module through a group of PCIE×4 interfaces;

[0127] The SAR target detection module adopts the AI ​​chip + FPGA architecture design and runs the first target detection model to perform target detection and recognition on SAR images;

[0128] The optical target detection module adopts the AI ​​chip + FPGA architecture design and runs the second target detection model to detect and identify targets in optical images.

[0129] The SAR target detection module and the optical target detection module are composed of SoC (such as Lingxi KA200), FPGA (such as JFM7K325T), DDR (such as VD3D16G72RB), Flash, power circuit, interface circuit, high-speed data transmission circuit and target detection algorithm, which are used for intelligent detection of optical and SAR image targets. The target detection module (SAR) exchanges data with the main control module, high-speed interface module, SAR imaging module and storage module through a set of PCIE×4 interfaces; the target detection module (optical) exchanges data with the main control module, high-speed interface module and storage module through a set of PCIE×4 interfaces

[0130] High-speed interface module: It consists of system configuration FPGA (such as JFMRS01RH), Nand Flash (such as VDNF64G08), data processing FPGA (such as JFM7VX690T), DSP (such as FT6701-V400), power supply circuit, interface circuit, high-speed data transmission circuit and data access algorithm.

[0131] The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data;

[0132] The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module;

[0133] The power module consists of a primary power protection circuit, an anti-surge circuit, an overcurrent protection circuit, an anti-EMI circuit, a power-on and power-off remote control circuit, and a voltage telemetry circuit. The power module converts the primary power bus (42V / 28V) into 12V and 5.0V, providing the device with two power supply modes: 12V@130W and 5.0V@30W.

[0134] The backplane is used to connect the main control module, the high-speed interface module, the SAR target detection module, the SAR target detection module, the storage module and the power module.

[0135] This embodiment takes the detection of aircraft targets in optical and SAR images in the same area at similar times as an example to explain in detail the process of detecting on-orbit targets using optical data by the device of this embodiment. Figure 4 As shown, the specific steps are as follows:

[0136] When the device is powered on, each module processing program is automatically loaded;

[0137] The high-speed interface module accesses the optical and SAR raw data and determines the data source;

[0138] The SAR raw data is preprocessed in the SAR preprocessing unit of the high-speed interface module; the SAR raw data is imaged by the SAR imaging module; the high-speed interface module completes the slicing process and distributes it to the target detection module (SAR); the target detection and recognition is completed by the first target detection model in the target detection module (SAR); the detection and recognition results are sent to the main control module to complete the information post-processing;

[0139] After the optical data is preprocessed in the optical image preprocessing unit of the high-speed interface module, the slice processing is completed and distributed to the target detection module (optical); the target detection and recognition are completed through the second target detection model in the target detection module (optical), and the detection and recognition results are sent to the main control module to complete the information post-processing;

[0140] The main control module performs decision fusion processing on the optical and SAR image detection and recognition results of the same area at similar times to generate the final target detection and recognition results; the target detection and recognition results are transmitted to the high-speed interface module, and then to the storage module for storage, and then transmitted out of the device through the high-speed data interface.

[0141] It should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, once the basic creative concept of the present invention is known, a number of improvements and modifications can be made by a person skilled in the art without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for detecting and identifying aircraft targets based on optical and SAR remote sensing data, characterized in that: The following steps are involved: Step S1, selecting fully annotated SAR images to form a data set A, and using the data set A to train a SAR image target detection model to obtain a first target detection model; Step S2, selecting an optical image in the same area and at the same time as the SAR image in the data set A to form a data set B, and using the data set B to train an optical image target test model to obtain a second target detection model; Step S3, acquiring a SAR image and an optical image of the same area at the same time, and inputting the images into the first target detection model and the second target detection model for detection respectively, and performing rotation transformation on the detection results of the first target detection model and the second target detection model respectively according to the attitude angle of the detection target, to obtain a first detection result and a second detection result; Step S4: performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm.

2. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 1, characterized in that: In step S3, the first target detection model outputs a first detection result, which specifically includes: Step S311, using the Canny edge detection operator to extract edge information from the input SAR image, extracting the strong scattering points of the aircraft, using the positional relationship of the scattering points to build a model, obtaining the structural relationship characteristics of the aircraft, and forming an edge detection map; Step S312: using a pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map; Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center ); Step S314, extracting the corresponding SAR suspected target slice according to the center point of the suspected target; Step S315: extracting geometric features, invariant moment features, and electromagnetic scattering features of the SAR suspected target slices, and classifying the suspected targets using a single-classification support vector machine to obtain a first aircraft target; Step S316: Physically modeling the first aircraft target according to the scattering parameters and in combination with the attitude angle of the first aircraft target, performing a rotation transformation on the first aircraft target according to the wing deployment direction information of the first aircraft target, and obtaining a SAR slice of the first aircraft target as the first detection result and outputting it.

3. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equation (1) and equation (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtaining the first detection result: Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

4. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In the step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum point of the suspected target in the edge saliency map to obtain the SAR suspected target slice.

5. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In the step S3, the second target detection model outputs the second detection result, which specifically includes: Step S321, using ResNet-50 as the backbone network, performing feature extraction on the input optical image, and obtaining features of the optical image downsampled by 4, 8, 16, and 32 times, which are recorded as P1, P2, P3, and P4 respectively; Step S322, divide the features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After the Transformer structure, the feature maps T1, T2, T3, and T4 of the same size are output respectively; Step S323, performing feature fusion on the feature map obtained in step S322 through a bidirectional fusion feature pyramid network, and using a skip residual connection method at the initial position and final position of the feature of each layer to fuse the shallow features with the deep features; Step S324, performing aircraft target detection in the feature map obtained by the position sub-network after the feature fusion in step S323, obtaining the center point of the second aircraft target, regressing the width w and height h of the bounding box from the center point of the second aircraft target, and obtaining the attitude angle θ2 of the second aircraft target; Step S325: rotating the optical image of the second aircraft target according to the unfolding direction information of the wing of the aircraft target so that the angle of rotation is uniform; the calculation formula of the rotation change is as follows: in, represents the transformed coordinates of the center point of the second aircraft target, represents the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the origin of the coordinate axis before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°); Step S326, pass each channel in the optical image of the second aircraft target after the rotation change through a Sigmoid function, and the heat map value of each point in the obtained result is the confidence of the second aircraft target with the point as the center point, obtain the target category through the classification subnetwork, obtain the optical image slice of the second aircraft target, and output it as the second target detection result.

6. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 5, characterized in that: The attention-based decision fusion algorithm includes: Step S41: According to the first aircraft target, obtain the scattering key features, divide the scattering key points into cluster center points through the clustering algorithm, strengthen the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is F SAR ; Step S42: according to the second target detection result, ResNet is used to extract the target features in the optical image slice, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ; Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain a position feature set P of the SAR target and the optical target. SAR and P optics , the sizes are N SAR and N optics ; Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij : in, And there is γ and σ represent the aggregation functions of the image target features and position features respectively, and are calculated as follows: Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector, Step S45: The obtained attention value a ij Normalized using the Softmax algorithm Step S46: construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the category, bounding box and angle information of the target; Y is the optical image target detection set, where any detection result y j Contains the category, bounding box and angle information of the target; according to any result pair in the X, Y set (x i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible match result, and adds an (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E; Step S47, using the maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection; Step S48: Based on the matching result, decision fusion is performed on the matched first target detection result and the second target detection result, and a final target detection and recognition result is output.

7. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 6, characterized in that: In step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including: Remember C plane Class aircraft targets The detection result of the first target and the second target involved in the fusion are Z = [z SAR , z optics ], where z SAR represents the first target detection result, z optics represents the second target detection result, z k The posterior probability of belonging to class k is: Among them are different types of posterior probabilities of the two target detection results. The decisions of the first target detection model and the second target detection model are independent of each other. The joint probability distribution based on Bayesian theory is: P(T c |Y)=P(T c |y SAR )P(T c |y optics ) The target category is determined according to the maximum a posteriori probability criterion to obtain the final aircraft target category: Identity(Y)=argmax c P(T c |Y)。 8. An aircraft target recognition device based on optical and SAR remote sensing data, used to implement the aircraft target detection and recognition method based on optical and SAR remote sensing data as claimed in any one of claims 1 to 7, the whole device adopts VPX architecture design, including: The main control module adopts the CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing; High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing and data distribution of the device; SAR imaging module, which adopts FPGA+DSP architecture design, is used for imaging processing of SAR raw data; The SAR target detection module adopts the AI ​​chip + FPGA architecture design and runs the first target detection model to perform target detection and recognition on the SAR image; The optical target detection module adopts the AI ​​chip + FPGA architecture design and runs the second target detection model to perform target detection and recognition on the optical image; The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data; The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module; A backplane is used to connect the main control module, the high-speed interface module, the SAR target detection module, the SAR target detection module, the storage module and the power module.

Images