Aircraft target detection and recognition method and device based on optical and SAR remote sensing data

Abstract

The present invention relates to a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data. The method comprises the following steps: S1, selecting fully annotated SAR images to form a data set A for training a first target detection model; S2, selecting optical images in the same area and at the same time as data set A to form a data set B for training a second target detection model; S3, obtaining SAR and optical images in the same area and at the same time, and inputting them into the first and second target detection models for detection, respectively, and performing rotation transformation on the detection results of the first and second target detection models according to the attitude angle of the detection target to obtain first and second detection results; S4, performing decision fusion on the first and second detection results through an attention-based decision fusion detection algorithm. The present invention adopts a deep learning image processing method and uses optical and SAR multi-sensor data fusion detection technology to effectively improve the accuracy of target recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing satellite data processing, and in particular to a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data. Background Art

[0002] Intelligent on-orbit processing of remote sensing satellite data is a current research hotspot. Optical data and synthetic aperture radar (SAR) data are the two most common data types in satellite remote sensing. Due to their different imaging principles, each has its own advantages in Earth observation. SAR sensors have all-day, all-weather detection capabilities, can penetrate clouds and fog, and are unaffected by shadows and illumination duration. However, they lack sufficient texture and ground radiation information, making interpretation difficult. Optical remote sensing images can intuitively convey texture, color, and shape information to users, but due to lighting and weather restrictions, data acquisition capabilities are limited.

[0003] Therefore, how to improve the efficiency of satellite remote sensing target detection and recognition, combine the respective advantages of SAR and optics, utilize the advantages of SAR remote sensing images in geometric characteristics, and the advantages of optical remote sensing images in radiation characteristics that can extract rich spectral information for classification and interpretation, and construct on-orbit intelligent target detection methods and devices based on optical and SAR remote sensing data has become one of the hot issues in the field of on-orbit intelligent processing of remote sensing satellite data. Summary of the Invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and device for aircraft target detection and identification based on optical and SAR remote sensing data, which can realize aircraft target detection and identification based on optical and SAR satellite remote sensing data.

[0005] To achieve the above-mentioned object of the invention, the present invention provides a method for detecting and identifying aircraft targets based on optical and SAR remote sensing data, comprising the following steps:

[0006] Step S1: selecting fully annotated SAR images to form a data set A, and using the data set A to train a SAR image target detection model to obtain a first target detection model;

[0007] Step S2: selecting an optical image in the same area and at the same time as the SAR image in the dataset A to form a dataset B, and using the dataset B to train an optical image target test model to obtain a second target detection model;

[0008] Step S3: Acquire a SAR image and an optical image of the same area at the same time, input them into the first target detection model and the second target detection model for detection, respectively, and perform rotation transformation on the detection results of the first target detection model and the second target detection model according to the attitude angle of the detection target to obtain a first detection result and a second detection result;

[0009] Step S4: performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm.

[0010] According to a technical solution of the present invention, in step S3, the first target detection model outputs a first detection result, specifically including:

[0011] Step S311: Use the Canny edge detection operator to extract edge information from the input SAR image, extract the strong scattering points of the aircraft, use the positional relationship of the scattering points to build a model, obtain the structural relationship characteristics of the aircraft, and form an edge detection map;

[0012] Step S312: using a pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map;

[0013] Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center );

[0014] Step S314: extracting the corresponding SAR suspected target slice according to the center point of the suspected target;

[0015] Step S315: extracting geometric features, moment invariant features, and electromagnetic scattering features of the SAR suspected target slice, and classifying the suspected target using a single-class support vector machine to obtain a first aircraft target.

[0016] Step S316: Physically model the first aircraft target based on the scattering parameters and the attitude angle of the first aircraft target, perform a rotation transformation on the first aircraft target based on the wing deployment direction information of the first aircraft target, obtain a SAR slice of the first aircraft target, and output it as the first detection result.

[0017] According to a technical solution of the present invention, in step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equations (1) and (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtaining the first detection result:

[0018]

[0019] Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

[0020] According to a technical solution of the present invention, in step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum point of the suspected target in the edge saliency map to obtain the SAR suspected target slice.

[0021] According to a technical solution of the present invention, in step S3, the second target detection model outputs the second detection result, specifically including:

[0022] Step S321: ResNet-50 is used as the backbone network to perform feature extraction on the input optical image, and the features of the optical image downsampled by 4, 8, 16, and 32 times are obtained and recorded as P1, P2, P3, and P4 respectively;

[0023] Step S322: Divide features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After passing through the Transformer structure, feature maps T1, T2, T3, and T4 of the same size are output respectively;

[0024] Step S323: performing feature fusion on the feature map obtained in step S22 through a bidirectional fusion feature pyramid network. By using multiple upsampling, downsampling, and cross-layer connections, a skip residual connection method is used at the initial and final positions of the features of each layer to fuse the shallow features with the deep features.

[0025] Step S324: Perform aircraft target detection in the feature map obtained by the feature fusion in step S323 through the position sub-network to obtain the center point of the second aircraft target, regress the width w and height h of the bounding box from the center point of the second aircraft target, and obtain the attitude angle θ2 of the second aircraft target;

[0026] Step S325: Rotate the optical image of the second aircraft target according to the wing deployment direction information of the aircraft target so that the angle of rotation is uniform. The calculation formula for the rotation change is as follows:

[0027]

[0028] in, represents the transformed coordinates of the center point of the second aircraft target, represents the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the coordinate axis origin before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°);

[0029] Step S326: Pass each channel of the rotated optical image of the second aircraft target through a Sigmoid function. The heat map value of each point in the obtained result is the confidence level of the second aircraft target centered at that point. The target category is obtained through the classification subnetwork, and an optical image slice of the second aircraft target is obtained and output as the second target detection result.

[0030] According to a technical solution of the present invention, the attention-based decision fusion algorithm includes:

[0031] Step S41: According to the first target detection result, obtain the scattering key features, divide the scattering key points into cluster centers through the clustering algorithm, strengthen the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is N SAR ;

[0032] Step S42: According to the second target detection result, ResNet is used to extract the target features in the optical image slice, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ;

[0033] Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain the position feature set F of the SAR target and the optical target. SAR and P optics , the sizes are F SAR and N optics ;

[0034] Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij :

[0035]

[0036] in, And there is

[0037] γ and σ represent the aggregation functions of image target features and position features, respectively, and are calculated as follows:

[0038]

[0039] Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector,

[0040] Step S45: The obtained attention value a ij Normalized using the Softmax algorithm

[0041] Step S46: Construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the target category, bounding box and angle information; Y is the optical image target detection set, where any detection result y j Contains the target category, bounding box and angle information; according to any result pair (x i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible matching result, and adds a (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E;

[0042] Step S47: Using a maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection;

[0043] Step S48: Based on the matching result, perform decision fusion on the matched first target detection result and the second target detection result, and output a final target detection and recognition result.

[0044] According to a technical solution of the present invention, in step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including:

[0045] Remember C plane Class aircraft targets The first target detection result and the second target detection result involved in the fusion are Z=[zSAR , z optics ], where z SAR Indicates the first target detection result, z optics Represents the second target detection result, z k The posterior probability of belonging to class k is:

[0046]

[0047] Among them are the different types of posterior probabilities of the two target detection results. The decisions of the first target detection model and the second target detection model are independent of each other. The joint probability distribution obtained based on Bayesian theory is:

[0048] P(T c ||Y)=P(T c |y SAR )P(T c |y optics )

[0049] The target category is determined according to the maximum posterior probability criterion to obtain the final aircraft target category:

[0050] Identity(Y)=argmax c P(T c ||Y).

[0051] According to one aspect of the present invention, a device for aircraft target recognition based on optical and SAR remote sensing data is provided for implementing the above-mentioned aircraft target recognition method. The entire device adopts a VPX architecture design and includes:

[0052] The main control module adopts a CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing;

[0053] High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing, and data distribution of the device;

[0054] SAR imaging module, designed with FPGA+DSP architecture, is used for imaging processing of SAR raw data;

[0055] The SAR target detection module uses an AI chip + FPGA architecture design and runs the first target detection model to perform target detection and recognition on SAR images.

[0056] The optical target detection module adopts the AI ​​chip + FPGA architecture design and runs the second target detection model to detect and identify targets in optical images;

[0057] The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data;

[0058] The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module;

[0059] A backplane is used to connect the main control module, the high-speed interface module, the SAR target detection module, the SAR target detection module, the storage module and the power module.

[0060] The present invention proposes a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data, which has the following advantages:

[0061] (1) Target detection and recognition are performed on optical remote sensing data and SAR remote sensing data respectively, and decision analysis is performed on the optical and SAR target detection and recognition results, making full use of the advantages of optical and SAR image data, expanding the data acquisition capability of optical remote sensing image recognition, and eliminating the problems of insufficient texture and geographic radiation information and difficulty in interpretation in SAR recognition, thereby improving the accuracy of target detection and recognition;

[0062] (2) An attention-based fusion module is designed to calculate the attention value between the target detection results from two different sources, construct a target matching bipartite graph, and use the maximum matching algorithm to obtain the matching results of target detection. On this basis, Bayesian decision fusion is used to make decision fusion on the category recognition results output by the first target detection model and the second target detection model, effectively eliminating the shortcomings of different recognition methods when applied to aircraft target recognition, correcting the accuracy of the recognition results, and effectively improving the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0064] Figure 1 A schematic diagram schematically illustrates an aircraft target detection device based on optical and SAR remote sensing data according to an embodiment of the present invention;

[0065] Figure 2 A flow chart schematically illustrating a method for detecting a target using SAR data according to an embodiment of the present invention;

[0066] Figure 3 A flowchart schematically illustrating an optical data target detection method according to an embodiment of the present invention;

[0067] Figure 4 A flowchart schematically illustrating a method for detecting aircraft targets using optical and SAR remote sensing data according to an embodiment of the present invention;

[0068] Figure 5 The flowchart of the aircraft target detection and recognition method based on optical and SAR remote sensing data of the present invention is schematically shown. DETAILED DESCRIPTION

[0069] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0070] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0071] This embodiment is based on the solution of the present invention, and designs a method and device for detecting and identifying aircraft targets based on optical and SAR remote sensing data, such as Figure 1-5 shown.

[0072] This embodiment is a method for detecting and identifying aircraft targets based on optical and SAR remote sensing data, such as Figure 5 As shown, the following steps are included:

[0073] Step S1: select a complete set of annotated SAR images to form a data set A, and use the data set A to train a SAR image target detection model to obtain a first target detection model;

[0074] Step S2: select optical images that are in the same area and at the same time as the SAR images in dataset A to form dataset B, and use dataset B to train an optical image target test model to obtain a second target detection model;

[0075] Step S3: Acquire a SAR image and an optical image of the same area at the same time, input them into a first target detection model and a second target detection model for detection, respectively, and perform rotation transformation on the detection results of the first target detection model and the second target detection model according to the attitude angle of the detection target to obtain a first detection result and a second detection result;

[0076] Figure 2This is a flow chart of the SAR data target detection method of this embodiment;

[0077] Figure 3 This is a flow chart of the optical data target detection method of this embodiment;

[0078] In step S3, the first target detection model outputs a first detection result, which specifically includes:

[0079] Step S311: Use the Canny edge detection operator to extract edge information from the input SAR image, extract the strong scattering points of the aircraft, use the positional relationship of the scattering points to build a model, obtain the structural relationship characteristics of the aircraft, and form an edge detection map;

[0080] Step S312: using the pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map;

[0081] Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center );

[0082] Step S314: extracting the corresponding SAR suspected target slice according to the center point of the suspected target;

[0083] In step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum point of the suspected target in the edge saliency map to obtain a SAR suspected target slice.

[0084] Step S315: For the SAR suspected target slice, extract the geometric features, invariant moment features, and electromagnetic scattering features of the SAR suspected target slice, and classify the suspected target using a single-class support vector machine or a trained single-class convolutional neural network. The suspected target is divided into an aircraft target and a non-aircraft target, and the classified aircraft target is used as the first aircraft target.

[0085] Step S316: Physically model the aircraft target based on the scattering parameters and the attitude angle of the first aircraft target, perform a rotation transformation on the first aircraft target based on the wing deployment direction information of the first aircraft target, obtain a SAR slice of the first aircraft target, and output it as a first detection result.

[0086] In step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equations (1) and (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtaining the first detection result:

[0087]

[0088] Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

[0089] In step S3, the second target detection model outputs a second detection result, which specifically includes:

[0090] Step S321: ResNet-50 is used as the backbone network to perform feature extraction on the input optical image, and the features of the optical image downsampled by 4, 8, 16, and 32 times are obtained and recorded as P1, P2, P3, and P4 respectively;

[0091] Step S322: Divide features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After passing through the Transformer structure, feature maps T1, T2, T3, and T4 of the same size are output respectively;

[0092] Step S323: perform feature fusion on the feature map obtained in step S322 through a bidirectional fusion feature pyramid network. By using multiple upsampling, downsampling and cross-layer connections, a skip residual connection method is used at the initial and final positions of the features of each layer to fuse the shallow features with the deep features.

[0093] Step S324: Perform aircraft target detection in the feature map obtained by the feature fusion in step S323 through the position sub-network to obtain the center point of the second aircraft target, regress the width w and height h of the bounding box from the center point of the second aircraft target, and obtain the attitude angle θ2 of the second aircraft target;

[0094] Step S325: Rotate the optical image of the second aircraft target according to the wing deployment direction information of the aircraft target so that the angle of rotation is uniform. The calculation formula for the rotation change is as follows:

[0095]

[0096] in, Indicates the transformed coordinates of the center point of the second aircraft target, Indicates the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the coordinate axis origin before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°);

[0097] Step S326: Pass each channel of the rotated optical image of the second aircraft target through a Sigmoid function. The heat map value of each point in the obtained result is the confidence level of the second aircraft target centered at that point. The target category is obtained through the classification subnetwork, and an optical image slice of the second aircraft target is obtained and output as the second target detection result.

[0098] Step S4: performing decision fusion on the first detection result and the second detection result through an attention-based decision fusion detection algorithm.

[0099] The target slices of the same-direction detection results output by the first target detection model and the second target detection model are matched to output the final target detection matching result. The Bayesian decision fusion is then used to make a decision fusion on the category recognition results between the results to output the final target detection and recognition result.

[0100] Attention-based decision fusion algorithms include:

[0101] Step S41: According to the first target detection result, obtain the scattering key features, divide the scattering key points into cluster centers through the clustering algorithm, enhance the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is F SAR ;

[0102] Step S42: According to the second target detection result, ResNet is used to extract the target features in the optical image slice, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ;

[0103] Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain the position feature set P of the SAR target and the optical target. SAR and P optics , the sizes are N SAR and N optics ;

[0104] Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij :

[0105]

[0106] in, And there is

[0107] γ and σ represent the aggregation functions of image target features and position features, respectively, and are calculated as follows:

[0108]

[0109] Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector,

[0110] Step S45: The obtained attention value a ij Normalized using the Softmax algorithm

[0111] Step S46: Construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the target category, bounding box and angle information; Y is the optical image target detection set, where any detection result y j Contains the target category, bounding box and angle information. According to any result pair (x i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible matching result, and adds a (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E;

[0112] Step S47: Using a maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection;

[0113] Step S48: Based on the matching result, a decision fusion is performed on the matched first target detection result and the second target detection result, and a final target detection and recognition result is output.

[0114] In step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including:

[0115] Remember C plane Class aircraft targets The first target detection result and the second target detection result involved in the fusion are Z=[z SAR , z optics ], where zSAR Indicates the first target detection result, z optics Represents the second target detection result, z k The posterior probability of belonging to class k is:

[0116]

[0117] Among them are the different types of posterior probabilities of the two target detection results. The decisions of the first target detection model and the second target detection model are independent of each other. The joint probability distribution obtained based on Bayesian theory is:

[0118] P(T c |Y)=P(T c |y SAR )P(T c |y optics )

[0119] The target category is determined according to the maximum posterior probability criterion to obtain the final aircraft target category:

[0120] Identity(Y)=argmax c P(T c |Y).

[0121] This embodiment also provides an aircraft target recognition device based on optical and SAR remote sensing data, which is used to implement the above-mentioned aircraft target recognition method. The whole device adopts VPX architecture design. Figure 1 As shown in the figure, it includes a main control module, high-speed interface module, target detection module, storage module, power module, and backplane. Each module is dynamically interconnected based on the Space VPX standard topology architecture, and high-speed data transmission between modules is achieved through the PCIE high-speed interface.

[0122] The main control module adopts a CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing;

[0123] The main control module consists of a CPU (e.g., FT-D2000), an FPGA (e.g., JFM7K325T), a DDR memory (e.g., VD3D16G72RB), Flash memory, power supply circuits, interface circuits, high-speed data transmission circuits, and main control software. This module implements device operation, monitoring, control, partial information processing, and inter-module and external communication. The main control module exchanges data with the SAR imaging module, target detection module (SAR), target detection module (optical), high-speed interface module, and storage module via a set of PCIE x4 interfaces. Synchronous control is provided via 16-channel LVDS with the high-speed interface module, SAR imaging module, target detection module (SAR), target detection module (optical), and storage module. The main control module uses the CAN bus as a management bus to monitor each module's operating voltage, current, temperature, operating status, and other parameters, and to control module resets and restarts.

[0124] High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing, and data distribution of the device;

[0125] SAR imaging module, designed with FPGA+DSP architecture, is used for imaging processing of SAR raw data;

[0126] The SAR imaging module consists of an FPGA (such as the JFM7VX690T-RT), a DSP (such as the FT-M6678H), a DDR (such as the VD3D16G72RB), Flash, a power supply circuit, an interface circuit, a high-speed data transmission circuit, and a SAR imaging processing algorithm. It is used to receive raw SAR data in real time and perform imaging processing through the FPGA + DSP array. The imaging results can be transmitted to the target detection module, the high-speed interface module, or the main control module for subsequent processing. The SAR imaging module exchanges data with the main control module, the high-speed interface module, the target detection module (SAR), and the storage module via a set of PCIE × 4 interfaces.

[0127] The SAR target detection module uses an AI chip + FPGA architecture design and runs the first target detection model to detect and identify targets in SAR images.

[0128] The optical target detection module uses an AI chip + FPGA architecture design and runs a second target detection model to detect and identify targets in optical images.

[0129] The SAR target detection module and the optical target detection module are both composed of SoC (such as Lingxi KA200), FPGA (such as JFM7K325T), DDR (such as VD3D16G72RB), Flash, power supply circuit, interface circuit, high-speed data transmission circuit and target detection algorithm, and are used for intelligent detection of optical and SAR image targets. The target detection module (SAR) exchanges data with the main control module, high-speed interface module, SAR imaging module and storage module through a set of PCIE×4 interfaces; the target detection module (optical) exchanges data with the main control module, high-speed interface module and storage module through a set of PCIE×4 interfaces.

[0130] High-speed interface module: It consists of a system configuration FPGA (such as JFMRS01RH), Nand Flash (such as VDNF64G08), a data processing FPGA (such as JFM7VX690T), a DSP (such as FT6701-V400), a power supply circuit, an interface circuit, a high-speed data transmission circuit, and a data access algorithm.

[0131] The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data;

[0132] The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module;

[0133] The power module consists of a primary power protection circuit, an anti-surge circuit, an overcurrent protection circuit, an anti-EMI circuit, a remote power-on / off control circuit, and a voltage telemetry circuit. The power module converts the primary power bus (42V / 28V) into 12V and 5.0V, providing the device with two power modes: 12V@130W and 5.0V@30W.

[0134] The backplane is used to connect the main control module, high-speed interface module, SAR target detection module, SAR target detection module, storage module and power module.

[0135] This embodiment takes the detection of aircraft targets in optical and SAR images at similar times in the same area as an example to explain in detail the process of detecting on-orbit targets using optical data by the device of this embodiment. Figure 4 The specific steps are as follows:

[0136] When the device is powered on, each module processing program is automatically loaded;

[0137] The high-speed interface module accesses optical and SAR raw data and determines the data source;

[0138] The SAR raw data is pre-processed in the SAR pre-processing unit of the high-speed interface module; the SAR raw data is imaged by the SAR imaging module; the high-speed interface module then performs slicing processing and distributes it to the target detection module (SAR); target detection and recognition are completed using the first target detection model in the target detection module (SAR); the detection and recognition results are sent to the main control module to complete information post-processing;

[0139] After the optical data is pre-processed by the optical image pre-processing unit of the high-speed interface module, it is sliced ​​and distributed to the target detection module (optical). The target detection and recognition are completed by the second target detection model in the target detection module (optical). The detection and recognition results are sent to the main control module to complete the information post-processing.

[0140] The main control module performs decision fusion processing on the optical and SAR image detection and recognition results of the same area at similar times to generate the final target detection and recognition results; the target detection and recognition results are transmitted to the high-speed interface module, and then transferred to the storage module for storage, and then transmitted out of the device through the high-speed data interface.

[0141] It should be noted that the above is a preferred embodiment of the present invention. It should be noted that although the preferred embodiment of the present invention has been described, it is clear that those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered as within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for detecting and identifying aircraft targets based on optical and SAR remote sensing data, characterized in that: The following steps are involved: Step S1: selecting fully annotated SAR images to form a data set A, and using the data set A to train a SAR image target detection model to obtain a first target detection model; Step S2: selecting an optical image in the same area and at the same time as the SAR image in the dataset A to form a dataset B, and using the dataset B to train an optical image target test model to obtain a second target detection model; Step S3: Acquire a SAR image and an optical image of the same area at the same time, input them into the first target detection model and the second target detection model for detection, respectively, and perform rotation transformation on the detection results of the first target detection model and the second target detection model according to the attitude angle of the detection target to obtain a first detection target result and a second detection target result; The first detection target result is a SAR slice of the first aircraft target; The second detection target result is an optical image slice of the second aircraft target; Step S4: performing decision fusion on the first detection target result and the second detection target result through an attention-based decision fusion detection algorithm; The attention-based decision fusion algorithm includes: Step S41: According to the first aircraft target, obtain the scattering key features, divide the scattering key points into cluster centers through the clustering algorithm, strengthen the connection between the scattering key points, form clustered scattering key features, and obtain the clustered scattering feature set F SAR , the size of the cluster scattering feature set is N SAR ; Step S42: According to the second target detection result, ResNet is used to extract the target features in the optical image slice of the second aircraft target, and the image target feature set F is obtained. optics , the size of the image target feature set is N optics ; Step S43: Serialize the target center point coordinates of the first target detection result and the second target detection result according to the clustered scattering feature set and the image target feature set to obtain the position feature set P of the SAR target and the optical target. SAR and P optics , the sizes are N SAR and N optics ; Step S44: Calculate the attention value a between the first target detection result and the second target detection result according to the attention-based fusion algorithm. ij : in, And there is γ and σ represent the aggregation functions of image target features and position features, respectively, and are calculated as follows: Among them, W SAR and W optics is a learnable weight parameter, σ calculates the Euclidean distance of the input value, W postion is a learnable weight vector, Step S45: The obtained attention value a ij Normalized using the Softmax algorithm Step S46: Construct a target matching bipartite graph G(X, E, Y) based on the normalized attention value, where X is the target detection result set of the SAR image, and any detection result x i Contains the target category, bounding box and angle information; Y is the optical image target detection set, where any detection result y j Contains the target category, bounding box and angle information; according to any result pair (x i ,y j ), if the corresponding The result is considered to be (x i ,y j ) is a possible matching result, and adds a (x i ,y j ) to the target matching bipartite graph, the set of these edges constitutes E; Step S47: Using the maximum matching algorithm on the constructed target matching bipartite graph G(X, E, Y) to obtain a matching result of target detection; Step S48: Based on the matching result, perform decision fusion on the matched first target detection result and the second target detection result, and output a final target detection and recognition result.

2. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 1, characterized in that: In step S3, the first target detection model outputs a first target detection result, specifically including: Step S311: Use the Canny edge detection operator to extract edge information from the input SAR image, extract the strong scattering points of the aircraft, use the positional relationship of the scattering points to build a model, obtain the structural relationship characteristics of the aircraft, and form an edge detection map; Step S312: using a pre-trained real aircraft target edge map as a convolution operator, performing convolution filtering on the edge detection map to form an edge saliency map; Step S313: Use the non-maximum suppression algorithm to obtain the local maximum point of the edge saliency map, which is considered to be the center point of the suspected target (x center ,y center ); Step S314: extracting the corresponding SAR suspected target slice according to the center point of the suspected target; Step S315: extracting geometric features, moment invariant features, and electromagnetic scattering features of the SAR suspected target slice, and classifying the suspected target using a single-class support vector machine to obtain a first aircraft target. Step S316: Physically model the first aircraft target based on the scattering parameters and the attitude angle of the first aircraft target, perform a rotation transformation on the first aircraft target based on the wing deployment direction information of the first aircraft target, obtain a SAR slice of the first aircraft target, and output it as the first target detection result.

3. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In step S316, for any key point of the first aircraft target According to the attitude angle θ1 of the first aircraft target, equations (1) and (2), the coordinates of the key points of the first aircraft target after rotation are calculated: Obtain the first target detection result: Among them, x center represents the x-axis coordinate of the center point of the first aircraft target, y center Indicates the y-axis coordinate of the center point of the first aircraft target, key i It is the key point number of the first aircraft target.

4. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In step S314, a sliding window with a step length of 10 pixels is performed around the non-maximum point of the suspected target in the edge saliency map to obtain the SAR suspected target slice.

5. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 2, characterized in that: In step S3, the second target detection model outputs the second target detection result, specifically including: Step S321: ResNet-50 is used as the backbone network to perform feature extraction on the input optical image, and the features of the optical image downsampled by 4, 8, 16, and 32 times are obtained and recorded as P1, P2, P3, and P4 respectively; Step S322: Divide features P1, P2, P3, and P4 into 8×8, 4×4, 2×2, and 1×1 partitions respectively, and perform self-attention within the partitions. After passing through the Transformer structure, feature maps T1, T2, T3, and T4 of the same size are output respectively; Step S323: performing feature fusion on the feature map obtained in step S322 through a bidirectional fusion feature pyramid network. By using multiple upsampling, downsampling, and cross-layer connections, a skip residual connection method is used at the initial and final positions of the features of each layer to fuse the shallow features with the deep features. Step S324: Perform aircraft target detection in the feature map obtained by the feature fusion in step S323 through the position sub-network to obtain the center point of the second aircraft target, regress the width w and height h of the bounding box from the center point of the second aircraft target, and obtain the attitude angle θ2 of the second aircraft target; Step S325: Rotate the optical image of the second aircraft target according to the wing deployment direction information of the aircraft target so that the angle of rotation is uniform. The calculation formula for the rotation change is as follows: in, represents the transformed coordinates of the center point of the second aircraft target, represents the coordinates of the center point of the second aircraft target before transformation, represents the relative position offset of the coordinate axis origin before and after the center point of the second aircraft target is transformed; θ2 represents the attitude angle of the second aircraft target, ranging from (0, 360°); Step S326: Pass each channel of the rotated optical image of the second aircraft target through a Sigmoid function. The heat map value of each point in the obtained result is the confidence level of the second aircraft target centered at that point. The target category is obtained through the classification subnetwork, and an optical image slice of the second aircraft target is obtained and output as the second target detection result.

6. The method for detecting and identifying aircraft targets based on optical and SAR remote sensing data according to claim 1, characterized in that: In step S48, Bayesian decision fusion is used to perform decision fusion on the matched first target detection result and the second target detection result, specifically including: Note C plane Class aircraft targets The first target detection result and the second target detection result involved in the fusion are Z=[z SAR , z optics ], where z SAR Indicates the first target detection result, z optics Represents the second target detection result, z k The posterior probability of belonging to class k is: Among them is the posterior probability of the two target detection results of different types, c = 1, 2, ..., C plane , the decisions of the first target detection model and the second target detection model are independent of each other, and the joint probability distribution obtained based on Bayesian theory is: P(T c |Y)=P(T c |y SAR )P(T c |y optics ) The target category is determined according to the maximum posterior probability criterion to obtain the final aircraft target category: Identity(T)=argma c P(T c |Y)。 7. An aircraft target recognition device based on optical and SAR remote sensing data, for implementing the aircraft target detection and recognition method based on optical and SAR remote sensing data as described in any one of claims 1 to 6, wherein the entire device adopts a VPX architecture design and comprises: The main control module adopts a CPU+FPGA architecture design and is used for device health management, system control, image fusion processing, and target detection post-processing; High-speed interface module, designed with FPGA core architecture, is used for external communication, data access, data preprocessing, image slicing, and data distribution of the device; SAR imaging module, designed with FPGA+DSP architecture, is used for imaging processing of SAR raw data; The SAR target detection module uses an AI chip + FPGA architecture design and runs the first target detection model to perform target detection and recognition on SAR images. The optical target detection module adopts the AI ​​chip + FPGA architecture design and runs the second target detection model to detect and identify targets in optical images; The storage module adopts the core architecture design of SSD, which is used for fast storage and reading of large amounts of data; The power module adopts the core architecture design of DCDC+LDO to convert the external power supply into the voltage required by each module; A backplane is used to connect the main control module, the high-speed interface module, the SAR target detection module, the SAR target detection module, the storage module and the power module.

Images