A method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementarity

Abstract

A method for target detection and recognition of multi-source remote sensing images based on multi-branch information complementarity, the method comprising: signal-level fusion detection and recognition based on cross-domain information guidance. Signal-level fusion detection and recognition achieves full fusion of complementary information of multi-source data through a cross-domain information-guided fusion module, and then outputs target position and category information through a target detection and recognition module. Multi-branch fusion recognition result association based on intersection-over-union ratio. A single-source target detection and recognition module is used to process a single-source image, output a single-source detection and recognition result, and combine the signal-level fusion recognition result to calculate the intersection-over-union ratio of multi-branch target prediction frames and associate them. Decision-level fusion of multi-branch recognition results based on step confidence threshold screening. For the multi-branch prediction results after association, the targets are screened and the final target type is determined based on the number of prediction frame associations, target confidence and category. This method can achieve high-probability and low false alarm rate detection and recognition of targets in complex scenes and environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a target detection and recognition method for multi-source remote sensing images, and specifically to a multi-source remote sensing image target detection and recognition method based on multi-branch information complementarity. Background Art

[0002] As a key technology in the field of remote sensing image interpretation, remote sensing image target detection and recognition can effectively classify and accurately locate the region of interest or target instance by extracting the differential features between the target and the background in aerospace remote sensing data. It plays a key role in military and civilian applications such as timely rescue at sea, intelligent traffic management, and real-time regional monitoring.

[0003] However, single payload imaging is easily affected by its own imaging mode and external imaging environment, and has certain limitations in application under low-quality imaging conditions and complex environmental interference conditions. Optical payload imaging has high resolution and rich image texture details, but it is easily affected by environmental factors such as clouds and fog and is difficult to image at night; infrared payloads can achieve day and night imaging, but the image resolution is low and texture detail information is missing; SAR payloads are not easily affected by environmental factors, but the noise distribution is serious and the imaging quality is poor. Therefore, in view of the problem that existing single-source target detection methods such as YOLO and Faster RCNN are difficult to meet the needs of all-day and all-weather target detection, it is urgent to use complementary information through multi-source data fusion to achieve high-precision detection and recognition of targets in complex scenes.

[0004] Existing fusion detection and recognition methods are mainly divided into signal-level fusion and decision-level fusion. Signal-level fusion adds or splices the features extracted from image pixels, which can fully fuse the complementary information between multi-source data. However, interference factors such as noise in single-source images will still affect the image quality and the accuracy of detection and recognition. Decision-level fusion detects and recognizes multi-source images separately, and fuses the results through decision criteria. Although it can reduce the impact of poor quality of single-source images, it cannot fully fuse the complementary information of multi-source data, and has little effect on improving detection and recognition in complex scenes.

[0005] In order to solve the limitation of the above-mentioned single fusion process, the present invention combines signal-level and decision-level fusion to achieve accurate detection and recognition of targets in multi-source remote sensing images through multi-branch information complementation. Summary of the invention

[0006] In view of the fact that single load information is difficult to meet the high-precision detection and identification of targets in complex scenes, the present invention proposes a multi-source remote sensing image target detection and identification method based on multi-branch information complementarity. This method fully exploits the complementary information between multi-source data through signal-level fusion guided by cross-domain information, reduces the interference of false alarms on target detection and identification in complex scenes, and improves the recognition ability of easily confused targets; then the signal-level fusion and single-source fusion detection and identification results are fused at the decision level to reduce the impact of data with poor image quality on signal-level fusion identification, and finally outputs the location, category, and confidence results of the target.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A multi-source remote sensing image target detection and recognition method based on multi-branch information complementarity, the method is:

[0009] Step 1: Signal-level fusion detection and recognition based on cross-domain information guidance; wherein, the signal-level fusion detection and recognition generates the fusion weights of multi-source images adaptively through the cross-domain information-guided fusion module, enhances the effective target features in the complementary information and suppresses the invalid environmental interference, avoids the problem of insufficient data fusion caused by direct addition or splicing operations, realizes the full fusion of complementary information of multi-source data, generates a signal-level fusion image, and then outputs the target position and category information through the target detection and recognition module;

[0010] Step 2: Correlation of multi-branch fusion recognition results based on intersection-over-union ratio; Use the single-source target detection and recognition module to process the single-source image, output the single-source detection and recognition result, combine it with the signal-level fusion detection and recognition result in step 1, and calculate the intersection-over-union ratio of the multi-branch target prediction box to achieve correlation matching of the single-source and signal-level fusion detection and recognition results;

[0011] Step 3: Decision-level fusion of multi-branch recognition results based on step confidence threshold screening; for the multi-branch prediction results after association in step 2, the step confidence threshold is calculated based on the number of prediction box associations, and the target confidence and one-hot category encoding are combined to screen the target and determine the final target type, so as to reduce the impact of false alarms and missed alarms in a single data source on the results and improve the recognition accuracy.

[0012] Furthermore, in step 1, the cross-domain information guided fusion module and the target detection and recognition module are end-to-end cascade networks, the parameters of both are optimized simultaneously during the training process, and the data used in the training and reasoning processes are multi-source image data.

[0013] Furthermore, the specific steps of step 1 are:

[0014] Step 1-1: The signal-level fusion of multi-source images is realized by guiding the fusion module through cross-domain information. If there are n data sources in total, the image of the i-th data source is recorded as I i , its length, width, and number of channels are recorded as (H, W, 1), and the image after signal-level fusion is recorded as I signal ∈(H,W,n);

[0015] Step 1-2: Detect and identify the signal-level fused data through the target detection and recognition module. The target detection and recognition module inputs the cross-domain information to guide the fusion module to obtain the fused data I signal , output signal level fusion detection and recognition results, represents the horizontal coordinate of the center of the jth target identified by signal-level fusion detection; represents the vertical coordinate of the center of the jth target identified by signal-level fusion detection; represents the center width of the jth target identified by signal-level fusion detection; represents the center length of the jth target identified by signal-level fusion detection; represents the jth target category identified by signal-level fusion detection; Represents the confidence of the jth target identified by signal-level fusion detection;

[0016] Furthermore, the specific steps of step 1-1 are:

[0017] Step 1-1-1: Use cross-domain information to guide the mining of spatial attention of different source data to adaptively generate the fusion weights of multi-source data. The method of generating weights is to first splice the multi-source image data along the channel dimension to obtain I cat ∈(H,W,n), each branch realizes cross-domain information interaction through 3×3 convolution, and then generates the i-th fusion weight ω through the Sigmoid activation function i , written as:

[0018] ω i =σ(conv 3×3 (I cat ))

[0019] In the formula, conv 3×3 (·) is a 3×3 convolution, σ(·) is a Sigmoid activation function;

[0020] Step 1-1-2: Based on the weights adaptively generated in step 1-1-1, adjust the spatial importance of multi-source data, and combine channel splicing and 3×3 convolution to achieve the fusion of multi-source data; first, multiply the image points of different branches by the corresponding generated weights and then splice them along the channel dimension, and then fuse the weighted data through 3×3 convolution. The fused data Ifusion Denoted as:

[0021] I fusion =conv 3×3 (cat(ω1·I1,ω2·I2,...ω n I n ,))

[0022] Where cat(·) is the channel concatenation operation;

[0023] Step 1-1-3: Adjust the importance of the channel domain of multi-source fusion data based on the channel attention mechanism. First, summarize the spatial global information in each channel through spatial global pooling to obtain an n-dimensional spatial pooling vector g that matches the number of channels. ave , where the i-th channel weight The calculation method is expressed as:

[0024]

[0025] In the formula, For I fusion The value of the i-th channel at position (j1,k1);

[0026] The spatial pooling vector is sequentially passed through the fully connected layer and the Sigmoid activation function to obtain the channel weight coefficient vector c, which is then combined with I fusion The multiplication of each channel is used to adjust the importance of the channel, and finally the adjusted fusion data I is obtained. signal :

[0027]

[0028] Where FCN(·) is a fully connected layer.

[0029] Furthermore, in step 1-2, the target detection and recognition module is any one-stage or two-stage target detection and recognition network, such as YOLOv5, FasterRCNN, CenterNet and other networks, and only the confidence conf is retained. signal Greater than the basic confidence threshold conf base The target, the basic confidence threshold conf base The default is 0.25.

[0030] Furthermore, the specific steps of step 2 are:

[0031] Step 2-1: Detect and identify the single-source data separately through the single-source target detection and recognition module; for the i-th data source image I input into the single-source target detection and recognition module i , output detection and recognition results Represents the central horizontal coordinate of the kth target detected and identified in the i-th data source; Represents the central ordinate of the kth target detected and identified in the i-th data source; Represents the center width of the kth target detected and identified in the i-th data source; Represents the center length of the kth target detected and identified in the i-th data source; Represents the category of the kth target detected and identified in the i-th data source; Represents the confidence of the kth target detected and identified in the i-th data source;

[0032] Step 2-2: Correlate the signal level fusion and single source detection results obtained in step 1-2 and step 2-1 by intersection and union comparison.

[0033] Furthermore, in step 2-1, the single-source target detection and recognition module is any one-stage or two-stage target detection and recognition network, such as YOLOv5, Faster RCNN, CenterNet and other networks. The single-source target detection and recognition module needs to train weights separately for different data sources. The data used in the training and reasoning process are the corresponding single-source image data, and only the confidence conf is retained. i Greater than the basic confidence threshold conf base The target, the basic confidence threshold conf base The default is 0.25.

[0034] Further, in step 2-2:

[0035] Step 2-2-1: Initialize the association sequence for the jth target detection and recognition result in signal level fusion Calculate the intersection over union (IoU) of the detection and recognition results of the first data source. If the intersection over union (IoU) of the lth target is greater than the intersection over union (IoU) threshold, base , then add the associated sequence Then, the remaining single-source detection and recognition results from the second to the nth are associated in turn; the intersection over union threshold IoU base The default value is 0.5.

[0036] Step 2-2-2: For targets that are not associated in the single-source detection and recognition results, sequence initialization and sequence association are performed in sequence starting from the first data source, so that all targets are eventually associated.

[0037] Furthermore, the specific steps of step 3 are:

[0038] Step 3-1: Filter the associated sequence by using the step confidence threshold; since the target may not be detected in all sources, and in order to reduce the impact of false alarms from a single data source on the results, the associated sequence is filtered by calculating the step confidence threshold. There is a linear relationship between the number of data sources where the detection and recognition results appear and the screening threshold. For details, see the following screening threshold calculation formula, so that the detection and recognition results that appear only in a few data sources are filtered by a higher threshold, and the detection and recognition results that appear in more data sources are filtered by a lower threshold. For the associated sequence with detection and recognition results in s data sources (including signal-level fusion data and other single-source data), the step confidence threshold conf s The calculation method is:

[0039]

[0040] Where n is the number of single-source data sources, and the basic confidence threshold conf is base The default is 0.25, the high confidence threshold conf base The default value is 0.75. If all detection and recognition results (confidence) in a sequence are greater than the step confidence threshold, the sequence is retained, otherwise it is deleted.

[0041] Step 3-2: Perform decision-level fusion on the filtered sequence to determine the final target type; if there are m types of targets in total, first convert the category information in different data sources into one-hot encoding, that is, m-dimensional vector; if the detection result is the i1th type of target, then the i1th element value in the corresponding m-dimensional vector is 1, and the others are 0; weight the category onehot encoding with the corresponding confidence level, and select the corresponding index at the maximum value as the final target category cls final :

[0042]

[0043] In the formula, conf j2 and cls j2 The target confidence and category one-hot encoding predicted by the j2th data source are respectively.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] (1) The present invention can effectively solve the problem of low detection and recognition accuracy due to the constraints of the respective imaging methods and the external imaging environment on the processing of single payload information. By fully integrating the complementary information of multi-source data, it can achieve high-probability and low-false-alarm-rate detection and recognition of targets in complex environments.

[0046] (2) The present invention combines signal-level fusion and decision-level fusion to avoid the limitations of a single fusion method. Signal-level fusion can fully tap into the complementary information of multi-source data, reduce the interference of false alarms on target detection and recognition in complex scenes, and improve the recognition ability of easily confused targets; decision-level fusion further improves performance based on multi-branch prediction, which can reduce the impact of data with poor image quality on signal-level fusion recognition.

[0047] (3) The present invention adaptively generates fusion weights of multi-source images through signal-level fusion guided by cross-domain information, enhances effective target features in complementary information and suppresses invalid environmental interference, thereby avoiding the problem of insufficient data fusion caused by direct addition or splicing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of target detection and recognition in multi-source remote sensing images based on multi-branch information complementarity.

[0049] Figure 2 Schematic diagram of the cross-domain information guided fusion module.

[0050] Figure 3 This is the example 1 of multi-source fusion recognition result.

[0051] Figure 4 for Figure 3 Corresponding true value annotation diagram.

[0052] Figure 5 This is the example 2 of multi-source fusion recognition result.

[0053] Figure 6 for Figure 5 Corresponding true value annotation diagram.

[0054] Figure 7 This is the example 3 of multi-source fusion recognition result.

[0055] Figure 8 for Figure 7 Corresponding true value annotation diagram. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0057] The present invention is based on signal-level fusion detection and recognition guided by cross-domain information. Among them, the signal-level fusion detection and recognition realizes the full fusion of complementary information of multi-source data through the cross-domain information-guided fusion module, and then realizes the output of target position and category information through the target detection and recognition module. Multi-branch fusion recognition result association based on intersection-over-union ratio. The single-source image is processed by the single-source target detection and recognition module, and the single-source detection and recognition result is output. Combined with the signal-level fusion recognition result, the intersection-over-union ratio of the multi-branch target prediction box is calculated and associated. Decision-level fusion of multi-branch recognition results based on step confidence threshold screening. For the multi-branch prediction results after association, the targets are screened and the final target type is determined based on the number of prediction box associations, target confidence and category. This method can effectively solve the problem of low detection and recognition accuracy due to the constraints of the respective imaging methods and the external imaging environment in single-load information processing. By fully fusing the complementary information of multi-source data, high-probability and low-false-alarm rate detection and recognition of targets in complex scenes and environments can be achieved. The overall flow chart is as follows Figure 1 shown.

[0058] Embodiment 1:

[0059] A multi-source remote sensing image target detection and recognition method based on multi-branch information complementarity, the method is:

[0060] Step 1: Signal-level fusion detection and recognition based on cross-domain information guidance; wherein, the signal-level fusion detection and recognition adaptively generates the fusion weights of multi-source images through the cross-domain information guided fusion module, enhances the effective target features in the complementary information and suppresses invalid environmental interference, avoids the problem of insufficient data fusion due to direct addition or splicing operations, realizes the full fusion of complementary information of multi-source data, generates a signal-level fusion image, and then outputs the target position and category information through the target detection and recognition module; in step 1, the cross-domain information guided fusion module and the target detection and recognition module are end-to-end cascade networks, and the parameters of both are optimized simultaneously during the training process, and the data used in the training and reasoning processes are all multi-source image data. The schematic diagram of the cross-domain guided fusion module is shown in the figure below. Figure 2 shown.

[0061] The specific steps of step 1 are:

[0062] Step 1-1: The signal-level fusion of multi-source images is realized by guiding the fusion module through cross-domain information. If there are n data sources in total, the image of the i-th data source is recorded as I i , its length, width, and number of channels are recorded as (H, W, 1), and the image after signal-level fusion is recorded as I signal ∈(H,W,n);

[0063] The specific steps of step 1-1 are:

[0064] Step 1-1-1: Use cross-domain information to guide the mining of spatial attention of different source data to adaptively generate the fusion weights of multi-source data. The method of generating weights is to first splice the multi-source image data along the channel dimension to obtain I cat ∈(H,W,n), each branch realizes cross-domain information interaction through 3×3 convolution, and then generates the i-th fusion weight ω through the Sigmoid activation function i , written as:

[0065] ω i =σ(conv 3×3 (I cat ))

[0066] In the formula, conv 3×3 (·) is a 3×3 convolution, σ(·) is a Sigmoid activation function;

[0067] Step 1-1-2: Based on the weights adaptively generated in step 1-1-1, adjust the spatial importance of multi-source data, and combine channel splicing and 3×3 convolution to achieve the fusion of multi-source data; first, multiply the image points of different branches by the corresponding generated weights and then splice them along the channel dimension, and then fuse the weighted data through 3×3 convolution. The fused data I fusion Denoted as:

[0068] I fusion =conv 3×3 (cat(ω1·I1,ω2·I2,...ω n I n ,))

[0069] Where cat(·) is the channel concatenation operation;

[0070] Step 1-1-3: Adjust the importance of the channel domain of multi-source fusion data based on the channel attention mechanism. First, summarize the spatial global information in each channel through spatial global pooling to obtain an n-dimensional spatial pooling vector g that matches the number of channels. ave , where the i-th channel weight The calculation method is expressed as:

[0071]

[0072] In the formula, For I fusion The value of the i-th channel at position (j1,k1);

[0073] The spatial pooling vector is sequentially passed through the fully connected layer and the Sigmoid activation function to obtain the channel weight coefficient vector c, which is then combined with I fusionThe multiplication of each channel is used to adjust the importance of the channel, and finally the adjusted fusion data I is obtained. signal :

[0074] I signal =c·I fusion =σ(FCN(g ave ))·I fusion

[0075] Where FCN(·) is a fully connected layer.

[0076] Step 1-2: Detect and identify the signal-level fused data through the target detection and recognition module. The target detection and recognition module inputs the cross-domain information to guide the fusion module to obtain the fused data I signal , output signal level fusion detection and recognition results, represents the horizontal coordinate of the center of the jth target identified by signal-level fusion detection; represents the vertical coordinate of the center of the jth target identified by signal-level fusion detection; represents the center width of the jth target identified by signal-level fusion detection; represents the center length of the jth target identified by signal-level fusion detection; represents the jth target category identified by signal-level fusion detection; Represents the confidence of the jth target identified by signal-level fusion detection;

[0077] In step 1-2, the target detection and recognition module is any one-stage or two-stage target detection and recognition network, such as YOLOv5, FasterRCNN, CenterNet, etc., and only the confidence conf is retained. signal Greater than the basic confidence threshold conf base The target, the basic confidence threshold conf base The default is 0.25.

[0078] Step 2: Correlation of multi-branch fusion recognition results based on intersection-over-union ratio; Use the single-source target detection and recognition module to process the single-source image, output the single-source detection and recognition result, combine it with the signal-level fusion detection and recognition result in step 1, and calculate the intersection-over-union ratio of the multi-branch target prediction box to achieve correlation matching of the single-source and signal-level fusion detection and recognition results;

[0079] The specific steps of step 2 are:

[0080] Step 2-1: Detect and identify the single-source data separately through the single-source target detection and recognition module; for the i-th data source image I input into the single-source target detection and recognition module i , output detection and recognition results Represents the central horizontal coordinate of the kth target detected and identified in the i-th data source; Represents the central ordinate of the kth target detected and identified in the i-th data source; Represents the center width of the kth target detected and identified in the i-th data source; Represents the center length of the kth target detected and identified in the i-th data source; Represents the category of the kth target detected and identified in the i-th data source; Represents the confidence of the kth target detected and identified in the i-th data source;

[0081] In step 2-1, the single-source target detection and recognition module is any one-stage or two-stage target detection and recognition network, such as YOLOv5, FasterRCNN, CenterNet and other networks. The single-source target detection and recognition module needs to train weights separately for different data sources. The data used in the training and reasoning process are the corresponding single-source image data, and only the confidence conf is retained. i Greater than the basic confidence threshold conf base The target, the basic confidence threshold conf base The default is 0.25.

[0082] Step 2-2: Correlate the signal level fusion and single source detection and recognition results obtained in step 1-2 and step 2-1 by intersection and union comparison. In step 2-2:

[0083] Step 2-2-1: Initialize the association sequence for the jth target detection and recognition result in signal level fusion Calculate the intersection over union (IoU) of the detection and recognition results of the first data source. If the intersection over union (IoU) of the lth target is greater than the intersection over union (IoU) threshold, base , then add the associated sequence Then, the remaining single-source detection and recognition results from the second to the nth are associated in turn; the intersection over union threshold IoU base The default value is 0.5.

[0084] Step 2-2-2: For targets that are not associated in the single-source detection and recognition results, sequence initialization and sequence association are performed in sequence starting from the first data source, so that all targets are eventually associated.

[0085] Step 3: Decision-level fusion of multi-branch recognition results based on step confidence threshold screening; for the multi-branch prediction results after association in step 2, the step confidence threshold is calculated based on the number of prediction box associations, and the target confidence and one-hot category encoding are combined to screen the target and determine the final target type, so as to reduce the impact of false alarms and missed alarms in a single data source on the results and improve the recognition accuracy. The specific steps of step 3 are:

[0086] Step 3-1: Filter the associated sequence by using the step confidence threshold; since the target may not be detected in all sources, and in order to reduce the impact of false alarms from a single data source on the results, the associated sequence is filtered by calculating the step confidence threshold. There is a linear relationship between the number of data sources where the detection and recognition results appear and the screening threshold. For details, see the following screening threshold calculation formula, so that the detection and recognition results that appear only in a few data sources are filtered by a higher threshold, and the detection and recognition results that appear in more data sources are filtered by a lower threshold. For the associated sequence with detection and recognition results in s data sources (including signal-level fusion data and other single-source data), the step confidence threshold conf s The calculation method is:

[0087]

[0088] Where n is the number of single-source data sources, and the basic confidence threshold conf is base The default is 0.25, the high confidence threshold conf base The default value is 0.75. If all detection and recognition results (confidence) in a sequence are greater than the step confidence threshold, the sequence is retained, otherwise it is deleted.

[0089] Step 3-2: Perform decision-level fusion on the filtered sequence to determine the final target type; if there are m types of targets in total, first convert the category information in different data sources into one-hot encoding, that is, m-dimensional vector; if the detection result is the i1th type of target, then the i1th element value in the corresponding m-dimensional vector is 1, and the others are 0; weight the category onehot encoding with the corresponding confidence level, and select the corresponding index at the maximum value as the final target category cls final :

[0090]

[0091] In the formula, conf j2 and cls j2 The target confidence and category one-hot encoding predicted by the j2th data source are respectively.

[0092] like Figure 3-Figure 8By fusing the recognition results and the corresponding true values ​​through this method, it can be seen that all targets can be detected and identified without false alarms.

Claims

1. A multi-source remote sensing image target detection and recognition method based on multi-branch information complementation, characterized by: The method is: Step 1: Signal-level fusion detection and recognition based on cross-domain information guidance; wherein, the signal-level fusion detection and recognition realizes the full fusion of complementary information of multi-source data through the cross-domain information guidance fusion module, generates a signal-level fusion image, and then realizes the output of target position and category information through the target detection and recognition module; The specific steps of step 1 are: Step 1-1: The signal-level fusion of multi-source images is realized by guiding the fusion module through cross-domain information. If there are n data sources in total, the image of the i-th data source is recorded as I i , its length, width, and number of channels are recorded as (H, W, 1), and the image after signal-level fusion is recorded as I signal ∈(H,W,n); Step 1-2: Detect and identify the signal-level fused data through the target detection and recognition module. The target detection and recognition module inputs the cross-domain information to guide the fusion module to obtain the fused data I signal , output signal level fusion detection and recognition results, represents the horizontal coordinate of the center of the jth target identified by signal-level fusion detection; represents the vertical coordinate of the center of the jth target identified by signal-level fusion detection; represents the center width of the jth target identified by signal-level fusion detection; represents the center length of the jth target identified by signal-level fusion detection; represents the jth target category identified by signal-level fusion detection; Represents the confidence of the jth target identified by signal-level fusion detection; The specific steps of step 1-1 are: Step 1-1-1: Use cross-domain information to guide the mining of spatial attention of different source data to adaptively generate the fusion weights of multi-source data. The method of generating weights is to first splice the multi-source image data along the channel dimension to obtain I cat ∈(H,W,n), each branch realizes cross-domain information interaction through 3×3 convolution, and then generates the i-th fusion weight ω through the Sigmoid activation function i , written as: oh i =σ(conv 3×3 (I cat )) In the formula, conv 3×3 (·) is a 3×3 convolution, σ(·) is a Sigmoid activation function; Step 1-1-2: Based on the weights adaptively generated in step 1-1-1, adjust the spatial importance of multi-source data, and combine channel splicing and 3×3 convolution to achieve the fusion of multi-source data; first, multiply the image points of different branches by the corresponding generated weights and then splice them along the channel dimension, and then fuse the weighted data through 3×3 convolution. The fused data I fusion Denoted as: I fusion =conv 3×3 (cat(ω1·I1,ω2·I2,...ω n ·I n ,)) Where cat(·) is the channel concatenation operation; Step 1-1-3: Adjust the importance of the channel domain of multi-source fusion data based on the channel attention mechanism. First, summarize the spatial global information in each channel through spatial global pooling to obtain an n-dimensional spatial pooling vector g that matches the number of channels. ave , where the i-th channel weight The calculation method is expressed as: In the formula, For I fusion The value of the i-th channel at position (j1,k1); The spatial pooling vector is sequentially passed through the fully connected layer and the Sigmoid activation function to obtain the channel weight coefficient vector c, which is then combined with I fusion The multiplication of each channel is used to adjust the importance of the channel, and finally the adjusted fusion data I is obtained. signal : I signal =c·I fusion =σ(FCN(g ave ))·I fusion In the formula, FCN(·) is a fully connected layer; Step 2: Correlation of multi-branch fusion recognition results based on intersection-over-union ratio; Use the single-source target detection and recognition module to process the single-source image, output the single-source detection and recognition result, combine it with the signal-level fusion detection and recognition result in step 1, and calculate the intersection-over-union ratio of the multi-branch target prediction box to achieve correlation matching of the single-source and signal-level fusion detection and recognition results; Step 3: Decision-level fusion of multi-branch recognition results based on step confidence threshold screening; For the multi-branch prediction results after association in step 2, the step confidence threshold is calculated based on the number of prediction box associations, and the target confidence and one-hot category encoding are combined to screen the target and determine the final target type.

2. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 1, characterized in that: In step 1, the cross-domain information-guided fusion module and the target detection and recognition module are end-to-end cascade networks, and the parameters of both are optimized simultaneously during the training process. The data used in the training and reasoning processes are multi-source image data.

3. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 1, characterized in that: In step 1-2, the target detection and recognition module is any one-stage or two-stage target detection and recognition network, and only the confidence conf is retained. signal Greater than the basic confidence threshold conf base goal.

4. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 1, characterized in that: The specific steps of step 2 are: Step 2-1: Detect and identify the single-source data separately through the single-source target detection and recognition module; for the i-th data source image I input into the single-source target detection and recognition module i , output detection and recognition results Represents the central horizontal coordinate of the kth target detected and identified in the i-th data source; Represents the central ordinate of the kth target detected and identified in the i-th data source; Represents the center width of the kth target detected and identified in the i-th data source; Represents the center length of the kth target detected and identified in the i-th data source; Represents the category of the kth target detected and identified in the i-th data source; conf i k Represents the confidence of the kth target detected and identified in the i-th data source; Step 2-2: Correlate the signal level fusion and single source detection results obtained in step 1-2 and step 2-1 by intersection and union comparison.

5. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 4, characterized in that: In step 2-1, the single-source target detection and recognition module is any one-stage or two-stage target detection and recognition network. The single-source target detection and recognition module needs to train weights separately for different data sources. The data used in the training and reasoning process are the corresponding single-source image data, and only the confidence conf is retained. i Greater than the basic confidence threshold conf base goal.

6. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 4, characterized in that: In step 2-2: Step 2-2-1: Initialize the association sequence for the jth target detection and recognition result in signal level fusion Calculate the intersection over union (IoU) of the detection and recognition results of the first data source. If the intersection over union (IoU) of the lth target is greater than the intersection over union (IoU) threshold, base , then add the associated sequence Then, the remaining single-source detection and recognition results from the second to the nth are associated in sequence; Step 2-2-2: For targets that are not associated in the single-source detection and recognition results, sequence initialization and sequence association are performed in sequence starting from the first data source, so that all targets are eventually associated.

7. The method for target detection and recognition in multi-source remote sensing images based on multi-branch information complementation according to claim 1, characterized in that: The specific steps of step 3 are: Step 3-1: Filter the associated sequence by using the step confidence threshold; since the target may not be detected in all sources, and in order to reduce the impact of false alarms from a single data source on the results, the associated sequence is filtered by calculating the step confidence threshold. There is a linear relationship between the number of data sources where the detection and recognition results appear and the screening threshold. For details, see the following screening threshold calculation formula, so that the detection and recognition results that appear only in fewer data sources are filtered by a higher threshold, and the detection and recognition results that appear in more data sources are filtered by a lower threshold. For s data sources, the s data sources include signal-level fusion data and other single-source data, and there is an associated sequence of detection and recognition results, the step confidence threshold conf s The calculation method is: Where n is the number of single-source data sources; if all detection and recognition results in the sequence, i.e., confidence, are greater than the step confidence threshold, the sequence is retained, otherwise the sequence is deleted; Step 3-2: Perform decision-level fusion on the filtered sequence to determine the final target type; if there are m types of targets in total, first convert the category information in different data sources into one-hot encoding, that is, m-dimensional vector; if the detection result is the i1th type of target, then the i1th element value in the corresponding m-dimensional vector is 1, and the others are 0; weight the category onehot encoding with the corresponding confidence level, and select the corresponding index at the maximum value as the final target category cls final : In the formula, and The target confidence and category one-hot encoding predicted by the j2th data source are respectively.

Images