Target information fusion method based on spaceflight reconnaissance image data and electronic data

Abstract

The invention provides a target information fusion method based on spaceflight reconnaissance image data and electronic data. Accurate positioning and association fusion of a target are realized. The method comprises the following steps: performing image fusion on a panchromatic image and a multispectral image of spaceflight reconnaissance, performing target detection and positioning by using a deep learning method, and performing association fusion on attribute features and motion features of an image target and an electronic target by using an association fusion algorithm. And meanwhile, the current target and the historical target are subjected to association analysis to realize historical target fusion. Compared with the prior art, the method has the advantages that image fusion, image and electronic target fusion and historical target fusion algorithms are applied to target detection and fusion of spaceflight reconnaissance data, target information fusion is carried out from multiple levels, and the positioning precision, attribute richness and reliability of the target are remarkably improved; and the method has a certain practical value in a battlefield target intelligence analysis and processing process.

Description

Technical Field

[0001] The present invention relates to the fields of image fusion, target fusion, deep learning and computer vision, and in particular to a target information fusion method based on aerospace reconnaissance image data and electronic data. Background Art

[0002] With the continuous development of technology, aerospace reconnaissance data plays an important role in many application fields. The multi-level target information fusion analysis technology of aerospace reconnaissance data uses aerospace reconnaissance images for target detection and fuses them with aerospace reconnaissance electronic target data, thereby improving the positioning accuracy, attribute richness and reliability of the target. This technology targets two typical targets on the battlefield: command posts and radar stations. Combining image data and electronic data can obtain intelligence information of important targets, which has high research and practical value.

[0003] Space reconnaissance data has significant advantages such as wide coverage, multiple data source types, and large amount of data information. Target identification such as radars and command posts based on space reconnaissance data has huge military application value.

[0004] In the existing technology, the target recognition method based on aerospace reconnaissance image data has high positioning accuracy but has great limitations and is easily affected by factors such as shooting time, angle, fog, occlusion, camouflage, etc.; the target recognition method based on aerospace reconnaissance electronic data obtains rich information and is less affected by environmental factors, but has large positioning errors.

[0005] Therefore, there is an urgent need for a target information fusion method based on aerospace reconnaissance image data and electronic data to achieve accurate positioning and information acquisition of important targets in aerospace reconnaissance data. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention provides a target information fusion method based on aerospace reconnaissance image data and electronic data. The method first fuses the panchromatic image and the multispectral image through an image fusion algorithm to increase the information content of the image, then uses a deep learning algorithm to realize image target detection and recognition and output the target geometric features, and then uses a target fusion algorithm to realize the fusion of image target information and electronic target information, thereby improving the positioning accuracy, attribute richness and reliability of the target.

[0007] The technical solution adopted by the present invention is: a target information fusion method based on aerospace reconnaissance image data and electronic data, comprising the following steps:

[0008] Step 1: Image fusion stage: pixel-level fusion of panchromatic and multispectral images of space reconnaissance;

[0009] Step 2: Target detection stage: Manually label typical battlefield targets such as command posts and radar stations in the fused image, select the target detection model, adjust it and train it until convergence, complete the target detection and positioning of the aerospace reconnaissance image, and generate target geometric features;

[0010] Step 3: Target association and fusion stage: Analyze the space reconnaissance electronic data, and use the attribute feature association technology, motion feature association technology and association judgment technology to compare, supplement, remove false and duplicate data from different satellites to achieve target-level fusion of optical images and electronic information multi-source data.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] (1) The present invention fuses the panchromatic image and the multispectral image through an image fusion algorithm, which can enrich the texture details and spectral information of the target in the image;

[0013] (2) The present invention utilizes a target fusion algorithm to achieve the fusion of image target information and electronic target information, which can achieve the de-falsification, de-duplication and information completion of target information, thereby improving the positioning accuracy, attribute richness and reliability of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the target information fusion method based on aerospace reconnaissance image data and electronic data;

[0015] Figure 2 Panchromatic images, multispectral images and their fusion results for aerospace reconnaissance;

[0016] Figure 3 For annotated space reconnaissance images;

[0017] Figure 4 is the target detection result image;

[0018] Figure 5 Take a screenshot of the target detection result list;

[0019] Figure 6 Take screenshots for electronic target data;

[0020] Figure 7 Screenshot of the image target and electronic target fusion result list. DETAILED DESCRIPTION

[0021] In order to make the technical means and creative features of the present invention easy to understand, the following is a brief description of the present invention in conjunction with the attached Figure 1-7 The technical scheme of the present invention is further illustrated with embodiments.

[0022] A target information fusion method based on aerospace reconnaissance image data and electronic data, the specific steps of which are as follows:

[0023] S1: Image fusion stage: Select two remote sensing images to be fused, first check the image band, format and coordinate range, then use RPB and DEM files for orthorectification, and finally use the open source Pansharpen algorithm for pixel-level image fusion.

[0024] S1.1: Read the panchromatic and multispectral remote sensing images of GF-1, and verify the image band, format, and coordinate range.

[0025] S1.2: Use the RPB files and DEM files contained in the panchromatic and multispectral images of GF-1 to perform orthorectification using rational polynomial coefficients.

[0026] S1.3: Use Pansharpen algorithm to perform image fusion on panchromatic and full-color images.

[0027] S2: Target detection stage: Manually label the command post and radar station targets in the space reconnaissance image dataset, select the target detection model for adjustment and train it until convergence, and complete the target detection of the space reconnaissance image;

[0028] S2.1: Collect Gaofen-1 satellite data as space reconnaissance image data, and the image size is unified to 608*608 by cropping. Use labelme software to annotate the command post and radar station targets in the image, and the labels are recorded as [radar, command]. After the annotation is completed, each image will get a json file with the same name, and then run the labelme2voc script file to batch convert the annotated files into voc format. Space reconnaissance images and their annotation forms are as follows Figure 3 shown.

[0029] S2.2: Use Baidu PaddlePaddle deep learning framework for training and inference, select the deep neural network Yolov3 as the target detection model to realize target detection in space reconnaissance images, and use darknet53 as its backbone network structure. In order to match it with the format of the annotated dataset, adjust the corresponding parameters in the configuration file, change num_class to 2, bach_size to 4, iters to 60000, crop_size to [608,608], learning_rate to 0.01, loss to Yolov3Loss, and optimize_type to Momentum.

[0030] S2.3: Use the adjusted target detection model Yolov3 in S2.2 to train the data set in S2.1. After training to convergence, save the corresponding weight file. The target detection model can load the trained weights to perform target detection on the input space reconnaissance image and obtain the image after target detection (such as Figure 4 ) and target detection result list (as shown in Figure 5 shown).

[0031] S2.4: Convert the relative coordinate information of the image into geographic coordinate information, and estimate the geometric features of the target (length and width feature information of the target).

[0032] S3: Target association and fusion stage: Analyze the space reconnaissance electronic data, and through the three steps of attribute feature association technology, motion feature association technology and association judgment technology, compare, supplement, remove false and duplicate data from different satellites to achieve target-level fusion of multi-source data such as optical images and electronic information.

[0033] S3.1: Attribute feature association.

[0034] The key attribute characteristics of electronic reconnaissance data are: country and region, air and sea category, enemy and friend attributes, radiation source model, frequency type, operating frequency, pulse repetition period type, pulse repetition period, pulse width type, pulse width, scanning type, scanning period and target identification results based on the above characteristics;

[0035] The key attribute characteristics of image reconnaissance data are: state information such as the target's geographic location, imaging time, and movement direction. It can also provide information on shape, size, and its distribution characteristics, as well as target recognition results based on the above characteristics.

[0036] The association rules of key attributes are divided into two categories: one is the attributes that can directly determine the success of association; the other is the attributes that support the success of association to a certain extent. For example, through the combination of artificial intelligence and target recognition results, the success of association can be directly determined;

[0037] S3.2: Motion feature association. The present invention adopts the association judgment based on position points. According to the motion features of the target position points obtained by the sensor, the association degree components based on the distance difference, speed difference and direction difference are calculated with all the fused target data, and the association degree is obtained by weighting.

[0038] (1) Calculation of correlation components based on distance difference

[0039] For the measurement value x received at time t r，i，t , using the estimated value of the possibly associated secondary track j at time t0, predict the position x of the track at time j，t , let Δ=x r，i，t -xj，t , the Δ value is standardized and calculated, and the formula is as follows.

[0040]

[0041] P j =P j,t +F r,i,t

[0042] Where t represents the time, r represents the sensor number (corresponding to the track characteristics), i and j represent the tracks respectively, (r, i) represents a track and its track characteristics, P j represents the estimated error covariance of the secondary track, P j，t is the estimated error covariance of the secondary track j at time t, F r，i，t For x r，i，t The measurement error variance is defined as r，i，t and the estimated value x of the secondary track j at time t j，t The distance d between them is:

[0043] d (r,i),j =Δ T Δ

[0044] d (r，i)，j =(x r，i，t -x j，t ) T (P j，t -F r，i，t )(x r，i，t -x j，t )

[0045] Define the distance difference correlation value u1 = exp(-d (r，i)，j ), the smaller u1 is, the smaller the measurement value x of a track (r, i) is. r，i，t The smaller the distance between the estimated value of the secondary track j at time t, the greater the correlation between the primary track and the secondary track.

[0046] (2) Calculation of correlation components based on velocity difference

[0047] Calculate the velocity v at time t r，i，t , calculate the velocity v of the secondary track j at time t j，t Let the difference Δv = v r,i,t -v j,t ; Normalization processing, set the maximum speed v of the target max With the minimum speed value v min , define the speed difference correlation u2 = Δv / (v max -v min ), represents the speed v of a track (r,i) r,i,tand the estimated velocity v of the secondary track j at time t j,t The smaller the speed difference correlation is, the greater the correlation between the primary track and the secondary track is.

[0048] (3) Calculation of correlation components based on direction difference

[0049] For a track i, the direction h at time t is calculated by filtering or three-point solution of the motion direction. r,i,t , calculate the direction h of the secondary track j at time t j,t . Define the direction difference correlation u3 = Δh = h r,i,t -h j,t ; represents the direction of motion of a track (r,i) h r,i,t and the estimated direction h of the secondary track j at time t j,t The smaller the difference between them, the greater the correlation between the primary track and the secondary track.

[0050] Assign a weight w to each calculated correlation component k , k = 1, 2, 3, so that the correlation degree reaches the maximum. The definition of correlation degree is as follows:

[0051]

[0052] For N times that meet the relevant indicators of the measurement track, through N times u(t n ) value to calculate the average value of correlation

[0053]

[0054] The final generated track association pairing combination, C = (sensor number r, primary track i, secondary track j, average correlation degree ).

[0055] S3.3: Comprehensive association judgment. The comprehensive association value is obtained by comprehensively calculating the attribute feature association judgment and the motion feature association, and the comprehensive association judgment is made according to the comprehensive association value:

[0056] The comprehensive correlation values ​​of the candidates that meet the threshold requirements are sorted, and the original measurement value is associated with the fusion track corresponding to the maximum correlation value to achieve data association judgment of the same target by the same and different sensors;

[0057] When the comprehensive correlation with all fusion target data does not reach the threshold, the original target data is used as a new fusion target;

[0058] When the comprehensive correlation value with the associated fusion target data does not reach the threshold, the association relationship with the fusion target data is released.

[0059] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A target information fusion method based on aerospace reconnaissance image data and electronic data, characterized in that: The following steps are involved: Step 1: Image fusion stage: pixel-level fusion of panchromatic and multispectral images of space reconnaissance; Step 2: Target detection stage: Manually label typical battlefield targets such as command posts and radar stations in the fused image, select a target detection model, adjust it, and train it until convergence, complete target detection and positioning of the aerospace reconnaissance image, and generate target geometric features; Step 3: Target association and fusion stage: Analyze the space reconnaissance electronic data, and use the attribute feature association technology, motion feature association technology and association judgment technology to compare, supplement, remove false and duplicate data from different satellites to achieve target-level fusion of optical images and electronic information multi-source data.

2. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 1, characterized in that: Step 1 The image fusion stage includes the following steps: Step 1.1: Read the panchromatic and multispectral remote sensing images of GF-1 and verify the image band, format, and coordinate range; Step 1.2: Use the RPB files and DEM files contained in the panchromatic and multispectral images of GF-1 to perform orthorectification using rational polynomial coefficients; Step 1.3: Use Pansharpen algorithm to perform image fusion on the panchromatic image and the full-color image.

3. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 1, characterized in that: The target detection phase in step 2 includes the following steps: Step 2.1: Select aerospace reconnaissance image data, use labelme software to annotate battlefield targets with electronic reconnaissance features and image features in the image, and after the annotation is completed, each image will get a json file with the same name, and then run the labelme2voc script file to convert the annotated file into voc format; Step 2.2: Select the deep neural network Yolov3 as the target detection model to realize the target detection of space reconnaissance images. Its backbone network structure is darknet53, and the parameters are adjusted; Step 2.3: Use the target detection model to train the data set. After training to convergence, save the corresponding weight file. The target detection model loads the trained weights to perform target detection on the input aerospace reconnaissance image, and outputs the detected target category information and image relative coordinate information; Step 2.4: Convert the image relative coordinate information into geographic coordinate information and estimate the geometric features of the target.

4. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 1, characterized in that: Step 3 The target association fusion stage includes the following steps: Step 3.1: Attribute feature association,The association rules of key attributes are divided into two categories: one is the attribute that can directly determine the success of association; the other is the attribute that supports the success of association to a certain extent; Step 3.2: Motion feature association, taking the association judgment based on the position point and the position point; according to the motion features of the target position point obtained by the sensor, the correlation degree components based on the distance difference, speed difference and direction difference are calculated with all the fused target data, and the correlation degree is obtained by weighting; Step 3.3: comprehensive association judgment: the attribute feature association judgment situation and the motion feature association degree are comprehensively calculated to obtain a comprehensive association degree value, and a comprehensive association judgment is made based on the comprehensive association degree value.

5. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: Step 3.1 includes: the key attribute characteristics of electronic reconnaissance data are: country and region, air and sea category, enemy and friend attributes, radiation source model, frequency type, operating frequency, pulse repetition period type, pulse repetition period, pulse width type, pulse width, scanning type, scanning period and target recognition results obtained based on the above characteristics; The key attribute characteristics of image reconnaissance data are: target geographic location, imaging time and movement direction status information, providing shape, size and distribution characteristic information, and target recognition results based on the above characteristics.

6. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: In step 3.2, the correlation component based on the distance difference is calculated: Define the distance difference correlation component u 1 =exp(-d (r，i)，j ), where u1 is the distance difference correlation component, d (r,i),j is the measurement value x of a track (r, i) r，i，t and the estimated value x of the secondary track j at time t j，t The distance between The smaller u1 is, the smaller the measurement value x of a track (r, i) is. r，i，t The smaller the distance between the estimated value of the secondary track j at time t, the greater the correlation between the primary track and the secondary track.

7. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: In step 3.2, the correlation component based on the velocity difference is calculated: Define the velocity difference correlation component u2 = Δv / (v max -v min) , where u2 is the correlation component of the speed difference, Δv is the speed difference between the two tracks, and v max The maximum speed v of the target min is the target minimum speed value; u2 represents the velocity v of a track (r,i) r，i，t and the estimated velocity v of the secondary track j at time t j，t The smaller the speed difference u2 is, the greater the correlation between the primary track and the secondary track is.

8. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: In step 3.2, the correlation component based on the direction difference is calculated: Define the direction difference correlation component u3 = Δh = h r,i,t -h j,t ; In the formula, u3 is the direction difference correlation component, h r,i,t is the direction of motion of a track (r,i) at time t, h j,t represents the moving direction of the secondary track j at time t; u3 represents the direction of motion of a track (r,i) r,i,t and the estimated direction h of the secondary track j at time t j,t The smaller the difference between them, the greater the correlation between the primary track and the secondary track.

9. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: In step 3.2, the calculation of correlation is: Defining Relevance In the formula, u is the correlation degree, u k is the above correlation component, w k , k = 1, 2, 3 are the weights of each correlation component; For N times that meet the relevant indicators of the measurement track, through N times u(t n ), n = 1, 2, ..., N correlation calculation correlation average n=1, 2, ..., N; The final generated track association pairing combination is C = (sensor number r, primary track i, secondary track j, average correlation degree u).

10. A target information fusion method based on aerospace reconnaissance image data and electronic data according to claim 4, characterized in that: In step 3.3, The comprehensive correlation values ​​of the candidates that meet the threshold requirements are sorted, and the original measurement value is associated with the fusion track corresponding to the maximum correlation value to achieve data association judgment of the same target by the same and different sensors; When the comprehensive correlation with all fusion target data does not reach the threshold, the original target data is used as a new fusion target; When the comprehensive correlation value with the associated fusion target data does not reach the threshold, the association relationship with the fusion target data is released.

Images