Multi-source remote sensing image fusion and target perception method and device based on multi-level cooperation

Abstract

This invention provides a method and apparatus for multi-source remote sensing image fusion and target perception based on multi-level collaborative processing. It acquires original heterogeneous images from the same geographical location; fuses these images using a multi-source fusion method, then inputs the fused images into a trained target detection model for target detection, resulting in an output image containing target perception boxes. Constraints are used to determine whether to process the target perception boxes, thereby determining whether the labeled targets are credible and marking them in the visible light image. This invention uses different fusion algorithms to process the original heterogeneous images, enhancing target features and enriching scene information. It also learns the spatial-spectral features of the fused image, improving target perception accuracy. Finally, a voting mechanism is used to determine credible targets from the output results, further improving target perception accuracy.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology and image target detection technology, specifically relating to a method and device for multi-source remote sensing image fusion and target perception based on multi-level collaboration. Background Technology

[0002] Remote sensing images mainly refer to various films (or photographs) that record the electromagnetic wave intensity of ground objects, such as aerial photographs and satellite photographs. They contain rich features, such as spectral features, spatial features, and textural features. By appropriately processing different target images, they can be applied to various fields, such as land cover monitoring, forest cover monitoring, and land change monitoring. Image fusion is an enhancement technique that processes images obtained from different sensors in the same scene to obtain a more information-rich image, facilitating subsequent processing. Infrared images are obtained by "measuring" the heat radiated outward by objects. Because infrared images are grayscale images, they have characteristics such as low image resolution, low contrast, low signal-to-noise ratio, and poor image quality, resulting in blurred visual effects and limited information. Visible light images, on the other hand, have high spatial resolution and rich scene information, but target features are not prominent. Therefore, using either of these two types of data alone will result in unsatisfactory target perception. Thus, research on fusing these two types of data and fully utilizing their feature information for target perception has significant practical application value.

[0003] Currently, pixel-level fusion methods are commonly used to fuse infrared and visible light images. Pixel-level image fusion is the lowest level of fusion, but it can make full use of the input data. It directly processes the initial data information without feature extraction and classification, aiming to fuse the images into an image with richer information than before. Compared to infrared and visible light image fusion methods, panchromatic and multispectral image fusion methods have developed more rapidly and are therefore more numerous and comprehensive. The simplest fusion method is weighted pixel-level fusion, but this method may reduce the contrast between images, resulting in poor fusion effects. Pixel-level fusion based on traditional transform domain methods, such as HIS and PCA, replaces the intensity or first principal component of the visible light image with the infrared image. This may cause significant color distortion in the fused image, thus affecting the accuracy of subsequent target perception. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for multi-source remote sensing image fusion and target perception based on multi-level collaborative processing. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] In a first aspect, the present invention provides a method for multi-source remote sensing image fusion and target perception based on multi-level collaboration, comprising:

[0006] Step 1: Obtain raw heterogeneous images from the same geographical location;

[0007] The original heterogeneous image includes infrared images and visible light images;

[0008] Step 2: The infrared image and the visible light image are fused using a multi-element fusion method to obtain a fused image for each fusion method.

[0009] Step 3: Take the fused image from each fusion method, the infrared image, and the visible light image as input images, and input them into the trained target detection model to perform feature extraction, feature enhancement, and target detection, and obtain an output image containing a target perception box labeled with at least one target location;

[0010] Step 4: Determine whether the targets marked in the target perception boxes are the same target based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box to the boundary.

[0011] Step 5: Identify all credible targets based on the number of targets with the same objective;

[0012] Step 6: Mark all credible targets in the visible light image as target-aware bounding boxes.

[0013] The multi-element fusion methods include: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS, and Wavelet fusion methods.

[0014] Optionally, step 4 includes:

[0015] Step 41: For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of the target sensing box containing the target and the boundary.

[0016] Step 42: For two feature maps, if the first distance is less than the shortest second distance, then the target in the target perception box in the two maps is determined to be the same target.

[0017] Optionally, step 5 includes:

[0018] Step 51: Count the number of targets identified as having the same objective;

[0019] Step 52: Determine if the number of targets exceeds half of the targets at the same target location. If so, determine that the target is a credible target; otherwise, remove the target.

[0020] The target detection models include: YOLO network model, Faster RCNN network model, and SSD network model.

[0021] Secondly, the present invention provides a multi-source remote sensing image fusion and target perception device based on multi-level collaborative methods, comprising:

[0022] The acquisition module is configured to acquire raw heterogeneous images from the same geographic location;

[0023] The original heterogeneous image includes infrared images and visible light images;

[0024] The fusion module is configured to fuse the infrared image and the visible light image using a multi-fusion method to obtain a fused image under each fusion method.

[0025] The recognition module is configured to take the fused image under each fusion method, the infrared image and the visible light image as input images, input them into the trained target detection model, and perform feature extraction, feature enhancement and target detection to obtain an output image containing a target perception box labeled with at least one target location.

[0026] The judgment module is configured to determine whether the targets marked in the target perception box are the same target based on the distance between the center points of the target perception box and the shortest distance from the center point of the target perception box to the boundary.

[0027] The determination module is configured to determine all trustworthy targets based on the number of identical targets.

[0028] The annotation module is configured to annotate all trustworthy targets in the visible light image as target-aware bounding boxes.

[0029] The multi-element fusion methods include: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS, and Wavelet fusion methods.

[0030] Optionally, the determination module is configured as follows:

[0031] For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of each target sensing box and the boundary.

[0032] For two feature maps, if the first distance is less than the smallest second distance, then the target in the target perception box in the two maps is determined to be the same target.

[0033] Optionally, the determining module is configured as follows:

[0034] The number of targets identified as having the same objective;

[0035] Determine if the number of targets exceeds half of the targets at the same location. If so, determine that target as a credible target; otherwise, remove the target.

[0036] Optionally, the target detection model includes: YOLO network model, Faster RCNN network model, and SSD network model.

[0037] This invention provides a method and apparatus for multi-source remote sensing image fusion and target perception based on multi-level collaboration. It acquires original heterogeneous images from the same geographical location; fuses these images using a multi-source fusion method; and then inputs the fused images into a trained target detection model for feature extraction, feature enhancement, and target detection, resulting in an output image containing target perception boxes. A decision is made regarding whether to label the target within the perception box based on the distance between the center points of the perception boxes and the shortest distance from the center point of the target perception box to the boundary, thus determining whether the target is a credible target. If credible, it is labeled in the visible light image. This invention uses different fusion algorithms to process the original heterogeneous images, enhancing the features of targets in the images and enriching scene information; it learns the spectral and spatial features of the fused image to improve target perception accuracy; and a voting mechanism is used to determine credible targets based on the perception results, further improving target perception accuracy.

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a multi-source remote sensing image fusion and target perception method based on multi-level collaboration provided by the present invention;

[0040] Figure 2 The infrared and visible light images used in this invention;

[0041] Figure 3 This is a detailed schematic diagram of a multi-source remote sensing image fusion and target perception method based on multi-level collaboration according to the present invention;

[0042] Figure 4 This is a schematic diagram of the YOLO v4 target detection network structure used in the method of the present invention;

[0043] Figure 5 This is a fusion result diagram of the multiple fusion methods used in the method of this invention;

[0044] Figure 6The results of target perception in infrared and visible light images using the YOLO v4 model employed in the method of this invention are shown. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0046] like Figure 1 As shown, the present invention provides a method for multi-source remote sensing image fusion and target perception based on multi-level collaboration, comprising:

[0047] Step 1: Obtain raw heterogeneous images from the same geographical location;

[0048] The original heterogeneous image includes infrared images and visible light images;

[0049] refer to Figure 2 As shown, the infrared and visible light images acquired in this step were not imaged by the same sensor and have different resolutions. To ensure that the two images are the same size and do not affect subsequent fusion operations, the two images are cropped and the data is resampled; in this embodiment, the resolutions of the infrared and visible light images are 0.0443 μm and 0.37 μm, respectively. Registration, resampling, and cropping operations were performed on the two images sequentially.

[0050] Step 2: The infrared image and the visible light image are fused using a multi-element fusion method to obtain a fused image for each fusion method.

[0051] Combination Figure 3 The multi-element fusion method of the present invention includes: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS and Wavelet fusion methods.

[0052] This invention can also sharpen the preprocessed image; different sharpening methods have different effects on the subsequent fusion effect. The following section describes some of the methods used in the fusion process.

[0053] IHS (Intensity, Hue, Saturation) image fusion steps: ① Upsample the images to ensure they are the same size and perform histogram matching; ② Perform a forward IHS transform on the visible light image; ③ Replace the intensity component I with the infrared image; ④ Perform an inverse IHS transform on the replaced three components. PCA (Principal Components Analysis) image fusion steps: ① Register the source images; ② Calculate the eigenvalues ​​and corresponding eigenvectors of the principal component transformation matrix of the visible light image; ③ Sort the eigenvalues ​​in descending order, and the corresponding eigenvectors change accordingly, denoted as λ1, λ2, λ3…λ…n , ④ Each principal component is arranged according to Calculation; ⑤ Perform histogram matching between the infrared image and the first principal component, where the first principal component is the component replaced by the infrared image, and then replace the first principal component with the infrared image; ⑥ Perform inverse principal component transformation to obtain the fused image. Wavelet transform image fusion steps; ① Preprocess and register the original image; ② Perform wavelet decomposition on the processed image to obtain low-frequency and high-frequency components; ③ Apply different fusion rules to the low-frequency and high-frequency components (low-frequency fusion rule: weighted average; high-frequency fusion rule: take the larger absolute value); ④ Perform inverse wavelet transform; ⑤ Obtain the fused image.

[0054] Step 3: Take the fused image from each fusion method, the infrared image, and the visible light image as input images, and input them into the trained target detection model to perform feature extraction, feature enhancement, and target detection, and obtain an output image containing a target perception box labeled with at least one target location;

[0055] The target detection models include the YOLO network model, the Faster R-CNN network model, and the SSD network model.

[0056] Combination Figure 3 as well as Figure 4 The object detection model mainly consists of three parts: Backbone, Neck, and Prediction, which respectively realize feature extraction, feature enhancement, and prediction results for object detection perception.

[0057] Taking YOLO v4 as an example, refer to Figure 4 , Figure 4 The figure shows a schematic diagram of the YOLO v4 target detection network structure used in this invention; as shown Figure 4As shown, in the Backbone, the image to be detected is input. At this stage, the input image is scaled to the network's input size and normalized. During network training, YOLO v4 uses Mosaic data augmentation to improve training speed and network accuracy; cmBN and SAT adversarial training are used to improve generalization performance; the Mish activation function replaces the original ReLU activation function; and Dropblock is added to further enhance generalization. In the Neck, the Neck network is used to further improve feature diversity and robustness; the SPP module is used to fuse feature maps of different scales; and a top-down FPN feature pyramid and a bottom-up PAN feature pyramid are used to improve feature extraction capabilities. In Prediction, the three feature maps used for prediction are 76*76*255, 38*38*255, and 19*19*255. The CIOU_Loss loss function is used for regression during training. The prediction box selection function is DIOU_nms.

[0058] refer to Figure 5 , Figure 5 This is a fusion result diagram of the multiple fusion methods used in the present invention. Figure 5 Each input image is processed by the detection and perception model, and the output image contains the target perception box.

[0059] This invention allows setting parameters during the training of the YOLO v4 model, using publicly available multi-scene, multi-feature images and their labels for training and validation, and selecting the model with the minimum validation loss as the trained object detection perception model.

[0060] refer to Figure 6 This represents the target perception results of the YOLO v4 model used in the method of this invention for infrared and visible light images. Figure 6 As can be seen, some targets were not marked by the perception box, while some non-targets were marked by the perception box, so further judgment is needed.

[0061] Step 4: Determine whether the targets marked in the target perception boxes are the same target based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box to the boundary.

[0062] Step 5: Identify all credible targets based on the number of targets with the same objective;

[0063] It is worth noting that this invention uses Bagging decision to judge the targets marked in the target perception box, to determine whether the targets need to be removed, and to further determine whether the targets are credible targets, rather than targets misidentified by the target detection model.

[0064] Step 6: Mark all credible targets in the visible light image as target-aware bounding boxes.

[0065] As an optional embodiment of the present invention, step 4 includes:

[0066] Step 41: For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of the target sensing box containing the target and the boundary.

[0067] Step 42: For two feature maps, if the first distance is less than the smallest second distance, then the target in the target perception box in the two maps is determined to be the same target.

[0068] It is worth noting that: the first center point of the target perception box is calculated, and the distance between the first center point of each target in each feature map and the center point of each target in the remaining images is calculated. If the distance between the center points of two targets is less than the shortest distance among the distances from the center point of this target to the target perception boxes, then the targets corresponding to the two feature maps are considered to be the same target, thus obtaining the result of whether the same target exists in the remaining feature maps.

[0069] As an optional embodiment of the present invention, step 5 includes:

[0070] Step 51: Count the number of targets identified as having the same objective;

[0071] Step 52: Determine if the number of targets exceeds half of the targets at the same target location. If so, determine that the target is a credible target; otherwise, remove the target.

[0072] This invention provides a multi-source remote sensing image fusion and target perception method based on multi-level collaboration. It acquires original heterogeneous images from the same geographical location; employs a multi-fusion method to fuse the infrared and visible light images, obtaining fused images under each fusion method; uses the fused images under each fusion method, the infrared image, and the visible light image as input images, and inputs them into a trained target detection model for feature extraction, feature enhancement, and target detection, resulting in an output image containing target perception boxes labeled with at least one target location; determines whether the targets labeled in the target perception boxes are the same target based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box containing the target to its boundary; identifies all credible targets based on the number of identical targets; and labels all credible targets in the visible light image as target perception boxes. This invention enhances the features of targets in the images and enriches the scene information by using different fusion algorithms to process infrared and visible light images. It improves target perception accuracy by learning the spectral and spatial features of the fused image through a deep learning YOLOv4 network; and further improves target perception accuracy by using a decision fusion and voting mechanism to determine credible targets through decision-making fusion.

[0073] The present invention provides a multi-source remote sensing image fusion and target perception device based on multi-level collaboration, comprising:

[0074] The acquisition module is configured to acquire raw heterogeneous images from the same geographic location;

[0075] The original heterogeneous image includes infrared images and visible light images;

[0076] The fusion module is configured to fuse the infrared image and the visible light image using a multi-fusion method to obtain a fused image under each fusion method.

[0077] The recognition module is configured to take the fused image under each fusion method, the infrared image and the visible light image as input images, input them into the trained target detection model, and perform feature extraction, feature enhancement and target detection to obtain an output image containing a target perception box labeled with at least one target location.

[0078] The judgment module is configured to determine whether the targets marked in the target perception box are the same target based on the distance between the center points of the target perception box and the shortest distance from the center point of the target perception box to the boundary.

[0079] The determination module is configured to determine all trustworthy targets based on the number of identical targets.

[0080] The annotation module is configured to annotate all trustworthy targets in the visible light image as target-aware bounding boxes.

[0081] The multi-element fusion methods include: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS, and Wavelet fusion methods.

[0082] Optionally, the determination module is configured as follows:

[0083] For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of each target sensing box and the boundary.

[0084] For two feature maps, if the first distance is less than the smallest second distance, then the target in the target perception box in the two maps is determined to be the same target.

[0085] Optionally, the determining module is configured as follows:

[0086] The number of targets identified as having the same objective;

[0087] Determine if the number of targets exceeds half of the targets at the same location. If so, determine that target as a credible target; otherwise, remove the target.

[0088] Optionally, the target detection model includes: YOLO network model, Faster RCNN network model, and SSD network model.

[0089] This invention provides a multi-source remote sensing image fusion and target perception device based on multi-level collaboration. It acquires original heterogeneous images from the same geographical location; employs a multi-fusion method to fuse the infrared and visible light images, obtaining fused images under each fusion method; uses the fused images under each fusion method, the infrared image, and the visible light image as input images, and inputs them into a trained target detection model for feature extraction, feature enhancement, and target detection, resulting in an output image containing target perception boxes labeled with at least one target location; determines whether the targets labeled in the target perception boxes are the same target based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box to the boundary; determines all credible targets based on the number of the same target; and labels all credible targets in the visible light image as target perception boxes. This invention enhances the features of targets in the images and enriches the scene information by using different fusion algorithms to process infrared and visible light images. It improves target perception accuracy by learning the spectral and spatial features of the fused image through a deep learning YOLOv4 network; and further improves target perception accuracy by using a decision fusion and voting mechanism to determine credible targets through decision-making fusion.

[0090] This invention, through experimental verification, demonstrates that its method can enhance the features of targets in images and enrich scene information. By using a deep learning YOLOv4 network to learn the fused image's spectral and spatial features, the accuracy of target perception is improved. Furthermore, through decision fusion, a voting mechanism is applied to the results, thereby further enhancing the accuracy of target perception.

[0091] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0092] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0093] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for multi-source remote sensing image fusion and target perception based on multi-level collaborative methods, characterized in that, include: Step 1: Obtain raw heterogeneous images from the same geographical location; The original heterogeneous image includes infrared images and visible light images; Step 2: The infrared image and the visible light image are fused using a multi-element fusion method to obtain a fused image for each fusion method. Step 3: Take the fused image from each fusion method, the infrared image, and the visible light image as input images, and input them into the trained target detection model to perform feature extraction, feature enhancement, and target detection, and obtain an output image containing a target perception box labeled with at least one target location; Step 4: Based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box containing the target to the boundary, determine whether the targets marked in the target perception boxes are the same target; Step 4 includes: Step 41: For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of the target sensing box containing the target and the boundary. Step 42: For two feature maps, if the first distance is less than the shortest second distance, then the target in the target perception box in the two maps is determined to be the same target; Step 5: Identify all credible targets based on the number of targets with the same objective; Step 6: Mark all credible targets in the visible light image as target-aware bounding boxes.

2. The method for multi-source remote sensing image fusion and target perception based on multi-level collaboration according to claim 1, characterized in that, The multi-element fusion methods include: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS, and Wavelet fusion methods.

3. The method for multi-source remote sensing image fusion and target perception based on multi-level collaboration according to claim 1, characterized in that, Step 5 includes: Step 51: Count the number of targets identified as having the same objective; Step 52: Determine if the number of targets exceeds half of the targets at the same target location. If so, determine that the target is a credible target; otherwise, remove the target.

4. The method for multi-source remote sensing image fusion and target perception based on multi-level collaboration according to claim 1, characterized in that, The target detection models include: YOLO network model, Faster RCNN network model, and SSD network model.

5. A device for multi-source remote sensing image fusion and target perception based on multi-level collaborative methods, characterized in that, include: The acquisition module is configured to acquire raw heterogeneous images from the same geographic location; The original heterogeneous image includes infrared images and visible light images; The fusion module is configured to fuse the infrared image and the visible light image using a multi-fusion method to obtain a fused image under each fusion method. The recognition module is configured to take the fused image under each fusion method, the infrared image and the visible light image as input images, input them into the trained target detection model, and perform feature extraction, feature enhancement and target detection to obtain an output image containing a target perception box labeled with at least one target location. The judgment module is configured to determine whether the targets marked in the target perception boxes are the same target based on the distance between the center points of the target perception boxes and the shortest distance from the center point of the target perception box to the boundary; the judgment module is configured to: For target sensing boxes at the same target location, calculate the first distance between the center point of each target sensing box and the center point of other target sensing boxes, and the second distance between the center point of each target sensing box and the boundary. For two feature maps, if the first distance is less than the smallest second distance, then the target in the target perception box in the two maps is determined to be the same target; The determination module is configured to determine all trustworthy targets based on the number of identical targets. The annotation module is configured to annotate all trustworthy targets in the visible light image as target-aware bounding boxes.

6. The multi-source remote sensing image fusion and target perception device based on multi-level collaboration according to claim 5, characterized in that, The multi-element fusion methods include: Closed, Gram-Schmidt, Guided, IHS, MMP, PCA, P+XS, and Wavelet fusion methods.

7. The method for multi-source remote sensing image fusion and target perception based on multi-level collaboration according to claim 5, characterized in that, The determining module is configured as follows: The number of targets identified as having the same objective; Determine if the number of targets exceeds half of the targets at the same location. If so, determine that target as a credible target; otherwise, remove the target.

8. The multi-source remote sensing image fusion and target perception device based on multi-level collaboration according to claim 5, characterized in that, The target detection models include: YOLO network model, Faster RCNN network model, and SSD network model.

Images