A multi-stage cooperative heterogeneous image fusion system and method based on DSP

Abstract

The application relates to a multi-stage cooperative heterogeneous and heterogeneous image fusion system and method based on a DSP. The image fusion method comprises the following steps: step 1, a DSP module performs a plurality of pixel-level fusion processes on at least two received original images to obtain a plurality of pixel fusion images; step 2, the pixel fusion images and the original images are transmitted to an embedded AI device by using a TCP protocol; step 3, a target recognition network configured on the embedded AI device is used to perform target recognition on the pixel fusion images and the original images to obtain a recognition result corresponding to each image, and the recognition result comprises a target recognition frame and a target classification result; and step 4, the recognition result is transmitted to the DSP module by using the TCP protocol to perform decision-level image fusion processing and obtain a final recognition result corresponding to the original image. The image fusion method can avoid errors caused by using a single fusion method, and the recognition result is more accurate.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a multi-level collaborative heterogeneous image fusion system and method based on DSP. Background Technology

[0002] Image fusion is a technique that uses various algorithms to combine two or more images that are simultaneously in phase. The fusion result leverages the spatiotemporal correlation and information complementarity of two or more images, providing a more comprehensive and clear description of the scene, thus enhancing human recognition and machine detection. Existing image fusion methods include pixel-level, feature-level, and decision-level methods. Pixel-level fusion uses pixels as the basis for image fusion, feature-level fusion extracts individual features, and decision-level fusion performs fusion after each image has been processed to obtain decision information. From pixel-level to feature-level to decision-level, the required image quality decreases sequentially, but the error tolerance increases, and the system openness also improves.

[0003] Current image fusion methods are mostly pixel-level fusion algorithms, requiring high image quality. The completeness of image information acquisition depends on the suitability of subsequent algorithms for the fused image. Different fusion algorithms have their own advantages and disadvantages; some algorithms may only be suitable for certain images, while having a negative effect on others. In terms of hardware implementation, existing image fusion methods are not implemented on mobile devices, limiting their application scenarios. Among current hardware implementations, single DSP (Digital Signal Processor) or multiple DSP solutions face challenges in task allocation. DSP+FPGA solutions encounter communication and data transmission issues between the DSP and FPGA. Large-scale FPGA solutions face challenges in the efficient utilization of FPGA internal resources. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a multi-level collaborative heterogeneous image fusion system and method based on DSP.

[0005] This invention provides a DSP-based multi-level collaborative heterogeneous image fusion method, comprising:

[0006] Step 1: The DSP module performs various pixel-level fusion processes on at least two received original images to obtain multiple pixel-fused images;

[0007] Step 2: Use the TCP protocol to transmit the pixel-fused image and the original image to the embedded AI device;

[0008] Step 3: Using the target recognition network configured on the embedded AI device, target recognition is performed on the pixel fusion image and the original image to obtain the recognition result corresponding to each image. The recognition result includes the target recognition box and the target classification result.

[0009] Step 4: Use the TCP protocol to transmit the recognition result to the DSP module for decision-level image fusion processing to obtain the final recognition result corresponding to the original image.

[0010] In one embodiment of the present invention, the original image is a multispectral image, a hyperspectral image, a radar image, a near-infrared image, or a three-channel color original image.

[0011] In one embodiment of the present invention, step 1 includes:

[0012] After the DSP module performs image registration and denoising on at least two received original images, it performs pixel-level fusion processing using various pixel-level fusion processing methods to obtain multiple pixel-fused images.

[0013] In one embodiment of the present invention, the pixel-level fusion processing method is at least two of the following: lost-pansharp fusion method, guided-pansharp fusion method, his-pansharp fusion method, mmp-pansharp fusion method, PCA fusion method, P+XS fusion method, wavelet transform fusion method, GS-NIR fusion method, and GS-RGB fusion method.

[0014] In one embodiment of the present invention, the target recognition network is trained based on multiple training image samples and the labels corresponding to each training image sample, wherein the labels include target coordinate labels and target classification labels.

[0015] In one embodiment of the present invention, step 5, the decision-level image fusion processing, includes:

[0016] The identification results are voted on, and the identification results that exceed a preset threshold number of votes are retained as the final identification results.

[0017] This invention provides a DSP-based multi-level collaborative heterogeneous image fusion system, comprising: a DSP module and an embedded AI device, wherein,

[0018] The DSP module is used to perform various pixel-level fusion processing operations on at least two received original images to obtain multiple pixel-fused images, and to transmit the pixel-fused images and the original images to an embedded AI device using the TCP protocol.

[0019] The embedded AI device is used to perform target recognition on the pixel-fused image and the original image using a target recognition network configured thereon, to obtain the recognition result corresponding to each image, and to transmit the recognition result to the DSP module using the TCP protocol;

[0020] The DSP module is also used to perform decision-level image fusion processing on the received recognition results to obtain the final recognition result corresponding to the original image.

[0021] In one embodiment of the present invention, the DSP module includes one or more DSP devices. When multiple DSP devices are included, pixel-level image fusion processing and decision-level image fusion processing are performed on different DSP devices, and / or multiple pixel-level fusion processes are performed on different DSP devices.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. The DSP-based multi-level collaborative heterogeneous image fusion method of the present invention uses multiple pixel-level fusion methods to identify several results using a target recognition network, and finally uses decision-level fusion to perform decision fusion to obtain the final result. This avoids errors caused by using a single fusion method and avoids result deviations due to the quality problems of a single data source. Furthermore, using multiple pixel-level fusion methods can leverage the unique advantages of various images, and using an integrated discriminative method can make the recognition results more accurate.

[0024] 2. The DSP-based multi-level collaborative heterogeneous image fusion system of the present invention utilizes a DSP and embedded AI device architecture to achieve an automated process. It can be deployed in more mobile environments, freeing it from the environmental requirements of existing algorithms.

[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a multi-level collaborative heterogeneous image fusion method based on DSP provided in an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of a pixel-level image fusion process provided in an embodiment of the present invention;

[0028] Figure 3This is a flowchart of a decision-level image fusion process provided in an embodiment of the present invention;

[0029] Figure 4 This is a flowchart of the TCP handshake protocol provided in an embodiment of the present invention;

[0030] Figure 5 This is a flowchart of the processing of a DSP-based multi-level collaborative heterogeneous image fusion system provided in this embodiment of the invention.

[0031] Figure 6 This is a flowchart of the one-stage network and two-stage network provided in the embodiments of the present invention;

[0032] Figure 7 This is a flowchart of the YOLOv4 target recognition network provided in an embodiment of the present invention. Detailed Implementation

[0033] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a multi-level collaborative heterogeneous image fusion system and method based on DSP proposed in accordance with the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0034] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0035] Example 1

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of a DSP-based multi-level collaborative heterogeneous image fusion method provided in an embodiment of the present invention. As shown in the figure, the DSP-based multi-level collaborative heterogeneous image fusion method of this embodiment includes:

[0037] Step 1: The DSP module performs various pixel-level fusion processes on at least two received original images to obtain multiple pixel-fused images;

[0038] The original images are multispectral images, hyperspectral images, radar images, near-infrared images, or three-channel color original images.

[0039] Please see Figure 2The flowchart shown is a pixel-level image fusion flowchart. Pixel-level fusion is the fusion of the original images directly at the input level. Specifically, step 1 includes: after the DSP module performs image registration and denoising on at least two received original images, it performs pixel-level fusion processing using a variety of pixel-level fusion processing methods to obtain multiple pixel-fused images.

[0040] In this embodiment, the pixel-level fusion processing method is at least two of the following: lost-pansharp fusion, guided-pansharp fusion, his-pansharp fusion, mmp-pansharp fusion, PCA fusion, P+XS fusion, wavelet transform fusion, GS-NIR (Gram-NIR based on near-infrared) fusion, and GS-RGB (Gram-RGB based on RGB image) fusion. The above-mentioned pixel-level fusion methods are traditional pixel-level image fusion methods, and the specific fusion process will not be described in detail here.

[0041] In other embodiments, the pixel-level fusion processing method may also employ image fusion methods based on pyramid transform, image fusion methods based on Ridgelet transform, WMFGS (Schmidt orthogonal transform image fusion method with added weighted median filtering), image fusion methods based on Curvelet transform, and other variant algorithms based on traditional pixel-level image fusion.

[0042] Step 2: Use the TCP protocol to transmit the pixel-blended image and the original image to the embedded AI device;

[0043] TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based communication protocol. A connection must be established before data can be transmitted, i.e., a three-way handshake. Please refer to [further details omitted]. Figure 4 The TCP handshake protocol flowchart shown illustrates that before data transmission, the client first sends a trigger packet with SYN=1 (trigger flag) to the server, waiting for acknowledgment from the server. Upon receiving this trigger packet, the server responds by sending a packet with SYN=1 and ack=1 (acknowledgment flag) to confirm and further trigger the handshake. Finally, the client sends an acknowledgment packet with SYN=0 and ack=1 to confirm the handshake. Once confirmed, the three-way handshake is successfully established. Data communication can then commence. In step 2, the DSP module acts as the server, and the embedded AI device acts as the client, transmitting both the pixel-blended image and the original image to the embedded AI device.

[0044] Further, the TCP packet sending process is explained as follows: First, the application writes data to the send buffer. If the socket send buffer cannot hold all the application's data, and the application process is blocked, the application process will be suspended until all data is copied to the socket buffer. Then, TCP segments the data according to the MSS (Maximum Segment Size). MSS is the maximum data segment that a TCP packet can transmit at a time. To achieve optimal transmission performance, the TCP protocol usually negotiates the MSS value between the two parties when establishing a connection. In its implementation, the TCP protocol often uses the MTU (Maximum Transmission Unit, Ethernet MTU is 1500) value instead (subtracting the 20-byte size of the IP packet header and the 20-byte size of the TCP segment header). Therefore, MSS is 1460. The communicating parties will determine the maximum MSS value for this connection based on the minimum MSS value provided by both parties.

[0045] Step 3: Using the target recognition network that has been trained and configured on the embedded AI device, target recognition is performed on the pixel-fused image and the original image to obtain the recognition result for each image. The recognition result includes the target recognition box and the target classification result.

[0046] In this embodiment, the target recognition network is trained based on multiple training image samples and the corresponding labels for each training image sample. The labels include target coordinate labels and target classification labels.

[0047] Optionally, YOLOv4 can be used as the object recognition network. When training the recognition network, the original image type images to be fused can be used as the training dataset to train the YOLOv4 object recognition network to obtain the trained object recognition network. Alternatively, both the original image type images and the fused image type images can be used as the training dataset to train the YOLOv4 object recognition network to obtain the trained object recognition network. Only one network model needs to be trained.

[0048] Please refer to the above. Figure 6 and Figure 7 , Figure 6 This is a flowchart of the one-stage network and two-stage network provided in the embodiments of the present invention; Figure 7This is a flowchart of the YOLOv4 object recognition network provided in this embodiment of the invention. YOLOv4 achieves a perfect balance between speed and accuracy, improving the accuracy of convolutional neural networks (CNNs). The YOLO network mainly consists of a backbone, a neck, and a head connected in sequence, forming a one-stage network. The backbone is a convolutional neural network that aggregates and forms image features at different fine-grained levels; the neck is a series of network layers that mix and combine image features and pass them to the prediction layer; the head includes regression and classification branches for predicting image features, generating bounding boxes, and predicting categories. Specifically, after the YOLOv4 network is trained, the image to be detected is input into the network and sequentially processed through a CSPDarknet53 structure, an SPP structure, and a PANet structure for feature extraction. Then, the YOLOv3 head structure detects the input features to obtain the recognition result. Finally, the decoding network outputs the recognition result value, and non-maximum suppression is used to remove recognition results with low scores, resulting in the final recognition result, which is then displayed on the image to be detected.

[0049] For YOLOv4, optimization techniques such as Weighted Residual Connections (WRC), Cross-Stage Partial Connections (CSP), Cross-Mini-Batch Normalization (CmBN), Self-Adversarial Training (SAT), Mish Activation, Mosaic Data Augmentation, DropBlock Regularization, and CIoU Loss can be used for learning and training.

[0050] In other embodiments, the target recognition network may also be a one-stage or two-stage target recognition network such as YOLOv1, YOLOv2, YOLOv3, YOLOv5, RCNN, etc.

[0051] Step 4: Use the TCP protocol to transmit the recognition results to the DSP module for decision-level image fusion processing to obtain the final recognition result corresponding to the original image.

[0052] In step 4, the embedded AI device acts as the server, and the DSP module acts as the client, transmitting the recognition results to the embedded AI device. Specifically, how to establish a connection before data transmission and how to perform data transmission are similar to those described above and will not be repeated here.

[0053] In this embodiment, the DSP module performs decision-level image fusion processing on the received recognition results to obtain the final recognition result corresponding to the original image. Please refer to [link / reference]. Figure 3 , Figure 3This is a flowchart of a decision-level image fusion process provided by an embodiment of the present invention. As shown in the figure, in this embodiment, the decision-level image fusion processing includes: voting on the recognition results and retaining the recognition results that exceed a preset vote threshold as the final recognition results. That is, for the recognition results corresponding to multiple received images, if the number of votes for a target is greater than half, the target is considered credible and is retained; otherwise, it is not credible and is removed to obtain the final recognition result (target recognition box and target classification result).

[0054] It should be noted that in other embodiments, decision-level image fusion processing can also employ methods such as weighted average method, consensus theory, Bayesian inference, DS evidence theory, fuzzy integral method, NP criterion or expert system to perform decision-level image fusion on the received recognition results and obtain the final recognition result.

[0055] This embodiment of the DSP-based multi-level collaborative heterogeneous image fusion method uses multiple pixel-level fusion methods to identify several results using a target recognition network. Finally, decision-level fusion is used to perform decision fusion on the obtained results to obtain the final result. This method avoids errors caused by using a single fusion method and avoids result deviations due to the quality problems of a single data source. Moreover, using multiple pixel-level fusion methods can leverage the unique advantages of various images, and using an integrated discriminative approach can make the recognition results more accurate.

[0056] Example 2

[0057] This embodiment provides a DSP-based multi-level collaborative heterogeneous image fusion system, applicable to the DSP-based multi-level collaborative heterogeneous image fusion method described in Embodiment 1 above. This image fusion system includes a DSP module and an embedded AI device. Please refer to [reference needed]. Figure 5 , Figure 5 This is a flowchart of the processing of a DSP-based multi-level collaborative heterogeneous image fusion system provided in this embodiment of the invention. As shown in the figure, specifically, the DSP module performs various pixel-level fusion processing operations on at least two received original images to obtain multiple pixel-fused images, and transmits the pixel-fused images and the original images to an embedded AI device using the TCP protocol. The embedded AI device uses a trained target recognition network configured on it to perform target recognition on the pixel-fused images and the original images, obtaining a recognition result for each image, and transmits the recognition result to the DSP module using the TCP protocol. The DSP module also performs decision-level image fusion processing on the received recognition results to obtain the final recognition result corresponding to the original image. The recognition result includes a target bounding box and a target classification result.

[0058] In this embodiment, the DSP module includes one or more DSP devices. When multiple DSP devices are included, pixel-level image fusion processing and decision-level image fusion processing are performed on different DSP devices.

[0059] It should be noted that when multiple DSP devices are involved, various pixel-level fusion processes for two original images can also be performed on different DSP devices.

[0060] This embodiment of the DSP-based multi-level collaborative heterogeneous image fusion system utilizes a DSP and embedded AI device architecture to automate the process. It is independent of the environmental requirements of existing algorithms and can be deployed in more mobile environments, such as certain missile-borne environments, enabling precise target identification.

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0062] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-level collaborative heterogeneous image fusion method based on DSP, characterized in that, include: Step 1: The DSP module performs various pixel-level fusion processes on at least two received original images to obtain multiple pixel-fused images; Step 2: Use the TCP protocol to transmit the pixel-fused image and the original image to the embedded AI device; Step 3: Using the target recognition network configured on the embedded AI device, target recognition is performed on the pixel fusion image and the original image to obtain the recognition result corresponding to each image. The recognition result includes the target recognition box and the target classification result. Step 4: Use the TCP protocol to transmit the recognition result to the DSP module for decision-level image fusion processing to obtain the final recognition result corresponding to the original image.

2. The DSP-based multi-level collaborative heterogeneous image fusion method according to claim 1, characterized in that, The original image is a multispectral image, hyperspectral image, radar image, near-infrared image, or three-channel color original image.

3. The DSP-based multi-level collaborative heterogeneous image fusion method according to claim 1, characterized in that, Step 1 includes: After the DSP module performs image registration and denoising on at least two received original images, it performs pixel-level fusion processing using various pixel-level fusion processing methods to obtain multiple pixel-fused images.

4. The DSP-based multi-level collaborative heterogeneous image fusion method according to claim 3, characterized in that, The pixel-level fusion processing method is at least two of the following: lost-pansharp fusion method, guided-pansharp fusion method, his-pansharp fusion method, mmp-pansharp fusion method, PCA fusion method, P+XS fusion method, wavelet transform fusion method, GS-NIR fusion method, and GS-RGB fusion method.

5. The DSP-based multi-level collaborative heterogeneous image fusion method according to claim 1, characterized in that, The target recognition network is trained based on multiple training image samples and the corresponding labels for each training image sample. The labels include target coordinate labels and target classification labels.

6. The DSP-based multi-level collaborative heterogeneous image fusion method according to claim 1, characterized in that, In step 4, the decision-level image fusion processing includes: The identification results are voted on, and the identification results that exceed a preset threshold number of votes are retained as the final identification results.

7. A multi-level collaborative heterogeneous image fusion system based on DSP, characterized in that, include: DSP modules and embedded AI devices, among which, The DSP module is used to perform various pixel-level fusion processing operations on at least two received original images to obtain multiple pixel-fused images, and to transmit the pixel-fused images and the original images to an embedded AI device using the TCP protocol. The embedded AI device is used to perform target recognition on the pixel-fused image and the original image using a target recognition network configured thereon, to obtain the recognition result corresponding to each image, and to transmit the recognition result to the DSP module using the TCP protocol; The DSP module is also used to perform decision-level image fusion processing on the received recognition results to obtain the final recognition result corresponding to the original image.

8. The DSP-based multi-level collaborative heterogeneous image fusion system according to claim 7, characterized in that, The DSP module includes one or more DSP devices. When multiple DSP devices are included, pixel-level image fusion processing and decision-level image fusion processing are performed on different DSP devices, and / or multiple pixel-level fusion processes are performed on different DSP devices.

Images