Multimodal information fusion reasoning method and device for remote sensing interpretation integrated machine

Abstract

The present invention provides a multi-modal information fusion reasoning method and device for a remote sensing interpretation integrated machine, which relates to the field of remote sensing image processing technology, and can solve the problem of insufficient accuracy and credibility of interpretation of traditional single-modal information interpretation methods. The method includes: obtaining two remote sensing images to be inferred, the two remote sensing images are in the same scene and belong to different modes; inputting each remote sensing image into a dual-branch encoder for processing, and obtaining the general branch features and the exclusive branch features of each remote sensing image; using a feature fusion module, the two general branch features are fused, and the two exclusive branch features are fused; the fused general branch features and the fused exclusive branch features are summed and input into the downstream task detection head to obtain the reasoning results of the two remote sensing images in the scene. The present invention not only realizes edge deployment on the remote sensing interpretation integrated machine, but also realizes the fusion of multi-modal observation information, thereby improving the reasoning accuracy of ground observation.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a multi-modal information fusion reasoning method and device for a remote sensing interpretation integrated machine. Background Art

[0002] With the continuous development of remote sensing technology, the number of remote sensing satellites in orbit and the means of earth observation have been greatly enriched. The remote sensing images obtained cover multimodal information obtained by various sensors, such as optical images, radar images, infrared images, etc. At the same time, the remote sensing image interpretation task has gradually developed towards the edge. How to effectively use this multimodal information on the terminal device and organically integrate the information obtained by different sensors to improve the accuracy and credibility of remote sensing image interpretation requires considering the relationship between different information sources. The traditional single-modal information interpretation method is no longer applicable. Summary of the invention

[0003] In view of the problem that the traditional single-modal information interpretation method has insufficient accuracy and credibility in interpreting remote sensing images, the present invention provides a multi-modal information fusion reasoning method and device for a remote sensing interpretation all-in-one machine.

[0004] On the one hand, the present invention provides a multimodal information fusion reasoning method for a remote sensing interpretation integrated machine, including: step S1, obtaining two remote sensing images to be inferred, the two remote sensing images are in the same scene and belong to different modes; step S2, inputting each of the remote sensing images into a pre-built dual-branch encoder for processing, and obtaining the general branch features and specific branch features of each of the remote sensing images, wherein the general branch features reflect the common information of the two remote sensing images, and the specific branch features reflect the specific information of each remote sensing image; step S3, using a feature fusion module to fuse the two general branch features and the two specific branch features; step S4, summing the fused general branch features and the fused specific branch features and inputting them into a downstream task detection head to obtain the reasoning results of the two remote sensing images in the scene.

[0005] Further, in step S1, the two remote sensing images are acquired by different sensors and include any two of optical images, radar images and infrared images.

[0006] Furthermore, in step S2, the dual-branch encoder adopts a lightweight CNN architecture based on dilated separable convolution, including: a universal branch feature extractor for extracting universal branch features of each of the remote sensing images; a proprietary branch feature extractor for extracting proprietary branch features of each of the remote sensing images; wherein the universal branch feature extractor and the proprietary branch feature extractor use the same model structure, but the model parameters of the two are different.

[0007] Furthermore, in the step S3, the feature fusion module is trained in the following manner: step S311, in each round of training, respectively calculating the first cosine similarity of the two common branch features and the second cosine similarity of the two specific branch features; step S312, determining a loss value based on the first cosine similarity and the second cosine similarity, and using the loss value to constrain the parameter update of the feature fusion module in each round of training to obtain a trained feature fusion module.

[0008] Further, in step S312, the loss value is determined according to the following principle: the first cosine similarity is controlled to be as small as possible, and the second cosine similarity is controlled to be as large as possible.

[0009] Further, in step S3, a feature fusion module is used to fuse the two general branch features, specifically including: connecting the two general branch features along the channel dimension and performing 1×1 convolution to obtain the fused general branch feature.

[0010] Further, in step S3, a feature fusion module is used to fuse the two proprietary branch features, specifically including: step S321, inputting each of the proprietary branch features into a parameter-shared spatial activation function, and calculating an activation score at each position in the space; step S322, according to the two activation scores, the two proprietary branch features are fused through spatial selection to obtain the fused proprietary branch feature.

[0011] Furthermore, the step S322 specifically includes: according to the activation score of each of the two proprietary branch features at each spatial position, selecting a proprietary branch feature with a higher activation score at each spatial position for combination to obtain the fused proprietary branch feature.

[0012] On the other hand, the present invention provides a multimodal information fusion reasoning device for a remote sensing interpretation integrated machine, comprising: a multimodal remote sensing image acquisition module, used to obtain two remote sensing images to be inferred, the two remote sensing images are in the same scene and belong to different modalities; a dual-branch feature extraction module, used to input each of the remote sensing images into a pre-built dual-branch encoder for processing, and obtain the general branch features and specific branch features of each of the remote sensing images, wherein the general branch features reflect the common information of the two remote sensing images, and the specific branch features reflect the specific information of each remote sensing image; a feature fusion module, used to fuse the two general branch features and the two specific branch features; a multimodal remote sensing image reasoning module, used to sum the fused general branch features and the fused specific branch features and input them into a downstream task detection head to obtain the reasoning results of the two remote sensing images in the scene.

[0013] Compared with the prior art, the multimodal information fusion reasoning method and device for remote sensing interpretation integrated machine provided by the present invention has at least the following beneficial effects:

[0014] After acquiring multimodal remote sensing images, the present invention not only realizes edge deployment on the remote sensing interpretation integrated machine through the pre-built dual-branch encoder and feature fusion module, but also has a good extraction capability for multi-domain multimodal data features, effectively aggregates each modal feature in a unified space, fuses multimodal information, and improves the reasoning accuracy of earth observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of a multimodal information fusion reasoning method for a remote sensing interpretation integrated machine according to an embodiment of the present invention is schematically shown;

[0017] Figure 2 A schematic diagram of a network structure of a dual-branch encoder according to an embodiment of the present invention is shown;

[0018] Figure 3 A flowchart schematically illustrates a training process of a feature fusion module according to an embodiment of the present invention;

[0019] Figure 4 A network structure diagram of a feature fusion module according to an embodiment of the present invention is schematically shown;

[0020] Figure 5 A flowchart of using a feature fusion module to fuse proprietary branch features according to an embodiment of the present invention is schematically shown;

[0021] Figure 6 The structure block diagram of a multi-modal information fusion and reasoning device for a remote sensing and interpretation integrated machine according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] Figure 1 The flowchart of the multimodal information fusion reasoning method for the remote sensing interpretation integrated machine according to an embodiment of the present invention is schematically shown.

[0026] like Figure 1 As shown, according to the multimodal information fusion reasoning method for the remote sensing interpretation integrated machine of this embodiment, it can include steps S1 to S4.

[0027] Step S1, obtaining two remote sensing images to be inferred, where the two remote sensing images are in the same scene and belong to different modalities.

[0028] Step S2, input each remote sensing image into a pre-built dual-branch encoder for processing to obtain common branch features and specific branch features of each remote sensing image, wherein the common branch features reflect the common information of the two remote sensing images, and the specific branch features reflect the specific information of each remote sensing image.

[0029] Step S3, using a feature fusion module, fuses the two common branch features and fuses the two specific branch features.

[0030] In step S4, the fused common branch features and the fused specific branch features are summed and input into the downstream task detection head to obtain the inference results of the two remote sensing images in the scene.

[0031] Through the embodiments of the present invention, after acquiring multimodal remote sensing images, the pre-built dual-branch encoder and feature fusion module not only realize edge deployment on the remote sensing interpretation integrated machine, but also have a good extraction capability for multi-domain multimodal data features, effectively aggregate each modal feature in a unified space, fuse multimodal information, and improve the reasoning accuracy of earth observation.

[0032] In the embodiment of the present invention, in step S1, the two remote sensing images are acquired by different sensors and include any two of optical images, radar images and infrared images.

[0033] It is understandable that, since optical imaging remote sensing sensors are easily affected by weather, they cannot effectively reflect the situation in the sensing area in many cases, while SAR sensors can work in all weather conditions and penetrate certain ground objects, providing rich geometric information about ground objects. Based on this, the present invention can combine multi-modal information to obtain more comprehensive and accurate ground observation results.

[0034] Figure 2 The network structure diagram of a dual-branch encoder according to an embodiment of the present invention is schematically shown.

[0035] like Figure 2 As shown, in this embodiment of the present invention, in step S2, the dual-branch encoder adopts a lightweight CNN architecture based on dilated separable convolution, including:

[0036] A universal branch feature extractor, used to extract universal branch features of each remote sensing image;

[0037] A proprietary branch feature extractor, used to extract the proprietary branch features of each remote sensing image;

[0038] Among them, the general branch feature extractor and the dedicated branch feature extractor use the same model structure, but their model parameters are different.

[0039] Taking two remote sensing images, optical image and SAR image, as examples, the two modal remote sensing image inputs are processed separately using a two-branch encoder (i.e., a dual-branch encoder), and the two encoders use the same model structure. The dual-branch encoder adopts a lightweight CNN architecture based on dilated separable convolution, which includes a general branch feature extractor and a dedicated branch feature extractor. The general branch feature extractor is responsible for extracting the common information reflected by each modal data; the dedicated branch feature extractor extracts the dedicated information of different modal data respectively, reflecting the differentiated information of different modalities. Before training begins, the model parameters of the dual-branch encoder are randomly initialized, but the parameters of each path are different.

[0040] Figure 3 The flowchart schematically shows the training process of the feature fusion module according to the embodiment of the present invention.

[0041] Please combine Figure 2 and Figure 3 In the embodiment of the present invention, in step S3, the feature fusion module is trained by the following steps S311 to S312.

[0042] Step S311, in each round of training, respectively calculate the first cosine similarity of the two common branch features and the second cosine similarity of the two specific branch features.

[0043] Step S312, jointly determining a loss value based on the first cosine similarity and the second cosine similarity, and using the loss value to constrain parameter updates of the feature fusion module in each round of training to obtain a trained feature fusion module.

[0044] Furthermore, the loss value is determined according to the following principle: the first cosine similarity is controlled to be as small as possible, and the second cosine similarity is controlled to be as large as possible. For example, the cosine similarity of the common branch features of the optical image and the SAR image is controlled to be as small as possible, and the cosine similarity of the exclusive branch features of the optical image and the SAR image is controlled to be as large as possible, and the loss value is transmitted back to update the network parameters of the feature fusion module, so that the two similarity losses are integrated to jointly constrain the update of the model parameters.

[0045] Figure 4 The network structure diagram of the feature fusion module according to an embodiment of the present invention is schematically shown.

[0046] like Figure 4 As shown, in step S3, a feature fusion module is used to fuse the two general branch features, specifically including: connecting the two general branch features along the channel dimension and performing 1×1 convolution to obtain a fused general branch feature.

[0047] Figure 5The flowchart of using the feature fusion module to fuse the dedicated branch features according to an embodiment of the present invention is schematically shown.

[0048] Please combine Figure 4 and Figure 5 In step S3, the two proprietary branch features are fused using a feature fusion module, which may specifically include steps S321 to S322.

[0049] In step S321, each dedicated branch feature is input into a parameter-sharing spatial activation function, and an activation score is calculated at each position in the space.

[0050] Step S322: According to the two activation scores, the two specific branch features are fused through spatial selection to obtain a fused specific branch feature.

[0051] For example, using the feature fusion module, the common branch features of the optical image and the SAR image are concat (connected) along the channel dimension and restored to the original dimension through 1×1 convolution. At the same time, the specific branch features of the optical image and the SAR image are respectively input into an activation network, and an activation score is calculated at each position in the space, and the fusion of the specific branch features is completed through spatial selection.

[0052] Furthermore, according to the activation score of each of the two exclusive branch features at each spatial position, the exclusive branch features with higher activation scores can be selected at each spatial position for combination to obtain the fused exclusive branch features.

[0053] Finally, the fused common branch features and the fused specific branch features are summed and input into the downstream task detection head to output the final inference result.

[0054] Through the embodiments of the present invention, by separating the common information and proprietary information between different modalities, the common branch features of different modalities are strengthened, that is, the common knowledge is strengthened, and the proprietary branch features of different modalities are fused in spatial positions, that is, the information complementarity of the proprietary knowledge of multimodal data is achieved. Finally, the fusion of multimodal observation information is achieved, and the maximum information gain is achieved, so as to obtain more complete and reliable reasoning results than a single terminal.

[0055] Based on the method disclosed in the above embodiment, the present invention also provides a multi-modal information fusion reasoning device for remote sensing interpretation integrated machine. Figure 6 The device is described in detail.

[0056] Figure 6 The structure block diagram of a multi-modal information fusion and reasoning device for a remote sensing and interpretation integrated machine according to an embodiment of the present invention is schematically shown.

[0057] like Figure 6 As shown, according to this embodiment, the multimodal information fusion and reasoning device 600 for the remote sensing interpretation integrated machine includes a multimodal remote sensing image acquisition module 610, a dual-branch feature extraction module 620, a feature fusion module 630 and a multimodal remote sensing image reasoning module 640.

[0058] The multimodal remote sensing image acquisition module 610 is used to acquire two remote sensing images to be inferred, where the two remote sensing images are in the same scene and belong to different modalities.

[0059] The dual-branch feature extraction module 620 is used to input each remote sensing image into a pre-built dual-branch encoder for processing to obtain the common branch features and specific branch features of each remote sensing image, wherein the common branch features reflect the common information of the two remote sensing images, and the specific branch features reflect the specific information of each remote sensing image.

[0060] The feature fusion module 630 is used to fuse two common branch features and two specific branch features.

[0061] The multimodal remote sensing image reasoning module 640 is used to sum the fused common branch features and the fused specific branch features and input them into the downstream task detection head to obtain the reasoning results of the two remote sensing images in the scene.

[0062] It should be noted that the implementation method of the device part is similar to the implementation method of the method part, and the technical effects achieved are also similar. For specific details, please refer to the above-mentioned method implementation method part, which will not be repeated here.

[0063] According to an embodiment of the present invention, any of the multimodal remote sensing image acquisition module 610, the dual-branch feature extraction module 620, the feature fusion module 630, and the multimodal remote sensing image reasoning module 640 can be combined in one module for implementation, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the multimodal remote sensing image acquisition module 610, the dual-branch feature extraction module 620, the feature fusion module 630, and the multimodal remote sensing image reasoning module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the multimodal remote sensing image acquisition module 610, the dual-branch feature extraction module 620, the feature fusion module 630 and the multimodal remote sensing image reasoning module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0064] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0065] In addition, the terms "first" and "second" are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the word "one" or "an" preceding an element does not exclude the presence of multiple such elements.

[0066] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal information fusion reasoning method for remote sensing interpretation integrated machine, characterized in that: include: Step S1, obtaining two remote sensing images to be inferred, wherein the two remote sensing images are in the same scene and belong to different modalities; Step S2, inputting each of the remote sensing images into a pre-built dual-branch encoder for processing, obtaining a common branch feature and a specific branch feature of each of the remote sensing images, wherein the common branch feature reflects the common information of the two remote sensing images, and the specific branch feature reflects the specific information of each of the remote sensing images; Step S3, using a feature fusion module to fuse the two common branch features and to fuse the two specific branch features; Step S4, summing the fused common branch features and the fused specific branch features and inputting the sum into a downstream task detection head to obtain the inference results of the two remote sensing images in the scene; Wherein, in step S3, using a feature fusion module, the two common branch features are fused, and the two specific branch features are fused, specifically including: The two general branch features are connected along the channel dimension and then subjected to 1×1 convolution to obtain the fused general branch feature; Step S321, input each of the dedicated branch features into a parameter-sharing spatial activation function, and calculate an activation score at each position in the space; Step S322: According to the two activation scores, the two specific branch features are fused through spatial selection to obtain the fused specific branch feature.

2. The multimodal information fusion reasoning method for remote sensing interpretation integrated machine according to claim 1 is characterized in that: In step S1, the two remote sensing images are acquired by different sensors and include any two of optical images, radar images and infrared images.

3. The multimodal information fusion reasoning method for remote sensing interpretation integrated machine according to claim 1 is characterized in that: In step S2, the dual-branch encoder adopts a lightweight CNN architecture based on dilated separable convolution, including: A universal branch feature extractor, used for extracting universal branch features of each of the remote sensing images; A dedicated branch feature extractor, used to extract dedicated branch features of each remote sensing image; The general branch feature extractor and the proprietary branch feature extractor use the same model structure, but their model parameters are different.

4. The multimodal information fusion reasoning method for remote sensing interpretation integrated machine according to claim 1 is characterized in that: In step S3, the feature fusion module is trained in the following manner: Step S311, in each round of training, respectively calculating the first cosine similarity of the two common branch features and the second cosine similarity of the two specific branch features; Step S312, jointly determining a loss value according to the first cosine similarity and the second cosine similarity, and using the loss value to constrain parameter updates of the feature fusion module in each round of training to obtain a trained feature fusion module.

5. The multimodal information fusion reasoning method for remote sensing interpretation integrated machine according to claim 4 is characterized in that: In step S312, the loss value is determined according to the following principles: The first cosine similarity is controlled to be as small as possible, and the second cosine similarity is controlled to be as large as possible.

6. The multimodal information fusion reasoning method for remote sensing interpretation integrated machine according to claim 1 is characterized in that: The step S322 specifically includes: According to the activation score of each of the two exclusive branch features at each spatial position, the exclusive branch feature with a higher activation score is selected at each spatial position for combination to obtain the fused exclusive branch feature.

7. A multi-modal information fusion reasoning device for a remote sensing interpretation integrated machine, characterized in that: include: A multimodal remote sensing image acquisition module, used to acquire two remote sensing images to be inferred, wherein the two remote sensing images are in the same scene and belong to different modalities; A dual-branch feature extraction module, used for inputting each of the remote sensing images into a pre-built dual-branch encoder for processing, to obtain a universal branch feature and a specific branch feature of each of the remote sensing images, wherein the universal branch feature reflects the common information of the two remote sensing images, and the specific branch feature reflects the specific information of each of the remote sensing images; A feature fusion module, used to fuse the two common branch features and the two specific branch features; The feature fusion module is used to fuse the two common branch features and the two specific branch features, specifically including: The two general branch features are connected along the channel dimension and then subjected to 1×1 convolution to obtain the fused general branch feature; Step S321, input each of the dedicated branch features into a parameter-sharing spatial activation function, and calculate an activation score at each position in the space; Step S322, according to the two activation scores, the two specific branch features are fused by spatial selection to obtain the fused specific branch feature; The multimodal remote sensing image reasoning module is used to sum the fused common branch features and the fused proprietary branch features and input them into the downstream task detection head to obtain the reasoning results of the two remote sensing images in the scene.

Images