Map reconstruction method and system based on multi-source information fusion

Abstract

The present application discloses a map reconstruction method and system based on multi-source information fusion, which relates to the field of intelligent construction. The method obtains an overhead image of a target area collected by a drone and extracts a satellite image of the target area from a database. The image processing and analysis technology based on computer vision is used to perform image analysis and feature extraction on the overhead image of the target area and the satellite image of the target area, so as to automatically generate a three-dimensional model of the target area based on the multi-modal fusion representation features of the overhead image of the target area and the satellite image of the target area. In this way, more accurate terrain and feature information can be provided, reducing the errors that may exist in a single data source, and at the same time expanding the coverage of the map, so that the generated results are more accurate and detailed.

Description

Technical Field

[0001] The present application relates to the field of intelligent construction, and more specifically, to a map reconstruction method and system based on multi-source information fusion. Background Art

[0002] In areas such as autonomous driving, robotic navigation, and intelligent transportation systems, there is a growing demand for automated map reconstruction to support accurate localization and path planning. Map reconstruction is a computer vision technique that involves extracting three-dimensional information from a series of images to build a detailed model of the environment. This process usually requires the use of specific sensors and algorithms to analyze the images and extract depth and position information from them. Map reconstruction has applications in many fields, such as robotic navigation, augmented reality, urban planning, game development, etc.

[0003] However, traditional map reconstruction may rely on a single type of data source, such as using only optical images or terrain data, which may not provide comprehensive geographical environment information. In addition, a single data source may not cover all areas, especially in areas with complex terrain or difficult to access, resulting in incomplete map information.

[0004] Therefore, a map reconstruction scheme based on multi-source information fusion is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a map reconstruction method based on multi-source information fusion is provided, which includes:

[0007] Obtaining overhead images of the target area collected by the drone;

[0008] Extracting a satellite image of the target area from a database;

[0009] Extracting target area image features from the overhead image of the target area and the satellite image of the target area to obtain a target area overhead image feature map and a target area satellite image feature map;

[0010] Inputting the target area overhead image feature map and the target area satellite image feature map into a collaborative attention module to obtain a collaborative attention enhanced target area overhead image feature map and a collaborative attention enhanced target area satellite image feature map;

[0011] Inputting the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature;

[0012] Based on the multimodal fusion representation features of the target area, a three-dimensional model of the target area is obtained.

[0013] In the above-mentioned map reconstruction method based on multi-source information fusion, target area image features are extracted from the overhead image of the target area and the satellite image of the target area respectively to obtain a target area overhead image feature map and a target area satellite image feature map, including: passing the overhead image of the target area and the satellite image of the target area through a target area image feature extractor based on a void convolutional neural network model respectively to obtain the target area overhead image feature map and the target area satellite image feature map.

[0014] In the above-mentioned map reconstruction method based on multi-source information fusion, the target area overhead image feature map and the target area satellite image feature map are input into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map, including: calculating the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map; normalizing the full-channel target area feature similarity matrix to obtain a normalized full-channel target area feature similarity matrix; calculating the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector based on the normalized full-channel target area feature similarity matrix; using the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector as weight vectors, respectively calculating the channel-by-channel product of the target area overhead image feature map and the target area satellite image feature map to obtain the target area collaborative attention weight vector. Obtain the aerial image feature map of the collaborative attention enhanced target area and the satellite image feature map of the collaborative attention enhanced target area; wherein, the full-channel target area feature similarity matrix is ​​normalized to obtain a normalized full-channel target area feature similarity matrix, including: taking each eigenvalue in the full-channel target area feature similarity matrix as an exponent of a natural constant to calculate a position-based exponential function value with the natural constant as the base to obtain a target area full-channel feature similarity class support matrix; calculating the sum of each column vector in the target area full-channel feature similarity class support matrix to obtain a target area full-channel feature similarity class support column global vector composed of multiple column vector global sum values; calculating the position-based division between each eigenvalue in the target area full-channel feature similarity class support matrix and the corresponding column vector global sum value in the target area full-channel feature similarity class support column global vector to obtain the normalized full-channel target area feature similarity matrix.

[0015] In the above-mentioned map reconstruction method based on multi-source information fusion, calculating the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map includes: reshaping the target area overhead image feature map and the target area satellite image feature map to obtain a target area overhead image feature matrix and a target area satellite image feature matrix; calculating the product between the transposed matrices of the target area overhead image feature matrix and the target area satellite image feature matrix to obtain the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map.

[0016] In the above-mentioned map reconstruction method based on multi-source information fusion, the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector are calculated based on the normalized full-channel target area feature similarity matrix, including: performing global mean pooling along the channel dimension on the target area overhead image feature map and the target area satellite image feature map to obtain the target area overhead image full-channel feature vector and the target area satellite image full-channel feature vector; calculating the matrix product between the normalized full-channel target area feature similarity matrix and the target area overhead image full-channel feature vector to obtain the target area overhead image collaborative attention weight vector; calculating the matrix product between the normalized full-channel target area feature similarity matrix and the target area satellite image full-channel feature vector to obtain the target area satellite image collaborative attention weight vector.

[0017] In the above-mentioned map reconstruction method based on multi-source information fusion, the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map are input into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature, including: cascading the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image to obtain a target area overhead-target area satellite tandem feature map; performing convolution encoding on the target area overhead-target area satellite tandem feature map to obtain a target area overhead-target area satellite association feature map; performing batch normalization processing on the target area overhead-target area satellite association feature map to obtain a normalized target area overhead-target area satellite association feature map; performing global pooling processing based on the maximum value on the normalized target area overhead-target area satellite association feature map. Global pooling processing of random values ​​and global pooling processing based on average values ​​are performed to obtain the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector; the position-weighted sum between the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector is calculated to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector; the eigenvalue of each position in the target area overhead shot-target area satellite associated multi-scale pooling representation vector is used as a weight to weight each feature matrix along the channel dimension of the collaborative attention enhanced target area overhead shot image feature map to obtain the target area multimodal fusion representation feature map.

[0018] In the above-mentioned map reconstruction method based on multi-source information fusion, the position-weighted sum of the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector is calculated to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector, including: the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector are added. The feature vectors are respectively multiplied by the corresponding adjustment parameters according to position to obtain the target area overhead shot-target area satellite global maximum modulation feature vector, the target area overhead shot-target area satellite global random value modulation feature vector and the target area overhead shot-target area satellite global mean modulation feature vector; the target area overhead shot-target area satellite global maximum modulation feature vector, the target area overhead shot-target area satellite global random value modulation feature vector and the target area overhead shot-target area satellite global mean modulation feature vector are added according to position to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector.

[0019] In the above-mentioned map reconstruction method based on multi-source information fusion, a three-dimensional model of the target area is obtained based on the multimodal fusion representation features of the target area, including: passing the multimodal fusion representation feature map of the target area through a three-dimensional modeler based on a generative adversarial network to obtain a three-dimensional model of the target area.

[0020] According to another aspect of the present application, a map reconstruction system based on multi-source information fusion is provided, comprising:

[0021] The overhead image acquisition module of the target area is used to obtain the overhead image of the target area collected by the drone;

[0022] A satellite image acquisition module for the target area, used to extract the satellite image of the target area from the database;

[0023] A target area image feature extraction module is used to extract target area image features from the overhead image of the target area and the satellite image of the target area to obtain a target area overhead image feature map and a target area satellite image feature map;

[0024] A collaborative attention enhancement module, used for inputting the target area overhead image feature map and the target area satellite image feature map into the collaborative attention module to obtain a collaborative attention enhancement target area overhead image feature map and a collaborative attention enhancement target area satellite image feature map;

[0025] A multi-channel feature fusion module, used for inputting the feature map of the aerial image of the collaborative attention enhanced target area and the feature map of the satellite image of the collaborative attention enhanced target area into the multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature;

[0026] The target area three-dimensional model generation module is used to obtain the target area three-dimensional model based on the multimodal fusion representation features of the target area.

[0027] In the above-mentioned map reconstruction system based on multi-source information fusion, the target area image feature extraction module is used to: respectively pass the overhead image of the target area and the satellite image of the target area through a target area image feature extractor based on a hole convolutional neural network model to obtain a feature map of the overhead image of the target area and a feature map of the satellite image of the target area.

[0028] Compared with the prior art, the present application provides a map reconstruction method and system based on multi-source information fusion, which obtains an overhead image of the target area collected by a drone and extracts a satellite image of the target area from a database. Image processing and analysis technology based on computer vision is used to perform image analysis and feature extraction on the overhead image of the target area and the satellite image of the target area, so as to automatically generate a three-dimensional model of the target area based on the multimodal fusion representation features of the overhead image of the target area and the satellite image of the target area. In this way, more accurate terrain and feature information can be provided, reducing the errors that may exist in a single data source, and at the same time expanding the coverage of the map, so that the generated results are more accurate and detailed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0030] Figure 1 Flow chart of a map reconstruction method based on multi-source information fusion according to an embodiment of the present application.

[0031] Figure 2 Schematic diagram of the architecture of a map reconstruction method based on multi-source information fusion according to an embodiment of the present application.

[0032] Figure 3The present invention is a flowchart of inputting the target area overhead image feature map and the target area satellite image feature map into a collaborative attention module in a map reconstruction method based on multi-source information fusion according to an embodiment of the present application to obtain a collaborative attention enhanced target area overhead image feature map and a collaborative attention enhanced target area satellite image feature map.

[0033] Figure 4 The present invention is a flowchart of inputting the collaborative attention enhanced target area aerial image feature map and the collaborative attention enhanced target area satellite image feature map into a multi-channel feature fusion module in a map reconstruction method based on multi-source information fusion according to an embodiment of the present application to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature.

[0034] Figure 5 The present invention is a flowchart of calculating the position-weighted sum of the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector in a map reconstruction method based on multi-source information fusion according to an embodiment of the present application to obtain a target area overhead shot-target area satellite associated multi-scale pooling representation vector.

[0035] Figure 6 4 is a block diagram of a map reconstruction system based on multi-source information fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0037] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0038] In the description of the embodiments of the present application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] In areas such as autonomous driving, robotic navigation, and intelligent transportation systems, there is a growing demand for automated map reconstruction to support accurate localization and path planning. Map reconstruction is a computer vision technique that involves extracting three-dimensional information from a series of images to build a detailed model of the environment. This process usually requires the use of specific sensors and algorithms to analyze the images and extract depth and position information from them. Map reconstruction has applications in many fields, such as robotic navigation, augmented reality, urban planning, game development, etc.

[0041] However, traditional map reconstruction may rely on a single type of data source, such as using only optical images or terrain data, which may not provide comprehensive geographical environment information. In addition, a single data source may not cover all areas, especially in areas with complex terrain or difficult to access, resulting in incomplete map information.

[0042] Based on this, the present application proposes a map reconstruction method based on multi-source information fusion, which obtains an overhead image of the target area collected by a drone and extracts a satellite image of the target area from a database. Image processing and analysis technology based on computer vision is used to perform image analysis and feature extraction on the overhead image of the target area and the satellite image of the target area, so as to automatically generate a three-dimensional model of the target area based on the multimodal fusion representation features of the overhead image of the target area and the satellite image of the target area. In this way, more accurate terrain and feature information can be provided, reducing the errors that may exist in a single data source, and at the same time expanding the coverage of the map, so that the generated results are more accurate and detailed.

[0043] Figure 1 Flow chart of a map reconstruction method based on multi-source information fusion according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of a map reconstruction method based on multi-source information fusion according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the map reconstruction method based on multi-source information fusion according to the embodiment of the present application, the method includes: S110, acquiring an overhead image of the target area collected by a drone; S120, extracting a satellite image of the target area from a database; S130, performing target area image feature extraction on the overhead image of the target area and the satellite image of the target area respectively to obtain a target area overhead image feature map and a target area satellite image feature map; S140, inputting the target area overhead image feature map and the target area satellite image feature map into a collaborative attention module to obtain a collaborative attention enhanced target area overhead image feature map and a collaborative attention enhanced target area satellite image feature map; S150, inputting the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature; S160, obtaining a target area three-dimensional model based on the target area multimodal fusion representation feature.

[0044] In step S110, an overhead image of the target area collected by a drone is obtained. In step S120, a satellite image of the target area is extracted from a database. It should be understood that the overhead image of the target area is taken by an unmanned aerial vehicle (UAV) at a lower flight altitude, and usually provides a high-resolution ground view that can show the details of the target area. The satellite image of the target area is taken by a satellite in Earth orbit, and usually covers a wider area, but the resolution may be lower than that of the image taken by the drone. Based on this, in order to provide more comprehensive geographic information and improve the accuracy and breadth of map reconstruction, in the technical solution of the present application, an overhead image of the target area collected by a drone is obtained, and a satellite image of the target area is extracted from a database.

[0045] In step S130, target area image feature extraction is performed on the overhead image of the target area and the satellite image of the target area respectively to obtain a target area overhead image feature map and a target area satellite image feature map. Specifically, in an embodiment of the present application, target area image feature extraction is performed on the overhead image of the target area and the satellite image of the target area respectively to obtain a target area overhead image feature map and a target area satellite image feature map, including: passing the overhead image of the target area and the satellite image of the target area through a target area image feature extractor based on a hole convolutional neural network model to obtain the target area overhead image feature map and the target area satellite image feature map. Accordingly, considering that both the overhead image of the target area and the satellite image of the target area contain implicit feature information of different scales about the target area, the hole convolutional neural network can effectively expand the receptive field, allowing the network to capture a wider range of information without increasing parameters and computational complexity, thereby being able to extract multi-scale image features. Therefore, in the technical solution of the present application, the overhead image of the target area and the satellite image of the target area are respectively passed through a target area image feature extractor based on a hole convolutional neural network model to capture and mine implicit feature information of different scales in the overhead image and the satellite image, thereby generating more representative and rich target area overhead image feature map and target area satellite image feature map.

[0046] In step S140, the target area overhead image feature map and the target area satellite image feature map are input into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map. It should be understood that, considering that the target area overhead image feature map provides a detailed view of the target area, it can show the characteristics of the ground to highlight the important visual information of the target area. The target area satellite image feature map provides a macroscopic view of the target area for describing a wider range of geographical features. At the same time, there is similarity information about the characteristics of the target area between the target area overhead image feature map and the target area satellite image feature map. Therefore, in order to better enhance the target area overhead image feature map and the target area satellite image feature map based on the similarity between the features of the two different sources, in the technical solution of the present application, the target area overhead image feature map and the target area satellite image feature map are input into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map. Specifically, the collaborative attention module calculates the similarity between the feature map of the overhead image of the target area and the feature map of the satellite image of the target area in all channels to determine which features have similar representations in the two images from different sources, so as to help the system pay more attention to the interrelated features in the overhead image and the satellite image of the target area, and then calculates the collaborative attention weight vectors of the feature map of the overhead image of the target area and the feature map of the satellite image of the target area based on the similarity information, and weights the feature map of the overhead image of the target area and the feature map of the satellite image of the target area accordingly, so that the model pays more attention to those image feature information that plays a key role in map reconstruction, so as to improve the importance and representation ability of the target area features, thereby generating a more accurate and comprehensive feature representation, and providing a better foundation for the generation of a three-dimensional model.

[0047] Figure 3 The present invention is a flowchart of inputting the target area overhead image feature map and the target area satellite image feature map into the collaborative attention module in the map reconstruction method based on multi-source information fusion according to the embodiment of the present application to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map. Specifically, in the embodiment of the present application, Figure 3As shown, the target area overhead image feature map and the target area satellite image feature map are input into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map, including: S210, calculating the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map; S220, normalizing the full-channel target area feature similarity matrix to obtain a normalized full-channel target area feature similarity matrix; S230, calculating the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector based on the normalized full-channel target area feature similarity matrix; S240, using the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector as weight vectors, respectively calculating the channel-by-channel product of the target area overhead image feature map and the target area satellite image feature map to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map.

[0048] More specifically, in an embodiment of the present application, calculating the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map includes: reshaping the target area overhead image feature map and the target area satellite image feature map to obtain the target area overhead image feature matrix and the target area satellite image feature matrix; calculating the product between the transposed matrices of the target area overhead image feature matrix and the target area satellite image feature matrix to obtain the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map. In particular, the size of the target area overhead image feature map and the target area satellite image feature map is (C×H×W), and the size of the target area overhead image feature matrix and the target area satellite image feature matrix is , where C, H, and W represent the number of channels, height, and width of the feature, respectively.

[0049] More specifically, in an embodiment of the present application, the full-channel target area feature similarity matrix is ​​normalized to obtain a normalized full-channel target area feature similarity matrix, including: taking each eigenvalue in the full-channel target area feature similarity matrix as an exponent of a natural constant to calculate a position-based exponential function value with the natural constant as the base to obtain a target area full-channel feature similarity class support matrix; calculating the sum of each column vector in the target area full-channel feature similarity class support matrix to obtain a target area full-channel feature similarity class support column global vector composed of multiple column vector global sum values; calculating the position-based division between each eigenvalue in the target area full-channel feature similarity class support matrix and the corresponding column vector global sum value in the target area full-channel feature similarity class support column global vector to obtain the normalized full-channel target area feature similarity matrix.

[0050] More specifically, in an embodiment of the present application, the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector are calculated based on the normalized full-channel target area feature similarity matrix, including: performing global mean pooling along the channel dimension on the target area overhead image feature map and the target area satellite image feature map to obtain the target area overhead image full-channel feature vector and the target area satellite image full-channel feature vector; calculating the matrix product between the normalized full-channel target area feature similarity matrix and the target area overhead image full-channel feature vector to obtain the target area overhead image collaborative attention weight vector; calculating the matrix product between the normalized full-channel target area feature similarity matrix and the target area satellite image full-channel feature vector to obtain the target area satellite image collaborative attention weight vector.

[0051] In an embodiment of the present application, specifically, inputting the target area overhead image feature map and the target area satellite image feature map into a collaborative attention module to obtain a collaborative attention enhanced target area overhead image feature map and a collaborative attention enhanced target area satellite image feature map, including: using the collaborative attention module to process the target area overhead image feature map and the target area satellite image feature map with the following attention formula to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map; wherein the attention formula is: ; ; ; ; ; ; ; ; ; ;in, and are respectively a feature map of the overhead image of the target area and a feature map of the satellite image of the target area, For shape reshaping, and are respectively the target area overhead image feature matrix and the target area satellite image feature matrix, is the full-channel target region feature similarity matrix, is the similarity matrix of the full channel target area features The characteristic value of is the similarity matrix of the full channel target area features The characteristic value of represents the number of columns of the full-channel target area feature similarity matrix, The natural constant The exponential operation with base , is the eigenvalue of each position in the normalized full-channel target region feature similarity matrix, It is to perform global mean pooling on the feature map. and are respectively the full-channel feature vector of the overhead image of the target area and the full-channel feature vector of the satellite image of the target area, is the normalized full-channel target region feature similarity matrix, and are respectively the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector, and They are respectively the aerial image feature map of the collaborative attention and intensification target area and the satellite image feature map of the collaborative attention and intensification target area, Represents matrix multiplication.

[0052] In step S150, the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map are input into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature. Accordingly, considering that the collaborative attention enhanced target area overhead image feature map overhead image usually has a higher resolution, it can provide more detail information, but the understanding of the semantic information of the overall scene is not as comprehensive as that of the satellite image. The collaborative attention enhanced target area satellite image feature map usually has a lower resolution, so it may not be as rich in details as the overhead image, but it can provide more semantic information. Based on this, in order to make full use of the respective advantages of the two, the target area is described more comprehensively to improve the accuracy and effect of map reconstruction. In the technical solution of the present application, the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map are input into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map. That is to say, the multi-channel feature fusion module can effectively fuse the feature map of the aerial image of the collaboratively focused and enhanced target area with different feature information from different sources and the feature map of the satellite image of the collaboratively focused and enhanced target area, and make full use of the advantages of the features of the two image sources, so as to better retain their respective characteristics and make up for their respective shortcomings, so as to obtain a more comprehensive and representative multimodal fusion feature representation of the target area, thereby improving the quality of map reconstruction.

[0053] Figure 4 The present invention is a flowchart of inputting the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map into a multi-channel feature fusion module in a map reconstruction method based on multi-source information fusion according to an embodiment of the present application to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature. Specifically, in the embodiment of the present application, Figure 4As shown, the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map are input into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature, including: S310, cascading the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image to obtain a target area overhead-target area satellite tandem feature map; S320, performing convolution coding on the target area overhead-target area satellite tandem feature map to obtain a target area overhead-target area satellite association feature map; S330, performing batch normalization processing on the target area overhead-target area satellite association feature map to obtain a normalized target area overhead-target area satellite association feature map; S340, performing global pooling processing based on the maximum value and random value based global pooling processing on the normalized target area overhead-target area satellite association feature map. Global pooling processing and average-based global pooling processing are performed to obtain the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector; S350, calculating the position-weighted sum between the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector; S360, using the eigenvalue of each position in the target area overhead shot-target area satellite associated multi-scale pooling representation vector as a weight, weighting each feature matrix along the channel dimension of the collaborative attention enhanced target area overhead shot image feature map to obtain the target area multimodal fusion representation feature map.

[0054] Figure 5 The flowchart is a method for calculating the weighted sum of the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector according to the map reconstruction method based on multi-source information fusion in the embodiment of the present application to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector. More specifically, in the embodiment of the present application, Figure 5As shown, the position-weighted sum of the target area overhead-target area satellite global maximum pooling feature vector, the target area overhead-target area satellite global random value pooling feature vector and the target area overhead-target area satellite global mean pooling feature vector is calculated to obtain a target area overhead-target area satellite associated multi-scale pooling representation vector, including: S410, the target area overhead-target area satellite global maximum pooling feature vector, the target area overhead-target area satellite global random value pooling feature vector and the target area overhead-target area satellite global mean pooling feature vector are respectively compared with The corresponding adjustment parameters are multiplied by position to obtain the target area overhead shot-target area satellite global maximum modulation feature vector, the target area overhead shot-target area satellite global random value modulation feature vector and the target area overhead shot-target area satellite global mean modulation feature vector; S420, the target area overhead shot-target area satellite global maximum modulation feature vector, the target area overhead shot-target area satellite global random value modulation feature vector and the target area overhead shot-target area satellite global mean modulation feature vector are added by position to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector.

[0055] In an embodiment of the present application, specifically, the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map are input into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature, including: using the multi-channel feature fusion module to process the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map using the following multimodal fusion formula to obtain the target area multimodal fusion representation feature map; wherein the multimodal fusion formula is: ; ; ; ;in, and They are respectively the aerial image feature map of the collaborative attention and intensification target area and the satellite image feature map of the collaborative attention and intensification target area, It is to perform cascade processing on the feature map. is the target area aerial view-target area satellite series characteristic map, It is convolutional encoding of feature maps. represents batch normalization processing, is the normalized target area aerial view-target area satellite correlation feature map, , and The feature matrices of the feature map along the channel dimension are subjected to global pooling processing based on the average value, global pooling processing based on the maximum value, and global pooling processing based on random values. , and are the corresponding adjustment parameters, is the target area aerial view-target area satellite associated multi-scale pooling representation vector, is the multimodal fusion representation feature map of the target area, Indicates that a vector is used to perform a weighted operation on the feature map.

[0056] In step S160, based on the multimodal fusion representation features of the target area, a three-dimensional model of the target area is obtained. Specifically, in an embodiment of the present application, based on the multimodal fusion representation features of the target area, a three-dimensional model of the target area is obtained, including: passing the multimodal fusion representation feature map of the target area through a three-dimensional modeler based on a generative adversarial network to obtain the three-dimensional model of the target area. That is, the multimodal fusion representation features of the target area obtained by multi-channel fusion of the collaborative attention-enhanced target area overhead image feature map and the collaborative attention-enhanced target area satellite image feature map are classified and processed to automatically generate a three-dimensional model of the target area. In this way, more accurate terrain and feature information can be provided, and possible errors in a single data source can be reduced. At the same time, the coverage of the map can be expanded to make the generated results more accurate and detailed.

[0057] It should be understood that, here, considering that the target area overhead image feature map and the target area satellite image feature map respectively represent the image semantic features of the overhead image and the satellite image of the target area, when performing collaborative attention enhancement and feature fusion, local fusion overflow features will be caused due to the difference in attention weights, thereby affecting the accuracy of the target area three-dimensional model obtained by the target area multimodal fusion representation feature map through the three-dimensional modeler based on the adversarial generative network.

[0058] Based on this, preferably, the target area multimodal fusion representation feature map is passed through a three-dimensional modeler based on a generative adversarial network to obtain a target area three-dimensional model, which includes the following steps: expanding the target area multimodal fusion representation feature map into a target area multimodal fusion representation feature vector; calculating the target area multimodal fusion and representation matrix and the target area multimodal fusion difference representation matrix of the target area multimodal fusion representation feature vector, wherein the first The value of the position is the first and The mean of the eigenvalues ​​at the position, and the multimodal fusion difference representation matrix of the target area The value of the position is the first and The absolute value of the difference between the eigenvalues ​​of the positions; performing matrix multiplication on the transposed vector of the target region multimodal fusion representation feature vector and the target region multimodal fusion and representation matrix to obtain the target region multimodal fusion and representation vector, and performing matrix multiplication on the target region multimodal fusion difference representation matrix and the target region multimodal fusion representation feature vector to obtain the target region multimodal fusion difference representation vector, wherein the target region multimodal fusion representation feature vector is a column vector; calculating the matrix product of the target region multimodal fusion and representation matrix and the target region multimodal fusion difference representation matrix, and performing matrix multiplication on the target region multimodal fusion representation feature vector; After matrix multiplication of the transposed vector of the modal fusion representation feature vector and the matrix product, dot addition is performed with the transposed vectors of the target area multimodal fusion sum representation vector and the target area multimodal fusion difference representation vector to obtain a corrected target area multimodal fusion representation feature vector; the corrected target area multimodal fusion representation feature vector is restored to a corrected target area multimodal fusion representation feature map according to the expansion of the target area multimodal fusion representation feature map; the corrected target area multimodal fusion representation feature map is passed through the three-dimensional modeler based on the generative adversarial network to obtain the three-dimensional model of the target area.

[0059] That is, by using the sum representation matrix and the difference representation matrix of the group aggregation statistical evaluation of the local distribution of the feature value granularity of the target area multimodal fusion representation feature map as the retrieval and response distribution enhancement of the target area multimodal fusion representation feature map, and constructing a reference-free distribution retrieval response framework under the open domain of the group aggregation sum-difference feature distribution of the target area multimodal fusion representation feature map, the distribution response redundancy caused by the local overflow characteristics of the target area multimodal fusion representation feature map is avoided through response superposition, so as to realize the faithfulness constraint of the self-aggregation statistics correlation of the response of the target area multimodal fusion representation feature map to the target generation regression domain, and improve the accuracy of the target area three-dimensional model obtained by the target area multimodal fusion representation feature map through the three-dimensional modeler based on the adversarial generation network. In this way, more accurate terrain and feature information can be provided, the possible errors of a single data source can be reduced, and the coverage of the map can be expanded to make the generated results more accurate and detailed.

[0060] In summary, a map reconstruction method based on multi-source information fusion based on an embodiment of the present application is explained, which obtains an overhead image of the target area collected by a drone and extracts a satellite image of the target area from a database. Image processing and analysis technology based on computer vision is used to perform image analysis and feature extraction on the overhead image of the target area and the satellite image of the target area, so as to automatically generate a three-dimensional model of the target area based on the multimodal fusion representation features of the overhead image of the target area and the satellite image of the target area. In this way, more accurate terrain and feature information can be provided, reducing possible errors in a single data source, and at the same time expanding the coverage of the map so that the generated results are more accurate and detailed.

[0061] Figure 6 FIG. 1 is a block diagram of a map reconstruction system based on multi-source information fusion according to an embodiment of the present application. Figure 6 As shown, according to the map reconstruction system 100 based on multi-source information fusion according to the embodiment of the present application, it includes: an overhead image acquisition module 110 of the target area, which is used to obtain the overhead image of the target area acquired by the drone; a satellite image acquisition module 120 of the target area, which is used to extract the satellite image of the target area from the database; a target area image feature extraction module 130, which is used to extract the target area image features from the overhead image of the target area and the satellite image of the target area respectively to obtain the target area overhead image feature map and the target area satellite image feature map; a collaborative attention enhancement module 140, which is used to extract the target area The overhead image feature map and the target area satellite image feature map are input into the collaborative attention module to obtain the collaborative attention-enhanced target area overhead image feature map and the collaborative attention-enhanced target area satellite image feature map; the multi-channel feature fusion module 150 is used to input the collaborative attention-enhanced target area overhead image feature map and the collaborative attention-enhanced target area satellite image feature map into the multi-channel feature fusion module to obtain the target area multimodal fusion representation feature map as the target area multimodal fusion representation feature; the target area three-dimensional model generation module 160 is used to obtain the target area three-dimensional model based on the target area multimodal fusion representation feature.

[0062] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned map reconstruction system based on multi-source information fusion have been described in the above reference. Figures 1 to 5 The description of the map reconstruction method based on multi-source information fusion has been introduced in detail, and therefore, its repeated description will be omitted.

[0063] As described above, the map reconstruction system 100 based on multi-source information fusion according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a map reconstruction algorithm based on multi-source information fusion. In a possible implementation, the map reconstruction system 100 based on multi-source information fusion according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the map reconstruction system 100 based on multi-source information fusion can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the map reconstruction system 100 based on multi-source information fusion can also be one of the many hardware modules of the wireless terminal.

[0064] Alternatively, in another example, the map reconstruction system 100 based on multi-source information fusion and the wireless terminal may also be separate devices, and the map reconstruction system 100 based on multi-source information fusion may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0065] The above is only an example of the principles of the present application, and various modifications may be made by those skilled in the art without departing from the scope of the present application. The above embodiments are presented for the purpose of illustration rather than limitation. The present application may also take many forms other than those explicitly described herein. Therefore, it should be emphasized that the present application is not limited to the methods, systems and devices explicitly disclosed, but is intended to include variations and modifications within the spirit and scope of the appended claims.

Claims

1. A map reconstruction method based on multi-source information fusion, characterized in that: include: Obtaining overhead images of the target area collected by the drone; Extracting a satellite image of the target area from a database; Extracting target area image features from the overhead image of the target area and the satellite image of the target area to obtain a target area overhead image feature map and a target area satellite image feature map; Inputting the target area overhead image feature map and the target area satellite image feature map into a collaborative attention module to obtain a collaborative attention enhanced target area overhead image feature map and a collaborative attention enhanced target area satellite image feature map; Inputting the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature; Based on the multimodal fusion representation features of the target area, a three-dimensional model of the target area is obtained; The step of inputting the target area overhead image feature map and the target area satellite image feature map into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map comprises: Calculating a full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map; Normalizing the full-channel target region feature similarity matrix to obtain a normalized full-channel target region feature similarity matrix; Calculate the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector based on the normalized full-channel target area feature similarity matrix; Taking the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector as weight vectors, respectively calculating the channel-by-channel product of the target area overhead image feature map and the target area satellite image feature map to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map; Among them, normalizing the full-channel target area feature similarity matrix to obtain a normalized full-channel target area feature similarity matrix includes: Taking each eigenvalue in the full-channel target region feature similarity matrix as an exponent of a natural constant, calculating an exponential function value based on the natural constant according to position to obtain a target region full-channel feature similarity class support matrix; Calculate the sum of each column vector in the target region full channel feature similarity class support matrix to obtain a target region full channel feature similarity class support column global vector composed of a plurality of column vector global sum values; Calculate the positional division between each eigenvalue in the target region full channel feature similarity class support matrix and the corresponding column vector global sum value in the target region full channel feature similarity class support column global vector to obtain the normalized full channel target region feature similarity matrix; The method includes inputting the aerial image feature map of the collaborative attention enhanced target area and the satellite image feature map of the collaborative attention enhanced target area into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature, including: Cascading the overhead image feature map of the collaborative attention and enhanced target area and the satellite image of the collaborative attention and enhanced target area to obtain a target area overhead-target area satellite tandem feature map; Convolution encoding is performed on the target area overhead shot-target area satellite tandem feature map to obtain a target area overhead shot-target area satellite associated feature map; Batch normalization processing is performed on the target area overhead shot-target area satellite correlation feature map to obtain a normalized target area overhead shot-target area satellite correlation feature map; Performing maximum-based global pooling processing, random-value-based global pooling processing, and average-based global pooling processing on the normalized target area overhead-target area satellite association feature map to obtain a target area overhead-target area satellite global maximum pooling feature vector, a target area overhead-target area satellite global random value pooling feature vector, and a target area overhead-target area satellite global average pooling feature vector; Calculate the position-weighted sum of the target area overhead-target area satellite global maximum pooling feature vector, the target area overhead-target area satellite global random value pooling feature vector and the target area overhead-target area satellite global mean pooling feature vector to obtain a target area overhead-target area satellite associated multi-scale pooling representation vector; Taking the eigenvalues ​​of each position in the target area overhead view-target area satellite associated multi-scale pooling representation vector as weights, each feature matrix along the channel dimension of the collaborative attention enhanced target area overhead view image feature map is weighted to obtain the target area multimodal fusion representation feature map.

2. The map reconstruction method based on multi-source information fusion according to claim 1, characterized in that: The overhead image of the target area and the satellite image of the target area are respectively subjected to target area image feature extraction to obtain a target area overhead image feature map and a target area satellite image feature map, including: the overhead image of the target area and the satellite image of the target area are respectively subjected to a target area image feature extractor based on a hole convolutional neural network model to obtain the target area overhead image feature map and the target area satellite image feature map.

3. The map reconstruction method based on multi-source information fusion according to claim 2 is characterized in that: Calculating a full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map, including: Reshape the target area overhead image feature map and the target area satellite image feature map to obtain a target area overhead image feature matrix and a target area satellite image feature matrix; The product between the target area overhead image feature matrix and the transposed matrix of the target area satellite image feature matrix is ​​calculated to obtain the full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map.

4. The map reconstruction method based on multi-source information fusion according to claim 3 is characterized in that: Calculating the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector based on the normalized full-channel target area feature similarity matrix includes: Performing global mean pooling along the channel dimension on the target area overhead image feature map and the target area satellite image feature map to obtain a target area overhead image full-channel feature vector and a target area satellite image full-channel feature vector; Calculating the matrix product between the normalized full-channel target area feature similarity matrix and the full-channel feature vector of the target area overhead image to obtain the target area overhead image collaborative attention weight vector; The matrix product between the normalized full-channel target area feature similarity matrix and the full-channel feature vector of the target area satellite image is calculated to obtain the target area satellite image collaborative attention weight vector.

5. The map reconstruction method based on multi-source information fusion according to claim 4 is characterized in that: Calculating the weighted sum by position between the target area overhead shot-target area satellite global maximum pooling feature vector, the target area overhead shot-target area satellite global random value pooling feature vector and the target area overhead shot-target area satellite global mean pooling feature vector to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector, including: The target area overhead-target area satellite global maximum pooling feature vector, the target area overhead-target area satellite global random value pooling feature vector and the target area overhead-target area satellite global mean pooling feature vector are respectively multiplied by position with the corresponding adjustment parameters to obtain the target area overhead-target area satellite global maximum modulation feature vector, the target area overhead-target area satellite global random value modulation feature vector and the target area overhead-target area satellite global mean modulation feature vector; The target area overhead shot-target area satellite global maximum modulation feature vector, the target area overhead shot-target area satellite global random value modulation feature vector and the target area overhead shot-target area satellite global mean modulation feature vector are added according to position to obtain the target area overhead shot-target area satellite associated multi-scale pooling representation vector.

6. The map reconstruction method based on multi-source information fusion according to claim 5, characterized in that: Based on the multimodal fusion representation features of the target area, a three-dimensional model of the target area is obtained, including: passing the multimodal fusion representation feature map of the target area through a three-dimensional modeler based on a generative adversarial network to obtain the three-dimensional model of the target area.

7. A map reconstruction system based on multi-source information fusion, characterized in that: include: The overhead image acquisition module of the target area is used to obtain the overhead image of the target area collected by the drone; A satellite image acquisition module for the target area, used to extract the satellite image of the target area from the database; A target area image feature extraction module is used to extract target area image features from the overhead image of the target area and the satellite image of the target area to obtain a target area overhead image feature map and a target area satellite image feature map; A collaborative attention enhancement module, used for inputting the target area overhead image feature map and the target area satellite image feature map into the collaborative attention module to obtain a collaborative attention enhancement target area overhead image feature map and a collaborative attention enhancement target area satellite image feature map; A multi-channel feature fusion module, used for inputting the feature map of the aerial image of the collaborative attention enhanced target area and the feature map of the satellite image of the collaborative attention enhanced target area into the multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature; A target area three-dimensional model generation module, used to obtain a target area three-dimensional model based on the multi-modal fusion representation features of the target area; The step of inputting the target area overhead image feature map and the target area satellite image feature map into the collaborative attention module to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map comprises: Calculating a full-channel target area feature similarity matrix between the target area overhead image feature map and the target area satellite image feature map; Normalizing the full-channel target region feature similarity matrix to obtain a normalized full-channel target region feature similarity matrix; Calculate the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector based on the normalized full-channel target area feature similarity matrix; Taking the target area overhead image collaborative attention weight vector and the target area satellite image collaborative attention weight vector as weight vectors, respectively calculating the channel-by-channel product of the target area overhead image feature map and the target area satellite image feature map to obtain the collaborative attention enhanced target area overhead image feature map and the collaborative attention enhanced target area satellite image feature map; Among them, normalizing the full-channel target area feature similarity matrix to obtain a normalized full-channel target area feature similarity matrix includes: Taking each eigenvalue in the full-channel target region feature similarity matrix as an exponent of a natural constant, calculating an exponential function value based on the natural constant according to position to obtain a target region full-channel feature similarity class support matrix; Calculate the sum of each column vector in the target region full channel feature similarity class support matrix to obtain a target region full channel feature similarity class support column global vector composed of a plurality of column vector global sum values; Calculate the positional division between each eigenvalue in the target region full channel feature similarity class support matrix and the corresponding column vector global sum value in the target region full channel feature similarity class support column global vector to obtain the normalized full channel target region feature similarity matrix; The method includes inputting the aerial image feature map of the collaborative attention enhanced target area and the satellite image feature map of the collaborative attention enhanced target area into a multi-channel feature fusion module to obtain a target area multimodal fusion representation feature map as a target area multimodal fusion representation feature, including: Cascading the overhead image feature map of the collaborative attention and enhanced target area and the satellite image of the collaborative attention and enhanced target area to obtain a target area overhead-target area satellite tandem feature map; Convolution encoding is performed on the target area overhead shot-target area satellite tandem feature map to obtain a target area overhead shot-target area satellite associated feature map; Batch normalization processing is performed on the target area overhead shot-target area satellite correlation feature map to obtain a normalized target area overhead shot-target area satellite correlation feature map; Performing maximum-based global pooling processing, random-value-based global pooling processing, and average-based global pooling processing on the normalized target area overhead-target area satellite association feature map to obtain a target area overhead-target area satellite global maximum pooling feature vector, a target area overhead-target area satellite global random value pooling feature vector, and a target area overhead-target area satellite global average pooling feature vector; Calculate the position-weighted sum of the target area overhead-target area satellite global maximum pooling feature vector, the target area overhead-target area satellite global random value pooling feature vector and the target area overhead-target area satellite global mean pooling feature vector to obtain a target area overhead-target area satellite associated multi-scale pooling representation vector; Taking the eigenvalues ​​of each position in the target area overhead view-target area satellite associated multi-scale pooling representation vector as weights, each feature matrix along the channel dimension of the collaborative attention enhanced target area overhead view image feature map is weighted to obtain the target area multimodal fusion representation feature map.

8. The map reconstruction system based on multi-source information fusion according to claim 7, characterized in that: The target area image feature extraction module is used to: respectively pass the overhead image of the target area and the satellite image of the target area through a target area image feature extractor based on a hole convolutional neural network model to obtain a feature map of the overhead image of the target area and a feature map of the satellite image of the target area.

Images