Method for displaying geographical entity information based on real scene three-dimensional modeling

Abstract

The present application relates to the technical field of geographic information display, in particular to a geographic entity information display method based on real scene three-dimensional modeling, comprising: acquiring multi-angle remote sensing images of the geographic entity region, performing target detection on the multi-angle remote sensing images to obtain each geographic entity type in the multi-angle remote sensing images; matching the geographic entity type with a preset geographic knowledge graph to obtain geographic information corresponding to the geographic entity type. Through processing of the multi-angle remote sensing images, the present application can effectively identify geographic entities, and extract more accurate geographic information according to different perspectives. Through the process, the accuracy of geographic entity classification and identification can be improved, and the geographic entity type is matched with the geographic knowledge graph, so that each geographic entity not only has a description of the physical space position, but also can be combined with rich semantic information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geographic information display, and particularly relates to a geographic entity information display method based on real scene three-dimensional modeling. BACKGROUND

[0002] A geographic entity refers to various things or objects existing in geographic space, usually having a specific spatial position, shape and attribute, and can be positioned by geographic coordinates. The geographic entity not only includes geographic features in nature, but also includes various human facilities or phenomena produced by human activities.

[0003] Traditional methods usually rely on relatively simple image processing or manual measurement methods, and the recognition accuracy is limited by the quality, resolution and analysis algorithm of remote sensing images. It is often difficult to accurately identify geographic entities (such as buildings, roads, bridges, etc.) in complex terrain or different lighting conditions. Traditional methods may lack consideration of different perspectives, resulting in insufficient accuracy of geographic entity classification and recognition. Moreover, traditional methods usually only focus on the physical location and basic attributes (such as coordinates, size, etc.) of geographic entities, lacking semantic understanding (such as function, history, environmental impact, etc.) of geographic entities. The lack of such information makes traditional geographic data display difficult to meet the needs of in-depth analysis and practical application, especially in the fields of urban planning and environmental monitoring, which cannot provide comprehensive decision support. Moreover, traditional methods often only provide planar data or simple three-dimensional models of geographic entities, lacking deep spatial analysis capabilities, which makes it difficult for traditional methods to provide sufficient data support for complex spatial analysis, trend prediction and decision support, affecting the scientificity and effectiveness of decision-making. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art and provide a geographic entity information display method based on real scene three-dimensional modeling.

[0005] The technical solution adopted to solve the above technical problem is: a geographic entity information display method based on real scene three-dimensional modeling, comprising:

[0006] Obtaining multi-angle remote sensing images of the geographic entity region, and performing target detection on the multi-angle remote sensing images to obtain each geographic entity type in the multi-angle remote sensing images;

[0007] Matching the geographic entity type with a preset geographic knowledge graph to obtain geographic information corresponding to the geographic entity type;

[0008] Performing three-dimensional reconstruction on the geographic entity region based on the multi-angle remote sensing images to obtain a real scene three-dimensional model corresponding to the geographic entity region.

[0009] establishing a position mapping table between the real three-dimensional model and the multi-angle remote sensing image, mapping each geographic entity type in the multi-angle remote sensing image to the real three-dimensional model based on the position mapping table, to obtain a dynamic real three-dimensional model;

[0010] integrating the dynamic real three-dimensional model based on geographic information corresponding to the geographic entity type, to obtain a dynamic information real three-dimensional model, and displaying the dynamic information real three-dimensional model based on a three-dimensional rendering engine.

[0011] Preferably, target detection is performed on the multi-angle remote sensing image to obtain each geographic entity type in the multi-angle remote sensing image, including:

[0012] image enhancement is performed on the multi-angle remote sensing image based on a pre-trained image enhancement model, to obtain multi-angle standard remote sensing images;

[0013] target detection is performed on the multi-angle remote sensing image based on a pre-trained target detection model, to obtain each geographic entity type in the multi-angle remote sensing image.

[0014] Preferably, the target detection model adopts an improved YOLOv7-tiny model, and a visual channel attention mechanism is introduced into a backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used for global average pooling operation on an input feature map, and the size of the feature map is C*H*W, wherein C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map, and the expression of the visual channel attention mechanism is as follows:

[0015]

[0016] wherein z c represents the output of the visual channel attention mechanism, u c (i,j) represents the output of the feature map pixel (i,j).

[0017] Preferably, the image enhancement model comprises a shared encoding module, an edge decoding module, a color decoding module and a feature fusion module, the shared encoding module is used for feature extraction on the multi-angle remote sensing image to obtain edge features and color features corresponding to the multi-angle remote sensing image, the edge decoding module is used for decoding the edge features corresponding to the multi-angle remote sensing image based on the edge block and the image block to obtain image edge results of the multi-angle remote sensing image, the color decoding module is used for decoding the color features of the multi-angle remote sensing image based on the HSV color map to obtain image color results of the multi-angle remote sensing image, and the feature fusion module is used for splicing the image edge results and the image color results of the multi-angle remote sensing image to obtain a spliced feature map, and performing feature fusion on the spliced feature map to obtain a standard multi-angle remote sensing image.

[0018] Preferably, the three-dimensional reconstruction of the geographic entity region based on the multi-angle remote sensing image to obtain a real scene three-dimensional model corresponding to the geographic entity region comprises:

[0019] The multi-angle remote sensing image is subjected to distortion correction to obtain a multi-angle corrected remote sensing image.

[0020] The aerial triangulation in the geographic entity region is analyzed based on photogrammetry to obtain exterior orientation elements corresponding to the multi-angle remote sensing image.

[0021] The geometric information of each geographic entity in the multi-angle remote sensing image is extracted based on the exterior orientation elements according to a solution matrix of adjustment;

[0022] The geometric information of each geographic entity is converted into three-dimensional point cloud data based on a pre-trained three-dimensional reconstruction model, and the real scene three-dimensional model corresponding to the geographic entity region is obtained based on the three-dimensional point cloud data.

[0023] Preferably, the multi-angle remote sensing image is subjected to distortion correction to obtain a multi-angle corrected remote sensing image, comprising:

[0024] The multi-angle remote sensing image is subjected to camera distortion correction based on a point coordinate transformation model to obtain a multi-angle camera distortion corrected remote sensing image, wherein the calculation formula of the camera distortion correction is as follows:

[0025]

[0026] wherein (X, Y) represents the real pixel point coordinates of the multi-angle remote sensing image, (F X ,F Y ) represents the coordinate transformation function in the camera distortion correction, and (α, β) represents the multi-angle camera distortion corrected remote sensing image.

[0027] The multi-angle camera distortion rectification remote sensing image is subjected to geometric correction processing to obtain a multi-angle rectification remote sensing image, wherein the calculation formula of the geometric correction processing is as follows:

[0028] G Q (x Q ,y Q )=G N (x N ,y N )(X,Y);

[0029] wherein G Q (x Q ,y Q ) represents the gray value of the resampling point Q in the multi-angle camera distortion rectification remote sensing image, G N (x N ,y N ) represents the corresponding gray value of the nearest pixel to the resampling point Q, (x Q ,y Q ) represents the gray coordinate value of G Q (x Q ,y Q ), and (x N ,y N ) represents the gray coordinate value of G N (x N ,y N ).

[0030] Preferably, the three-dimensional reconstruction model comprises a feature extraction module, a feature aggregation module, a feature decomposition module and a feature regression module, the feature extraction module is used to extract first local features of geometric information of a preset multi-angle reference image and second local features of geometric information of the multi-angle remote sensing image, and convert the second local features into a view angle where the first local features are located to obtain third local features, and the first local features and the third local features are matched two by two to obtain a plurality of cost volumes.

[0031] Preferably, the feature aggregation module adopts a feature pyramid network, and the plurality of cost volumes are aggregated based on the feature pyramid network to obtain an aggregated cost volume, the feature decomposition module is used to down-sample the aggregated cost volume to obtain high-dimensional features, and then perform a capsule operation according to a feature channel dimension to decouple the aggregated cost volume into a plurality of capsule blocks, the feature regression module is used to encode high-dimensional features in the plurality of capsule blocks through a preset capsule convolutional layer to obtain a plurality of high-dimensional feature semantics, splice the plurality of high-dimensional feature semantics with the plurality of capsule blocks to obtain a plurality of spliced features, and regress the plurality of spliced features through a normalization operation to obtain a plurality of depth maps, and convert the plurality of depth maps into three-dimensional point cloud data.

[0032] Preferably, the position mapping table between the real three-dimensional model and the multi-angle remote sensing image is established, comprising:

[0033] Obtaining three-dimensional point cloud data of the real three-dimensional model, and extracting feature points from the three-dimensional point cloud data to obtain a first set of key feature points;

[0034] Extracting feature points from the multi-angle remote sensing image to obtain a second set of key feature points;

[0035] Matching the first set of key feature points and the second set of key feature points based on a scale-invariant feature transform method to obtain a matched feature point pair.

[0036] Preferably, the position mapping table between the real three-dimensional model and the multi-angle remote sensing image is established, further comprising:

[0037] Calculating a pose parameter of the multi-angle remote sensing image relative to the real three-dimensional model based on a visual positioning technology according to the matched feature point pair, wherein the pose parameter comprises a rotation matrix and a translation vector;

[0038] Establishing the position mapping table between the real three-dimensional model and the multi-angle remote sensing image based on the pose parameter, wherein the position mapping table comprises a corresponding relationship between an image coordinate and a three-dimensional coordinate.

[0039] The beneficial effects of the present application are as follows: (1) The present application can effectively identify geographical entities (such as buildings, roads, bridges, vegetation, etc.) by processing multi-angle remote sensing images, and extract more accurate geographical information according to different angles. Through this process, the accuracy of geographical entity classification and identification can be improved, and the geographical entity type is matched with the geographical knowledge graph, so that each geographical entity not only has a description of the physical space position, but also combines rich semantic information (such as function and purpose, historical background, environmental impact, etc.). The integration of such information helps to enhance the understanding of geographical entities and makes geographical data more valuable for application; (2) The present application realizes high-precision three-dimensional modeling of geographical entity regions by integrating multi-angle remote sensing images, thereby providing realistic and accurate spatial display. The real three-dimensional model can present the real spatial structure in different scenarios, enabling users to more intuitively understand and analyze the distribution and layout of geographical entities. Through the established position mapping table, each geographical entity type in the multi-angle remote sensing image can be accurately mapped into the real three-dimensional model, thereby forming a dynamic real three-dimensional model. This means that in the model display, the geographical entity position, shape and actual changes can be reflected synchronously, ensuring that the display content is synchronized with the reality and updated as time goes by and the environment changes; (3) The present application can form a dynamic information real three-dimensional model combining physical space and semantic information by integrating the geographical information corresponding to the geographical entity type. This model not only has spatial visual effects, but also carries real-time updated geographical data, enabling users to obtain more comprehensive and in-depth information when browsing. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A schematic diagram of the step flow of the overall method in an embodiment of the present application. DETAILED DESCRIPTION

[0041] Embodiment one, as shown in the figure, the geographical entity information display method based on real three-dimensional modeling proposed by the present application comprises: Figure 1

[0042] S1, obtaining multi-angle remote sensing images of a geographical entity region, and performing target detection on the multi-angle remote sensing images to obtain each geographical entity type in the multi-angle remote sensing images;

[0043] S2, matching the geographical entity type with a preset geographical knowledge graph to obtain geographical information corresponding to the geographical entity type;

[0044] S3, three-dimensional reconstruction of the geographical entity region based on the multi-angle remote sensing images to obtain a real three-dimensional model corresponding to the geographical entity region;

[0045] ​S4, a position mapping table between the real scene three-dimensional model and the multi-angle remote sensing image is established, each geographic entity type in the multi-angle remote sensing image is mapped to the real scene three-dimensional model based on the position mapping table, and a dynamic real scene three-dimensional model is obtained;

[0046] S5, the dynamic real scene three-dimensional model is integrated based on geographic information corresponding to the geographic entity type to obtain a dynamic information real scene three-dimensional model, and the dynamic information real scene three-dimensional model is displayed based on a three-dimensional rendering engine.

[0047] In the present application, remote sensing image refers to the ground image data obtained from a distance by a sensor (such as a sensor on a satellite, a drone, an airplane, etc.); multi-angle remote sensing image refers to remote sensing image data obtained from different angles or perspectives, which provides observation of the same geographic area from different perspectives, and usually helps to understand the form and structure of geographic entities more comprehensively, and using multi-angle images for analysis helps to improve the accuracy of three-dimensional reconstruction; target detection is a task in computer vision, aiming to automatically identify and locate objects or regions of interest in images, in geographic entity information display, target detection is used to identify different geographic entities (such as buildings, roads, trees, etc.) in images; geographic entity type refers to different categories of geographic entities identified in remote sensing images, each type usually corresponds to different geographic elements or features, such as buildings, roads, rivers, mountains, forests, etc.; geographic knowledge graph is a structured database that stores and represents geographic domain knowledge in the form of a graph, nodes in the graph usually represent geographic entities (such as cities, roads, buildings, etc.), and edges represent the relationships between these entities (such as adjacency, belonging, passing, etc.). In this method, the geographic knowledge graph is used to match the geographic entity types obtained by target detection with specific geographic information (such as coordinates, functions, attributes, etc.), providing detailed background information about geographic entities; three-dimensional rendering engine is a computer graphics tool used to convert three-dimensional models into two-dimensional images that can be presented on a display, the rendering engine processes geometric data, materials, lighting, shadows, etc. in the three-dimensional model to generate the final visual effect, in the dynamic information real scene three-dimensional model display, the rendering engine generates a realistic scene display according to real-time data (such as lighting changes, perspective changes, etc.).

[0048] In embodiment two, the geographic entity information display method based on real scene three-dimensional modeling proposed by the present application, compared with embodiment one, the present embodiment further comprises: performing target detection on the multi-angle remote sensing image to obtain each geographic entity type in the multi-angle remote sensing image, including:

[0049] Performing image enhancement on the multi-angle remote sensing image based on the pre-trained image enhancement model to obtain a multi-angle standard remote sensing image;

[0050] The target detection model is used for target detection on the multi-angle remote sensing image based on pre-training, to obtain each geographical entity type in the multi-angle remote sensing image.

[0051] In an optional embodiment, the target detection model adopts an improved YOLOv7-tiny model, and a visual channel attention mechanism is introduced into a backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used for global average pooling operation on an input feature map, and the size of the feature map is C*H*W, wherein C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map, and the expression of the visual channel attention mechanism is as follows:

[0052]

[0053] wherein z c represents the output of the visual channel attention mechanism, u c (i,j) represents the output of the feature map pixel (i,j).

[0054] It should be noted that YOLOv7-tiny is a lightweight version of YOLOv7, in order to improve the detection speed and reduce the model size, the network structure of YOLOv7 is simplified, and the number of layers and the number of parameters are reduced; in target detection, the backbone network is responsible for extracting features from the input image; the channel attention mechanism is an attention mechanism that pays attention to different channels of the input feature map to weight the importance of the channels, in target detection, each channel of the feature map represents a certain specific feature (such as color, texture, edge, etc.), the channel attention mechanism automatically adjusts the weight of each channel, thereby enhancing the response of key features, suppressing irrelevant or unimportant features, and improving the representation ability of the model.

[0055] In an optional embodiment, the image enhancement model includes a shared encoding module, an edge decoding module, a color decoding module, and a feature fusion module, the shared encoding module is used for feature extraction on the multi-angle remote sensing image to obtain edge features and color features corresponding to the multi-angle remote sensing image, the edge decoding module is used for decoding the edge features corresponding to the multi-angle remote sensing image based on edge blocks and image blocks to obtain image edge results of the multi-angle remote sensing image, the color decoding module is used for decoding the color features of the multi-angle remote sensing image based on an HSV color map to obtain image color results of the multi-angle remote sensing image, and the feature fusion module is used for splicing the image edge results and the image color results of the multi-angle remote sensing image to obtain a spliced feature map, and performing feature fusion on the spliced feature map to obtain a standard multi-angle remote sensing image.

[0056] It should be noted that the encoding module is usually used for feature extraction of the input image, which maps the original image to a latent space (usually a low-dimensional space), thereby extracting the high-level features of the image, and the sharing of the encoding module means that the module uses a shared structure to process images from different angles when processing multi-angle remote sensing images, which means that the module uses the same weights and network structure when extracting multi-angle image features, thereby improving the efficiency of the model and reducing the amount of calculation; the task of the edge decoding module is to restore or generate the edge information of the image from the features of the image, and the edge is the embodiment of the contour or structure of the object in the image, which is commonly used in image segmentation, object recognition and other tasks, and here, the edge decoding module converts the edge features extracted from the shared encoding module back to edge images or edge layers through a specific decoding mechanism (such as deconvolution, back propagation, etc.); the color decoding module is used to restore the color information of the image from the color features of the image, and in image processing, color is usually represented in different color spaces, such as RGB (red, green, and blue) or HSV (hue, saturation, and brightness), and here, the color decoding module decodes the color features of the image based on the HSV color map, thereby reconstructing the color information of the image; the function of the feature fusion module is to combine the edge results and color results of the image together to form a new feature map for further processing and enhancement, and this process usually needs to fuse different types of features (such as edge information and color information) into a new feature map to provide more comprehensive image information.

[0057] In an optional embodiment, a geographic entity region is reconstructed in three dimensions based on multi-angle remote sensing images to obtain a real scene three-dimensional model corresponding to the geographic entity region, comprising:

[0058] The multi-angle remote sensing images are rectified to obtain multi-angle rectified remote sensing images;

[0059] The aerial triangulation in the geographic entity region is analyzed based on photogrammetry to obtain exterior orientation elements corresponding to the multi-angle remote sensing images;

[0060] The geometric information of each geographic entity in the multi-angle remote sensing images is extracted based on the exterior orientation elements according to the adjustment matrix;

[0061] The geometric information of each geographic entity is converted into three-dimensional point cloud data based on a pre-trained three-dimensional reconstruction model, and a real scene three-dimensional model corresponding to the geographic entity region is obtained based on the three-dimensional point cloud data.

[0062] It should be noted that distortion correction refers to the correction of geometric distortion in remote sensing images caused by optical errors of the imaging system or other factors, so that the geometric shape of the image is more consistent with the actual ground situation. The distortion of remote sensing images may include lens distortion, projection distortion, perspective distortion, etc. After distortion correction, more accurate geographic information can be obtained. The exterior orientation elements refer to the parameters that describe the external position and attitude of the remote sensing image. In photogrammetry, the exterior orientation elements include the geographic coordinates (position) of the image and the orientation (attitude) when taking the picture. Aerial triangulation is a technique for determining the spatial position and attitude of images, commonly applied in aerial or satellite images. In this process, the relative positional relationship under different viewing angles is measured, and the exterior orientation elements of the camera are calculated using known ground control points (such as GPS positions or ground marker points). Simply put, aerial triangulation calculates the spatial coordinates and attitude information of the image through the geometric relationship (viewing angle and distance) of a triangle. Adjustment solution is a process used in photogrammetry to optimize measurement data, aiming to minimize errors through mathematical models and obtain optimal solutions. The adjustment solution matrix is a mathematical tool used in this process, usually a linear equation system or a least squares solution matrix. Geometric information refers to the spatial shape, position, and size of objects in the image. In remote sensing image processing, geometric information usually includes the spatial coordinates, shape outline, and size of geographic entities. By extracting geometric information from the image, the characteristics and attributes of the actual geographic area represented by the image can be understood. Three-dimensional point cloud data is a collection of points distributed in three-dimensional space obtained by measurement. Each point usually includes three-dimensional coordinates (X, Y, Z), which represent the specific position of the object surface in space. Through a large number of three-dimensional point clouds, the shape of the object or terrain can be reconstructed.

[0063] In an optional embodiment, the multi-angle remote sensing image is subjected to distortion correction to obtain a multi-angle corrected remote sensing image, comprising:

[0064] The multi-angle remote sensing image is subjected to camera distortion correction based on a point coordinate transformation model to obtain a multi-angle camera distortion corrected remote sensing image, wherein the calculation formula of the camera distortion correction is as follows:

[0065]

[0066] wherein (X, Y) represents the real pixel point coordinates of the multi-angle remote sensing image, (F X ,F Y ) represents the coordinate transformation function in the camera distortion correction, and (α, β) represents the multi-angle camera distortion corrected remote sensing image.

[0067] The multi-angle camera distortion corrected remote sensing image is subjected to geometric rectification processing to obtain a multi-angle corrected remote sensing image, wherein the calculation formula of the geometric rectification processing is as follows:

[0068] G Q (x Q ,y Q )=G N (x N ,y N )(X,Y);

[0069] wherein G Q (x Q ,y Q ) represents the gray value of the re-sampling point Q in the multi-angle camera distortion correction remote sensing image, G N (x N ,y N ) represents the corresponding gray value of the nearest pixel to the re-sampling point Q, (x Q ,y Q ) represents the gray coordinate value of G Q (x Q ,y Q ), and (x N ,y N ) represents the gray coordinate value of G N (x N ,y N ).

[0070] In an optional embodiment, the three-dimensional reconstruction model comprises a feature extraction module, a feature aggregation module, a feature decomposition module and a feature regression module, the feature extraction module is used to extract first local features of geometric information of a preset multi-angle reference image and second local features of geometric information of a multi-angle remote sensing image, and convert the second local features to a perspective angle where the first local features are located, to obtain third local features, and match the first local features and the third local features two by two, to obtain a plurality of cost volumes.

[0071] It should be noted that feature extraction is a basic step in computer vision and image processing, aiming to extract representative key points or local features from images. In three-dimensional reconstruction, the feature extraction module is responsible for extracting geometric information from multi-angle remote sensing images. The first local feature refers to the preliminary geometric feature extracted from the preset reference image. These features are usually local areas in the image with high recognition, stability and repeatability (such as corner points, edges, etc.). These features can be accurately identified in the image and used as a reference for matching in subsequent processing steps. The second local feature is the geometric feature extracted from multi-angle remote sensing images. It usually refers to the feature obtained from the target remote sensing image (i.e. different from the reference image). These features represent the local geometric information in the image, but due to different shooting angles, the second local feature usually has spatial displacement or rotation and other geometric changes compared with the first local feature. The view angle refers to the specific position and orientation of the camera or remote sensing sensor when shooting the image. Different view angles will affect the shape, scale and display method of geometric features in the image. In three-dimensional reconstruction, the geometric relationship between images under different view angles needs to be considered. The third local feature is a new feature obtained by converting the second local feature to the same view angle as the first local feature. This process is usually achieved through view angle conversion. The goal is to standardize features under different view angles for subsequent matching. Two-by-two matching refers to comparing and matching features in two images in image processing. Here, the first local feature and the third local feature are matched two by two, that is, through some distance measurement (such as Euclidean distance, angular distance, etc.), it is found out whether these feature points correspond in space. The cost volume is a data structure used to measure matching errors in the matching process. It represents the similarity or cost between each pair of matching feature points in the image. It is usually used to measure the quality of point cloud matching under different view angles in three-dimensional reconstruction. Feature aggregation refers to combining feature information from different images. It is usually used to fuse local features in multi-angle or multi-frame images into global features. Feature decomposition refers to decomposing aggregated features. The purpose is usually to extract more detailed local features or separate the contributions of different feature dimensions. The decomposed features help to understand the geometric structure of each part of the image. Feature regression refers to predicting or optimizing target values based on extracted feature information. In three-dimensional reconstruction, the feature regression module can be used to optimize the position, pose and other parameters of feature points to ensure the accuracy of the model.

[0072] In an optional embodiment, the feature aggregation module employs a feature pyramid network, the feature pyramid network is used to aggregate the plurality of cost volumes to obtain an aggregated cost volume, the feature decomposition module is configured to downsample the aggregated cost volume to obtain high-dimensional features, and then perform a capsule operation on the high-dimensional features in a feature channel dimension to decouple the aggregated cost volume into a plurality of capsule blocks, and the feature regression module is configured to encode the high-dimensional features in the plurality of capsule blocks through a preset capsule convolutional layer to obtain a plurality of high-dimensional feature semantics, and concatenate the plurality of high-dimensional feature semantics with the plurality of capsule blocks to obtain a plurality of concatenated features, and perform a regression operation on the plurality of concatenated features through a normalization operation to obtain a plurality of depth maps, and convert the plurality of depth maps into three-dimensional point cloud data.

[0073] It should be noted that the feature pyramid network is a neural network architecture commonly used for multi-scale object detection and feature extraction, the core idea of which is to effectively capture information of different scales by constructing feature maps of different resolutions (i.e., a feature pyramid), and the FPN extracts features from different levels of the network and fuses them, so that high-level features can contain rich semantic information, while low-level features retain detailed information; the capsule operation is derived from Capsule Networks, which is a deep learning method designed to overcome the problem of spatial information loss in traditional convolutional neural networks (CNNs), and Capsule Networks maintain spatial information by encapsulating the outputs of multiple neurons into a "capsule", and each capsule represents a specific attribute of an object (such as position, pose, rotation, etc.); capsuleization is an organization of high-dimensional features after feature decomposition in the "feature channel dimension" to form a plurality of capsule blocks, each of which captures feature information of a certain class or local region, and can effectively handle transformation information (such as rotation, translation, etc.) in an image; the capsule convolutional layer is a convolutional layer specially designed to handle capsules, unlike traditional convolutional layers, the capsule convolutional layer transmits information between multiple capsule blocks to capture spatial transformations and richer contextual relationships; feature concatenation refers to connecting multiple features of different sources (such as high-dimensional features of multiple capsule blocks) in the feature dimension to form a larger feature vector or feature map; the depth map is a two-dimensional image, and each pixel value represents the distance from the camera or observation point to the object surface, in three-dimensional reconstruction, the depth map is used to extract spatial position information of objects or scenes from two-dimensional images; the three-dimensional point cloud is a three-dimensional spatial data set composed of spatial coordinates (X, Y, Z) data obtained from multiple perspectives or depth maps, in three-dimensional reconstruction, the point cloud is an important data structure for describing the geometric shape of objects or scenes, and is usually used for further three-dimensional modeling, visualization, or object recognition, etc.

[0074] In an optional embodiment, the position mapping table between the real three-dimensional model and the multi-angle remote sensing image is established, comprising:

[0075] acquire three-dimensional point cloud data of a real scene three-dimensional model, and perform feature point extraction on the three-dimensional point cloud data to obtain a first set of key feature points;

[0076] perform feature point extraction on the multi-angle remote sensing images to obtain a second set of key feature points;

[0077] match the first set of key feature points and the second set of key feature points based on a scale-invariant feature transform method to obtain a matched feature point pair.

[0078] It should be noted that feature point extraction is to extract points with recognition, stability and saliency from images or point clouds, and these points are usually used for subsequent matching, registration and modeling tasks. Feature points have certain spatial uniqueness in images or three-dimensional space and can be used to describe local information of an object. SIFT (Scale-Invariant Feature Transform) is a feature extraction and description algorithm commonly used in image matching, object recognition and three-dimensional reconstruction. The SIFT algorithm can extract scale-invariant key points from images, and these key points have strong robustness to image rotation, scale change and illumination change. A matched feature point pair refers to a corresponding feature point pair found in different images or point cloud data based on a certain similarity measurement (such as SIFT descriptor matching). These feature point pairs represent the same point in the same actual object or scene observed from different perspectives or data sources (such as three-dimensional point cloud and remote sensing image).

[0079] In an optional embodiment, establishing a position mapping table between the real scene three-dimensional model and the multi-angle remote sensing images further includes:

[0080] calculating pose parameters of the multi-angle remote sensing images relative to the real scene three-dimensional model based on the matched feature point pair according to a visual positioning technology, wherein the pose parameters include a rotation matrix and a translation vector;

[0081] establishing a position mapping table between the real scene three-dimensional model and the multi-angle remote sensing images based on the pose parameters, wherein the position mapping table includes a corresponding relationship between image coordinates and three-dimensional coordinates.

[0082] It should be noted that the visual positioning technology refers to estimating the position and pose of the camera through image or video information. Generally, these technologies rely on feature points extracted from images (such as feature points extracted by SIFT, SURF, ORB, etc. algorithms) for position speculation, and the positioning process usually includes two parts of camera pose estimation and feature matching. Camera pose estimation refers to estimating the position (coordinates in space) and pose (i.e. the rotation direction of the camera) of the camera. Feature matching and registration refers to determining the specific position of the camera by matching the feature points in the image with the feature points in the known three-dimensional model or map. Rotation matrix refers to the direction of the rotation of an object, indicating the rotation transformation from the world coordinate system to the object coordinate system. Translation vector refers to the position of an object, indicating the translation from the origin of the world coordinate system to the origin of the object coordinate system.

[0083] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited thereto, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A method for displaying information of geographical entities based on real scene three-dimensional modeling, characterized in that, The method comprises the following steps: obtaining multi-angle remote sensing images of a geographical entity region, performing target detection on the multi-angle remote sensing images to obtain each geographical entity type in the multi-angle remote sensing images; matching the geographical entity type with a preset geographical knowledge graph to obtain geographical information corresponding to the geographical entity type; based on the multi-angle remote sensing images, performing three-dimensional reconstruction on the geographical entity region to obtain a real scene three-dimensional model corresponding to the geographical entity region; establishing a position mapping table between the real scene three-dimensional model and the multi-angle remote sensing images, and mapping each geographical entity type in the multi-angle remote sensing images to the real scene three-dimensional model based on the position mapping table to obtain a dynamic real scene three-dimensional model; based on the geographical information corresponding to the geographical entity type, integrating the dynamic real scene three-dimensional model to obtain a dynamic information real scene three-dimensional model, and displaying the dynamic information real scene three-dimensional model based on a three-dimensional rendering engine; performing target detection on the multi-angle remote sensing images to obtain each geographical entity type in the multi-angle remote sensing images, comprising: performing image enhancement on the multi-angle remote sensing images based on a pre-trained image enhancement model to obtain multi-angle standard remote sensing images; wherein the image enhancement model comprises a shared encoding module, an edge decoding module, a color decoding module and a feature fusion module, the shared encoding module is used for feature extraction on the multi-angle remote sensing images to obtain edge features and color features corresponding to the multi-angle remote sensing images, the edge decoding module is used for decoding the edge features corresponding to the multi-angle remote sensing images based on edge blocks and image blocks to obtain image edge results of the multi-angle remote sensing images, the color decoding module is used for decoding the color features of the multi-angle remote sensing images based on an HSV color graph to obtain image color results of the multi-angle remote sensing images, and the feature fusion module is used for splicing the image edge results and the image color results of the multi-angle remote sensing images to obtain a spliced feature map, and performing feature fusion on the spliced feature map to obtain standard multi-angle remote sensing images.

2. The method of claim 1, wherein the method further comprises: performing target detection on the multi-angle remote sensing images to obtain each geographical entity type in the multi-angle remote sensing images, further comprising: performing target detection on the multi-angle remote sensing images based on a pre-trained target detection model to obtain each geographical entity type in the multi-angle remote sensing images.

3. The method of claim 2, wherein the method further comprises: The target detection model adopts an improved YOLOv7-tiny model, and a visual channel attention mechanism is introduced into a backbone feature extraction network of the YOLOv7-tiny model, wherein the visual channel attention mechanism is used for performing a global average pooling operation on an input feature map, and a size of the feature map is wherein, represents a channel number of the feature map, represents a height of the feature map, represents a width of the feature map, and an expression of the visual channel attention mechanism is as follows: ； wherein, represents the output of the visual channel attention mechanism, represents the output of the feature image pixel .

4. The method of claim 1, wherein the method further comprises: based on the multi-angle remote sensing images, performing three-dimensional reconstruction on the geographical entity region to obtain a real scene three-dimensional model corresponding to the geographical entity region, comprising: performing distortion correction on the multi-angle remote sensing images to obtain multi-angle corrected remote sensing images; based on photogrammetry, analyzing aerial triangulation in the geographical entity region to obtain exterior orientation elements corresponding to the multi-angle remote sensing images; based on a matrix adjustment solution, extracting geometric information of each geographical entity in the multi-angle remote sensing images according to the exterior orientation elements; The geometric information of each geographic entity is converted into three-dimensional point cloud data based on a pre-trained three-dimensional reconstruction model, and a real three-dimensional model corresponding to the geographic entity region is obtained based on the three-dimensional point cloud data.

5. The method according to claim 4, wherein, The multi-angle remote sensing image is subjected to distortion correction to obtain a multi-angle corrected remote sensing image, including: The multi-angle remote sensing image is subjected to camera distortion correction based on a point coordinate transformation model to obtain a multi-angle camera distortion corrected remote sensing image; The multi-angle camera distortion corrected remote sensing image is subjected to geometric correction processing to obtain a multi-angle corrected remote sensing image.

6. The method of claim 5, wherein the method further comprises: The three-dimensional reconstruction model includes a feature extraction module, a feature aggregation module, a feature decomposition module, and a feature regression module. The feature extraction module is configured to extract first local features of geometric information of a preset multi-angle reference image and second local features of geometric information of the multi-angle remote sensing image, convert the second local features to a perspective angle of the first local features to obtain third local features, and match the first local features and the third local features two by two to obtain a plurality of cost volumes.

7. The method according to claim 6, wherein, The feature aggregation module uses a feature pyramid network to aggregate the plurality of cost volumes based on the feature pyramid network to obtain an aggregated cost volume. The feature decomposition module is configured to downsample the aggregated cost volume to obtain high-dimensional features, and then perform a capsule operation according to a feature channel dimension to decouple the aggregated cost volume into a plurality of capsule blocks. The feature regression module is configured to encode high-dimensional features in the plurality of capsule blocks through a preset capsule convolution layer to obtain a plurality of high-dimensional feature semantics, splice the plurality of high-dimensional feature semantics with the plurality of capsule blocks to obtain a plurality of spliced features, and regress the plurality of spliced features through a normalization operation to obtain a plurality of depth maps. The plurality of depth maps are converted into three-dimensional point cloud data.

8. The method of claim 7, wherein the method further comprises: A position mapping table between the real three-dimensional model and the multi-angle remote sensing image is established, including: Obtaining three-dimensional point cloud data of the real three-dimensional model, and extracting feature points from the three-dimensional point cloud data to obtain a first set of key feature points; Extracting feature points from the multi-angle remote sensing image to obtain a second set of key feature points; Matching the first set of key feature points and the second set of key feature points based on a scale-invariant feature transformation method to obtain a matching feature point pair.

9. The method of claim 8, wherein the method further comprises: Establishing the position mapping table between the real three-dimensional model and the multi-angle remote sensing image further includes: Calculating pose parameters of the multi-angle remote sensing image relative to the real three-dimensional model based on the matching feature point pair according to a visual positioning technology, wherein the pose parameters include a rotation matrix and a translation vector; Establishing the position mapping table between the real three-dimensional model and the multi-angle remote sensing image based on the pose parameters, wherein the position mapping table includes a corresponding relationship between image coordinates and three-dimensional coordinates.

Images