Geographic model construction method and device, equipment and storage medium
By combining drone aerial photography and continuous wave radar data, a high-precision geographical model is constructed, which solves the problems of low measurement accuracy and limitations of surface characteristics in traditional technology, and realizes a more comprehensive reconstruction of topographic and topographic features, providing smart mine management with Reliable data support.
Patent Information
- Application Number
- CN202411121532.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional digital construction technology has problems with low measurement accuracy and limited surface characteristics when building mining geographical models, making it difficult to obtain more accurate and comprehensive topographic and topographic characteristics.
Combining drone aerial photography data and continuous wave radar measurement data, a more accurate and comprehensive geographical model is built through the correlation between image feature extraction model and matching point feature information, and the measurement accuracy and accuracy are improved.
It realizes higher accuracy and comprehensive reconstruction of topographic and topographic features, provides reliable data support for the intelligent management of smart mines, and improves the accuracy and reliability of geographical models.
Smart Images

Figure CN118941734A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional geographic model construction, and in particular to a method, device, equipment and storage medium for constructing a geographic model. Background Art
[0002] Smart mines are based on the digitalization and informatization of mines, and actively perceive, automatically analyze, and quickly process mine production, occupational health and safety, technical support, and logistics. The ultimate goal of building smart mines is to achieve the goal of building safe mines, unmanned mines, efficient mines, and clean mines.
[0003] An important part of smart mines is to build a mine geographical model based on digital reconstruction technology. The current traditional digital construction technology mainly relies on remote sensing technology and geographic information system (GIS) technology. The mine geographical model constructed by it usually has problems such as low measurement accuracy and limited to surface features. Therefore, how to build a mine geographical model with higher measurement accuracy and more comprehensive and accurate terrain and geomorphic features has become an urgent problem to be solved. Summary of the invention
[0004] In view of this, the present invention proposes a method, device, equipment and storage medium for constructing a geographic model. The present invention combines the shooting data of the drone and the measurement data of the continuous wave radar to obtain more accurate and comprehensive geomorphic features of the geographic area, and then constructs a geographic model based on the more accurate and comprehensive geomorphic features, which has the characteristics of high measurement precision, high accuracy and more comprehensiveness, providing a prerequisite for the subsequent intelligent control and management of the geographic model.
[0005] A first aspect of the present invention provides a method for constructing a geographical model, the method comprising:
[0006] Obtain an image data set corresponding to the target area, where the image data set is obtained by taking aerial photos of the target area by a drone; the image data set includes multiple image data corresponding to the target area at different aerial photography angles;
[0007] Acquire multiple detection points in the target area, and radar feature information corresponding to each of the multiple detection points; each detection point corresponds to a continuous wave radar, and the radar feature information corresponding to the detection point is measured by the continuous wave radar; the radar feature information includes vertical height information and physical property information;
[0008] Input the image data set into the image feature extraction model to output key point feature information of multiple key points;
[0009] Matching multiple detection points and multiple key points, and determining multiple matching points and matching point feature information corresponding to each of the multiple matching points according to the matching results; the matching point feature information corresponding to each matching point is associated with the key point feature information and the radar feature information;
[0010] A geographic model is constructed based on the matching point feature information corresponding to each matching point.
[0011] In an optional embodiment, the image feature extraction model includes a fully convolutional backbone network module and a main regression head module connected in sequence; the image data set is input into the image feature extraction model to output key point feature information of multiple key points, including:
[0012] The image dataset is input into the fully convolutional backbone network module to obtain a predicted key point heat map; the fully convolutional backbone network module is used to predict multiple key points on multiple image data;
[0013] The predicted key point heat map is input into the main regression head module to obtain the key point feature information corresponding to each key point among multiple key points; the key point feature information is used to determine the two-dimensional bounding box and three-dimensional bounding box corresponding to each key point on the predicted key point heat map.
[0014] In an optional implementation, the key point feature information includes depth information; matching the multiple detection points with the multiple key points, and determining the multiple matching points according to the matching results, includes:
[0015] For any key point among the multiple key points, determine the three-dimensional region of interest ROI corresponding to the key point according to the two-dimensional bounding box corresponding to the key point, the depth feature corresponding to the key point and the three-dimensional bounding box corresponding to the key point;
[0016] Determine at least one detection point among the plurality of detection points that falls within the ROI;
[0017] Screening at least one detection point falling into the ROI, and determining a target detection point among the at least one detection point;
[0018] Determine the match between the target detection point and the key point, and associate the target detection point and the key point to determine the matching point.
[0019] In an optional embodiment, the method further comprises:
[0020] Expanding each of the plurality of detection points into a three-dimensional cube structure;
[0021] Determining at least one detection point among the plurality of detection points that falls within the ROI includes:
[0022] At least one detection point falling into the ROI is determined according to the multiple three-dimensional cube structures corresponding to the multiple detection points; wherein the three-dimensional cube structure corresponding to the at least one detection point falling into the ROI overlaps with the ROI.
[0023] In an optional embodiment, screening at least one detection point falling into the ROI to determine a target detection point in the at least one detection point includes:
[0024] Obtaining depth information corresponding to at least one detection point;
[0025] According to the depth information corresponding to the at least one detection point, a target detection point among the at least one detection point is determined; wherein the depth information corresponding to the target detection point indicates a minimum depth.
[0026] In an optional implementation, determining the matching point feature information corresponding to each matching point among the multiple matching points includes:
[0027] For each matching point, obtain the radar feature information corresponding to the matching point;
[0028] Generate a matching point heat map based on the radar feature information corresponding to the matching point;
[0029] Get the predicted key point heat map corresponding to the matching point;
[0030] Superimpose the matching point heat map and the predicted key point heat map corresponding to the matching point, and input the superposition result into the quadratic regression head model to obtain the matching point feature information corresponding to the matching point;
[0031] The matching point feature information corresponding to the matching point is determined according to the matching point feature information corresponding to the matching point and the key point feature information corresponding to the matching point.
[0032] In an optional implementation, constructing a geographic model according to the matching point feature information corresponding to each matching point includes:
[0033] Determine the center coordinates of the center point of the two-dimensional bounding box corresponding to each matching point, the information of the three-dimensional bounding box, the size information and the heading information according to the matching point feature information corresponding to each matching point;
[0034] Determine the relative coordinates of each corner point in the three-dimensional bounding box relative to the center point according to the size information and the position of the center point of the two-dimensional bounding box;
[0035] Determine the global coordinates corresponding to each corner point in the 3D bounding box according to the relative coordinates, the center coordinates of the center point and the heading information;
[0036] The geographic model is constructed based on the global coordinates corresponding to each corner point in the 3D bounding box.
[0037] A second aspect of the present invention provides a device for constructing a geographical model, the device comprising:
[0038] An acquisition unit is used to acquire an image data set corresponding to the target area; the image data set is obtained by aerial photography of the target area by a drone; the image data set includes a plurality of image data corresponding to the target area at different aerial photography angles;
[0039] The acquisition unit is further used to acquire multiple detection points in the target area and radar feature information corresponding to each of the multiple detection points; each detection point corresponds to a continuous wave radar, and the radar feature information corresponding to the detection point is measured by the continuous wave radar; the radar feature information includes vertical height information and physical property information;
[0040] A processing unit, for inputting the image data set into the image feature extraction model to output key point feature information of a plurality of key points;
[0041] The processing unit is further used to match the multiple detection points with the multiple key points, and determine the multiple matching points and the matching point feature information corresponding to each of the multiple matching points according to the matching results; the matching point feature information corresponding to each matching point is associated with the key point feature information and the radar feature information;
[0042] The construction unit is used to construct a geographic model according to the matching point feature information corresponding to each matching point.
[0043] In an optional embodiment, the image feature extraction model includes a fully convolutional backbone network module and a main regression head module connected in sequence;
[0044] The processing unit is specifically used to input the image data set into the full convolution backbone network module to obtain a predicted key point heat map; the full convolution backbone network module is used to predict multiple key points on multiple image data; the predicted key point heat map is input into the main regression head module to obtain key point feature information corresponding to each key point in the multiple key points; the key point feature information is used to determine the two-dimensional bounding box and the three-dimensional bounding box corresponding to each key point on the predicted key point heat map.
[0045] In an optional embodiment, the key point feature information includes depth information;
[0046] The processing unit is specifically used to determine, for any key point among multiple key points, a three-dimensional region of interest ROI corresponding to the key point according to a two-dimensional bounding box corresponding to the key point, a depth feature corresponding to the key point, and a three-dimensional bounding box corresponding to the key point; determine at least one detection point falling into the ROI among multiple detection points; screen at least one detection point falling into the ROI to determine a target detection point among the at least one detection point; determine that the target detection point matches the key point, and associate the target detection point with the key point to determine a matching point.
[0047] In an optional embodiment, the processing unit is further used to expand each detection point of the plurality of detection points into a three-dimensional cube structure;
[0048] The processing unit is specifically used to determine at least one detection point falling into the ROI according to multiple three-dimensional cube structures corresponding to multiple detection points; wherein the three-dimensional cube structure corresponding to at least one detection point falling into the ROI overlaps with the ROI.
[0049] In an optional implementation, the acquisition unit is further used to acquire depth information corresponding to at least one detection point;
[0050] The processing unit is specifically configured to determine a target detection point among the at least one detection point according to the depth information corresponding to the at least one detection point acquired by the acquisition unit; wherein the depth information corresponding to the target detection point indicates a minimum depth.
[0051] In an optional implementation, the acquisition unit is further configured to acquire, for each matching point, radar feature information corresponding to the matching point;
[0052] The processing unit is specifically used to generate a matching point heat map according to the radar feature information corresponding to the matching point; superimpose the matching point heat map and the predicted key point heat map corresponding to the matching point, and input the superposition result into the quadratic regression head model to obtain the matching point feature information corresponding to the matching point; determine the matching point feature information corresponding to the matching point according to the matching point feature information corresponding to the matching point and the key point feature information corresponding to the matching point.
[0053] In an optional embodiment, the construction unit is specifically used to determine the center coordinates of the center point of the two-dimensional bounding box corresponding to each matching point, the information of the three-dimensional bounding box, the size information and the heading information based on the matching point feature information corresponding to each matching point; determine the relative coordinates of each corner point in the three-dimensional bounding box relative to the center point based on the size information and the position of the center point of the two-dimensional bounding box; determine the global coordinates corresponding to each corner point in the three-dimensional bounding box based on the relative coordinates, the center coordinates of the center point and the heading information; and construct a geographic model based on the global coordinates corresponding to each corner point in the three-dimensional bounding box.
[0054] The third aspect of the present invention provides a device for building a geographic model, which includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor, so that the device for building a geographic model implements the method described in the first aspect and any one of the embodiments of the first aspect.
[0055] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions; the computer instructions enable the computer to execute the method described in the first aspect and any one of the embodiments of the first aspect.
[0056] The method, device, equipment and storage medium for constructing a geographic model provided by the present invention have the following beneficial effects compared with the prior art:
[0057] (1) The present invention obtains the image data set corresponding to the target area based on drone aerial photography and oblique photogrammetry technology, so that more image features can be obtained according to the image data set, thereby obtaining more comprehensive geomorphic features of the target area based on more image features. At the same time, high-resolution remote sensing images can also be obtained.
[0058] (2) The present invention also obtains the vertical height information and topographic change information of the target area based on the vertical displacement measurement of the continuous wave radar and the electromagnetic wave scattering mechanism. This combined processing method of drone aerial photography and oblique photogrammetry technology, vertical displacement measurement of the continuous wave radar and the electromagnetic wave scattering mechanism can improve the feature extraction capability for the target area, so as to obtain more refined and comprehensive topographic features, thereby improving the measurement accuracy of the topography of the target area.
[0059] (3) Reconstructing the geographic model based on more sophisticated and comprehensive geographic features can improve the accuracy and reliability of the geographic model and achieve accurate reconstruction of the topography of the target area, thus providing more accurate and reliable data support for the subsequent control and management of the target area. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 A schematic diagram of a process for constructing a geographic model provided by the present invention;
[0062] Figure 2A three-dimensional FFT method diagram corresponding to the continuous wave radar provided by the present invention;
[0063] Figure 3 A schematic diagram of the structure of the CenterFusion model provided by the present invention;
[0064] Figure 4 A network architecture diagram of the DLA network architecture provided by the present invention;
[0065] Figure 5 A schematic diagram of the structure of a device for constructing a geographic model provided in an embodiment of the present application;
[0066] Figure 6 A schematic diagram of the structure of a geographic model construction device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0067] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0068] Smart mines are based on the digitalization and informatization of mines, and actively perceive, automatically analyze, and quickly process mine production, occupational health and safety, technical support, and logistics. The ultimate goal of building smart mines is to achieve the goal of building safe mines, unmanned mines, efficient mines, and clean mines.
[0069] An important part of smart mines is to build a mine geographic model based on digital reconstruction technology. The mine geographic model needs to accurately construct the topography of the mine area in order to provide reliable data support for a series of subsequent control and management of the mine. However, the current traditional digital construction technology mainly relies on remote sensing technology and geographic information system (GIS) technology. The mine geographic models constructed by it usually have problems such as low measurement accuracy and are limited to surface features. Therefore, how to build a mine geographic model with higher measurement accuracy and more comprehensive and accurate topographic features has become an urgent problem to be solved.
[0070] In view of the above problems, the present invention proposes a method, device, equipment and storage medium for constructing a geographic model. The present invention combines drone aerial photography and oblique photogrammetry technology, vertical displacement measurement of continuous wave radar and a joint processing method of electromagnetic wave scattering mechanism to obtain more accurate and comprehensive geomorphic features of the geographic area, and then constructs a geographic model with higher measurement accuracy, higher accuracy and higher reliability based on the more accurate and comprehensive geomorphic features, so as to provide reliable data support for the subsequent intelligent management of the geographic area.
[0071] Figure 1 The following is a flow chart of a method for constructing a geographic model provided in an embodiment of the present application. Figure 1 As shown, the method for constructing the geographic model includes the following steps:
[0072] 101. Obtain an image dataset corresponding to the target area.
[0073] In this embodiment, it is necessary to use the drone's aerial photography technology and oblique photogrammetry technology to assist in the construction of the geographic model and achieve accurate measurement of the target area. Specifically, it is necessary to design the drone's route and shooting parameters to obtain multiple image data obtained after the drone shoots the target area at different aerial angles. It can be understood that drone aerial photography can cover a larger range of the target area's topography, and oblique photogrammetry technology can obtain images at different angles to provide more topographic details.
[0074] Exemplarily, the process of using a drone to take aerial photos of a target area is as follows:
[0075] First, determine the flight area. For example, the target area to be modeled can be determined based on the satellite map. If the satellite map is very different from the actual terrain, the drone can be used to fly to the boundary of the modeling area, and the target area can be further delineated according to the boundary range of the drone positioning. It can be understood that the altitude of the drone needs to be set according to the terrain of the target area. Specifically, the flight altitude of the drone must be higher than all geographical objects in the flight area. For example, if the target area is a mine, the flight altitude of the drone needs to be higher than all mountains in the delineated mining area.
[0076] Next, several control points can be set and the position information of the control points can be measured. It can be understood that the control point is a photo control point, which is used to obtain image data. Generally, the control point position needs to be selected on an object point that is clearly visible in the image, such as a fixed object such as the corner point of a building. For example, if the target area is a mine, the control point can be measured using mining network technology, and the horizontal coordinates and elevation coordinates of the control point need to be measured. When the drone is shooting, the control point needs to be clearly visible on each adjacent photo.
[0077] Then, it is necessary to plan the route and set the parameters of oblique photography according to the designated aerial photography range (target area). For example, two routes can be designed for aerial survey, and the heading overlap rate of the two routes can be 80%, and the lateral overlap rate can be 80%. Each route uses oblique photogrammetry technology to obtain 1 vertical lens and 4 oblique lenses. Before aerial photography, it is necessary to check the flight parameters and set the parameters of oblique photography, including flight altitude, flight speed, image shooting interval, heading spacing, lateral spacing, etc. Take over the drone after the parameters are normal, wait for the takeover to be successful, upload the aerial survey parameters, and start the drone to start aerial photography.
[0078] Next, control the drone to perform aerial photography according to the planned route and set parameters. Understandably, the drone may have a battery limit, so each route requires multiple shots, and after the battery is exhausted and returned, replace the new battery, and then automatically or manually select the flight progress and continue shooting.
[0079] Finally, the image data captured is obtained from the drone. For example, the integrity and reliability of the image data need to be further checked to screen the image data. If there is unqualified image data, the drone needs to be controlled to make up for the area corresponding to the unqualified image data and re-collect complete and reliable image data.
[0080] 102. Acquire multiple detection points in the target area and radar feature information corresponding to each of the multiple detection points.
[0081] It can be understood that this embodiment also needs to use multiple continuous wave radars to detect the target area; specifically, the multiple continuous wave radars correspond to multiple detection points respectively, and each continuous wave radar needs to perform displacement detection on its corresponding detection point, and combine the electromagnetic wave scattering mechanism to obtain physical property characteristics (topographic characteristics).
[0082] First, the vertical displacement of each detection point in the target area is measured by using continuous wave radar to monitor the deformation of the surface in real time. It can be understood that the vertical displacement information of the landform in the target area can be obtained through the measurement data of the continuous wave radar, and the risk of geological disasters can be discovered in time. In general, millimeter wave radar is often used to obtain the position and velocity information of the target object, so it has two-dimensional capabilities. It detects the target object by emitting millimeter wave signals and receiving echo signals. According to the time delay and frequency offset of the echo signal, the distance and speed between the target object and the radar can be calculated. Therefore, the vertical height information of the detection point, that is, the target object, can be obtained using the continuous wave radar.
[0083] Then, by analyzing the scattering characteristics of electromagnetic waves at the detection point, more detailed information about the topography of the detection point can be obtained, such as groundwater level, rock cracks, etc. Figure 2 As shown, this embodiment adopts Figure 2 The three-dimensional FFT method shown in the figure processes the echo signal of the MIMO-LFMCW radar to achieve the goals of distance measurement, speed measurement, and angle measurement. At the same time, a variety of spectrum data and point cloud data can be obtained in the process of signal processing. Through the measurement data of the continuous wave radar, the deformation and vertical displacement information between each detection point can be obtained, which reflects the changes in the geological structure. Then, combined with the electromagnetic wave scattering mechanism, the displacement measurement data of the continuous wave radar is further processed and analyzed. Since the propagation and scattering characteristics of electromagnetic waves in different media depend on the physical properties of these media, more detailed information about the topography of the target area can be obtained by analyzing the scattering characteristics of electromagnetic waves between different geographical objects (such as the surface and rock mass). For example, the scattering characteristics of electromagnetic waves above the groundwater level may be different from those below it. Using these differences, the location and changes of the groundwater level can be inferred.
[0084] 103. Input the image data set into the image feature extraction model to output key point feature information of multiple key points.
[0085] Next, the image data set acquired by the drone needs to be input into the image feature extraction model to obtain the image features corresponding to each image data, and determine multiple key points in the image data and the key point feature information of the key points based on the image features.
[0086] Exemplarily, the image feature extraction model includes a fully convolutional backbone network module and a main regression head module connected in sequence. Specifically, the image data set needs to be input into the fully convolutional backbone network module first, and the fully convolutional backbone network module outputs a predicted key point heat map; then the predicted key point heat map is input into the main regression head module, and the main regression head outputs the key point feature information corresponding to each key point.
[0087] For example, Figure 3As shown, the image feature extraction model can be the image feature extraction module in the CenterFuion model, and the CenterFuion model is improved from the CenterNet network. The CenterFuion model has been adaptively expanded for 3D scenes in terms of data dimension. Its input has been expanded from only image input to dimensions such as image, radar, transformation matrix, and depth of field information, while the output has been expanded to include information such as spatial coordinates, box size, heading, and posture. Specifically, the image feature extraction module is divided into two parts: a fully convolutional backbone network and a main regression head. In the CenterFusion model architecture, when performing preliminary detection of the image, the backbone network DLA (deep aggregation) is used as the fully convolutional backbone network to extract image features. Specifically, the DLA network greatly reduces the training time and improves the image feature extraction performance. Among them, the DLA network architecture is as follows Figure 4 shown.
[0088] The fully convolutional backbone network takes the image dataset (image) acquired by the drone as input and outputs the predicted key point heat map. Specifically, each image is used as input, and the predicted key point heat map is generated through downsampling (pooling), where the heat value of each pixel is between [0,1]. Then, the key points on the image are predicted by regressing the image features, and the two-dimensional size (width and height), center offset, three-dimensional size, depth and rotation of the key points are extracted.
[0089] Next, the predicted key point heat map is input to the main regression head, where the main regression head outputs key point feature information, which can include six branches, outputting key point heat map, key point offset, 2D width and height, 3D length, distance and rotation angle respectively. In this way, these key point feature information will determine an accurate 2D bounding box and a preliminary 3D bounding box for each key point.
[0090] 104. Match the multiple detection points with the multiple key points, and determine multiple matching points according to the matching results.
[0091] After obtaining the radar feature information of multiple detection points and the key point feature information of multiple key points, it is necessary to match the multiple detection points and the multiple key points based on this information, that is, the successfully matched detection points and key points are used to indicate the same geographic object, so that more comprehensive matching point feature information can be determined based on the radar feature information and key point feature information corresponding to the geographic object (i.e. matching point), and the fusion of radar feature information (radar data) and key point feature information (camera data) is realized. The following is a detailed introduction to the matching process of multiple detection points and the multiple key points:
[0092] In order to avoid the problems of inaccurate matching, unsuccessful matching and erroneous matching in traditional matching methods, this embodiment introduces a truncated cone association module and a pillar expansion module to better match detection points and key points.
[0093] (1) Cone association
[0094] Specifically, frustum association refers to determining the three-dimensional region of interest ROI corresponding to each key point based on the two-dimensional bounding box, depth feature and three-dimensional bounding box of the key point after obtaining the two-dimensional bounding box, depth feature and three-dimensional bounding box of the key point through the fully convolutional backbone network and the main regression head. For a certain key point, to determine the detection point that matches it, it is necessary to determine the detection point that falls into the ROI among multiple detection points. Among them, when obtaining the ROI, the real depth value of the key point is used in the training phase, and the depth prediction value generated by the image feature extraction model is used in the test phase. Exemplarily, the hyperparameter δ can be used to control the size of the ROI area to control the number of detection points that fall into the ROI area.
[0095] Next, the detection points falling into the ROI need to be screened to determine the target detection point that matches the key point. Exemplarily, when determining the target detection point corresponding to a key point, the detection point with the smallest depth in the ROI falling into the key point is determined as the target detection point, that is, the target detection point closest to the front is the matching result of the key point.
[0096] (2) Pillar expansion
[0097] It is understandable that in order to solve the impact of missing z-axis information on the matching results, it is necessary to introduce a pillar expansion module. Specifically, each of the multiple detection points corresponding to the continuous wave radar needs to be expanded into a three-dimensional cube structure, and then based on the three-dimensional cube structure, it is determined whether the detection point falls into the ROI corresponding to a key point. Exemplarily, for the ROI corresponding to a key point, if the three-dimensional cube structure corresponding to a detection point overlaps with the ROI, that is, all or part of it falls into the ROI, then it is determined that the detection point falls into the ROI corresponding to the key point. The detection point with the smallest depth that falls into the ROI of the key point is the target detection point that matches it.
[0098] like Figure 3As shown in the figure, in the CenterFusion model, the radar feature information (radar point cloud) corresponding to the continuous wave radar is passed through the pillar expansion module to determine the three-dimensional cube structure corresponding to each detection point. Then the frustum association module determines the ROI corresponding to each detection point based on the two-dimensional bounding box, depth feature and three-dimensional bounding box corresponding to each detection point output by the main regression head module, and then determines the matching result based on the three-dimensional cube structure corresponding to each detection point output by the pillar expansion module and the ROI corresponding to each detection point, and fuses the radar feature information with the key point feature information.
[0099] 105. Determine matching point feature information corresponding to each matching point among the multiple matching points.
[0100] After determining the matching result and obtaining the matching point (the matching point indicates a geographical object, such as a mountain, a building, a lake, etc.), it is necessary to obtain the radar feature information corresponding to the matching point and the key point feature information corresponding to the matching point to determine more comprehensive matching point feature information for the matching point.
[0101] Exemplarily, the radar feature information needs to be processed first. Specifically, the radar feature information needs to be converted into a tensor form and superimposed with the output of the fully convolutional backbone neural network.
[0102] Specifically, it is necessary to generate a three-channel heat map area (matching point heat map) based on the radar feature information corresponding to the detection point. The center point of the heat map area coincides with the center point of the two-dimensional bounding box corresponding to the matching point and is contained in the two-dimensional bounding box. The width and height of the heat map area are proportional to the two-dimensional bounding box and are controlled by parameters. Among them, the three channel values of the heat map are the normalized target depth, the x-axis component of the target radial velocity, and the y-axis component of the target radial velocity. The x-axis and y-axis here are taken from the local coordinate system.
[0103] Among them, the formula for the heat map value generated by radar feature information is taken as:
[0104]
[0105] in,
[0106] F: represents the normalized value of the current pixel, that is, the pixel value on the heat map.
[0107] i: represents the channel number of the heatmap, usually corresponding to different features or color channels, such as red, green, and blue channels.
[0108] x and y: represent the pixel coordinates of the current pixel, which are normalization factors relative to the entire heat map.
[0109] d: represents the pixel value of the current pixel, which is taken by channel separately, that is, different values are on different channels.
[0110] c x and c y : represents the x and y coordinates of the center point of target j, which is the detection point used to indicate the target location.
[0111] w j and h j : represents the width and height of target j, which are the parameters of target shape.
[0112] The above algorithm is used to generate a heat map where the value of each pixel depends on whether it is within a specified region of an object. If the pixel is within the specified region of the object, its value is set to the corresponding feature value; otherwise, its value is set to 0. The purpose of this is to highlight the object of interest and blur the rest.
[0113] Then, the matching point heat map is superimposed on the predicted key value heat map output by the fully convolutional backbone neural network as an expansion channel, and is input into the secondary regression head module as a fusion feature to calculate the matching point feature information of the matching points. The secondary regression head consists of four branches, which respectively predict the depth, rotation angle, speed and attributes of the geographic object corresponding to the matching point. The attributes here include whether the vehicle is stationary or driving, and whether the pedestrian is standing or sitting.
[0114] like Figure 2 As shown, the output results of the main regression head and the post-operative output results of the secondary regression head will be input into the 3DBOX decoder to decode the three-dimensional frame information. Exemplarily, the information obtained includes the size information of the two-dimensional bounding box corresponding to each matching point, the depth information of the center point of the two-dimensional bounding box, the length information of the two-dimensional bounding box, the target heading information, the target category information, the target state information, etc. Among them, according to the size information and the position of the center point of the two-dimensional bounding box, the relative coordinates of each corner point in the three-dimensional bounding box relative to the center point can be determined; then, according to the relative coordinates, the center coordinates of the center point and the heading information, the global coordinates corresponding to each corner point in the three-dimensional bounding box are determined. Finally, the global coordinates of the corner points are included in the structured data and returned. If a visualization result needs to be drawn, the corner points can be projected onto the image plane for drawing through the coordinate system projection relationship.
[0115] 106. Construct a geographic model according to the matching point feature information corresponding to each matching point.
[0116] Finally, based on the matching point feature information corresponding to each matching point, GIS technology can be used to model and reconstruct digital landforms, so as to create a high-precision digital earth model (DEM), terrain surface model, and geographic information database, etc., to achieve accurate reconstruction of the topography of the target area.
[0117] Exemplarily, the construction process includes data preparation, coordinate conversion, creation of digital elevation model (DEM), image processing, model building, texture mapping, data analysis and model verification. Specifically, in the data preparation stage, it is necessary to collect and organize the required data, including global coordinate data of the corner points of the 3D frame, elevation data, remote sensing images, geographic auxiliary data, etc. Then in the coordinate conversion stage, the global coordinates of the corner points of the 3D frame are converted into the coordinate system supported by the GIS software to ensure that all data are processed under the same coordinate reference. In the stage of creating a digital elevation model, elevation data, such as a digital elevation model (DEM), are used to create a surface elevation model through an interpolation algorithm. Exemplarily, tools in GIS software, such as triangulation or inverse distance weighted interpolation, can be used to generate a continuous surface elevation model. In the image processing stage, the remote sensing image is aligned with the DEM to obtain more accurate surface texture information. Exemplarily, image processing functions can be used to perform operations such as correction and cropping on the image to meet the needs of digital landform modeling. Next, the global coordinates of the corner points of the 3D frame and the DEM data are used to create a 3D scene in the GIS software. The corner points of the frame are used as the basic points of the terrain model, and the terrain height is determined according to the actual measurement value. Then texture mapping is performed. Specifically, the registered remote sensing image is applied to the terrain model to achieve real surface texture mapping. The image can be projected onto the surface model by mapping or material mapping, and the transparency, tone and other parameters of the texture can be adjusted to make it consistent with the actual landform. Finally, further data analysis and model verification can be performed as needed, such as evaluating the accuracy and credibility of the model by comparing the measured data and the modeling data.
[0118] Based on the above description, Figure 5 A schematic diagram of a device for constructing a geographic model provided in an embodiment of the present application. Figure 5 As shown, the device includes:
[0119] The acquisition unit 501 is used to acquire an image data set corresponding to the target area; the image data set is obtained by performing aerial photography of the target area by a drone; the image data set includes a plurality of image data corresponding to the target area at different aerial photography angles;
[0120] The acquisition unit 501 is further used to acquire multiple detection points in the target area, and radar feature information corresponding to each of the multiple detection points; each detection point corresponds to a continuous wave radar, and the radar feature information corresponding to the detection point is measured by the continuous wave radar; the radar feature information includes vertical height information and physical property information;
[0121] A processing unit 502 is used to input the image data set into the image feature extraction model to output key point feature information of multiple key points;
[0122] The processing unit 502 is further used to match the multiple detection points with the multiple key points, and determine the multiple matching points and the matching point feature information corresponding to each of the multiple matching points according to the matching results; the matching point feature information corresponding to each matching point is associated with the key point feature information and the radar feature information;
[0123] The construction unit 503 is used to construct a geographic model according to the matching point feature information corresponding to each matching point.
[0124] In an optional embodiment, the image feature extraction model includes a fully convolutional backbone network module and a main regression head module connected in sequence;
[0125] Processing unit 502 is specifically used to input the image data set into the full convolution backbone network module to obtain a predicted key point heat map; the full convolution backbone network module is used to predict multiple key points on multiple image data; the predicted key point heat map is input into the main regression head module to obtain key point feature information corresponding to each key point in the multiple key points; the key point feature information is used to determine the two-dimensional bounding box and the three-dimensional bounding box corresponding to each key point on the predicted key point heat map.
[0126] In an optional embodiment, the key point feature information includes depth information;
[0127] The processing unit 502 is specifically used to determine, for any one of the multiple key points, a three-dimensional region of interest ROI corresponding to the key point according to a two-dimensional bounding box corresponding to the key point, a depth feature corresponding to the key point, and a three-dimensional bounding box corresponding to the key point; determine at least one detection point falling into the ROI among the multiple detection points; screen at least one detection point falling into the ROI to determine a target detection point among the at least one detection point; determine that the target detection point matches the key point, and associate the target detection point with the key point to determine a matching point.
[0128] In an optional implementation, the processing unit 502 is further configured to expand each of the plurality of detection points into a three-dimensional cube structure;
[0129] The processing unit 502 is specifically configured to determine at least one detection point falling into the ROI according to multiple three-dimensional cube structures corresponding to multiple detection points; wherein the three-dimensional cube structure corresponding to at least one detection point falling into the ROI overlaps with the ROI.
[0130] In an optional implementation, the acquisition unit 501 is further configured to acquire depth information corresponding to at least one detection point;
[0131] The processing unit 502 is specifically configured to determine a target detection point among the at least one detection point according to the depth information corresponding to the at least one detection point acquired by the acquisition unit; wherein the depth information corresponding to the target detection point indicates a minimum depth.
[0132] In an optional implementation, the acquisition unit 501 is further configured to acquire, for each matching point, radar feature information corresponding to the matching point;
[0133] The processing unit 502 is specifically used to generate a matching point heat map according to the radar feature information corresponding to the matching point; superimpose the matching point heat map and the predicted key point heat map corresponding to the matching point, and input the superposition result into the quadratic regression head model to obtain the matching point feature information corresponding to the matching point; determine the matching point feature information corresponding to the matching point according to the matching point feature information corresponding to the matching point and the key point feature information corresponding to the matching point.
[0134] In an optional embodiment, the construction unit 503 is specifically used to determine the center coordinates of the center point of the two-dimensional bounding box corresponding to each matching point, the information of the three-dimensional bounding box, the size information and the heading information based on the matching point feature information corresponding to each matching point; determine the relative coordinates of each corner point in the three-dimensional bounding box relative to the center point based on the size information and the position of the center point of the two-dimensional bounding box; determine the global coordinates corresponding to each corner point in the three-dimensional bounding box based on the relative coordinates, the center coordinates of the center point and the heading information; and construct a geographic model based on the global coordinates corresponding to each corner point in the three-dimensional bounding box.
[0135] Exemplarily, the steps performed by the above-mentioned units are as follows: Figure 1 The process described in the illustrated embodiment will not be described in detail here.
[0136] See also Figure 6 The embodiment of the present application also shows a device 1000 for constructing a geographic model. The device 1000 may include one or more central processors 1001 and a communication interface 1003. Optionally, a memory 1002 may also be included.
[0137] The memory 1002 may be a temporary storage or a permanent storage. Furthermore, the CPU 1001 may be configured to communicate with the memory 1002 and execute a series of instruction operations in the memory 1002 on the prediction device.
[0138] In this embodiment, the CPU 1001 can execute the aforementioned Figure 1 The steps performed in the illustrated embodiment will not be described in detail here.
[0139] In this embodiment, the specific functional module division in the central processing unit 1001 can be the same as the above Figure 5The division method of each module described in is similar and will not be repeated here.
[0140] It should be noted that the information execution process of the central processor 1001 of the above-mentioned device 1000 and other contents can be specifically referred to in the aforementioned Figure 1 The description in the method embodiment shown will not be repeated here.
[0141] The present invention also provides a computer-readable storage medium, which stores computer instructions; the computer instructions enable the computer to execute the above Figure 1 The steps performed in any one of the embodiments shown.
[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for constructing a geographic model, characterized in that: The construction method comprises: Acquire an image data set corresponding to the target area, wherein the image data set is obtained by performing aerial photography of the target area by a drone; the image data set includes a plurality of image data corresponding to the target area at different aerial photography angles; Acquire multiple detection points in the target area and radar feature information of the multiple detection points; each of the multiple detection points corresponds to a continuous wave radar, and the radar feature information corresponding to each detection point is measured by the continuous wave radar; the radar feature information includes vertical height information and physical property information; Inputting the image data set into an image feature extraction model to output key point feature information of a plurality of key points; Matching the multiple detection points with the multiple key points, and determining multiple matching points and matching point feature information corresponding to each of the multiple matching points according to the matching results; the matching point feature information corresponding to each matching point is associated with the key point feature information and the radar feature information; A geographic model is constructed according to the matching point feature information corresponding to each matching point.
2. The construction method according to claim 1, characterized in that: The image feature extraction model includes a fully convolutional backbone network module and a main regression head module connected in sequence; the image data set is input into the image feature extraction model to output key point feature information of multiple key points, including: Inputting the image data set into the fully convolutional backbone network module to obtain a predicted key point heat map; the fully convolutional backbone network module is used to predict the multiple key points on the multiple image data; The predicted key point heat map is input into the main regression head module to obtain key point feature information corresponding to each of the multiple key points; the key point feature information is used to determine the two-dimensional bounding box and the three-dimensional bounding box corresponding to each of the key points on the predicted key point heat map.
3. The construction method according to claim 2, characterized in that: The key point feature information includes depth information; matching the multiple detection points with the multiple key points and determining multiple matching points according to the matching results includes: For any one of the multiple key points, determine a three-dimensional region of interest ROI corresponding to the key point according to a two-dimensional bounding box corresponding to the key point, a depth feature corresponding to the key point, and a three-dimensional bounding box corresponding to the key point; Determine at least one detection point among the plurality of detection points that falls within the ROI; Screening the at least one detection point falling into the ROI, and determining a target detection point among the at least one detection point; The target detection point and the key point are determined to match, and the target detection point and the key point are associated to determine the matching point.
4. The construction method according to claim 3, characterized in that: The method further comprises: Expanding each of the plurality of detection points into a three-dimensional cubic structure; The determining at least one detection point among the plurality of detection points that falls within the ROI comprises: According to the multiple three-dimensional cube structures corresponding to the multiple detection points, the at least one detection point falling into the ROI is determined; wherein the three-dimensional cube structure corresponding to the at least one detection point falling into the ROI overlaps with the ROI.
5. The construction method according to claim 4, characterized in that: The screening of the at least one detection point falling into the ROI to determine a target detection point among the at least one detection point includes: Acquiring depth information corresponding to the at least one detection point; According to the depth information corresponding to the at least one detection point, a target detection point among the at least one detection point is determined; wherein the depth information corresponding to the target detection point indicates a minimum depth.
6. The construction method according to any one of claims 3 to 5, characterized in that: The determining the matching point feature information corresponding to each matching point in the plurality of matching points includes: For each matching point, obtaining radar feature information corresponding to the matching point; Generate a matching point heat map according to the radar feature information corresponding to the matching point; Obtain a predicted key point heat map corresponding to the matching point; Superimposing the matching point heat map and the predicted key point heat map corresponding to the matching point, and inputting the superimposed result into the secondary regression head model to obtain the matching point feature information corresponding to the matching point; The matching point feature information corresponding to the matching point is determined according to the matching point feature information corresponding to the matching point and the key point feature information corresponding to the matching point.
7. The construction method according to claim 5, characterized in that: The step of constructing a geographic model according to the matching point feature information corresponding to each matching point includes: Determine the center coordinates of the center point of the two-dimensional bounding box corresponding to each matching point, the information of the three-dimensional bounding box, the size information and the heading information according to the matching point feature information corresponding to each matching point; Determine relative coordinates of each corner point in the three-dimensional bounding box relative to the center point according to the size information and the position of the center point of the two-dimensional bounding box; Determine the global coordinates corresponding to each corner point in the three-dimensional bounding box according to the relative coordinates, the center coordinates of the center point and the heading information; The geographic model is constructed according to the global coordinates corresponding to each corner point in the three-dimensional bounding box.
8. A device for constructing a geographical model, characterized in that: The construction device comprises: An acquisition unit is used to acquire an image data set corresponding to a target area; the image data set is obtained by performing aerial photography of the target area by a drone; the image data set includes a plurality of image data corresponding to the target area at different aerial photography angles; The acquisition unit is further used to acquire multiple detection points in the target area and radar feature information corresponding to each of the multiple detection points; each detection point corresponds to a continuous wave radar, and the radar feature information corresponding to the detection point is measured by the continuous wave radar; the radar feature information includes vertical height information and physical property information; A processing unit, used for inputting the image data set into an image feature extraction model to output key point feature information of a plurality of key points; The processing unit is further used to match the multiple detection points with the multiple key points, and determine multiple matching points and matching point feature information corresponding to each of the multiple matching points according to the matching results; the matching point feature information corresponding to each matching point is associated with the key point feature information and the radar feature information; A construction unit is used to construct a geographic model according to the matching point feature information corresponding to each matching point.
9. A device for constructing a geographical model, characterized in that: The geographic model construction device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor, so that the geographic model construction device implements the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions; the computer instructions enable the computer to execute the method according to any one of claims 1 to 7.
