A method, apparatus, and server for aligning a digital map with a 3D model.

Abstract

This application relates to the field of image processing technology, and provides a method, apparatus, and server for aligning a digital map with a 3D model. The method includes: acquiring a digital map; acquiring a 3D model corresponding to the digital map; and aligning the digital map with the 3D model according to preset target control points. Therefore, this application can align a digital map with a 3D model according to preset target control points, thereby meeting the user's need to view a digital map and a 3D model on the same screen.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a method, apparatus and server for aligning digital maps and 3D models. Background Technology

[0002] 3D reconstruction refers to establishing a mathematical model of a 3D object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the object's properties in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. Current technologies utilize computer vision methods to acquire 3D depth information from 2D images and reconstruct 3D models based on this information. However, this approach cannot meet the user's need to view digital maps and 3D models simultaneously on the same screen. Summary of the Invention

[0003] This application provides a method, apparatus, and server for aligning digital maps and 3D models, which can solve the technical problem in the prior art that cannot meet the user's need to view digital maps and 3D models on the same screen.

[0004] In a first aspect, embodiments of this application provide a method for aligning a digital map with a 3D model, including:

[0005] Get digital maps;

[0006] Obtain the 3D model corresponding to the digital map;

[0007] The digital map is aligned with the 3D model based on preset target control points.

[0008] In one possible implementation of the first aspect, obtaining the 3D model corresponding to the digital map includes:

[0009] Determine the target scene corresponding to the digital map;

[0010] The process involves acquiring panoramic image data and laser data corresponding to the target scene; wherein the panoramic image data and laser data are obtained by simultaneously capturing the target scene from a laser camera and a dome camera; and the relative positions of the laser camera and the dome camera are fixed.

[0011] A 3D model is reconstructed based on the panoramic image data to be processed and the laser data.

[0012] In one possible implementation of the first aspect, reconstructing a 3D model based on the panoramic image data to be processed and the laser data includes:

[0013] The panoramic image data to be processed and the laser data are registered so that the pixels in the panoramic image data to be processed correspond one-to-one with the depth points in the laser data.

[0014] Depth estimation is performed on the panoramic image to be processed to generate an initial point cloud;

[0015] The initial point cloud is optimized based on the laser data to obtain the target point cloud;

[0016] Geometric reconstruction is performed based on the target point cloud to obtain a 3D model.

[0017] In one possible implementation of the first aspect, depth estimation is performed based on the panoramic image to be processed to generate an initial point cloud, including:

[0018] Extract feature points from the panoramic image data to be processed;

[0019] The feature points are matched and tracked to obtain key points with matching relationships;

[0020] Estimate a first depth value of the key point based on the key point;

[0021] An initial point cloud is generated based on the first depth value.

[0022] In one possible implementation of the first aspect, optimizing the initial point cloud based on the laser data to obtain the target point cloud includes:

[0023] Based on the one-to-one correspondence between the pixels in the panoramic image data to be processed and the depth points in the laser data, a second depth value corresponding to the depth point is determined.

[0024] Each depth point corresponds to a second depth value;

[0025] The target point cloud is obtained by aligning the initial point cloud with the depth points.

[0026] In one possible implementation of the first aspect, the preset target control point includes real three-dimensional coordinates and virtual three-dimensional coordinates corresponding to the real three-dimensional coordinates;

[0027] Aligning the digital map with the 3D model according to preset target control points includes:

[0028] Projecting the actual three-dimensional coordinates onto a plane yields two-dimensional coordinates;

[0029] The mapping formula is determined based on the two-dimensional coordinates and the virtual three-dimensional coordinates;

[0030] Align the two-dimensional coordinate points of the digital map with the virtual three-dimensional coordinates of the three-dimensional model according to the mapping formula.

[0031] In one possible implementation of the first aspect, the mapping formula is:

[0032] ,in, It is the first vector corresponding to the two-dimensional coordinates in the digital map, the It is the second vector corresponding to the virtual three-dimensional coordinates. It is a translation matrix. It is a rotation matrix.

[0033] Secondly, embodiments of this application provide an alignment device for a digital map and a 3D model, comprising:

[0034] The first acquisition module is used to acquire digital maps;

[0035] The second acquisition module is used to acquire a three-dimensional model corresponding to the digital map;

[0036] The alignment module is used to align the digital map with the 3D model according to preset target control points.

[0037] In one possible implementation of the second aspect, the second acquisition module includes:

[0038] The first determining submodule is used to determine the target scene corresponding to the digital map;

[0039] The first acquisition submodule is used to acquire panoramic image data and laser data to be processed corresponding to the target scene; wherein, the panoramic image data and laser data to be processed are data obtained by simultaneously shooting the target scene with a laser camera and a dome camera; the relative positions of the laser camera and the dome camera are fixed;

[0040] The reconstruction submodule is used to reconstruct a 3D model based on the panoramic image data to be processed and the laser data.

[0041] In one possible implementation of the second aspect, the reconstruction submodule includes:

[0042] The registration unit is used to register the panoramic image data to be processed and the laser data so that the pixels in the panoramic image data to be processed correspond one-to-one with the depth points in the laser data.

[0043] The generation unit is used to perform depth estimation based on the panoramic image to be processed and generate an initial point cloud.

[0044] An optimization unit is used to optimize the initial point cloud based on the laser data to obtain a target point cloud;

[0045] The reconstruction unit is used to perform geometric reconstruction based on the target point cloud to obtain a 3D model.

[0046] In one possible implementation of the second aspect, the generating unit includes:

[0047] An extraction subunit is used to extract feature points from the panoramic image data to be processed;

[0048] The matching and tracking unit is used to match and track the feature points to obtain key points that have matching relationships;

[0049] An estimation unit is used to estimate a first depth value of the key point based on the key point;

[0050] The generation unit is used to generate an initial point cloud based on the first depth value.

[0051] In one possible implementation of the second aspect, the optimization unit includes:

[0052] A determining subunit is used to determine the second depth value corresponding to the depth point based on the one-to-one correspondence between the pixel points in the panoramic image data to be processed and the depth points in the laser data;

[0053] Each depth point corresponds to a second depth value;

[0054] An alignment subunit is used to align the initial point cloud according to the depth point to obtain the target point cloud.

[0055] In one possible implementation of the second aspect, the preset target control point includes real three-dimensional coordinates and virtual three-dimensional coordinates corresponding to the real three-dimensional coordinates;

[0056] The alignment module includes:

[0057] The projection submodule is used to project the actual three-dimensional coordinates onto a plane to obtain two-dimensional coordinates.

[0058] The mapping submodule is used to determine the mapping formula based on the two-dimensional coordinates and the virtual three-dimensional coordinates;

[0059] The alignment submodule is used to align the two-dimensional coordinate points of the digital map with the virtual three-dimensional coordinates of the three-dimensional model according to the mapping formula.

[0060] Thirdly, embodiments of this application provide a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the preceding claims.

[0061] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0062] The beneficial effects of the embodiments in this application compared with the prior art are:

[0063] In this embodiment, a digital map is acquired; a corresponding 3D model is acquired; and the digital map and the 3D model are aligned according to preset target control points, thereby meeting the user's need to view the digital map and the 3D model on the same screen. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the method for aligning a digital map with a 3D model provided in an embodiment of this application.

[0066] Figure 2 This is a structural block diagram of the alignment device for digital maps and 3D models provided in this application embodiment;

[0067] Figure 3 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0068] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0069] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0070] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0071] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0072] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0073] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0074] The technical solutions provided in the embodiments of this application will be described below through specific examples.

[0075] See Figure 1 This is a flowchart illustrating the method for aligning a digital map with a 3D model provided in an embodiment of this application. It is intended as an example and not a limitation. This method can be applied to a server and may include the following steps:

[0076] Step S101: Obtain the digital map.

[0077] The digital map can be Baidu Maps, Gaode Maps, Google Maps, etc.

[0078] Step S102: Obtain the 3D model corresponding to the digital map.

[0079] The data source for the 3D model can be obtained by the user based on photos taken with a depth camera.

[0080] For example, the depth camera can be a 3D camera, which includes a laser camera and a dome camera. The relative positions of the laser camera and the dome camera are fixed. The laser camera includes a first laser camera (i.e., an UpLidar laser camera), a second laser camera (i.e., a MidLidar laser camera), and a third laser camera (i.e., a DownLidar laser camera) with fixed relative positions. For example, the UpLidar laser camera, the MidLidar laser camera, and the DownLidar laser camera are arranged alternately in the order of top, middle, and bottom on the same vertical line. In addition, the 3D camera rotates 6 times at a 60-degree rotation angle at the same point to finally obtain a 360° panoramic image.

[0081] Optionally, the depth camera can also be an eight-lens camera, which consists of two groups, one above the other, each with four fisheye lenses. The four lenses capture images from the four groups and stitch them together to form a 360° panoramic image.

[0082] In practical applications, obtaining the 3D model corresponding to the digital map includes:

[0083] Step S201: Determine the target scene corresponding to the digital map.

[0084] It should be noted that the target scene corresponding to the digital map needs to be determined manually. The target scene can be a scene whose scene information is consistent with the geographical information represented by the digital map.

[0085] Step S202: Obtain the panoramic image data and laser data to be processed corresponding to the target scene.

[0086] The panoramic image data to be processed and the laser data are respectively obtained by the laser camera and the dome camera shooting the target scene from the same view; the relative positions of the laser camera and the dome camera are fixed.

[0087] Step S203: Reconstruct a 3D model based on the panoramic image data to be processed and the laser data.

[0088] In specific applications, a 3D model is reconstructed based on the panoramic image data and laser data to be processed, including:

[0089] Step S301: Register the panoramic image data to be processed and the laser data so that the pixels in the panoramic image data to be processed correspond one-to-one with the depth points in the laser data.

[0090] Step S302: Depth estimation is performed based on the panoramic image to be processed to generate an initial point cloud.

[0091] In practical applications, depth estimation is performed on the panoramic image to be processed to generate an initial point cloud, including:

[0092] Step S401: Extract feature points from the panoramic image data to be processed.

[0093] The extraction methods include, but are not limited to, Harris corner detection, FAST corner detection, SIFI extraction algorithm, and SURF extraction algorithm.

[0094] Step S402: Match and track the feature points to obtain key points with matching relationships.

[0095] Among them, the key points that have a matching relationship refer to the feature points that correspond to real points in the same world coordinate system.

[0096] Step S403: Estimate the first depth value of the keypoints based on the keypoints.

[0097] In practical applications, a visual odometry module is used to apply epipolar geometric constraints to key points, and then the singular value decomposition (SVD) method is used to triangulate the key points to obtain the first depth value of the key points.

[0098] Step S404: Generate an initial point cloud based on the first depth value.

[0099] In practical applications, the three-dimensional coordinates of the point cloud are obtained according to the following formula:

[0100] ,

[0101] Where (u, v) are the pixel coordinates of each keypoint in the image data, d is the first depth value of each keypoint in the image data, K is the intrinsic parameter of the dome camera, and (X, Y, Z) are the three-dimensional coordinates of the point cloud.

[0102] Step S303: Optimize the initial point cloud based on the laser data to obtain the target point cloud.

[0103] In practical applications, the initial point cloud is optimized based on laser data to obtain the target point cloud, including:

[0104] Step S501: Based on the one-to-one correspondence between pixels in the panoramic image data to be processed and depth points in the laser data, determine the second depth value corresponding to the depth point.

[0105] Each depth point corresponds to a second depth value.

[0106] Understandably, the purpose of registration is to establish the correspondence between the coordinate system of pixels in the image data and the coordinate system of depth points in the laser data. Specifically, the coordinate system of pixels in the image data is the image coordinate system, and the coordinate system of depth points in the laser data is the polar coordinate system.

[0107] Step S502: Align the initial point cloud with the depth points to obtain the target point cloud.

[0108] Preferably, the image data and laser data are registered so that the pixels in the image data correspond one-to-one with the depth points in the laser data.

[0109] Understandably, the purpose of registration is to establish the correspondence between the coordinate system of pixels in the image data and the coordinate system of depth points in the laser data. Specifically, the coordinate system of pixels in the image data is the image coordinate system, and the coordinate system of depth points in the laser data is the polar coordinate system.

[0110] Specifically, the target point cloud is obtained by aligning the initial point cloud with depth points, including:

[0111] (1) Based on the one-to-one correspondence between pixels in the image data and depth points in the laser data, determine the second depth value corresponding to each depth point. Each depth point corresponds to a second depth value.

[0112] The second depth value of the depth point is directly measured by the laser camera.

[0113] (2) Align the initial point cloud with the depth points to obtain the target point cloud.

[0114] In practical applications, a laser odometry module is used to extract the curvature detection edges and planar features of the initial point cloud. The distorted initial point cloud is determined based on the curvature detection edges and planar features. After removing the distorted initial point cloud, the initial point cloud is optimized for point cloud registration and motion estimation based on the second depth value of the depth point using the ICP algorithm.

[0115] In practical applications, point clouds are displayed on web pages using a layered loading method based on LOD (using an octree). The panoramic image in this application has a good roaming transition effect based on point clouds, which is achieved through WebGL shaders.

[0116] Step S304: Perform geometric reconstruction based on the target point cloud to obtain a 3D model.

[0117] The target point cloud is semantically recognized based on the pre-trained PointNet neural network architecture, thereby segmenting the point cloud corresponding to each object and obtaining a 3D model.

[0118] Step S103: Align the digital map with the 3D model according to the preset target control points.

[0119] The preset target control points include real 3D coordinates and virtual 3D coordinates corresponding to the real 3D coordinates. The real 3D coordinates are GIS coordinates, and the virtual 3D coordinates are local coordinates.

[0120] In practical applications, aligning the digital map with the 3D model according to preset target control points includes:

[0121] Step S601: Project the real three-dimensional coordinates onto the plane to obtain two-dimensional coordinates.

[0122] Two-dimensional coordinates refer to planar coordinates.

[0123] Step S602: Determine the mapping formula based on the two-dimensional coordinates and the virtual three-dimensional coordinates.

[0124] The mapping formula is as follows:

[0125] ,in, It is the first vector corresponding to the two-dimensional coordinates in the digital map. It is the second vector corresponding to the virtual three-dimensional coordinates. It is a translation matrix. It is a rotation matrix.

[0126] Step S603: Align the two-dimensional coordinate points of the digital map with the virtual three-dimensional coordinates of the three-dimensional model according to the mapping formula.

[0127] In practical applications, there are usually three preset target control points. Each preset target control point has two sets of coordinates: one set of GIS coordinates and one set of local coordinates. This is used to establish the mapping from GIS coordinates to local coordinates.

[0128] Understandably, the GIS coordinate system used in the application implementation is the Geodetic 2000 coordinate system (which also supports 84 coordinates). Through Mercator projection on the vertical axis, the GIS coordinates are converted into planar coordinates. Then, based on preset target control points, the mapping relationship between these planar coordinates and local coordinates is calculated, thus converting the GIS coordinates into local coordinates. Similarly, converting local coordinates into planar coordinates and then restoring them to the Geodetic 2000 coordinate system converts them back into GIS coordinates. This conversion requires both the front-end and server-side implementations.

[0129] In this embodiment, a digital map is acquired; a corresponding 3D model is acquired; and the digital map and the 3D model are aligned according to preset target control points, thereby meeting the user's need to view the digital map and the 3D model on the same screen.

[0130] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] Corresponding to the method described in the above embodiments, Figure 2 The diagram shows a structural block diagram of an alignment device for digital maps and 3D models provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0132] Reference Figure 2 The device includes:

[0133] The first acquisition module 21 is used to acquire digital maps;

[0134] The second acquisition module 22 is used to acquire a three-dimensional model corresponding to the digital map;

[0135] Alignment module 23 is used to align the digital map with the three-dimensional model according to preset target control points.

[0136] In one possible implementation, the second acquisition module includes:

[0137] The first determining submodule is used to determine the target scene corresponding to the digital map;

[0138] The first acquisition submodule is used to acquire panoramic image data and laser data to be processed corresponding to the target scene; wherein, the panoramic image data and laser data to be processed are data obtained by simultaneously shooting the target scene with a laser camera and a dome camera; the relative positions of the laser camera and the dome camera are fixed;

[0139] The reconstruction submodule is used to reconstruct a 3D model based on the panoramic image data to be processed and the laser data.

[0140] In one possible implementation, the reconstruction submodule includes:

[0141] The registration unit is used to register the panoramic image data to be processed and the laser data so that the pixels in the panoramic image data to be processed correspond one-to-one with the depth points in the laser data.

[0142] The generation unit is used to perform depth estimation based on the panoramic image to be processed and generate an initial point cloud.

[0143] An optimization unit is used to optimize the initial point cloud based on the laser data to obtain a target point cloud;

[0144] The reconstruction unit is used to perform geometric reconstruction based on the target point cloud to obtain a 3D model.

[0145] In one possible implementation, the generating unit includes:

[0146] An extraction subunit is used to extract feature points from the panoramic image data to be processed;

[0147] The matching and tracking unit is used to match and track the feature points to obtain key points that have matching relationships;

[0148] An estimation unit is used to estimate a first depth value of the key point based on the key point;

[0149] The generation unit is used to generate an initial point cloud based on the first depth value.

[0150] In one possible implementation, the optimization unit includes:

[0151] A determining subunit is used to determine the second depth value corresponding to the depth point based on the one-to-one correspondence between the pixel points in the panoramic image data to be processed and the depth points in the laser data;

[0152] Each depth point corresponds to a second depth value;

[0153] An alignment subunit is used to align the initial point cloud according to the depth point to obtain the target point cloud.

[0154] In one possible implementation, the preset target control point includes real three-dimensional coordinates and virtual three-dimensional coordinates corresponding to the real three-dimensional coordinates;

[0155] The alignment module includes:

[0156] The projection submodule is used to project the actual three-dimensional coordinates onto a plane to obtain two-dimensional coordinates.

[0157] The mapping submodule is used to determine the mapping formula based on the two-dimensional coordinates and the virtual three-dimensional coordinates;

[0158] The alignment submodule is used to align the two-dimensional coordinate points of the digital map with the virtual three-dimensional coordinates of the three-dimensional model according to the mapping formula.

[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0160] Figure 3 This is a schematic diagram of the structure of a server provided in one embodiment of this application. Figure 3 As shown, the server 3 in this embodiment includes: at least one processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30. When the processor 30 executes the computer program 32, it implements the steps in any of the above-described method embodiments.

[0161] The server 3 may be a computing device such as a cloud server. This server may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of server 3 and does not constitute a limitation on server 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0162] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0163] In some embodiments, the memory 31 may be an internal storage unit of the server 3, such as a hard drive or memory of the server 3. In other embodiments, the memory 31 may be an external storage device of the server 3, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the server 3. Furthermore, the memory 31 may include both internal storage units and external storage devices of the server 3. The memory 31 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0165] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a server, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0168] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method of aligning a digital map with a three-dimensional model, characterized in that, The method comprises the following steps: acquiring a digital map; acquiring a three-dimensional model corresponding to the digital map, comprising: determining a target scene corresponding to the digital map; acquiring to-be-processed panoramic image data and laser data corresponding to the target scene; wherein the to-be-processed panoramic image data and the laser data are data obtained by simultaneously photographing the target scene by a laser camera and a spherical screen camera; the relative positions of the laser camera and the spherical screen camera are fixed; reconstructing a 3D model according to the to-be-processed panoramic image data and the laser data, comprising: registering the to-be-processed panoramic image data and the laser data, so that the pixel points in the to-be-processed panoramic image data and the depth points of the laser data correspond one by one; performing depth estimation according to the to-be-processed panoramic image to generate an initial point cloud, comprising: extracting feature points of the to-be-processed panoramic image data; performing matching tracking on the feature points to obtain key points with a matching relationship; estimating a first depth value of the key points according to the key points; generating an initial point cloud according to the first depth value; optimizing the initial point cloud according to the laser data to obtain a target point cloud, comprising: determining a second depth value corresponding to the depth points based on the one-to-one correspondence relationship between the pixel points in the to-be-processed panoramic image data and the depth points of the laser data; wherein each depth point corresponds to a second depth value; aligning the initial point cloud according to the depth points to obtain a target point cloud; performing geometric reconstruction according to the target point cloud to obtain a 3D model; the data source of the three-dimensional model is obtained by photographing by a depth camera; the depth camera is a 3D camera, the 3D camera comprises a laser camera and a spherical screen camera, the relative positions of the laser camera and the spherical screen camera are fixed, the laser camera comprises UpLidar laser cameras, MidLidar laser cameras and DownLidar laser cameras with fixed relative positions, the UpLidar laser cameras, the MidLidar laser cameras and the DownLidar laser cameras are arranged in the order of up, middle and down on the same vertical line, and the 3D camera rotates 6 times at a rotation angle of 60 degrees at the same point to finally obtain a 360° panoramic image; aligning the digital map and the three-dimensional model according to a preset target control point.

2. The method of aligning a digital map with a three-dimensional model of claim 1, wherein, The preset target control point comprises a real three-dimensional coordinate and a virtual three-dimensional coordinate corresponding to the real three-dimensional coordinate; aligning the digital map and the three-dimensional model according to a preset target control point, comprising: projecting the real three-dimensional coordinate to a plane to obtain a two-dimensional coordinate; determining a mapping formula according to the two-dimensional coordinate and the virtual three-dimensional coordinate; aligning the two-dimensional coordinate points of the digital map and the virtual three-dimensional coordinates of the three-dimensional model according to the mapping formula.

3. The method of aligning a digital map with a three-dimensional model of claim 2, wherein, The mapping formula is: wherein, is a first vector corresponding to a two-dimensional coordinate pair in the digital map, the is a second vector corresponding to the virtual three-dimensional coordinate, is a translation matrix, is a rotation matrix.

4. An apparatus for aligning a digital map with a three-dimensional model, characterized by: comprising: a first acquisition module for acquiring a digital map; The second acquisition module is configured to acquire a three-dimensional model corresponding to the digital map, including: determining a target scene corresponding to the digital map; acquiring to-be-processed panoramic image data and laser data corresponding to the target scene; wherein the to-be-processed panoramic image data and the laser data are data obtained by simultaneously photographing the target scene by a laser camera and a spherical screen camera; the laser camera and the spherical screen camera have a fixed relative position; a 3D model is reconstructed according to the to-be-processed panoramic image data and the laser data, including: registering the to-be-processed panoramic image data and the laser data, so that a pixel point in the to-be-processed panoramic image data and a depth point of the laser data correspond to each other; performing depth estimation according to the to-be-processed panoramic image to generate an initial point cloud, including: extracting feature points of the to-be-processed panoramic image data; performing matching tracking on the feature points to obtain key points having a matching relationship; estimating a first depth value of the key points according to the key points; generating an initial point cloud according to the first depth value; optimizing the initial point cloud according to the laser data to obtain a target point cloud, including: determining a second depth value corresponding to the depth point based on the one-to-one correspondence relationship between the pixel point in the to-be-processed panoramic image data and the depth point of the laser data; wherein each depth point corresponds to a second depth value; aligning the initial point cloud according to the depth point to obtain a target point cloud; performing geometric reconstruction according to the target point cloud to obtain a 3D model; the data source of the three-dimensional model is obtained by photographing by a depth camera; the depth camera is a 3D camera, the 3D camera includes a laser camera and a spherical screen camera, the laser camera and the spherical screen camera have a fixed relative position, the laser camera includes UpLidar laser cameras, MidLidar laser cameras, and DownLidar laser cameras having a fixed relative position, the UpLidar laser cameras, the MidLidar laser cameras, and the DownLidar laser cameras are arranged in an upper, middle, and lower order on a same vertical line, and the 3D camera rotates 6 times at a rotation angle of 60 degrees at a same point to finally obtain a 360° panoramic image; The alignment module is configured to align the digital map and the three-dimensional model according to a preset target control point.

5. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the method of any one of claims 1 to 3.

Images