A 3D reconstruction method and system for large-scale scene maps

Abstract

The present application relates to a three-dimensional reconstruction method for a large-scale scene map. The method includes: dividing a target scene into multiple sub-regions according to a first preset rule, configuring multiple shooting paths in the sub-regions, and configuring multiple shooting sequences in the shooting paths, where the shooting sequences include movement route information and viewing angle information; constructing a reconstruction process tree of the target scene with the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes; acquiring image data collected according to the shooting sequences, and reconstructing a three-dimensional map of the target scene based on the image data and the reconstruction process tree. Through the present application, the problems of low efficiency and poor reliability in the reconstruction method of large-scale three-dimensional scene maps in related technologies are solved, and the reconstruction rate and system robustness are improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional map reconstruction, and particularly to a method and system for three-dimensional reconstruction of large-scale scene maps. Background Art

[0002] Scene three-dimensional reconstruction is to reconstruct the three-dimensional space information of a real scene as the three-dimensional map of the scene. With the rise of technical concepts such as augmented reality, robotics, and digital twins, the demand for constructing three-dimensional map information of a specific scene (usually expressed by a scene three-dimensional model) is also continuously expanding. A high-quality scene three-dimensional model can provide an accurate spatial reference and a reliable digital archive for the above applications, and is an important foundation for the implementation of these application technologies. Generally speaking, the larger the scale of the scene, the greater the challenge of three-dimensional reconstruction, and the higher the costs such as time, manpower, and equipment.

[0003] In related technologies, there are two categories of technical routes for supporting large-scale scene Figure 3 three-dimensional reconstruction as follows:

[0004] 1. Obtain data by scanning an offline scene with a lidar (stationary or handheld), and reconstruct a three-dimensional map based on the lidar data; however, the price of lidar equipment is expensive and the maintenance cost is relatively high. In addition, color information of the scene cannot be directly obtained;

[0005] 2. Obtain data by taking RGB pictures (such as oblique photography) of an offline scene, and reconstruct a three-dimensional map based on the image data; this method has simple steps and low costs in the data acquisition link. However, the algorithm reconstruction link is more difficult. Especially for some difficult scenes (such as large scale, many mismatches, and many weak textures), it is difficult for common three-dimensional reconstruction software to complete the reconstruction at one time with low manpower and time costs at present, and the reliability of this method is relatively low.

[0006] Currently, no effective solution has been proposed for the problem of relatively low reliability of the three-dimensional scene map reconstruction method in related technologies. Summary of the Invention

[0007] Embodiments of this application provide a method, a system, a computer device, and a computer-readable storage medium for three-dimensional reconstruction of a large-scale scene map, so as to at least solve the problem of relatively low reliability of the three-dimensional scene map reconstruction method.

[0008] In a first aspect, embodiments of this application provide a method for three-dimensional reconstruction of a large-scale scene map, and the method includes:

[0009] According to the first preset rule, divide the target scene into multiple sub-regions, configure multiple shooting paths in the sub-regions, and configure multiple shooting sequences in the shooting paths, where the shooting sequences include movement route information and perspective information;

[0010] Taking the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes, construct the reconstruction process tree of the target scene;

[0011] Obtain the image data collected according to the shooting sequences, and reconstruct the three-dimensional map of the target scene based on the image data and the reconstruction process tree.

[0012] In some embodiments, the reconstructing the three-dimensional map of the target scene based on the scene data and the reconstruction process tree includes:

[0013] According to the second preset rule, configure the association method of the image data within the third branch node and the association method between each branch node, and define the association information according to the association method, where the association method includes one or more combinations of brute-force matching, sequence matching, spatial matching, and manual matching;

[0014] According to the third preset rule, configure the mapping method of each branch node, where the mapping method includes one or more combinations of cross-layer mapping, incremental mapping, and splicing mapping;

[0015] Reconstruct the three-dimensional map of the target scene according to the association information and the mapping method.

[0016] In some embodiments, the reconstructing the three-dimensional map of the target scene based on the image data and the reconstruction process tree includes:

[0017] In the case of adopting the incremental mapping, the three-dimensional map reconstruction algorithm combines one or more of the post-order traversal principle, the hierarchical traversal principle, and the manual traversal rule according to the fourth preset rule to merge the mapping data of each branch node to obtain the three-dimensional map of the target scene, where the mapping data includes the image data and the association information;

[0018] In the case of adopting the splicing mapping, the three-dimensional map reconstruction algorithm independently reconstructs the node three-dimensional maps of each branch node, and respectively determines the pose information of each node three-dimensional map in the global map coordinate system,

[0019] According to the pose information, transform the node three-dimensional map from the node map coordinate system to the global map coordinate system, and splice the node three-dimensional maps according to the fourth preset rule to obtain the three-dimensional map of the target scene;

[0020] In the case of adopting the cross-layer mapping, the three-dimensional map reconstruction algorithm performs global map reconstruction based on the image data subordinate to all the third branch nodes to obtain the three-dimensional map of the target scene.

[0021] In some embodiments, when an error occurs in reconstructing the three-dimensional map of the target scene by using one or a combination of the incremental mapping and the stitching mapping, the method further includes:

[0022] Interrupting the reconstruction task, determining the fault point of the reconstruction task, and repairing the mapping data corresponding to the fault point;

[0023] Obtaining the merging result of the reconstruction task before reaching the fault point, and restarting the reconstruction task again based on the merging result and the repaired mapping data.

[0024] In some embodiments, after the three-dimensional map of the target scene is reconstructed, the method further includes:

[0025] In the case where the three-dimensional map needs to supplement new branch nodes,

[0026] Determining the mounting position of the node to be updated in the reconstruction process tree according to the relevance between the mapping data of the node to be updated and the mapping data of the existing nodes;

[0027] Configuring the association mode and the mapping mode between the node to be updated and the existing nodes according to the second preset rule and the third preset rule, and defining the association information through the association mode;

[0028] Performing the supplement task of the three-dimensional map of the target scene based on the mapping data of the branch node to be updated through the three-dimensional map reconstruction algorithm.

[0029] In some embodiments, after the three-dimensional map of the target scene is reconstructed, the method further includes:

[0030] In the case where the three-dimensional map needs to delete existing branch nodes,

[0031] Determining the node to be deleted according to the manual interaction information, and removing the node to be deleted, the mapping data and the association information related to the node to be deleted in the reconstruction process tree.

[0032] In some embodiments, when the image data is video data, the method further includes:

[0033] Performing redundancy elimination on the video data to obtain picture data, and reconstructing the three-dimensional map of the target scene based on the picture data and the reconstruction process tree, where the redundancy elimination includes:

[0034] Obtain the moving speed of the imaging device when the video data is captured, and determine the sampling interval according to the moving speed;

[0035] In the video data, perform periodic sampling according to the sampling interval to obtain the picture data.

[0036] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction system for a large-scale scene map. The system includes: a process tree construction module and a map reconstruction module, where,

[0037] The process tree construction module is used to divide the target scene into multiple sub-regions according to a first preset rule, configure multiple shooting paths in the sub-regions, and configure multiple shooting sequences in the shooting paths. Among them, the shooting sequences include movement route information and viewing angle information, and

[0038] Taking the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes, construct the reconstruction process tree of the target scene;;

[0039] The map reconstruction module is used to obtain the image data collected according to the shooting sequence, and reconstruct the three-dimensional map of the target scene based on the image data and the reconstruction process tree.

[0040] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements a three-dimensional reconstruction method for a large-scale scene map as described in the first aspect above.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements a three-dimensional reconstruction method for a large-scale scene map as described in the first aspect above.

[0042] Compared with the related art, a three-dimensional reconstruction method for a large-scale scene map provided by an embodiment of the present application has the following beneficial effects:

[0043] 1. Select a suitable mapping method in combination with the specific situation of the target scene. For example, when the scene is relatively complex, select incremental mapping to reduce or avoid errors; when the scene is relatively simple, select cross-layer mapping to improve efficiency.

[0044] 2. Freely define the association information of each component data in the reconstruction task. Compared with the method of directly using brute-force matching to define the association information of component data in the related art, it can reduce the redundancy of matching relationships and incorrect matches, improve the speed of map reconstruction, and reduce the error probability.

[0045] 3. When using incremental mapping or stitching mapping, if the mapping process is interrupted, after fixing the problem, the merged results can be directly reused instead of starting the merging from scratch.

[0046] 4. When it is necessary to update the scene map, for example, when supplementing a new target area, only need to define the data corresponding to the area to be updated as a new area node and specify the association information between the new node and the original node. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0048] Figure 1 is a schematic diagram of the application environment of a three-dimensional reconstruction method for a large-scale scene map according to an embodiment of the present application;

[0049] Figure 2 is a flowchart of a three-dimensional reconstruction method for a large-scale scene map according to an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of the division result of a target scene according to an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of a reconstruction process tree according to an embodiment of the present application;

[0052] Figure 5 is a structural block diagram of a three-dimensional reconstruction system for a large-scale scene map according to an embodiment of the present application;

[0053] Figure 6 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be described and explained below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided in the present application without creative efforts fall within the scope of protection of the present application.

[0055] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. In addition, it can also be understood that although the efforts made in such a development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing, or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be understood as the content disclosed in the present application being insufficient.

[0056] The mention of "embodiment" in this application means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of this application. The appearance of this phrase at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments without conflict.

[0057] Unless otherwise defined, the technical terms or scientific terms involved in this application should have the ordinary meaning understood by those with ordinary skills in the technical field to which this application belongs. The words such as "a", "an", "one kind", "the" and the like involved in this application do not indicate a quantity limitation and can represent a singular or plural number. The terms "including", "comprising", "having" and any variations thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include unlisted steps or units, or may also include other steps or units inherent to these processes, methods, products or devices. The terms "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0058] A three-dimensional reconstruction method for a large-scale scene map provided by an embodiment of this application can be applied in an application environment as Figure 1 shown. Figure 1Schematic diagram of the application environment of a three-dimensional reconstruction method for a large-scale scene map according to an embodiment of the present application, as shown in Figure 1 As shown, the terminal 10 acquires image data of the target scene, which can be a station-type device set in the target scene or a movable portable device. Further, a three-dimensional map reconstruction algorithm is deployed in the server 11. Through this three-dimensional map reconstruction algorithm, it can reconstruct a three-dimensional map of the target scene according to the image data sent by the terminal 10. This three-dimensional map can be used on platforms such as augmented reality and digital twin to provide an implementation basis for them. It should be noted that the terminal 10 in this embodiment can be an oblique photography device, a handheld camera, a pan-tilt camera, a smartphone, etc., and the server 11 can be a single server or a cluster composed of multiple servers.

[0059] Figure 2 Flowchart of a three-dimensional reconstruction method for a large-scale scene map according to an embodiment of the present application, as shown in Figure 2 As shown, the process includes the following steps:

[0060] S201, according to the first preset rule, divide the target scene into multiple sub-regions, configure multiple shooting paths in the sub-regions, and configure multiple shooting sequences in the shooting paths, where the shooting sequence includes movement route information and viewing angle information;

[0061] In this embodiment, the first preset rule is formulated manually according to the actual situation of the target scene. For example, in the top view of the target scene, divide the target scene according to the building shape, fixed interval distance, etc.; configure the shooting paths and shooting sequences according to the regional complexity, fixed interval distance, etc.

[0062] Further, when configuring the shooting sequence, the movement route of the camera device and the viewing angle of the camera device should be marked when shooting the scene image, so as to obtain comprehensive and accurate on-site images, thereby improving the authenticity of the subsequent generated scene map.

[0063] S202, taking the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes, construct the reconstruction process tree of the target scene;

[0064] In the reconstruction process tree, except for the top-level scene node, each node has its own parent node, that is, the "subordinate relationship"; except for the bottom-level shooting sequence node, the map nodes at other levels have their own branch nodes, that is, the "inclusion relationship". According to the subordinate relationship and the inclusion relationship, each node can find the shooting image data divided within the scope of this node, and in the subsequent map reconstruction process, each node can also find which upper-level nodes the map data reconstructed by itself can be merged with.

[0065] S203. Obtain the image data collected according to the shooting sequence, and reconstruct the three-dimensional map of the target scene based on the image data and the reconstruction process tree.

[0066] The image data can be obtained by on-site personnel using a handheld camera device; when shooting the image data, according to the movement route and perspective indicated in step S201 above, the image data corresponding to each shooting sequence is sequentially shot.

[0067] It should be noted that in order to obtain more comprehensive image data of the target scene, for each shooting path, multiple sets of image data need to be obtained under the condition of changing the shooting movement route, the shooting perspective or the rotation angle of the camera device, and each set of image data corresponds to a shooting sequence.

[0068] Furthermore, when the above image data is video data, since there are a large number of similar or repeated images in a set of video data, directly using the video data for scene map reconstruction will slow down the map reconstruction rate. Therefore, redundancy elimination is required to obtain the effective image data that needs to be added to the mapping process, and redundancy elimination can be performed by manual screening or automatic sampling.

[0069] Execute the above reconstruction task through a three-dimensional map reconstruction algorithm. Optionally, use the SFM (Structure From Motion) algorithm to reconstruct the three-dimensional map of the target scene based on the above reconstruction process tree and image data. This SFM algorithm can automatically restore the scene structure based on two or more scenes, and it is an algorithm that can achieve camera tracking and motion matching.

[0070] Through the above steps S201 to S204, compared with the method of directly reconstructing the three-dimensional map of a large-scale scene based on laser data and RGB data in the related art, in the embodiment of the present application, a large-scale target scene is divided into multiple levels of branch nodes, and further a reconstruction process tree is constructed based on the branch nodes.

[0071] During the process of reconstructing the three-dimensional map of the target scene, due to the inclusion relationship and subordination relationship between nodes, refined hierarchical management can be carried out, which can freely specify the mapping method and association information of each branch node; when abnormal problems such as mapping errors occur, the existing merged results can also be reused after repairing the abnormal problems without starting over; in addition, when a certain sub-region needs to be updated, only a new map node needs to be defined in the process tree and its association relationship is defined, which will not affect other existing regions. Through the three-dimensional reconstruction method of the large-scale scene map provided by this embodiment, the reconstruction efficiency and reliability of the three-dimensional map are greatly improved.

[0072] The scenario division described in the above step S201 is illustrated by taking a specific application scenario as an example. Figure 3 is a schematic diagram of a target scene division result according to an embodiment of the present application, such as Figure 3 As shown in FIG. 1 , a three-dimensional map of an underground garage of a large shopping mall is reconstructed, and the division based on the floor plan of the garage includes the following steps:

[0073] The first step is to use the entire parking lot scene as the largest map node. Optionally, this layer of map nodes is called a "scene layer", where the "scene layer" can be named "parking";

[0074] The second step is, Figure 3 As shown by the dotted line in the middle, the garage scene is divided into three sub-areas, each of which is a map node, and the map nodes of this layer are called "location layer", where each sub-area can be named in the form of "C1, C2, C3...Cn";

[0075] The third step is to plan multiple shooting paths in each sub-area, and plan multiple shooting sequences in each shooting path. Figure 3 As shown, the shaded area is the shooting path. In the shooting path, the line segment connected by multiple small circles is the shooting sequence. Optionally, the direction of the fan-shaped area in the small circle indicates the camera's viewing angle when shooting the image. The shooting path corresponds to a map node, and this layer is called the "route layer". Each shooting path can be named in the form of "R1, R2, R3...Rn"; the shooting sequence corresponds to a map node, and this layer is called the "sequence layer". Each shooting sequence can be named in the form of "seq1, seq2, seq3...seqn";

[0076] In the fourth step, the acquisition personnel collect multiple sets of video data according to each shooting sequence, and remove the redundancy of these video data to obtain image data. Each set of image data corresponds to a sequence node, and each sequence node contains a set of image data.

[0077] The map nodes at all levels obtained after the target scene is divided are represented in the form of a tree diagram; for example, the division result of the above underground parking lot can be expressed as follows Figure 4 express, Figure 4 is a schematic diagram of a process tree reconstruction according to an embodiment of the present application, such as Figure 4 As shown, the topmost root node is the scene node (parking), and from top to bottom they are: the sub-area corresponding to the first branch node, the shooting path corresponding to the second branch node, and the shooting sequence corresponding to the third branch node.

[0078] In some of these embodiments, in the three-dimensional mapping method, the association relationship between data is a key piece of information. After obtaining the association relationship between each piece of data, the reconstruction task can be executed;

[0079] Considering the problems such as redundant matching relationships and matching errors caused by directly using brute-force matching during the mapping process, the embodiments of the present application can configure the association method for mapping data hierarchically and precisely according to the reconstruction process tree, including:

[0080] According to the second preset rule, configure the association method for the image data subordinate to the third branch node, and configure the association method between each branch node, and define the association information through the association method. Among them, the association method includes one or a combination of more of: brute-force matching, sequence matching, spatial matching, and manual matching.

[0081] Brute-force matching: That is, pair each object in two data sets one by one. Sequence matching: That is, in a set of image data, each picture is only paired with several pictures before and after. Spatial matching: That is, in a set of image data, each picture is only paired with the images within a specific distance range (such as the distance from the optical center of the imaging device). Manual matching: That is, specify the pairing relationship or prohibit the pairing relationship by manually outputting an interaction instruction. When an association between certain data may cause mapping errors, these data can be specified as irrelevant through this manual matching method, thereby avoiding mapping errors caused by incorrect matching relationships.

[0082] In this embodiment, the second preset rule is formulated manually according to the actual situation of the target scene. Based on each map node, the association method between each layer of map nodes and the matching method of the data within each map node can be customized.

[0083] Optionally, an example of configuring the association method and generating the scene association information is as follows:

[0084] For the image data subordinate to the third branch, match it by combining the sequence matching method and the spatial matching method;

[0085] For the shooting path corresponding to the second branch node, specify the association information between the routes according to the manual matching method to obtain a route pair by matching two routes; further, generate the association relationship of each image data in the route pair according to the brute-force matching method; finally, generate the association relationship of each picture data in the image data again according to the brute-force matching method.

[0086] In some of these embodiments, through the reconstructed process tree, the embodiments of the present application can also set multiple mapping methods for each branch node, which can be flexibly selected according to the target site conditions. Specifically, according to the third preset rule, the mapping methods of each node are configured, where the mapping method includes one or a combination of more of the following: cross-layer mapping, incremental mapping, and splicing mapping. It should be noted that the third preset rule is also formulated manually according to the actual situation of the target scenario.

[0087] For some scenarios with large scale, complex content or prone to errors, incremental mapping can be selected. According to the fourth preset rule through the SFM algorithm, the mapping data (including image data and associated information) of each branch node is sequentially merged to obtain the three-dimensional map of the target scenario.

[0088] It should be noted that the fourth preset rule is used to indicate the traversal method of the algorithm during map reconstruction. In this embodiment, the fourth preset rule is not limited to a specific rule, and it can be one or a combination of more of the following: post-order traversal principle, level traversal principle, and manually specified traversal rules.

[0089] Taking a specific scenario as an example to illustrate incremental mapping. For example, reconstructing the three-dimensional map of the C6_R1 node in the above embodiment includes the following steps:

[0090] First step, use the SFM algorithm to perform three-dimensional reconstruction on the seq1 node to obtain the three-dimensional map result of the seq1 node, and use this three-dimensional map result as the initial merge result, that is, the 0th merge;

[0091] Second step, taking the result of the 0th merge and the mapping data of the current branch node seq2 to be merged as the input, perform three-dimensional reconstruction again through the SFM algorithm, that is, obtain the 1st merge result;

[0092] Third step, if there are other branch nodes in C6_R1, then sequentially traverse other branch nodes, and continue to merge other branch nodes according to the second step above until all branch nodes are merged to obtain the three-dimensional map of the C6_R1 node.

[0093] Based on the above embodiment, if the incremental mapping method is fully used to obtain the three-dimensional map of the entire target scenario, then according to the post-order traversal principle, all branch nodes under the root node are sequentially merged. For example, complete the incremental mapping process of a certain scenario recursively in the following order: Seq1, Seq2, C6_R1,..., Seq5, Seq6, C6_R3, C6,..., C7_R2, C7,..., C8_R3, C8, Parking.

[0094] For some target scenarios with small scale, simple content or low mapping risks, cross-layer mapping can be directly selected, that is, directly collect all the image data under the scene nodes and perform node map reconstruction. For example, when directly reconstructing the 3D map of the scene node Parking, all the image data under the Parking node (e.g., seq1-seq16) can be obtained according to the inclusion relationship, and then these data are used as the input of the 3D map reconstruction algorithm to obtain the 3D map of the entire scene node of the underground parking lot.

[0095] In the case of stitching mapping, the 3D maps of the branch nodes are reconstructed separately and completely, and then the 3D maps of the individual branch nodes are stitched into the same global coordinate system. In the case of using stitching mapping, the SFM algorithm independently reconstructs the node 3D maps of each branch node and respectively determines the pose information from the node 3D maps to the global map coordinate system;

[0096] Furthermore, according to the pose information, the node 3D maps are transformed from the node map coordinate system to the global map coordinate system, and according to the fourth preset rule, the node 3D maps are stitched in sequence to obtain the 3D map of the target scene.

[0097] Taking a specific scene as an example to illustrate the stitching mapping. For example, the steps for reconstructing the 3D map of the C6_R1 node in the above embodiment are as follows:

[0098] First step, for the third branch nodes Seq1 and Seq2, the 3D maps of Seq1 and Seq2 are respectively obtained using the SFM algorithm;

[0099] Second step, the 6DOF poses of the Seq1 map and the Seq2 map to the global map coordinate system are respectively determined;

[0100] Third step, the Seq1 map and the Seq2 map are respectively transformed to the global unified coordinate system using the 6DOF poses to obtain the 3D map of the C6_R1 node.

[0101] It should be noted that the mapping method in this embodiment is not limited to a specific one. In the actual mapping process, the above several mapping methods can be flexibly combined to complete the 3D reconstruction of specific nodes. Figure 3 3D reconstruction.

[0102] In some of these implementations, in the case where an error occurs when performing the reconstruction task using one or a combination of incremental mapping and stitching mapping, the reconstruction task needs to be interrupted, and the fault point of the reconstruction task is determined, and the mapping data corresponding to the fault point is repaired;

[0103] Furthermore, the merging result before the reconstruction task reaches the fault point is obtained, and based on the merging result and the repaired mapping data, the reconstruction task is restarted again.

[0104] It should be noted that for each mapping method, if an error occurs during the intermediate process, it can be interrupted in a timely manner, and then the corresponding problems can be investigated and repaired. However, the reusability of the results of different mapping methods is different. For example, in the case of cross-layer mapping, if an interruption occurs, there is basically no reusable information when restarting the mapping.

[0105] A specific scenario is combined to illustrate the repair and restart processes after mapping interruption. For example, when reconstructing the Parking node map in an incremental mapping method, the following steps may exist:

[0106] The reconstruction of the C6 node is successful and serves as the initial merge result. Using the initial merge result and the mapping data of C7 as inputs, the merged mapping of C7 is successfully completed, and this result serves as the first merge result;

[0107] Using the first merge result and the mapping data of C8 as inputs, an error occurs during the merging of C8, and the mapping process is interrupted;

[0108] Checking the above result, it is found that the reason is that the mapping data of C8 itself is incorrect, and the mapping data error of C8 is corrected;

[0109] Using the first merge result and the repaired mapping data of C8 as inputs, the incremental mapping process is restarted again, the mapping data of C8 is merged, and the reconstruction is successful;

[0110] The mapping ends, and a three-dimensional map of the Parking node is obtained.

[0111] In this embodiment, in some mapping methods, since the mapping result of this node is gradually merged from the results of the branch nodes, if an interruption occurs during the mapping process, after repairing the problem, the already merged results can be directly reused instead of starting the merging from scratch, thereby improving the anti-interference ability and mapping efficiency.

[0112] In some of these embodiments, based on the reconstruction process tree provided in the embodiments of the present application, local updates of the three-dimensional map can be performed while maximizing the utilization of the existing map results, including: supplementing, deleting, and replacing.

[0113] When a new branch node needs to be supplemented, according to the relevance between the mapping data of the node to be updated and the mapping data of the existing node, the mounting position of the node to be updated in the reconstruction process tree is determined. Secondly, configure the association method between the node to be updated and the existing node and its mapping method, and define the association information through the association method. Finally, based on the mapping data of the branch node to be updated, the SFM algorithm is used to perform the supplement task of the three-dimensional map of the target scenario.

[0114] When it is necessary to delete an existing branch node, determine the node to be deleted according to the human interaction information, and remove the node to be deleted, as well as the mapping data and association information related to the node to be deleted, from the reconstructed process tree.

[0115] When it is necessary to replace an existing branch node, that is, on the basis of deleting the branch node, the method of supplementing a new branch node in the above-mentioned embodiment can be adopted.

[0116] In some embodiments, when the collected on-site image data is video data, redundancy elimination needs to be performed on it.

[0117] Redundancy elimination can be performed on the video data through periodic sampling to obtain image data. For example: obtain the movement speed of the imaging device when shooting the video data, determine the sampling interval according to the movement speed; in the video data, perform periodic sampling according to the sampling interval to obtain image data. In this embodiment, the sampling interval can be determined by the following formula 1:

[0118] Formula 1:

[0119] where T is the sampling interval, f is the number of frames captured by the imaging device per second, m is the number of frames expected to be retained per unit distance of movement of the imaging device, and v is the average moving speed of the imaging device during shooting. Optionally, the units of f, m, and v are frames / s, frames / m, and m / s respectively.

[0120] This embodiment also provides a three-dimensional reconstruction system for a large-scale scene map. This system is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, terms such as "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also conceivable.

[0121] Figure 5 is a structural block diagram of a three-dimensional reconstruction system for a large-scale scene map according to an embodiment of the present application. As Figure 5 shown, the system includes: a process tree construction module 51 and a map reconstruction module 52, where

[0122] The process tree construction module 51 is used to divide the target scene into multiple sub-regions according to a first preset rule, configure multiple shooting paths in the sub-regions, and configure multiple shooting sequences in the shooting paths. Among them, the shooting sequence includes movement route information and viewing angle information, and

[0123] taking the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes, construct a reconstruction process tree of the target scene;

[0124] The map reconstruction module 52 is configured to obtain the image data collected according to the shooting sequence, and reconstruct the three-dimensional map of the target scene based on the image data and the reconstruction process tree.

[0125] It should be noted that, in this embodiment, the map reconstruction algorithm does not directly reconstruct the map based on each mapping data. Before that, steps such as feature extraction and feature matching need to be performed. These steps can be executed in the map reconstruction algorithm or in the external environment. After the external environment extracts and matches the features, the image information, feature information, and matching information of the branch node are used as mapping data to participate in the map reconstruction task.

[0126] In addition, in combination with the three-dimensional reconstruction method of a large-scale scene map in the above embodiment, the embodiment of the present application can provide a storage medium to implement. A computer program is stored on the storage medium; when the computer program is executed by a processor, the three-dimensional reconstruction method of any one of the above embodiments is implemented.

[0127] In one embodiment, a computer device is provided, and the computer device may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional reconstruction method of a large-scale scene map is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0128] In one embodiment, Figure 6 is a schematic internal structure diagram of an electronic device according to an embodiment of the present application. As Figure 6 shown, an electronic device is provided, and the electronic device may be a server. The internal structure diagram thereof may be as Figure 6As shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected by an internal bus. Among them, the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, the computer program, when executed by the processor, is used to implement a method for three-dimensional reconstruction of a large-scale scene map, and the database is used to store data.

[0129] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0130] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or an external cache. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0132] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A three-dimensional reconstruction method for a large-scale scene map, characterized in that, the method includes: Dividing the target scene into multiple sub-regions according to a first preset rule, configuring multiple shooting paths in the sub-regions, and configuring multiple shooting sequences in the shooting paths, wherein the shooting sequences include movement route information and viewing angle information; Constructing a reconstruction process tree of the target scene with the target scene as the root node, the sub-region as the first branch node, the shooting path as the second branch node, and the shooting sequence as the third branch node; Obtaining the image data collected according to the shooting sequence, and reconstructing the three-dimensional map of the target scene based on the image data and the reconstruction process tree, specifically including: Configuring the association method of the image data in the third branch node and the association method between each branch node according to a second preset rule, and defining association information according to the association method, wherein the association method includes one or a combination of brute-force matching, sequence matching, spatial matching, and manual matching; Configuring the mapping method of each branch node according to a third preset rule, wherein the mapping method includes one or a combination of cross-layer mapping, incremental mapping, and splicing mapping; Reconstructing the three-dimensional map of the target scene based on the association information and the mapping method; In the case of adopting the incremental mapping, the three-dimensional map reconstruction algorithm merges the mapping data of each branch node to obtain the three-dimensional map of the target scene according to a fourth preset rule, wherein the mapping data includes the image data and the association information, and the fourth preset rule includes one or a combination of post-order traversal principle, level traversal principle, and manual traversal rule; In the case of adopting the splicing mapping, the three-dimensional map reconstruction algorithm independently reconstructs the node three-dimensional maps of each branch node, and respectively determines the pose information of each node three-dimensional map in the global map coordinate system, According to the pose information, transforming the node three-dimensional map from the node map coordinate system to the global map coordinate system, and splicing the node three-dimensional maps according to the fourth preset rule to obtain the three-dimensional map of the target scene; In the case of adopting the cross-layer mapping, the three-dimensional map reconstruction algorithm performs global map reconstruction based on the image data under all the third branch nodes to obtain the three-dimensional map of the target scene.

2. The method according to claim 1, characterized in that, in the case where an error occurs in reconstructing the three-dimensional map of the target scene by using one or a combination of the incremental mapping and the splicing mapping, the method further includes: Interrupting the reconstruction task, determining the fault point of the reconstruction task, and repairing the mapping data corresponding to the fault point; Obtaining the merging result before the reconstruction task reaches the fault point, and restarting the reconstruction task again based on the merging result and the repaired mapping data.

3. The method according to claim 2, characterized in that, after the three-dimensional map of the reconstructed target scene is completed, the method further includes: in the case where new branch nodes need to be supplemented in the three-dimensional map, Determine the mounting position of the node to be updated in the reconstruction process tree according to the relevance between the mapping data of the node to be updated and the mapping data of the existing nodes; Configure the association method and mapping method between the node to be updated and the existing nodes according to the second preset rule and the third preset rule, and define the association information through the association method; Execute the supplementary task of the three-dimensional map based on the mapping data of the node to be updated through the three-dimensional map reconstruction algorithm.

4. The method according to claim 3, wherein, after the three-dimensional map of the reconstructed target scene is completed, the method further includes: in the case that an existing branch node needs to be deleted from the three-dimensional map, determine the node to be deleted according to the manual interaction information, and remove the node to be deleted, the mapping data related to the node to be deleted, and the association information from the reconstruction process tree.

5. The method according to claim 1, wherein, in the case that the image data is video data, the method further includes: perform redundancy elimination on the video data to obtain picture data, and reconstruct the three-dimensional map of the target scene based on the picture data and the reconstruction process tree, wherein the redundancy elimination includes: obtain the movement speed of the camera device when the video data is captured, and determine the sampling interval according to the movement speed; in the video data, perform periodic sampling according to the sampling interval to obtain the picture data.

6. A three-dimensional reconstruction system for a large-scale scene map, wherein, the system includes: a process tree construction module and a map reconstruction module, wherein, the process tree construction module is used to divide the target scene into multiple sub-regions according to the first preset rule, configure multiple shooting paths in the sub-regions, and configure multiple shooting sequences in the shooting paths, wherein the shooting sequences include movement route information and viewing angle information, and construct the reconstruction process tree of the target scene with the target scene as the root node, the sub-regions as the first branch nodes, the shooting paths as the second branch nodes, and the shooting sequences as the third branch nodes; the map reconstruction module is used to obtain the image data collected according to the shooting sequence, and reconstruct the three-dimensional map of the target scene based on the image data and the reconstruction process tree, specifically including: configure the association method of the image data in the third branch node and the association method between each branch node according to the second preset rule, and define the association information according to the association method, wherein the association method includes one or a combination of more of: brute-force matching, sequence matching, spatial matching, and manual matching; configure the mapping method of each branch node according to the third preset rule, wherein the mapping method includes one or a combination of more of: cross-layer mapping, incremental mapping, and splicing mapping; reconstruct the three-dimensional map of the target scene according to the association information and the mapping method. In the case of adopting the incremental mapping, the three-dimensional map reconstruction algorithm combines the mapping data of each branch node according to the fourth preset rule to obtain the three-dimensional map of the target scene, where the mapping data includes the image data and the association information, and the fourth preset rule includes one or more combinations of the post-order traversal principle, the level-order traversal principle, and the manual traversal rule; In the case of adopting the stitching mapping, the three-dimensional map reconstruction algorithm independently reconstructs the node three-dimensional maps of each branch node, and respectively determines the pose information of each of the node three-dimensional maps in the global map coordinate system. According to the pose information, the node three-dimensional map is transformed from the node map coordinate system to the global map coordinate system, and the node three-dimensional maps are stitched according to the fourth preset rule to obtain the three-dimensional map of the target scene; In the case of adopting the cross-layer mapping, the three-dimensional map reconstruction algorithm performs global map reconstruction based on the image data subordinate to all the third branch nodes to obtain the three-dimensional map of the target scene.

7. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, when the processor executes the computer program, it implements a three-dimensional reconstruction method for a large-scale scene map according to any one of claims 1 to 5.

8. A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, it implements a three-dimensional reconstruction method for a large-scale scene map according to any one of claims 1 to 5.

Images