Method for collecting three-dimensional reconstruction data, electronic device, and storage medium

Abstract

The present application provides a method for collecting three-dimensional reconstruction data, an electronic device, and a storage medium, belonging to the technical field of three-dimensional reconstruction. The method includes: obtaining three-dimensional reconstruction data collected in real time, where the three-dimensional reconstruction data includes camera data, attitude measurement data, and point cloud data; fusing the attitude measurement data and the point cloud data to determine the initial pose of the point cloud data; generating a local point cloud map based on the initial pose and the camera data; determining an evaluation result of the three-dimensional reconstruction data based on the local point cloud map according to a preset evaluation index, where the preset evaluation index includes at least one of a data integrity evaluation index, a data quality evaluation index, and a geometric consistency evaluation index; and determining a collection instruction for the three-dimensional reconstruction data based on the evaluation result to collect the three-dimensional reconstruction data based on the collection instruction. The present application can improve the collection efficiency of three-dimensional reconstruction data under limited computing power and memory conditions.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular, to a method for collecting three-dimensional reconstruction data, an electronic device, and a storage medium. Background Art

[0002] In the development of modern technology, three-dimensional reconstruction technology has been widely used in many fields such as architecture, archaeology, industrial design, and urban planning. In the traditional three-dimensional reconstruction process, since the three-dimensional reconstruction data contains a large amount of data from multiple sensors, it has very high requirements for computing power and memory during map storage and display. When network communication cannot meet the transmission requirements of cloud computing, the existing technology usually generates a global map using high-performance computer devices after the three-dimensional reconstruction data collection is completed, so as to verify the validity of the collected data. This means that any errors or omissions in the collection need to be recollected, which greatly affects the data collection efficiency.

[0003] Therefore, how to improve the collection efficiency of three-dimensional reconstruction data under limited computing power and memory conditions is a technical problem that needs to be solved urgently at present. Summary of the Invention

[0004] The purpose of this application is to provide a method for collecting three-dimensional reconstruction data, an electronic device, and a storage medium to solve the above problems.

[0005] To achieve the above purpose, in the first aspect, this application proposes a method for collecting three-dimensional reconstruction data, and the method includes:

[0006] Obtain real-time collected three-dimensional reconstruction data, where the three-dimensional reconstruction data includes camera data, attitude measurement data, and point cloud data;

[0007] Fuse the attitude measurement data and the point cloud data to determine the initial pose of the point cloud data;

[0008] Generate a local point cloud map based on the initial pose and the camera data;

[0009] Based on the local point cloud map, determine the evaluation result of the three-dimensional reconstruction data according to a preset evaluation index, where the preset evaluation index includes at least one of a data integrity evaluation index, a data quality evaluation index, and a geometric consistency evaluation index;

[0010] Based on the evaluation result, determine a collection instruction for the three-dimensional reconstruction data, so as to collect the three-dimensional reconstruction data based on the collection instruction.

[0011] In some embodiments, the generating a local point cloud map based on the initial pose and the camera data further includes:

[0012] Obtain a real-time local map and transform the point cloud data into the real-time local map based on the initial pose;

[0013] In the real-time local map, determine the matching point cloud corresponding to the point cloud data;

[0014] Determine the optimized pose of the point cloud data by minimizing the spatial distance of the matching point cloud;

[0015] Generate a local point cloud map based on the optimized pose and the camera data.

[0016] In some embodiments, the generating a local point cloud map based on the optimized pose and the camera data further includes:

[0017] Obtain the latest key frame saved in the sliding window;

[0018] Calculate the difference value between the optimized pose and the pose of the latest key frame, and determine whether the difference value exceeds a preset difference value;

[0019] If so, determine that the point cloud data is the latest key frame, and generate a local point cloud map based on the optimized pose and the camera data;

[0020] If not, execute the step of obtaining the three-dimensional reconstruction data collected in real time.

[0021] In some embodiments, after determining that the point cloud data is the latest key frame, it includes:

[0022] Calculate the density distribution and feature distribution of the point cloud data;

[0023] Based on the density distribution and the feature distribution, determine the sampling rate distribution of the point cloud data;

[0024] Downsample the point cloud data according to the sampling rate distribution;

[0025] Save the downsampled point cloud data to the global map.

[0026] In some embodiments, the camera data includes image information and image pose, and the generating a local point cloud map based on the optimized pose and the camera data includes:

[0027] Based on the optimized pose, the image pose and the downsampled point cloud data, determine the point cloud within the camera view;

[0028] Based on the image information, color the point cloud within the camera view to generate a local point cloud map.

[0029] In some embodiments, the camera data includes image information and image pose. Generating a local point cloud map based on the initial pose and the camera data includes:

[0030] Based on the initial pose and the image pose, determining the point cloud within the camera's view;

[0031] Back-projecting the point cloud within the camera's view onto the image information to determine the back-projected points;

[0032] Coloring the point cloud within the camera's view based on the color of the back-projected points to generate a local point cloud map.

[0033] In some embodiments, based on the local point cloud map, determining an evaluation result of the three-dimensional reconstruction data according to a preset evaluation metric, where the preset evaluation metric includes at least one of a data integrity evaluation metric, a data quality evaluation metric, and a geometric consistency evaluation metric, and includes:

[0034] Based on the data integrity evaluation metric, calculating the point cloud coverage range of the local point cloud map to determine the point cloud missing area; and / or

[0035] Based on the data quality evaluation metric, calculating the noise level and resolution of the local point cloud map to determine the area where the data quality does not meet the standard; and / or

[0036] Based on the geometric consistency evaluation metric, comparing the geometric consistency within the local point cloud map to determine the geometric deviation area.

[0037] In some embodiments, based on the evaluation result, determining an acquisition instruction for the three-dimensional reconstruction data includes:

[0038] Determining a supplementary acquisition instruction according to the point cloud missing area; and / or

[0039] Determining a sensor parameter adjustment instruction and a supplementary acquisition instruction according to the area where the data quality does not meet the standard; and / or

[0040] Determining a sensor correction instruction and a supplementary acquisition instruction according to the geometric deviation area.

[0041] In a second aspect, the present application also proposes an electronic device, including:

[0042] One or more processors;

[0043] A memory for storing one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the acquisition method of the three-dimensional reconstruction data as described above.

[0044] In a third aspect, the present application also provides a storage medium storing executable instructions that, when executed by a processor, cause the processor to execute the method for acquiring three-dimensional reconstruction data as described above.

[0045] Compared with the prior art, the beneficial effects of the present application include:

[0046] In a first aspect, by acquiring and fusing attitude measurement data and point cloud data in real time to determine the initial pose of the point cloud data, accurate registration of the point cloud data can be achieved during the data acquisition process. This registration can improve the accuracy of positioning, reduce redundant data caused by positioning errors, further reduce the data volume, and improve the processing speed. At the same time, it helps with subsequent data evaluation and determination of acquisition instructions, thereby improving the data acquisition efficiency. In a second aspect, by generating a local point cloud map in real time during the data acquisition process and performing evaluation and decision-making based on the local point cloud map, the dependence on the global map can be reduced, thus reducing the requirements for high-performance computing devices and large-capacity storage devices, and saving computing power and memory resources. In a third aspect, by performing evaluation based on the local point cloud map, errors or omissions in the acquired three-dimensional reconstruction data can be detected in a timely manner to improve the data acquisition efficiency. In a fourth aspect, by determining the acquisition instructions based on the evaluation results, the sensor can be guided to perform more accurate and efficient acquisition. This feedback-based adjustment mechanism further improves the data acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope of the present application.

[0048] Figure 1 It is a schematic flowchart of an embodiment of the method for acquiring three-dimensional reconstruction data of the present application;

[0049] Figure 2 It is a partial schematic flowchart of an embodiment of the method for acquiring three-dimensional reconstruction data of the present application;

[0050] Figure 3 It is a partial schematic flowchart of an embodiment of the method for acquiring three-dimensional reconstruction data of the present application;

[0051] Figure 4 It is a partial schematic flowchart of an embodiment of the method for acquiring three-dimensional reconstruction data of the present application;

[0052] Figure 5 It is a full schematic flowchart of an embodiment of the method for acquiring three-dimensional reconstruction data of the present application;

[0053] Figure 6Schematic diagram of the electronic device involved in the method for collecting three-dimensional reconstruction data in any embodiment of the present application. Detailed implementation manners

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] All terms used in the present application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0056] For example, terms such as "first" and "second" used in the present application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0057] For another example, terms such as "including" and "comprising" used in the present application indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0058] As mentioned above, in the traditional three-dimensional reconstruction process, since the three-dimensional reconstruction data contains a large amount of data from multiple sensors, this places very high requirements on computing power and memory during map storage and display. When network communication cannot meet the transmission requirements of cloud computing, the prior art usually generates a global map using high-performance computer devices after the three-dimensional reconstruction data is collected, so as to verify the validity of the collected data. This means that any errors or omissions during collection need to be re-collected, which greatly affects the data collection efficiency. Therefore, how to improve the collection efficiency of three-dimensional reconstruction data under limited computing power and memory conditions is a technical problem that needs to be solved urgently at present. For this reason, the present application proposes a method for collecting three-dimensional reconstruction data, an electronic device, and a storage medium, which can improve the collection efficiency of three-dimensional reconstruction data under limited computing power and memory conditions.

[0059] As Figure 1 shown, the method for collecting three-dimensional reconstruction data in this embodiment includes the following steps:

[0060] Step S10, obtaining the three-dimensional reconstruction data collected in real time, where the three-dimensional reconstruction data includes camera data, attitude measurement data, and point cloud data.

[0061] In this embodiment, the application scenarios of the method for collecting three-dimensional reconstruction data cover a variety of intelligent terminals, including but not limited to smart phones, personal computers, robots, and other electronic devices with data processing, network communication, and program running functions. After a data collection device equipped with a point cloud sensor (such as a lidar, depth sensor, etc.), an attitude sensor (such as an IMU, inertial measurement unit), and a camera sensor (such as a binocular fisheye camera) is turned on, the electronic device acquires the three-dimensional reconstruction data that is collected in real time by the data collection device and is time-hard synchronized. Here, time-hard synchronization refers to the synchronization in time achieved through hardware.

[0062] The three-dimensional reconstruction data in this embodiment includes camera data, attitude measurement data, and point cloud data. It should be understood that camera data refers to the image data obtained from the camera sensor, and the image data includes image information and image pose in some embodiments. Attitude measurement data refers to the acceleration data and angular velocity data obtained from the attitude sensor. Among them, the acceleration data is provided by a three-axis accelerometer to measure the linear acceleration of the attitude sensor on the X, Y, and Z axes. The angular velocity data is provided by a three-axis gyroscope to measure the angular velocity (rotation speed) of the attitude sensor on the X, Y, and Z axes. In some embodiments, the attitude measurement data may also include magnetic force data provided by a three-axis magnetometer to measure the magnetic field strength in the environment. Point cloud data refers to a data set composed of a large number of points with three-dimensional coordinates obtained from point cloud sensors such as lidars and depth sensors, and these points can reflect the geometric structure of the environment.

[0063] In some embodiments, the three-dimensional reconstruction data collected in real time is acquired and preprocessed to improve the quality and accuracy of the three-dimensional reconstruction data. Specifically, the attitude measurement data is filtered to remove noise and outliers to improve the stability and accuracy of the attitude measurement data. The point cloud data is filtered to remove outliers and noise to improve the purity and quality of the point cloud data.

[0064] In some embodiments, the attitude measurement data or the preprocessed attitude measurement data is pre-integrated to obtain a pose state. Specifically, the prior pose estimate of the attitude measurement data (the pose state predicted at the previous time point ) is obtained, and according to the prior pose estimate and the motion model function , the pose state is determined . During the pre-integration process, the electronic device will perform time synchronization and cumulative error correction on the attitude measurement data to generate a smooth and accurate pose trajectory. Thereby improving the accuracy, stability, and efficiency of subsequent fusion with the point cloud data.

[0065]

[0066] It should be understood that represents the pose state at time i + 1, represents the sampling time interval of the pose sensor, u represents the measurement input of the IMU (attitude measurement data), and w represents Gaussian noise.

[0067]

[0068]

[0069]

[0070]

[0071] It should be understood that x represents the pose state, R represents the pose orientation, p represents the three-dimensional position, v is the velocity, and b represents the attitude bias. represents the bias of the gyroscope, represents the bias of the accelerometer, represents the gravitational acceleration. represents the noise measured by the gyroscope. represents the noise measured by the accelerometer. represents the noise of the gyroscope bias. represents the noise of the accelerometer bias. represents the angular velocity, represents the acceleration.

[0072] Step S20: fuse the attitude measurement data and the point cloud data to determine the initial pose of the point cloud data.

[0073] In this embodiment, the initial pose refers to the pose of the point cloud sensor when collecting the point cloud data. In some embodiments, the pose state x and the point cloud data can be directly fused to determine the initial pose of the point cloud data. The specific calculation method can be based on Bayes' theorem, . Among them, is the posterior probability, that is, the probability estimate of the initial pose after observing the point cloud data ; is the likelihood probability, that is, the probability of observing the point cloud data under the pose state ; is the prior probability, that is, the probability estimate of the initial pose before observing new point cloud data.

[0074] In some embodiments, the structural information of the point cloud data can be extracted. The structural information includes line segment structural information and surface structural information. Based on Bayes' theorem, the pose measurement data (or pose state x) and the structural information of the point cloud data are fused to determine the initial pose of the point cloud data.

[0075] Step S30: Generate a local point cloud map based on the initial pose and the camera data.

[0076] In this embodiment, the camera data includes image information and image pose. The image pose is calculated from the pose state obtained by pre-integrating the extrinsic parameters of the pre-calibrated image sensor and the pose sensor.

[0077] In some embodiments, based on the initial pose and the image pose, the point cloud within the camera's view is determined. Based on the image information, the point cloud within the camera's view is colored to generate a colored local point cloud map.

[0078] In some embodiments, based on the initial pose and the image pose, the point cloud within the camera's view is determined; the point cloud within the camera's view is back-projected onto the image information to determine the back-projected points; based on the colors of the back-projected points, the point cloud within the camera's view is colored to generate a local point cloud map.

[0079] In some embodiments, a real-time local map is obtained, and the point cloud data is matched and registered with the real-time local map to determine the optimized pose of the point cloud data. The optimized pose takes into account the actual matching situation between the point cloud data and the real-time local map and can be used to update the initial pose of the point cloud data to more accurately reflect the position and orientation of the point cloud sensor in the global coordinate system.

[0080] It should be noted that the construction of the local point cloud map in this embodiment is a dynamic and cyclic process. The real-time local map refers to the local point cloud map currently being displayed and generated based on the previous round of 3D reconstruction data.

[0081] In some embodiments, based on the optimized pose and the image pose, the point cloud within the camera's view is determined. Based on the image information, the point cloud within the camera's view is colored to generate a colored local point cloud map.

[0082] In some embodiments, based on the optimized pose and the image pose, the point cloud within the camera's view is determined; the point cloud within the camera's view is back-projected onto the image information to determine the back-projected points; based on the colors of the back-projected points, the point cloud within the camera's view is colored to generate a local point cloud map.

[0083] In addition, since the local point cloud map has an intuitive visualization effect, real-time display of the local point cloud map on mobile devices allows the acquisition personnel to immediately see the effect of the currently acquired 3D reconstruction data and make adjustments according to the actual situation. For example, if it is found that the data quality in a certain area is not high, the acquisition personnel can immediately rescan that area to ensure the validity and integrity of the data. Another example is that in a complex indoor environment, the acquisition personnel can select the best scanning route based on the real-time displayed point cloud map to avoid repeated scanning and missed areas.

[0084] Step S40: Based on the local point cloud map, determine the evaluation result of the 3D reconstruction data according to a preset evaluation index, where the preset evaluation index includes at least one of a data integrity evaluation index, a data quality evaluation index, and a geometric consistency evaluation index.

[0085] In this embodiment, the data integrity evaluation index is used to measure whether the 3D reconstruction data completely covers the target object or scene. It focuses on the coverage range and missing situation of the data. The data quality evaluation index is used to measure the accuracy and fineness of the 3D reconstruction data. It focuses on the noise level, resolution, and accuracy of the data. The geometric consistency evaluation index is used to measure the consistency and correctness of the 3D reconstruction data in terms of geometric shape. It focuses on whether the data faithfully reflects the actual geometric features of the target object. In practical applications, one or more evaluation indexes can be selected according to specific requirements to comprehensively evaluate the 3D reconstruction data. For example, if the application scenario has high requirements for the coverage range of the data, the data integrity evaluation index can be focused on; if high requirements are placed on the accuracy and details of the data, the data quality evaluation index can be focused on; if it is necessary to ensure the accuracy of the geometric shape of the reconstructed data, the geometric consistency evaluation index can be focused on.

[0086] In some implementation manners, according to the environmental complexity of the local point cloud map, determine the calculation accuracy, and determine the evaluation result of the 3D reconstruction data according to the calculation accuracy and the preset evaluation index. The determining factors of the environmental complexity include at least one of the following factors: the data volume of the local point cloud map, the available computing resource information of the electronic device, and the system load of the electronic device. Among them, the data volume of the local point cloud map is directly proportional to the calculation accuracy, the available computing resource information of the electronic device is directly proportional to the calculation accuracy, and the system load of the electronic device is inversely proportional to the calculation accuracy. This embodiment can dynamically adjust the calculation accuracy according to the complexity of the environment, so as to fully utilize the computing resources and improve the calculation efficiency while ensuring the accuracy of the evaluation result.

[0087] In some embodiments, based on the data integrity evaluation index, calculate the point cloud coverage of the local point cloud map and determine the point cloud missing area. Specifically, use the multi-scale analysis technology to calculate the point cloud coverage of the local point cloud map, and determine and mark the point cloud missing area by comparing the point cloud coverage with the expected coverage.

[0088] In some embodiments, based on the data quality evaluation index, calculate the noise level and resolution of the local point cloud map and determine the area where the data quality does not meet the standard. Specifically, apply the adaptive noise filtering algorithm to calculate the noise level of the local point cloud map and calculate the resolution of the local point cloud map; based on the noise level and resolution, determine and mark the area where the data quality does not meet the standard.

[0089] In some embodiments, based on the geometric consistency evaluation index, compare the geometric consistency inside the local point cloud map and determine the geometric deviation area. Specifically, extract feature points from the local point cloud map; based on the feature points, perform geometric consistency comparison on the local point cloud map; based on the comparison result, determine and mark the geometric deviation area.

[0090] Step S50, based on the evaluation result, determine the acquisition instruction for the three-dimensional reconstruction data, and acquire the three-dimensional reconstruction data based on the acquisition instruction.

[0091] In this embodiment, the evaluation result includes at least one of the following: the marked point cloud missing area, the marked area where the data does not meet the standard, and the marked geometric deviation area. The acquisition instruction determined according to the evaluation result includes at least one of the following: the supplementary acquisition instruction, the sensor parameter adjustment instruction, and the sensor correction instruction. Among them, the supplementary acquisition instruction includes the data acquisition path and / or the position and angle of the sensor on the data acquisition device. The sensor parameter adjustment instruction is used to adjust the sensor parameters of the data acquisition device, such as the exposure time, resolution, gain, etc., to improve the data acquisition quality. The sensor correction instruction is used to calibrate or correct the sensor on the data acquisition device to eliminate geometric deviation.

[0092] In some embodiments, according to the point cloud missing area, determine the supplementary acquisition instruction, and the supplementary acquisition instruction is used to instruct the data acquisition device to supplement and acquire the three-dimensional reconstruction data of the point cloud missing area. Specifically, according to the position and range of the point cloud missing area, determine the supplementary acquisition instruction to supplement and acquire the three-dimensional reconstruction data of the point cloud missing area.

[0093] In some embodiments, according to the area where the data quality does not meet the standard, determine the sensor parameter adjustment instruction and the supplementary acquisition instruction, and supplement and acquire the three-dimensional reconstruction data of the area where the data quality does not meet the standard according to the sensor parameter adjustment instruction and the supplementary acquisition instruction.

[0094] In some embodiments, according to the geometric deviation region, a sensor correction instruction and a supplementary acquisition instruction are determined to supplement the acquisition of three-dimensional reconstruction data of the geometric deviation region according to the sensor correction instruction and the supplementary acquisition instruction.

[0095] In this embodiment, in the first aspect, by acquiring and fusing attitude measurement data and point cloud data in real time, the initial pose of the point cloud data is determined, and accurate registration of the point cloud data can be achieved during the data acquisition process. This registration can improve the accuracy of positioning, reduce redundant data caused by positioning errors, further reduce the data volume, and improve the processing speed. At the same time, it helps with subsequent data evaluation and determination of acquisition instructions, thereby improving the data acquisition efficiency. In the second aspect, by generating a local point cloud map in real time during the data acquisition process and performing evaluation and decision-making based on the local point cloud map, the dependence on the global map can be reduced, thereby reducing the requirements for high-performance computing devices and large-capacity storage devices, and saving computing power and memory resources. In the third aspect, by performing evaluation based on the local point cloud map, errors or omissions in the acquired three-dimensional reconstruction data can be detected in a timely manner to improve the data acquisition efficiency. In the fourth aspect, by determining the acquisition instruction based on the evaluation result, the sensor can be guided to perform more accurate and efficient acquisition, and this feedback-based adjustment mechanism further improves the data acquisition efficiency.

[0096] As Figure 2 shown, the step S30 includes:

[0097] Step A10, obtaining a real-time local map and converting the point cloud data into the real-time local map based on the initial pose.

[0098] Step A20, determining the matching point cloud corresponding to the point cloud data in the real-time local map.

[0099] In this embodiment, a real-time local map is obtained, and the point cloud data is converted into the coordinate system of the real-time local map based on the initial pose. The matching point cloud corresponding to or most similar to the point cloud data is found in the real-time local map.

[0100] Step A30, determining the optimized pose of the point cloud data by minimizing the spatial distance of the matching point cloud.

[0101] In this embodiment, through optimization algorithms (such as gradient descent, Levenberg-Marquardt algorithm, etc.), the spatial distance between the point cloud data and the matching point cloud is minimized. The spatial distance refers to the distance between each point in the point cloud data and the matching point in the corresponding matching point cloud. During the process of minimizing the spatial distance, the position and pose of the point cloud data are adjusted to obtain the optimized pose.

[0102] Step A40: Generate a local point cloud map based on the optimized pose and the camera data.

[0103] For the specific implementation in this embodiment, refer to step S30, and no further elaboration will be provided here.

[0104] In this embodiment, on the one hand, by acquiring the real-time local map and performing point cloud data conversion and registration, real-time registration can be achieved during the data acquisition process, enabling 3D reconstruction in a dynamic environment and allowing for timely updating and adjustment of the reconstruction results. On the other hand, by minimizing the spatial distance between the matching point clouds, a more accurate point cloud data registration result can be obtained. This helps improve the accuracy of 3D reconstruction and makes the generated local point cloud map more accurate.

[0105] As Figure 3 shown, step A40 may further include:

[0106] Step B10: Obtain the latest key frame saved in the sliding window.

[0107] In this embodiment, the sliding window is used to store a number of the latest key frames. The key frame that was added to the sliding window most recently is the latest key frame.

[0108] Step B20: Calculate the difference value between the optimized pose and the pose of the latest key frame.

[0109] In this embodiment, the pose difference value refers to the difference between the current optimized pose and the pose of the latest key frame. This difference value can be represented by the rotation angle and the translation distance.

[0110] Step B30: Determine whether the difference value exceeds a preset difference value.

[0111] In this embodiment, the preset difference value is a threshold used to determine whether the pose difference between the current optimized pose and the pose of the latest key frame is large enough.

[0112] Step B40: If so, determine that the point cloud data is the latest key frame, and generate a local point cloud map based on the optimized pose and the camera data.

[0113] In this embodiment, if the difference value exceeds the preset difference value, the current point cloud data is determined to be the latest key frame, and the local point cloud map corresponding to the latest key frame is generated. If the difference value does not exceed the preset difference value, the point cloud data is determined to be a redundant frame, and the redundant frame is not used to generate a local point cloud map. Return to step S10. By using the sliding window and key frame technology in this embodiment, the data processing flow can be simplified. Only processing key frames instead of all frames can significantly improve the processing efficiency.

[0114] In some embodiments, when a latest key frame is generated, it is detected whether the sliding window has reached its maximum capacity. If so, the key frame that was earliest added to the sliding window is removed, and the latest key frame is added to the sliding window. This ensures that only the latest several key frames are included in the sliding window. In this way, the electronic device can efficiently manage and utilize key frame data, avoid storing too many redundant frames, reduce the burden of data storage and processing, and improve the processing speed and accuracy.

[0115] In this embodiment, by calculating the pose difference between the optimized pose and the latest key frame and determining whether the difference value exceeds a preset difference value to determine the latest key frame, it is not only possible to effectively avoid frequent updates caused by small pose changes, but also to avoid storing too many redundant frames, thereby reducing the burden of data storage and processing and improving the processing speed and accuracy.

[0116] As Figure 4 and Figure 5 shown, step B40 further includes:

[0117] Step B41, calculating the density distribution and feature distribution of the point cloud data.

[0118] In this embodiment, the density distribution refers to the distribution of points in the point cloud data, usually represented by the number of points and the spatial distribution. The feature distribution refers to the distribution of feature points in the point cloud data, usually represented by the number of feature points and the spatial distribution. In the point cloud data, feature points refer to points with significant geometric characteristics, which can be points with higher curvature and larger gradient changes, and these points have unique identifiability and stability in the point cloud data.

[0119] Step B42, based on the density distribution and the feature distribution, determining the sampling rate distribution of the point cloud data.

[0120] In this embodiment, the sampling rate distribution means that in different regions, the sampling rate of the point cloud data is different. Specifically, in regions with higher density or more features, the sampling rate will be lower; in regions with lower density or fewer features, the sampling rate will be higher. That is, the density of points and feature points in the point cloud data is inversely proportional to the sampling rate.

[0121] Step B43, downsampling the point cloud data according to the sampling rate distribution.

[0122] In this embodiment, downsampling means sparsifying the point cloud data, that is, reducing the number of points in the point cloud data to reduce the data volume and computational complexity. By downsampling the point cloud data according to the sampling rate distribution, the scale of the point cloud data can be reduced while retaining the main geometric features, improving the processing efficiency.

[0123] Step B44, saving the downsampled point cloud data to the global map.

[0124] In this embodiment, the global map is not rendered, but all point cloud data is reserved for the global map. Only the downsampled data is saved, which can significantly reduce the storage space and management complexity.

[0125] Step B45: Based on the optimized pose, image pose, and downsampled point cloud data, determine the point cloud within the camera's view.

[0126] Step B46: Based on the image information, color the point cloud within the camera's view to generate a local point cloud map.

[0127] In this embodiment, except for replacing the point cloud data with downsampled point cloud data, the specific implementation is the same as that in step S30, and will not be elaborated here.

[0128] In this embodiment, on the one hand, by calculating the density distribution and feature distribution, the sampling rate distribution can be determined, thereby downsampling the point cloud data. This helps reduce the data volume and computational complexity, and improve the data processing efficiency. On the other hand, the downsampling process can filter out some noise and redundant information, making the point cloud data cleaner and smoother, which helps improve the accuracy and quality of the constructed local point cloud map. On the third hand, the downsampled point cloud data is clearer and easier to understand during visualization, and will not appear cluttered due to excessive points. On the fourth hand, downsampling can reduce the computational instability and error accumulation caused by overly dense data, thereby improving the robustness of the algorithm.

[0129] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above method embodiments.

[0130] In one embodiment, an electronic device is further provided, including one or more processors; a memory, and one or more programs are stored in the memory. When the one or more programs are executed by the one or more processors, the one or more processors execute the steps in the above method embodiments.

[0131] In one embodiment, as Figure 6As shown, it shows a schematic structural diagram of an electronic device for implementing an embodiment of the present application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0132] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 based on need. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 based on need, so that a computer program read from it can be installed into the storage section 708 based on need.

[0133] Specifically, based on the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product including a computer-readable medium carrying instructions. In such an embodiment, the instructions can be downloaded and installed from a network through the communication section 709, and / or installed from the removable medium 711. When the instructions are executed by the central processing unit (CPU) 701, the various method steps described in the present application are executed.

[0134] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0135] In addition, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means that it is within the scope of the present application and forms different embodiments. For example, in the claims above, any one of the claimed embodiments can be used in any combination. The information disclosed in this background section is only intended to deepen the understanding of the overall background art of the present application, and should not be regarded as an admission or any form of implication that this information constitutes the prior art known to those skilled in the art.

Claims

1. A method for collecting three-dimensional reconstruction data, characterized in that: The method comprises: Acquire three-dimensional reconstruction data collected in real time, wherein the three-dimensional reconstruction data includes camera data, posture measurement data and point cloud data; Fusing the posture measurement data and the point cloud data to determine an initial posture of the point cloud data; Acquire a real-time local map, and based on the initial pose, convert the point cloud data into the real-time local map; In the real-time local map, determining a matching point cloud corresponding to the point cloud data; Determining an optimized pose of the point cloud data by minimizing a spatial distance of matching point clouds; Get the latest key frame saved in the sliding window; Calculating the difference between the optimized pose and the pose of the latest key frame, and determining whether the difference exceeds a preset difference value; If so, determining that the point cloud data is the latest key frame, calculating the density distribution and feature distribution of the point cloud data, determining the sampling rate distribution of the point cloud data based on the density distribution and the feature distribution, downsampling the point cloud data according to the sampling rate distribution, saving the downsampled point cloud data to a global map, and generating a local point cloud map based on the optimized pose and the camera data; If not, then executing the step of acquiring the three-dimensional reconstruction data collected in real time; Based on the local point cloud map, determining an evaluation result of the three-dimensional reconstruction data according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator, and a geometric consistency evaluation indicator; Based on the evaluation result, an acquisition instruction for the three-dimensional reconstruction data is determined, so as to acquire the three-dimensional reconstruction data based on the acquisition instruction.

2. The method for collecting three-dimensional reconstruction data according to claim 1, characterized in that: The camera data includes image information and image pose, and generating a local point cloud map based on the optimized pose and the camera data includes: Determining a point cloud within a camera perspective based on the optimized pose, the image pose, and the downsampled point cloud data; Based on the image information, the point cloud within the camera's field of view is colored to generate a local point cloud map.

3. The method for collecting three-dimensional reconstruction data according to claim 1, characterized in that: The step of determining the evaluation result of the three-dimensional reconstruction data based on the local point cloud map according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator, and a geometric consistency evaluation indicator, including: Based on the data integrity evaluation index, calculating the point cloud coverage of the local point cloud map and determining the point cloud missing area; and / or Based on the data quality assessment index, calculate the noise level and resolution of the local point cloud map to determine the area where the data quality does not meet the standard; and / or Based on the geometric consistency evaluation index, the geometric consistency inside the local point cloud map is compared to determine the geometric deviation area.

4. The method for collecting three-dimensional reconstruction data according to claim 3, characterized in that: The step of determining the acquisition instruction of the three-dimensional reconstruction data based on the evaluation result includes: Determine a supplementary sampling instruction according to the missing area of ​​the point cloud; and / or Determine sensor parameter adjustment instructions and supplementary sampling instructions according to the area where the data quality does not meet the standards; and / or According to the geometric deviation area, a sensor correction instruction and a supplementary sampling instruction are determined.

5. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method for collecting three-dimensional reconstruction data as described in any one of claims 1 to 4.

6. A storage medium, characterized in that: The storage medium stores executable instructions, and when the instructions are executed by the processor, the processor executes the method for collecting three-dimensional reconstruction data according to any one of claims 1 to 4.

Images