Three-dimensional reconstruction data acquisition method, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional reconstruction data acquisition method, electronic equipment and a storage medium, and belongs to the technical field of three-dimensional reconstruction, the method comprises the following steps: acquiring three-dimensional reconstruction data acquired in real time, the three-dimensional reconstruction data comprising camera data, attitude measurement data and point cloud data; fusing the attitude measurement data and the point cloud data, and determining an initial pose of the point cloud data; generating a local point cloud map based on the initial pose and the camera data; based on the local point cloud map, an evaluation result of the three-dimensional reconstruction data is determined according to preset evaluation indexes, and the preset evaluation indexes comprise at least one of a data integrity evaluation index, a data quality evaluation index and a geometric consistency evaluation index; and based on the evaluation result, determining an acquisition instruction of the three-dimensional reconstruction data, so as to acquire the three-dimensional reconstruction data based on the acquisition instruction. According to the invention, the acquisition efficiency of the three-dimensional reconstruction data can be improved under the condition of limited computing power and memory.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional reconstruction, and in particular to a method for collecting three-dimensional reconstruction data, an electronic device and a storage medium. Background Art

[0002] With the development of modern science and technology, 3D reconstruction technology has been widely used in many fields such as architecture, archaeology, industrial design, and urban planning. In the traditional 3D reconstruction process, since the 3D reconstruction data contains a large amount of data from multiple sensors, the requirements for computing power and memory in the map storage and display process are very high. When network communication cannot meet the transmission requirements of cloud computing, the existing technology usually uses high-performance computer equipment to generate a global map after the 3D reconstruction data is collected, so as to verify the validity of the collected data. This means that any errors or omissions in the collection need to be re-collected, which greatly affects the efficiency of data collection.

[0003] Therefore, how to improve the efficiency of 3D reconstruction data acquisition under limited computing power and memory conditions is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of this application is to provide a method for collecting three-dimensional reconstruction data, an electronic device and a storage medium to solve the above-mentioned problems.

[0005] To achieve the above objectives, in a first aspect, the present application proposes a method for collecting three-dimensional reconstruction data, the method comprising: Acquire three-dimensional reconstruction data collected in real time, wherein the three-dimensional reconstruction data includes camera data, posture measurement data and point cloud data; Fusing the posture measurement data and the point cloud data to determine an initial posture of the point cloud data; Based on the initial pose and the camera data, generating a local point cloud map; Based on the local point cloud map, determining an evaluation result of the three-dimensional reconstruction data according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator, and a geometric consistency evaluation indicator; Based on the evaluation result, an acquisition instruction for the three-dimensional reconstruction data is determined, so as to acquire the three-dimensional reconstruction data based on the acquisition instruction.

[0006] In some implementations, generating a local point cloud map based on the initial pose and the camera data further includes: Acquire a real-time local map, and based on the initial pose, convert the point cloud data into the real-time local map; In the real-time local map, determining a matching point cloud corresponding to the point cloud data; Determining an optimized pose of the point cloud data by minimizing a spatial distance of matching point clouds; Based on the optimized pose and the camera data, a local point cloud map is generated.

[0007] In some implementations, generating a local point cloud map based on the optimized pose and the camera data further includes: Get the latest key frame saved in the sliding window; Calculating the difference between the optimized pose and the pose of the latest key frame, and determining whether the difference exceeds a preset difference value; If yes, determining that the point cloud data is the latest keyframe, and generating a local point cloud map based on the optimized pose and the camera data; If not, the step of acquiring the three-dimensional reconstruction data collected in real time is performed.

[0008] In some implementations, after determining that the point cloud data is the latest key frame, the method further includes: Calculating density distribution and feature distribution of the point cloud data; Determining a sampling rate distribution of the point cloud data based on the density distribution and the feature distribution; Downsampling the point cloud data according to the sampling rate distribution; Save the downsampled point cloud data to the global map.

[0009] In some implementations, the camera data includes image information and image pose, and generating a local point cloud map based on the optimized pose and the camera data includes: Determining a point cloud within a camera perspective based on the optimized pose, the image pose, and the downsampled point cloud data; Based on the image information, the point cloud within the camera's field of view is colored to generate a local point cloud map.

[0010] In some implementations, the camera data includes image information and image pose, and generating a local point cloud map based on the initial pose and the camera data includes: Determine a point cloud within a camera perspective based on the initial pose and the image pose; Back-projecting the point cloud within the camera's viewing angle onto the image information to determine a back-projection point; Based on the color of the back-projected point, the point cloud within the camera's field of view is colored to generate a local point cloud map.

[0011] In some embodiments, the evaluation result of the three-dimensional reconstruction data is determined based on the local point cloud map according to a preset evaluation index, wherein the preset evaluation index includes at least one of a data integrity evaluation index, a data quality evaluation index, and a geometric consistency evaluation index, including: Based on the data integrity evaluation index, calculating the point cloud coverage of the local point cloud map and determining the point cloud missing area; and / or Based on the data quality assessment index, calculate the noise level and resolution of the local point cloud map to determine the area where the data quality does not meet the standard; and / or Based on the geometric consistency evaluation index, the geometric consistency inside the local point cloud map is compared to determine the geometric deviation area.

[0012] In some implementations, determining the acquisition instruction of the three-dimensional reconstruction data based on the evaluation result includes: Determine a supplementary sampling instruction according to the missing area of ​​the point cloud; and / or Determine sensor parameter adjustment instructions and supplementary sampling instructions according to the area where the data quality does not meet the standards; and / or According to the geometric deviation area, a sensor correction instruction and a supplementary sampling instruction are determined.

[0013] In a second aspect, the present application further proposes an electronic device, comprising: one or more processors; A memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the three-dimensional reconstruction data acquisition method as described above.

[0014] In a third aspect, the present application further proposes a storage medium, wherein the storage medium stores executable instructions, and when the instructions are executed by a processor, the processor executes the three-dimensional reconstruction data acquisition method as described above.

[0015] Compared with the prior art, the beneficial effects of this application include: First, by acquiring and fusing attitude measurement data and point cloud data in real time and determining the initial pose of point cloud data, accurate point cloud data registration can be performed during data collection. This registration can improve positioning accuracy, reduce redundant data caused by positioning errors, further reduce the amount of data, and improve processing speed. At the same time, it helps to determine subsequent data evaluation and acquisition instructions, thereby improving data collection efficiency. Second, by generating local point cloud maps in real time during data collection, and evaluating and making decisions based on local point cloud maps, the reliance on global maps can be reduced, thereby reducing the demand for high-performance computing devices and large-capacity storage devices, saving computing power and memory resources. Third, by evaluating based on local point cloud maps, errors or omissions in the collected 3D reconstruction data can be discovered in a timely manner to improve data collection efficiency. Fourth, by determining acquisition instructions based on evaluation results, sensors can be guided to collect more accurately and efficiently. This feedback-based adjustment mechanism further improves data collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0017] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for collecting three-dimensional reconstruction data of the present application; Figure 2 A partial flow chart of an embodiment of a method for collecting three-dimensional reconstruction data of the present application; Figure 3 A partial flow chart of an embodiment of a method for collecting three-dimensional reconstruction data of the present application; Figure 4 A partial flow chart of an embodiment of a method for collecting three-dimensional reconstruction data of the present application; Figure 5 A schematic diagram of the entire process of an embodiment of a method for collecting three-dimensional reconstruction data of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device involved in the method for collecting three-dimensional reconstruction data in any embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0020] For example, the terms "first", "second", etc. used in this application may be used in this document to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element.

[0021] For another example, the terms “include”, “comprising”, etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0022] As mentioned above, in the traditional 3D reconstruction process, since the 3D reconstruction data contains a large amount of data from multiple sensors, the requirements for computing power and memory in the map storage and display process are very high. When network communication cannot meet the transmission requirements of cloud computing, the existing technology usually uses high-performance computer equipment to generate a global map after the 3D reconstruction data is collected, thereby verifying the validity of the collected data. This means that any errors or omissions in the collection need to be re-collected, which greatly affects the efficiency of data collection. Therefore, how to improve the collection efficiency of 3D reconstruction data under limited computing power and memory conditions is a technical problem that needs to be solved urgently. To this end, the present application proposes a method for collecting 3D reconstruction data, an electronic device and a storage medium, which can improve the collection efficiency of 3D reconstruction data under limited computing power and memory conditions.

[0023] like Figure 1 As shown, the method for collecting 3D reconstruction data in this embodiment includes the following steps: Step S10, acquiring three-dimensional reconstruction data collected in real time, wherein the three-dimensional reconstruction data includes camera data, posture measurement data and point cloud data.

[0024] In this embodiment, the application scenarios of the method for collecting 3D reconstruction data cover a variety of intelligent terminals, including but not limited to smart phones, personal computers, robots and other electronic devices with data processing, network communication and program running functions. After the data acquisition device with point cloud sensors (lidar, depth sensors, etc.), attitude sensors (such as IMU, inertial measurement unit) and camera sensors (such as binocular fisheye cameras) is turned on, the electronic device obtains the 3D reconstruction data collected in real time by the data acquisition device and hard-synchronized in time. Among them, hard synchronization in time refers to the synchronization in time achieved by hardware.

[0025] The three-dimensional reconstruction data in this embodiment includes camera data, attitude measurement data and point cloud data. It should be understood that camera data refers to image data obtained from a camera sensor, and the image data includes image information and image posture in some embodiments. Attitude measurement data refers to acceleration data and angular velocity data obtained from an attitude sensor. Among them, acceleration data is provided by a three-axis accelerometer to measure the linear acceleration of the attitude sensor on the three axes of X, Y, and Z. Angular velocity data is provided by a three-axis gyroscope to measure the angular velocity (rotation speed) of the attitude sensor on the three axes of X, Y, and Z. In some embodiments, attitude measurement data may also include magnetic data provided by a three-axis magnetometer to measure the magnetic field strength in the environment. Point cloud data refers to a data set consisting of a large number of points with three-dimensional coordinates obtained from point cloud sensors such as laser radars and depth sensors, which can reflect the geometric structure of the environment.

[0026] In some embodiments, the three-dimensional reconstruction data collected in real time is obtained, and the three-dimensional reconstruction data is preprocessed to improve the quality and accuracy of the three-dimensional reconstruction data. Specifically, the posture measurement data is filtered to remove noise and outliers to improve the stability and accuracy of the posture measurement data. The point cloud data is filtered to remove outliers and noise to improve the purity and quality of the point cloud data.

[0027] In some implementations, the posture measurement data or the preprocessed posture measurement data are pre-integrated to obtain the posture state. Specifically, the prior posture estimate of the posture measurement data (the posture state predicted at the previous time point) is obtained. ), based on the prior pose estimation and motion model function , determine the posture state During the pre-integration process, the electronic device will perform time synchronization and cumulative error correction on the attitude measurement data to generate a smooth and accurate attitude trajectory, thereby improving the accuracy, stability and efficiency of subsequent fusion with point cloud data.

[0028]

[0029] It is important to understand that represents the posture state at time i+1, represents the sampling time interval of the posture sensor, u represents the measurement input of the IMU (posture measurement data), and w represents Gaussian noise.

[0030]

[0031]

[0032]

[0033]

[0034] It is important to understand that x represents the posture state, R represents the posture orientation, p represents the three-dimensional position, v is the velocity, and b represents the posture bias. represents the bias of the gyroscope, represents the bias of the accelerometer, Represents the acceleration due to gravity. Represents the noise of the gyroscope measurement. Represents the noise of the accelerometer measurement. Represents the noise of the gyroscope bias. Represents the noise of the accelerometer bias. represents the angular velocity, Indicates acceleration.

[0035] Step S20: fusing the posture measurement data and the point cloud data to determine an initial posture of the point cloud data.

[0036] In this embodiment, the initial pose refers to the pose of the point cloud sensor when collecting the point cloud data. In some embodiments, the pose state x and the point cloud data can be directly fused to determine the initial pose of the point cloud data. The specific calculation method can be based on the Bayesian theorem, .in, is the posterior probability, that is, when observing the point cloud data Then the probability of the initial pose is estimated; is the likelihood probability, that is, in the posture state Point cloud data is observed probability; It is the prior probability, that is, the probability estimate of the initial pose before new point cloud data is observed.

[0037] In some embodiments, structural information of the point cloud data can be extracted, where the structural information includes line segment structure information and surface structure information. Based on Bayes' theorem, the posture measurement data (or posture state x) and the structural information of the point cloud data are fused to determine the initial posture of the point cloud data.

[0038] Step S30: generating a local point cloud map based on the initial pose and the camera data.

[0039] In this embodiment, the camera data includes image information and image pose. The image pose is calculated based on the pre-calibrated image sensor external parameters and the pose state obtained by pre-integration of the attitude sensor.

[0040] In some embodiments, a point cloud within the camera's view is determined based on the initial pose and the image pose, and the point cloud within the camera's view is colored based on the image information to generate a colored local point cloud map.

[0041] In some embodiments, based on the initial pose and the image pose, a point cloud within the camera's field of view is determined; the point cloud within the camera's field of view is back-projected onto the image information to determine the back-projection points; and based on the color of the back-projection points, the point cloud within the camera's field of view is colored to generate a local point cloud map.

[0042] In some embodiments, a real-time local map is obtained, the point cloud data is matched and registered with the real-time local map, and the optimized pose of the point cloud data is determined. The optimized pose takes into account the actual matching between the point cloud data and the real-time local map, and can be used to update the initial pose of the point cloud data to more accurately reflect the position and orientation of the point cloud sensor in the global coordinate system.

[0043] It should be noted that the construction of the local point cloud map in this embodiment is a dynamic and cyclic process. The real-time local map refers to the local point cloud map currently being displayed, which is generated based on the last round of 3D reconstruction data.

[0044] In some embodiments, a point cloud within the camera's view is determined based on the optimized pose and the image pose, and the point cloud within the camera's view is colored based on the image information to generate a colored local point cloud map.

[0045] In some embodiments, based on the optimized pose and the image pose, a point cloud within the camera's field of view is determined; the point cloud within the camera's field of view is back-projected onto the image information to determine the back-projection points; and based on the color of the back-projection points, the point cloud within the camera's field of view is colored to generate a local point cloud map.

[0046] In addition, since the local point cloud map has an intuitive visualization effect, the real-time display of the local point cloud map on the mobile device allows the collector to immediately see the effect of the currently collected 3D reconstruction data and make adjustments based on the actual situation. For example, if it is found that the data quality of a certain area is not high, the collector can immediately rescan the area to ensure the validity and integrity of the data. For another example, in a complex indoor environment, the collector can choose the best scanning route based on the real-time displayed point cloud map to avoid repeated scanning and missing areas.

[0047] Step S40, based on the local point cloud map, determining the evaluation result of the three-dimensional reconstruction data according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator and a geometric consistency evaluation indicator.

[0048] In this embodiment, the data integrity evaluation index is used to measure whether the three-dimensional reconstruction data completely covers the target object or scene. It focuses on the coverage and missingness of the data. The data quality evaluation index is used to measure the accuracy and precision of the three-dimensional reconstruction data. It focuses on the noise level, resolution and precision of the data. The geometric consistency evaluation index is used to measure the consistency and correctness of the three-dimensional reconstruction data in terms of geometry. It focuses on whether the data faithfully reflects the actual geometric features of the target object. In practical applications, one or more evaluation indicators can be selected according to specific needs to comprehensively evaluate the three-dimensional reconstruction data. For example, if the application scenario has high requirements for the coverage of the data, you can focus on the data integrity evaluation index; if you have high requirements for the accuracy and details of the data, you can focus on the data quality evaluation index; if you need to ensure that the geometry of the reconstructed data is accurate, you can focus on the geometric consistency evaluation index.

[0049] In some embodiments, the calculation accuracy is determined based on the environmental complexity of the local point cloud map, and the evaluation result of the three-dimensional reconstruction data is determined according to the calculation accuracy and the preset evaluation index. The factors determining the environmental complexity include at least one of the following factors: the data volume of the local point cloud map, the available computing resource information of the electronic device, and the system load of the electronic device. Among them, the data volume of the local point cloud map is proportional to the calculation accuracy, the available computing resource information of the electronic device is proportional to the calculation accuracy, and the system load of the electronic device is inversely proportional to the calculation accuracy. This embodiment can dynamically adjust the calculation accuracy according to the complexity of the environment, so as to make full use of computing resources and improve computing efficiency while ensuring the accuracy of the evaluation results.

[0050] In some embodiments, based on the data integrity evaluation index, the point cloud coverage of the local point cloud map is calculated to determine the point cloud missing area. Specifically, the point cloud coverage of the local point cloud map is calculated using a multi-scale analysis technique, and the point cloud missing area is determined and marked by comparing the point cloud coverage with the expected coverage.

[0051] In some embodiments, based on the data quality assessment index, the noise level and resolution of the local point cloud map are calculated, and the area where the data quality does not meet the standard is determined. Specifically, an adaptive noise filtering algorithm is applied to calculate the noise level of the local point cloud map and the resolution of the local point cloud map; based on the noise level and resolution, the area where the data quality does not meet the standard is determined and marked.

[0052] In some embodiments, based on the geometric consistency evaluation index, the geometric consistency within the local point cloud map is compared to determine the geometric deviation area. Specifically, feature points are extracted from the local point cloud map; based on the feature points, the local point cloud map is compared for geometric consistency; based on the comparison result, the geometric deviation area is determined and marked.

[0053] Step S50: determining a collection instruction for three-dimensional reconstruction data based on the evaluation result, so as to collect the three-dimensional reconstruction data based on the collection instruction.

[0054] In this embodiment, the evaluation result includes at least one of the following: a marked point cloud missing area, a marked data substandard area, and a marked geometric deviation area. The acquisition instructions determined according to the evaluation results include at least one of the following: a supplementary acquisition instruction, a sensor parameter adjustment instruction, and a sensor correction instruction. Among them, the supplementary acquisition instruction includes the data acquisition path and / or the position and angle of the sensor on the data acquisition device. The sensor parameter adjustment instruction is used to adjust the sensor parameters of the data acquisition device, such as exposure time, resolution, gain, etc., to improve the data acquisition quality. The sensor correction instruction is used to calibrate or correct the sensor on the data acquisition device to eliminate geometric deviations.

[0055] In some embodiments, a supplementary acquisition instruction is determined based on the missing area of ​​the point cloud, and the supplementary acquisition instruction is used to instruct the data acquisition device to supplement the acquisition of three-dimensional reconstruction data of the missing area of ​​the point cloud. Specifically, the supplementary acquisition instruction is determined based on the position and range of the missing area of ​​the point cloud to supplement the acquisition of three-dimensional reconstruction data of the missing area of ​​the point cloud.

[0056] In some implementations, sensor parameter adjustment instructions and supplementary acquisition instructions are determined based on areas where data quality does not meet standards, so as to supplement the 3D reconstruction data of the areas where data quality does not meet standards according to the sensor parameter adjustment instructions and supplementary acquisition instructions.

[0057] In some implementations, a sensor correction instruction and a supplementary sampling instruction are determined based on the geometric deviation area, so as to supplement the acquisition of three-dimensional reconstruction data of the geometric deviation area according to the sensor correction instruction and the supplementary sampling instruction.

[0058] In this embodiment, on the one hand, by acquiring and fusing posture measurement data and point cloud data in real time and determining the initial position and posture of point cloud data, accurate point cloud data registration can be achieved during data acquisition. This registration can improve the accuracy of positioning, reduce redundant data caused by positioning errors, further reduce the amount of data, and improve processing speed. At the same time, it is helpful for subsequent data evaluation and determination of acquisition instructions, thereby improving data acquisition efficiency. On the other hand, by generating a local point cloud map in real time during data acquisition, and evaluating and making decisions based on the local point cloud map, the dependence on the global map can be reduced, thereby reducing the demand for high-performance computing devices and large-capacity storage devices, saving computing power and memory resources. On the third hand, by evaluating based on the local point cloud map, errors or omissions in the collected three-dimensional reconstruction data can be discovered in time to improve data acquisition efficiency. On the fourth hand, by determining the acquisition instructions based on the evaluation results, the sensor can be guided to perform more accurate and efficient acquisition. This feedback-based adjustment mechanism further improves data acquisition efficiency.

[0059] like Figure 2 As shown, the step S30 includes: Step A10, obtaining a real-time local map, and based on the initial posture, converting the point cloud data into the real-time local map.

[0060] Step A20: determining, in the real-time local map, a matching point cloud corresponding to the point cloud data.

[0061] In this embodiment, a real-time local map is obtained, and based on the initial position and posture, the point cloud data is converted into a coordinate system in the real-time local map. A matching point cloud corresponding to or most similar to the point cloud data is found in the real-time local map.

[0062] Step A30, determining the optimized pose of the point cloud data by minimizing the spatial distance of the matching point clouds.

[0063] In this embodiment, the spatial distance between the point cloud data and the matching point cloud is minimized by an optimization algorithm (such as gradient descent, Levenberg-Marquardt algorithm, etc.). The spatial distance refers to the distance between each point in the point cloud data and the matching point in the corresponding matching point cloud. In the process of minimizing the spatial distance, the position and posture of the point cloud data are adjusted to obtain the optimized posture after optimization.

[0064] Step A40: Generate a local point cloud map based on the optimized posture and the camera data.

[0065] The specific implementation of this embodiment refers to step S30, and this embodiment will not be described in detail here.

[0066] In this embodiment, on the one hand, by acquiring a real-time local map and performing point cloud data conversion and registration, real-time registration can be achieved during the data acquisition process, thereby performing three-dimensional reconstruction in a dynamic environment, and being able to update and adjust the reconstruction results in a timely manner. On the other hand, by minimizing the spatial distance of the matching point cloud, a more accurate point cloud data registration result can be obtained. This helps to improve the accuracy of three-dimensional reconstruction and makes the generated local point cloud map more accurate.

[0067] like Figure 3 As shown, the step A40 may further include: Step B10, obtaining the latest key frame saved in the sliding window.

[0068] The sliding window in this embodiment is used to store the latest key frames. The key frame added to the sliding window at the latest is the latest key frame.

[0069] Step B20, calculating the difference between the optimized pose and the pose of the latest key frame.

[0070] In this embodiment, the posture difference value refers to the difference between the current optimized posture and the posture of the latest key frame. This difference value can be expressed by a rotation angle and a translation distance.

[0071] Step B30, determining whether the difference value exceeds a preset difference value.

[0072] In this embodiment, the preset difference value is a threshold value, which is used to determine whether the difference between the current optimized posture and the posture of the latest key frame is large enough.

[0073] Step B40: If yes, determine that the point cloud data is the latest key frame, and generate a local point cloud map based on the optimized posture and the camera data.

[0074] In this embodiment, if the difference value exceeds the preset difference value, the current point cloud data is determined as the latest key frame, and a local point cloud map corresponding to the latest key frame is generated. If the difference value does not exceed the preset difference value, the point cloud data is determined to be a redundant frame, and the redundant frame is not used to generate a local point cloud map, and the process returns to step S10. This embodiment can simplify the data processing process by using sliding windows and key frame technology. Processing only key frames instead of all frames can significantly improve processing efficiency.

[0075] In some embodiments, when the latest key frame is generated, it is detected whether the sliding window has reached the maximum capacity. If so, the key frame that was first added to the sliding window is removed, and the latest key frame is added to the sliding window. This ensures that only the latest key frames are included in the sliding window. In this way, the electronic device can efficiently manage and utilize key frame data, avoid storing too many redundant frames, reduce the burden of data storage and processing, and improve processing speed and accuracy.

[0076] In this embodiment, the latest key frame is determined by calculating the posture difference between the optimized posture and the latest key frame, and by judging whether the difference value exceeds the preset difference value. This can not only effectively avoid frequent updates due to small posture changes, but also avoid storing too many redundant frames, thereby reducing the burden of data storage and processing and improving processing speed and accuracy.

[0077] like Figure 4 and Figure 5 As shown, the step B40 further includes: Step B41, calculating the density distribution and feature distribution of the point cloud data.

[0078] In this embodiment, density distribution refers to the distribution of points in point cloud data, which is usually expressed by the number and spatial distribution of points. Feature distribution refers to the distribution of feature points in point cloud data, which is usually expressed by the number and spatial distribution of feature points. In point cloud data, feature points refer to points with significant geometric characteristics, which can be points with higher curvature and points with larger gradient changes. These points have unique identity and stability in point cloud data.

[0079] Step B42, determining the sampling rate distribution of the point cloud data based on the density distribution and the feature distribution.

[0080] In this embodiment, the sampling rate distribution means that the sampling rate of the point cloud data is different in different areas. Specifically, the sampling rate will be lower in areas with higher density or more features, and higher in areas with lower density or fewer features. That is, the density of points and feature points in the point cloud data is inversely proportional to the sampling rate.

[0081] Step B43, downsampling the point cloud data according to the sampling rate distribution.

[0082] In this embodiment, downsampling refers to sparse point cloud data, that is, reducing the number of points in the point cloud data to reduce the amount of data and computational complexity. By downsampling the point cloud data according to the sampling rate distribution, the size of the point cloud data can be reduced and the processing efficiency can be improved while retaining the main geometric features.

[0083] Step B44, saving the downsampled point cloud data to the global map.

[0084] In this embodiment, the global map is not rendered, but all point cloud data is retained for the global map. Only downsampled data is saved, which can significantly reduce storage space and management complexity.

[0085] Step B45, determining the point cloud within the camera's field of view based on the optimized pose, the image pose and the downsampled point cloud data.

[0086] Step B46, based on the image information, coloring the point cloud within the camera's field of view to generate a local point cloud map.

[0087] In this embodiment, except for replacing the point cloud data with downsampled point cloud data, the specific implementation method is consistent with that in step S30, and will not be described in detail in this embodiment.

[0088] In this embodiment, on the one hand, by calculating the density distribution and the feature distribution, the sampling rate distribution can be determined, so that the point cloud data can be downsampled. This helps to reduce the amount of data and the complexity of calculation, and improve the efficiency of data processing. On the other hand, the downsampling process can filter out some noise and redundant information, making the point cloud data cleaner and smoother, which helps to improve the accuracy and quality of the constructed local point cloud map. On the third hand, the downsampled point cloud data is clearer and easier to understand when visualized, and will not appear messy due to too many points. On the fourth hand, downsampling can reduce the computational instability and error accumulation caused by overly dense data, thereby improving the robustness of the algorithm.

[0089] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0090] In one embodiment, an electronic device is also provided, comprising one or more processors; a memory, wherein one or more programs are stored in the memory, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments.

[0091] In one embodiment, Figure 6, which shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0092] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0093] In particular, based on the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through the communication part 709, and / or installed from a removable medium 711. When the instructions are executed by the central processing unit (CPU) 701, the various method steps described in the present application are executed.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0095] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the above claims, any one of the claimed embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form of implication that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for collecting three-dimensional reconstruction data, characterized in that: The method comprises: Acquire three-dimensional reconstruction data collected in real time, wherein the three-dimensional reconstruction data includes camera data, posture measurement data and point cloud data; Fusing the posture measurement data and the point cloud data to determine an initial posture of the point cloud data; Based on the initial pose and the camera data, generating a local point cloud map; Based on the local point cloud map, determining an evaluation result of the three-dimensional reconstruction data according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator, and a geometric consistency evaluation indicator; Based on the evaluation result, an acquisition instruction for the three-dimensional reconstruction data is determined, so as to acquire the three-dimensional reconstruction data based on the acquisition instruction.

2. The method for collecting three-dimensional reconstruction data according to claim 1, characterized in that: The generating a local point cloud map based on the initial pose and the camera data further comprises: Acquire a real-time local map, and based on the initial pose, convert the point cloud data into the real-time local map; In the real-time local map, determining a matching point cloud corresponding to the point cloud data; Determining an optimized pose of the point cloud data by minimizing a spatial distance of matching point clouds; Based on the optimized pose and the camera data, a local point cloud map is generated.

3. The method for collecting three-dimensional reconstruction data according to claim 2, characterized in that: The generating a local point cloud map based on the optimized posture and the camera data further includes: Get the latest key frame saved in the sliding window; Calculating the difference between the optimized pose and the pose of the latest key frame, and determining whether the difference exceeds a preset difference value; If yes, determining that the point cloud data is the latest keyframe, and generating a local point cloud map based on the optimized pose and the camera data; If not, the step of acquiring the three-dimensional reconstruction data collected in real time is performed.

4. The method for collecting three-dimensional reconstruction data according to claim 3, characterized in that: After determining that the point cloud data is the latest key frame, the method further comprises: Calculating density distribution and feature distribution of the point cloud data; Determining a sampling rate distribution of the point cloud data based on the density distribution and the feature distribution; Downsampling the point cloud data according to the sampling rate distribution; Save the downsampled point cloud data to the global map.

5. The method for collecting three-dimensional reconstruction data according to claim 4, characterized in that: The camera data includes image information and image pose, and generating a local point cloud map based on the optimized pose and the camera data includes: Determining a point cloud within a camera perspective based on the optimized pose, the image pose, and the downsampled point cloud data; Based on the image information, the point cloud within the camera's field of view is colored to generate a local point cloud map.

6. The method for collecting three-dimensional reconstruction data according to claim 1, characterized in that: The camera data includes image information and image pose, and generating a local point cloud map based on the initial pose and the camera data includes: Determine a point cloud within a camera perspective based on the initial pose and the image pose; Back-projecting the point cloud within the camera's viewing angle onto the image information to determine a back-projection point; Based on the color of the back-projected point, the point cloud within the camera's field of view is colored to generate a local point cloud map.

7. The method for collecting three-dimensional reconstruction data according to claim 1, characterized in that: The step of determining the evaluation result of the three-dimensional reconstruction data based on the local point cloud map according to preset evaluation indicators, wherein the preset evaluation indicators include at least one of a data integrity evaluation indicator, a data quality evaluation indicator, and a geometric consistency evaluation indicator, including: Based on the data integrity evaluation index, calculating the point cloud coverage of the local point cloud map and determining the point cloud missing area; and / or Based on the data quality assessment index, calculate the noise level and resolution of the local point cloud map to determine the area where the data quality does not meet the standard; and / or Based on the geometric consistency evaluation index, the geometric consistency inside the local point cloud map is compared to determine the geometric deviation area.

8. The method for collecting three-dimensional reconstruction data according to claim 7, characterized in that: The step of determining the acquisition instruction of the three-dimensional reconstruction data based on the evaluation result includes: Determine a supplementary sampling instruction according to the missing area of ​​the point cloud; and / or Determine sensor parameter adjustment instructions and supplementary sampling instructions according to the area where the data quality does not meet the standards; and / or According to the geometric deviation area, a sensor correction instruction and a supplementary sampling instruction are determined.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the method for collecting three-dimensional reconstruction data as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium stores executable instructions, and when the instructions are executed by the processor, the processor executes the method for collecting three-dimensional reconstruction data according to any one of claims 1 to 8.

Images