Three-dimensional reconstruction method and device, equipment, storage medium and product

Abstract

The invention belongs to the field of three-dimensional reconstruction, and discloses a three-dimensional reconstruction method, device and equipment, a storage medium and a computer program product. The method comprises the following steps: selecting a key frame from a collected plane image; obtaining IMU data, and performing pre-integration on the IMU data based on the key frame to obtain a pre-integration result; performing visual inertia initialization according to the pre-integration result and the key frame to obtain a visual inertia initialization result; and determining a camera attitude according to the key frame and the visual inertia initialization result, and determining a three-dimensional reconstruction result based on the camera attitude. Compared with an existing mode of carrying out real-time three-dimensional reconstruction through combination of a monocular camera VO and Nerf, the mode provided by the invention can realize high-precision modeling.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, apparatus, device, storage medium and computer program product. Background Art

[0002] Most existing Nerf reconstruction technologies are still in the offline rendering stage and require pre-training, which is time-consuming and costly. There are methods that use monocular camera VO combined with Nerf for real-time 3D reconstruction, but the pose estimation and system robustness obtained by VO still need to be improved. Therefore, how to perform 3D reconstruction based on the collected map images and obtain a 3D real-time map has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the present invention is to provide a three-dimensional reconstruction method, device, equipment, storage medium and computer program product, aiming to solve the technical problem of low three-dimensional reconstruction efficiency when performing three-dimensional reconstruction on collected two-dimensional map images in the existing technology.

[0004] To achieve the above object, the present invention provides a three-dimensional reconstruction method, which includes the following steps:

[0005] Select key frames from the acquired planar image;

[0006] Acquire IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result;

[0007] Performing visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result;

[0008] A camera pose is determined according to the key frame and the visual-inertial initialization result, and a three-dimensional reconstruction result is determined based on the camera pose.

[0009] Optionally, the step of determining a camera pose according to the key frame and the visual-inertial initialization result, and determining a three-dimensional reconstruction result based on the camera pose includes:

[0010] determining a visual reprojection error based on the keyframe;

[0011] Determine an IMU pre-integration residual based on the key frame and the visual-inertial initialization result;

[0012] Determine the camera pose based on the visual reprojection error and the IMU pre-integration residual;

[0013] A three-dimensional reconstruction result is determined based on the camera pose.

[0014] Optionally, the step of determining a three-dimensional reconstruction result based on the camera posture includes:

[0015] The camera pose is used as the input of Nerf and combined with the skip voxel strategy to accelerate Nerf optimization to obtain a three-dimensional reconstruction result.

[0016] Optionally, the visual inertial initialization result includes gyroscope bias, accelerometer bias, gravitational acceleration, scale, and speed;

[0017] The step of performing visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result includes:

[0018] Determine a gyroscope bias, an accelerometer bias, a scale, and a gravitational acceleration according to the key frame and the pre-integration result;

[0019] A velocity is determined based on the gyroscope bias, the accelerometer bias, and the gravitational acceleration.

[0020] Optionally, the step of selecting a key frame from the acquired planar image includes:

[0021] Collecting a planar image, performing feature extraction on the planar image, and obtaining an ORB feature;

[0022] Determine the descriptor distance between the feature point and the feature point to be matched based on the ORB feature;

[0023] A key frame is selected from the planar image according to the descriptor distance.

[0024] Optionally, the step of selecting a key frame from the planar image according to the descriptor distance includes:

[0025] Determining the number of feature point matches based on the descriptor distance;

[0026] A key frame is selected from the planar image based on the number of feature point matches and a preset key frame selection strategy.

[0027] In addition, to achieve the above-mentioned object, the present invention further provides a three-dimensional reconstruction device, comprising:

[0028] An acquisition module, used for selecting key frames from the acquired planar images;

[0029] A pre-integration module is used to obtain IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result;

[0030] an initialization module, configured to perform visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result;

[0031] A three-dimensional reconstruction module is used to determine a camera pose according to the key frame and the visual-inertial initialization result, and to determine a three-dimensional reconstruction result based on the camera pose.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also proposes a three-dimensional reconstruction device, which includes: a memory, a processor, and a three-dimensional reconstruction program stored in the memory and executable on the processor, wherein the three-dimensional reconstruction program is configured to implement the steps of the three-dimensional reconstruction method described above.

[0033] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a three-dimensional reconstruction program is stored. When the three-dimensional reconstruction program is executed by a processor, the steps of the three-dimensional reconstruction method described above are implemented.

[0034] In addition, to achieve the above-mentioned object, the present invention further provides a computer program product, which includes a three-dimensional reconstruction program. When the three-dimensional reconstruction program is executed by a processor, the steps of the three-dimensional reconstruction method described above are implemented.

[0035] The present invention selects keyframes from a captured planar image; acquires IMU data, pre-integrates the IMU data based on the keyframes to obtain a pre-integration result; performs visual-inertial initialization based on the pre-integration result and the keyframes to obtain a visual-inertial initialization result; determines the camera pose based on the keyframes and the visual-inertial initialization result, and determines a three-dimensional reconstruction result based on the camera pose. Compared to existing methods that use a monocular VO camera combined with Nerf for real-time three-dimensional reconstruction, the above-mentioned method of the present invention has real-time operation. During the three-dimensional reconstruction process, it can detect in real time which aspects of the reconstructed map are flawed, requiring further data acquisition and three-dimensional reconstruction, and output a more complete model. Furthermore, because it incorporates an IMU sensor, the frequency of which is typically above 200Hz, it has better robustness against system rotation. Furthermore, since monocular vision has large errors in estimating scale, using an IMU for fusion can reduce scale errors and generate a better camera pose. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the structure of a three-dimensional reconstruction device in a hardware operating environment according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the process of the first embodiment of the three-dimensional reconstruction method of the present invention;

[0038] Figure 3 A schematic diagram of pre-integration of a first embodiment of a three-dimensional reconstruction method according to the present invention;

[0039] Figure 4 A schematic diagram of the angle of gravity acceleration direction of the first embodiment of the three-dimensional reconstruction method of the present invention;

[0040] Figure 5 2 is a flow chart of a second embodiment of a three-dimensional reconstruction method according to the present invention;

[0041] Figure 6 A schematic diagram of visual reprojection error of the second embodiment of the 3D reconstruction method of the present invention;

[0042] Figure 7 This is a schematic diagram of local map optimization in the second embodiment of the 3D reconstruction method of the present invention;

[0043] Figure 8 FIG. 4 is a structural block diagram of a first embodiment of a three-dimensional reconstruction device according to the present invention.

[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a 3D reconstruction device in the hardware operating environment involved in an embodiment of the present invention.

[0047] like Figure 1 As shown, the three-dimensional reconstruction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art will understand that Figure 1The structure shown in the figure does not constitute a limitation on the three-dimensional reconstruction device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a three-dimensional reconstruction program.

[0050] exist Figure 1 In the illustrated 3D reconstruction device, the network interface 1004 is primarily used for data communication with a network server; the user interface 1003 is primarily used for data interaction with a user; the processor 1001 and memory 1005 in the 3D reconstruction device of the present invention can be provided within the 3D reconstruction device. The 3D reconstruction device invokes a 3D reconstruction program stored in the memory 1005 via the processor 1001 and executes the 3D reconstruction method provided by an embodiment of the present invention.

[0051] Based on the above three-dimensional reconstruction device, an embodiment of the present invention provides a three-dimensional reconstruction method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a three-dimensional reconstruction method according to the present invention.

[0052] In this embodiment, the three-dimensional reconstruction method includes the following steps:

[0053] Step S10: Select key frames from the acquired planar image.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a mobile phone, tablet computer, or personal computer, or an electronic device or 3D reconstruction device capable of performing the aforementioned functions. This embodiment and the following embodiments will be described below using the 3D reconstruction device as an example.

[0055] It should be noted that the 3D reconstruction device is equipped with a camera, and the planar image may be an RGB image captured in real time by the camera in the 3D reconstruction device. The key frames selected from the captured planar images may be selected at a preset selection interval from the captured images as key frames. For example, one image may be selected as a key frame every five images.

[0056] Furthermore, in order to improve the efficiency of three-dimensional reconstruction, the step S10 may include: collecting a planar image, performing feature extraction on the planar image to obtain an ORB feature;

[0057] Determine the descriptor distance between the feature point and the feature point to be matched based on the ORB feature;

[0058] A key frame is selected from the planar image according to the descriptor distance.

[0059] The step of selecting a key frame from the planar image according to the descriptor distance comprises:

[0060] Determining the number of feature point matches based on the descriptor distance;

[0061] A key frame is selected from the planar image based on the number of feature point matches and a preset key frame selection strategy.

[0062] It should be understood that ORB feature points are achieved by combining two existing feature point detectors and descriptors. Specifically: Feature Point Detection: ORB uses the FAST (Features From Accelerated Segment Test) algorithm to detect feature points. The basic idea of ​​the FAST algorithm is to determine the location of a feature point by comparing the difference between the target point and other points in its surrounding neighborhood. It uses the density (grayscale value) of the target point and the surrounding pixels within a certain range to compare to determine whether the target point is a feature point. The key to the FAST algorithm is to select an appropriate threshold to filter out most non-feature points while retaining a small number of feature points. Feature Point Description: ORB uses the BRIEF (Binary Robust Independent Elementary Features) algorithm to describe feature points. The BRIEF algorithm constructs a descriptor by randomly selecting a set of point pairs and calculating their grayscale value differences. This method effectively reduces the impact of noise and improves the robustness of feature point detection. ORB feature points not only have a significant improvement in speed compared to SIFT and SURF, but also maintain rotation invariance and a certain degree of scale invariance. In addition, ORB feature points also include keypoint information, which provides detailed information about the feature point's orientation, size, and other aspects. In summary, ORB feature points are composed of two parts: keypoints and descriptors. Keypoints include information about the feature point's position in the image, its orientation, and size, while descriptors are calculated based on information about the pixels surrounding the keypoint. ORB feature points are designed to be fast to extract and rotationally and scale-invariant.

[0063] It should be noted that feature extraction from the planar image can obtain ORB features, which can be 800 feature points extracted from the image. ORB feature points are used for feature extraction. ORB features consist of keypoints and descriptors. The keypoints are FAST feature points, and the descriptors are BRIEF descriptors, which enhance the rotational invariance of the feature points. Feature matching correspondence can be derived from the similarity between descriptors. Given the high frequency of VIO and the limited range of feature points in a short period of time, a relatively small range can be searched for feature points. The descriptor distance between the feature points within the range and the feature points to be matched is then calculated to determine the optimal matching point. Keyframes selected from the planar image based on the descriptor distance can include selecting images with more than a preset number of matching feature points as keyframes. The preset keyframe selection strategy can include selecting images with more than a preset number of matching feature points or selecting no keyframes for more than a preset number of images. For example, if an image has more than a preset number of matching feature points, the image is selected as a keyframe; or if no keyframes are selected for more than ten images, the 11th image is selected as a keyframe. Both the preset number and the preset number of images are pre-set thresholds.

[0064] It should be understood that visual-inertial odometry (VIO) is a tightly coupled optimization of vision and IMU, which minimizes the error through reprojection error and IMU pre-integration residual calculation to provide more robust and accurate pose estimation; Neural Radiance Field (Nerf) is used for image rendering, which is to render the image from a given camera perspective through a deep learning model for three-dimensional implicit space modeling; Instant-ngp uses multi-resolution hash coding and CUD A platform to perform optimization based on stochastic gradient descent, which can speed up Nerf rendering.

[0065] Step S20: Acquire IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result.

[0066] It should be noted that IMU (Inertial Measurement Unit) data usually contains information about the rotation and linear motion of the device relative to its environment. In AirSim and ROS, the coordinate systems of the IMU output data are divided into two types: body coordinate system and world coordinate system. Body coordinate system: In this coordinate system, angular velocity and angular acceleration information are mainly provided. World coordinate system: In this coordinate system, the information provided includes linear velocity and linear acceleration. These data are recorded in the world coordinate system, that is, the North-East-Down (NED) coordinate system. The IMU data can be the IMU data of the three-dimensional reconstruction device. The pre-integration of the IMU data based on the key frame to obtain the pre-integration result can be referred to Figure 3 , Figure 3 A schematic diagram of pre-integration of a first embodiment of a three-dimensional reconstruction method according to the present invention; Figure 3 The hollow points in the image are used to represent the acquired plane image, and the solid points are used to represent the selected key frames. Since the IMU data frequency is a multiple of the camera image acquisition frequency, it is necessary to pre-integrate the IMU data to obtain the same time period as the image for optimization. For details, please refer to Figure 3 The pre-integration may adopt the pre-integration method in the prior art, which is not limited in this embodiment.

[0067] Step S30: performing visual inertial initialization according to the pre-integration result and the key frame to obtain a visual inertial initialization result.

[0068] It should be noted that the visual inertial initialization is performed according to the pre-integration result and the key frame to obtain the visual inertial initialization result, which may be to determine the gyroscope bias, gravity acceleration, accelerometer bias and speed according to the pre-integration result and the key frame.

[0069] Furthermore, in order to improve the accuracy of three-dimensional modeling, the visual inertial initialization result includes gyroscope bias, accelerometer bias, gravity acceleration, scale, and velocity; the step S30 may include: determining the gyroscope bias, accelerometer bias, scale, and gravity acceleration based on the key frame and the pre-integration result;

[0070] A velocity is determined based on the gyroscope bias, the accelerometer bias, and the gravitational acceleration.

[0071] It should be noted that the determination of the gyroscope bias, accelerometer bias, scale, and gravity acceleration according to the key frame and the pre-integration result may be:

[0072] 1) Gyroscope bias estimation value bg (i.e. gyroscope bias). When pure vision is run alone and the information required for initialization, i.e. the information of the key frame, is obtained, the rotation of the i-th key frame relative to the first frame can be obtained as Through the camera and IMU external reference R CB The carrier's posture can be further obtained During the initialization process, b can be estimated based on the rotation of consecutive key frames. g Assume that the IMU bias is zero and b g Remain unchanged. Obtain the gyroscope zero bias b g :

[0073]

[0074] Among them, argmin function is a common mathematical function used to calculate the minimum value of a function, N is used to represent the frame rate, and i is used to represent the current frame. Used to represent the relative posture of the previous frame and the next frame, Used to represent the Jacobian matrix, is used to represent the covariance, T is used to represent the matrix transpose, Used to represent the rotation of the i+1th frame.

[0075] 2) Without considering the accelerometer zero bias ba, roughly estimate the gravitational acceleration And scale s, consider the positional relationship between IMU and camera:

[0076] p WB =s*p WC +R WC t CB

[0077] In this embodiment, C is used to represent the camera coordinate system, W is used to represent the world coordinate system, B is used to represent the body coordinate system, p is used to represent translation, R is used to represent rotation, and p is used to represent the camera coordinate system. WB Used to represent the translation from the world coordinate system to the body coordinate system, p WC Used to represent the translation from the world coordinate system to the camera coordinate system, R WC The rotation from the world coordinate system to the camera coordinate system, t CB Used to represent the translation from the camera coordinate system to the body coordinate system.

[0078] Substituting the above formula into the pre-integration model, we get:

[0079]

[0080] The pre-integration model can be an existing pre-integration model, and the approximate value of the gravitational acceleration can be solved by the singular value decomposition (SVD) of the above formula: and scale s.

[0081] in, Used to represent the current frame rate, Used to characterize the offset between two adjacent IMU frames, p CB Used to represent the offset from the camera coordinate system to the body coordinate system, that is, the translation.

[0082] 3) Estimate the accelerometer zero bias ba. Considering the accelerometer zero bias, make a fine estimate of the gravity acceleration and scale s. According to the roughly calculated gravity acceleration, the gravity acceleration direction vector in the world coordinate system is obtained as Based on The rotation matrix R can be obtained WI :

[0083]

[0084] in, Used to represent the direction vector. For details, please refer to Figure 4 , Figure 4 Schematic diagram of the angle of gravity acceleration direction in the first embodiment of the 3D reconstruction method of the present invention.

[0085] Therefore, we can get the gravitational acceleration g W for:

[0086] It can be seen that δθ has only two degrees of freedom, and when considering When the accelerometer bias is included in , the gravitational acceleration can be expressed as:

[0087]

[0088] δθ xy =[δθ x δθ y ] T

[0089] Then using the first-order approximation:

[0090]

[0091] Considering the acceleration bias for pre-integrated measurement Δp ii+1 The correction of , we can get:

[0092]

[0093] Similarly, the relevant parameters can be solved based on the SVD of multi-frame data, that is, the gyroscope bias, accelerometer bias, scale and gravity acceleration can be obtained through the above formulas.

[0094] Then, based on the obtained gyroscope bias, accelerometer bias, and gravity acceleration and the pre-integration formula in step 2 above, the velocities of the first N-1 initialization keyframes can be obtained. The velocity of the last keyframe can be estimated through IMU integration. This is the velocity in the visual-inertial initialization result.

[0095] Step S40: determining a camera pose according to the key frame and the visual-inertial initialization result, and determining a three-dimensional reconstruction result based on the camera pose.

[0096] It should be noted that determining the camera pose based on the keyframes and the visual-inertial initialization results can include determining a visual reprojection error and an IMU pre-integration residual based on the keyframes and the visual-inertial initialization results, and solving the camera pose for each frame based on the visual reprojection error and the IMU pre-integration residual. The calculated camera pose is then used as input to Nerf, and Nerf is trained using the updated camera pose and map. Nerf optimizes by minimizing the photometric error through image color, obtaining a Nerf optimization result, and outputting a 3D reconstruction result based on the optimization result using Nerf.

[0097] This embodiment selects keyframes from the captured planar image; obtains IMU data, pre-integrates the IMU data based on the keyframes to obtain a pre-integration result; performs visual-inertial initialization based on the pre-integration result and the keyframes to obtain a visual-inertial initialization result; determines the camera pose based on the keyframes and the visual-inertial initialization result, and determines a three-dimensional reconstruction result based on the camera pose. Compared to existing methods of performing real-time three-dimensional reconstruction using a monocular camera VO combined with Nerf, the above-mentioned method of this embodiment has real-time operation. During the three-dimensional reconstruction process, it is possible to detect in real time which aspects of the reconstructed map have defects, requiring further data collection and three-dimensional reconstruction, and output a more complete model. Moreover, because the IMU sensor is integrated, the frequency of the sensor is usually above 200 Hz, which has better robustness against system rotation. In addition, since monocular vision has large errors in estimating scale, using the IMU for fusion can reduce the scale error and generate a better camera pose.

[0098] This embodiment uses visual extraction of ORB feature points, calculates the reprojection error of the feature points, and simultaneously combines the IMU pre-integration residual to perform tight coupling optimization of the two errors, providing more robust and accurate pose estimation, solving the pose estimation accuracy problem provided by VO, and can run in real time; the pose estimation of VIO is used as the input of Nerf, the photometric error is minimized by image color, and the Instant-ng p is used to implement the skip voxel strategy to achieve real-time and high-precision modeling of the Nerf system.

[0099] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a second embodiment of a three-dimensional reconstruction method according to the present invention.

[0100] Based on the above first embodiment, in this embodiment, step S40 includes:

[0101] Step S401: determining a visual reprojection error according to the key frame.

[0102] It should be noted that, please refer to Figure 6 , Figure 6 This is a schematic diagram of the visual reprojection error of the second embodiment of the 3D reconstruction method of the present invention; the pixels of the spatial point P between the two frames are p1 and p2 respectively. By projecting p1 onto the other frame, the rotation R and translation p of the two frames are calculated by minimizing the reprojection e. j,k is the reprojection error between landmark k and frame j, i.e., the visual reprojection error, which can be calculated by the following formula:

[0103]

[0104] Among them, e j,k Used to characterize visual reprojection error, It is used to represent the observed position of the image feature point of landmark k in frame j, Z C Used to represent landmarks in the camera coordinate system The third component, T BC is the external parameter matrix of the camera and body coordinate system, K is used to represent the internal parameter matrix of the camera, l k Used to represent the location of landmark k.

[0105] Step S402: Determine an IMU pre-integration residual based on the key frame and the visual-inertial initialization result.

[0106] It should be noted that the IMU pre-integration residual E IMU (i,j) is composed of the IMU pre-integrated measurement residual and the zero-bias random walk residual. It can be calculated by the following formula:

[0107]

[0108]

[0109] Among them, i and j are used to represent the i-th frame and j-th frame in the IMU, bg and ba are the zero bias of the gyroscope and accelerometer respectively. IMU (i, j) is used to represent the IMU pre-integration measurement residual, e b (i,j) is used to characterize the zero-bias random walk residual, Used to characterize the gyroscope zero bias value of the i-th frame and the j-th frame, The accelerometer zero bias value used to characterize the i-th frame and the j-th frame.

[0110] Step S403: Determine the camera pose according to the visual reprojection error and the IMU pre-integration residual.

[0111] It should be noted that the camera posture can be determined based on the visual reprojection error and the IMU pre-integration residual, referring to Figure 7 , Figure 7 This is a schematic diagram of local map optimization in the second embodiment of the 3D reconstruction method of the present invention. Specifically, local map optimization is performed using a sliding window. The window is set to a preset size, for example, 10. Then, the local map is jointly optimized using 10 keyframes. The constraints include the IMU constraints between all adjacent keyframes within the sliding window and the reprojection error constraints between the keyframes within the window and the landmarks. The pose of the camera in each frame is solved using the following formula:

[0112]

[0113]

[0114] Among them, the five parameters in λ are used to characterize the camera's posture information, and the camera posture can be determined by the above formula and the visual inertial initialization result.

[0115] Step S404: Determine a three-dimensional reconstruction result based on the camera posture.

[0116] It should be noted that the determination of the 3D reconstruction result based on the camera pose may be performed by using the calculated camera pose as input to Nerf, using the updated camera pose and map to train Nerf, and Nerf optimizing by minimizing the photometric error through image color:

[0117] r(d)=(-R T t+dR T K -1 [u,v,1] T ),d∈Z

[0118] Among them, r(d) is used to characterize the optimization result, t is used to characterize the translation, which can be understood as Pwc, d is used to characterize the viewing direction, R is used to characterize the camera rotation, which can be understood as Rwc, K is used to characterize the camera's intrinsic parameter matrix, u and v are used to characterize pixel values, and z is a real value.

[0119] At the same time, by skipping voxels, when the position is adopted along the projected ray, if the voxel is not occupied, the voxel is skipped, which speeds up the Nerf operation and finally outputs the reconstructed three-dimensional model, that is, the three-dimensional reconstruction result.

[0120] This embodiment determines the visual reprojection error based on the key frame; determines the IMU pre-integration residual based on the key frame and the visual-inertial initialization result; determines the camera pose based on the visual reprojection error and the IMU pre-integration residual; and determines the three-dimensional reconstruction result based on the camera pose. This embodiment is based on the visual-inertial odometry and combines the characteristics of the IMU sensor to have better robustness against system rotation, and during the visual-inertial initialization process, the generated scale s is more accurate than that of a monocular camera. The pose generated by VIO is used as the input of Nerf, and the updated camera pose and map are used to train Nerf, thereby improving the accuracy of Nerf's position and reducing the time of the entire three-dimensional modeling. Nerf does not require pre-training, and the entire system can run in real time.

[0121] Reference Figure 8 , Figure 8 FIG. 4 is a structural block diagram of a first embodiment of a three-dimensional reconstruction device according to the present invention.

[0122] like Figure 8 As shown, the three-dimensional reconstruction device proposed in the embodiment of the present invention includes:

[0123] The acquisition module 10 is used to select key frames from the acquired planar images;

[0124] A pre-integration module 20 is used to obtain IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result;

[0125] an initialization module 30, configured to perform visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result;

[0126] The 3D reconstruction module 40 is configured to determine a camera pose according to the key frame and the visual-inertial initialization result, and determine a 3D reconstruction result based on the camera pose.

[0127] This embodiment selects keyframes from a captured planar image; acquires IMU data, pre-integrates the IMU data based on the keyframes to obtain a pre-integration result; performs visual-inertial initialization based on the pre-integration result and the keyframes to obtain a visual-inertial initialization result; determines the camera pose based on the keyframes and the visual-inertial initialization result, and determines a 3D reconstruction result based on the camera pose. Compared to existing methods of real-time 3D reconstruction using a monocular VO camera combined with Nerf, this embodiment's method can achieve high-precision modeling.

[0128] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the embodiment scheme, and no limitation is made here.

[0129] In addition, for technical details not fully described in this embodiment, reference can be made to the three-dimensional reconstruction method provided in any embodiment of the present invention, and will not be repeated here.

[0130] Based on the first embodiment of the three-dimensional reconstruction device of the present invention, a second embodiment of the three-dimensional reconstruction device of the present invention is proposed.

[0131] In this embodiment, the 3D reconstruction module 40 is further configured to determine a visual reprojection error based on the key frame;

[0132] Determine an IMU pre-integration residual based on the key frame and the visual-inertial initialization result;

[0133] Determine the camera pose based on the visual reprojection error and the IMU pre-integration residual;

[0134] A three-dimensional reconstruction result is determined based on the camera pose.

[0135] Furthermore, the 3D reconstruction module 40 is further configured to use the camera posture as Nerf input and combine it with a voxel skipping strategy to accelerate Nerf optimization to obtain a 3D reconstruction result.

[0136] Furthermore, the initialization module 30 is further configured to determine the gyroscope bias, accelerometer bias, scale, and gravity acceleration based on the key frame and the pre-integration result;

[0137] A velocity is determined based on the gyroscope bias, the accelerometer bias, and the gravitational acceleration.

[0138] Furthermore, the acquisition module 10 is further used to acquire a planar image, perform feature extraction on the planar image, and obtain an ORB feature;

[0139] Determine the descriptor distance between the feature point and the feature point to be matched based on the ORB feature;

[0140] A key frame is selected from the planar image according to the descriptor distance.

[0141] Furthermore, the acquisition module 10 is further configured to determine the number of feature point matches based on the descriptor distance;

[0142] A key frame is selected from the planar image based on the number of feature point matches and a preset key frame selection strategy.

[0143] Other embodiments or specific implementations of the three-dimensional reconstruction device of the present invention can refer to the above-mentioned method embodiments and will not be described in detail here.

[0144] In addition, an embodiment of the present invention further provides a storage medium on which a three-dimensional reconstruction program is stored. When the three-dimensional reconstruction program is executed by a processor, the steps of the three-dimensional reconstruction method described above are implemented.

[0145] In addition, an embodiment of the present invention further provides a computer program product, including a three-dimensional reconstruction program, which implements the steps of the three-dimensional reconstruction method described above when executed by a processor.

[0146] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-mentioned three-dimensional reconstruction method, and will not be repeated here.

[0147] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0148] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0150] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method, characterized in that: The three-dimensional reconstruction method comprises the following steps: Select key frames from the acquired planar image; Acquire IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result; Performing visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result; A camera pose is determined according to the key frame and the visual-inertial initialization result, and a three-dimensional reconstruction result is determined based on the camera pose.

2. The three-dimensional reconstruction method according to claim 1, wherein: The step of determining the camera pose according to the key frame and the visual-inertial initialization result, and determining the three-dimensional reconstruction result based on the camera pose includes: determining a visual reprojection error based on the keyframe; Determine an IMU pre-integration residual based on the key frame and the visual-inertial initialization result; Determine the camera pose based on the visual reprojection error and the IMU pre-integration residual; A three-dimensional reconstruction result is determined based on the camera pose.

3. The three-dimensional reconstruction method according to claim 2, wherein: The step of determining a three-dimensional reconstruction result based on the camera posture comprises: The camera pose is used as the input of Nerf and combined with the skip voxel strategy to accelerate Nerf optimization to obtain a three-dimensional reconstruction result.

4. The three-dimensional reconstruction method according to any one of claims 1 to 3, wherein: The visual inertial initialization results include gyroscope bias, accelerometer bias, gravitational acceleration, scale, and velocity; The step of performing visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result includes: Determine a gyroscope bias, an accelerometer bias, a scale, and a gravitational acceleration according to the key frame and the pre-integration result; A velocity is determined based on the gyroscope bias, the accelerometer bias, and the gravitational acceleration.

5. The three-dimensional reconstruction method according to any one of claims 1 to 3, wherein: The step of selecting key frames from the acquired planar image includes: Collecting a planar image, performing feature extraction on the planar image, and obtaining an ORB feature; Determine the descriptor distance between the feature point and the feature point to be matched based on the ORB feature; A key frame is selected from the planar image according to the descriptor distance.

6. The three-dimensional reconstruction method according to claim 5, wherein: The step of selecting a key frame from the planar image according to the descriptor distance comprises: Determining the number of feature point matches based on the descriptor distance; A key frame is selected from the planar image based on the number of feature point matches and a preset key frame selection strategy.

7. A three-dimensional reconstruction device, characterized in that: The three-dimensional reconstruction device comprises: An acquisition module, used for selecting key frames from the acquired planar images; A pre-integration module is used to obtain IMU data, and pre-integrate the IMU data based on the key frame to obtain a pre-integration result; an initialization module, configured to perform visual-inertial initialization according to the pre-integration result and the key frame to obtain a visual-inertial initialization result; A three-dimensional reconstruction module is used to determine a camera pose according to the key frame and the visual-inertial initialization result, and to determine a three-dimensional reconstruction result based on the camera pose.

8. A three-dimensional reconstruction device, characterized in that: The device includes: a memory, a processor, and a three-dimensional reconstruction program stored in the memory and executable on the processor, wherein the three-dimensional reconstruction program is configured to implement the steps of the three-dimensional reconstruction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a three-dimensional reconstruction program, which, when executed by a processor, implements the steps of the three-dimensional reconstruction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a three-dimensional reconstruction program, and when the three-dimensional reconstruction program is executed by a processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 6 are implemented.

Images