A 3D scene reconstruction method and system based on adaptive keyframes

Abstract

The present disclosure provides a 3D scene reconstruction method and system based on adaptive keyframes, which relates to the field of computer vision technology. The method includes: obtaining multiple consecutive frames of images and inertial measurement unit data of the current scene; obtaining camera trajectory data and bias data of each inertial measurement component based on the inertial measurement unit data; performing target feature extraction and target tracking on each frame of the multiple frames to obtain feature point data of the target to be tracked; determining keyframes based on the scene complexity index and system resource utilization index of the current scene; performing visual-inertial joint optimization based on all feature point data, camera trajectory data, and bias data to generate point cloud data containing each feature point; generating a 3D Gaussian distribution of the current scene based on the point cloud data and keyframes, and rendering the distribution to obtain a first 3D rendered image of the current scene. The method can perform real-time 3D scene reconstruction in complex and changeable application scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a 3D scene reconstruction method and system based on adaptive keyframes. Background Art

[0002] In recent years, 3D scene reconstruction technology and simultaneous localization and mapping (SLAM) technologies have made significant progress in fields such as computer vision, augmented reality, and robotic navigation. Existing SLAM systems primarily rely on feature point extraction and matching, such as the simultaneous localization and mapping based on a fast feature point extraction and description algorithm (ORB-SLAM) and the monocular visual-inertial state estimator (VINS-Mono). While these systems offer excellent positioning accuracy, they struggle to produce high-quality, dense 3D reconstructions.

[0003] To address the limitations of sparse reconstruction, a series of direct methods have been proposed, such as monocular simultaneous localization and mapping (LSD-SLAM) and sparse direct visual odometry (DSO). These methods directly utilize pixel information in the image for optimization, producing semi-dense reconstructions. However, they suffer from low computational efficiency when processing large-scale scenes and are overly sensitive to changes in illumination.

[0004] Neural implicit representation methods, such as NeRF (Neural Radiance Fields), can generate high-quality rendering results from new perspectives, but the training and rendering processes are computationally expensive, making it difficult to achieve real-time performance. Currently, combining efficient neural rendering technology with real-time SLAM systems faces the following major challenges: First, keyframe selection. Existing SLAM systems select keyframes based on geometric information, but fail to fully consider scene complexity and rendering requirements. This results in too many keyframes in complex scenes and insufficient keyframes in simple scenes, affecting system performance and reconstruction quality. Second, computational resource allocation. Existing SLAM systems devote a large amount of computational resources to feature extraction and matching in the SLAM front-end, while ignoring the resource requirements of back-end optimization and rendering processes, resulting in uneven system performance. Third, there is a trade-off between rendering quality and real-time performance. High-quality 3D rendering usually requires a large amount of computational resources, which conflicts with the real-time requirements of SLAM systems, making it difficult to meet real-time requirements while ensuring rendering quality. Fourth, system robustness. In complex environments with dynamic and drastic lighting changes, the stability and accuracy of existing SLAM systems are often difficult to guarantee. Summary of the Invention

[0005] The main purpose of the present disclosure is to provide a 3D scene reconstruction method and system based on adaptive key frames to solve the technical problem that the related technology is unable to adaptively adjust key frames according to scene complexity and hardware conditions, resulting in limited real-time 3D scene reconstruction effects.

[0006] To achieve the above objectives, the first aspect of the present disclosure provides a 3D scene reconstruction method based on adaptive keyframes, comprising:

[0007] Acquire a plurality of consecutive frames of images of a current scene and inertial measurement unit data, wherein the inertial measurement unit is disposed inside a camera that captures the images and is equipped with a plurality of inertial measurement components;

[0008] Performing target feature extraction and target tracking on each frame of the multiple frames to obtain feature point data of the target to be tracked;

[0009] Acquire camera trajectory data and bias data of each of the inertial measurement components based on the inertial measurement unit data;

[0010] Determining a key frame based on a scene complexity index and a system resource utilization index of the current scene;

[0011] Performing visual-inertial joint optimization based on all of the feature point data, the camera trajectory data, and the bias data to generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate visual reprojection errors of the point cloud data by correcting the feature point data, the camera trajectory data, and the bias data; and

[0012] A 3D Gaussian distribution of the current scene is generated based on the point cloud data and the key frame, and the 3D Gaussian distribution is rendered to obtain a first 3D rendered image of the current scene.

[0013] Furthermore, the method further comprises:

[0014] Identifying similar scenes in historical scenes that match the current scene;

[0015] Obtaining the camera pose calculation model, and performing global consistency optimization on the camera pose calculation model based on the camera trajectory data of the similar scene and the camera trajectory data of the current scene;

[0016] Acquiring optimized camera trajectory data of the current scene based on the optimized camera pose calculation model; and

[0017] The visual-inertial joint optimization is performed based on the optimized camera trajectory data to generate a second 3D rendered image of the current scene.

[0018] Furthermore, the method further comprises:

[0019] generating a first sub-map based on the point cloud data of the current scene;

[0020] Identifying a second scene in the historical scene that forms a global map closed loop with the current scene;

[0021] Acquiring point cloud data of the second scene, and generating a second submap based on the point cloud data of the second scene;

[0022] Merge the first submap and the second submap to generate a global map.

[0023] Furthermore, determining the key frame based on the scene complexity index and the system resource utilization index of the current scene includes:

[0024] For each frame in the multi-frame image as the current frame, obtain the number of feature points N of the current frame f , feature point distribution variance V f , texture complexity T f ;

[0025] Based on the number of feature points N f , the feature point distribution variance V f , the texture complexity T f Obtaining a scene complexity index C of the current frame;

[0026] Determine the nearest key frame, and obtain the relative displacement A, overlap B, and information entropy H of the current frame relative to the nearest key frame;

[0027] Obtaining a system resource utilization index R, and dynamically adjusting a selection threshold T based on the scenario complexity index C and the system resource utilization index R;

[0028] Based on the information entropy H, the selection threshold T, the relative displacement A, and the overlap degree B, it is determined whether the current frame is the key frame.

[0029] Furthermore, the scene complexity index C, the information entropy H, and the selection threshold T are respectively configured as:

[0030] C=w1×N f +w2×V f +w3×T f

[0031]

[0032] T=T0×(1+α×C)×(1-β×R)

[0033] Where w1, w2, w3 are weight coefficients, p(x i ) is the feature point in the image area x iThe probability distribution in , T0 is the basic threshold, α is the adjustment coefficient, β is the correction coefficient; and,

[0034] The key frame is configured to simultaneously satisfy: the information entropy H exceeds the selection threshold T, and the relative displacement A exceeds a preset maximum allowable value, and the overlap degree B is lower than a preset overlap threshold.

[0035] Furthermore, the target feature extraction and target tracking are performed on each frame of the multiple frames to obtain feature point data of the target to be tracked, including:

[0036] For each frame of the multiple image frames as a current frame, perform feature tracking processing on the current frame using a KLT sparse optical flow algorithm to determine a first type of feature point of the target to be tracked, and obtain feature point data of the first type of feature point, wherein the feature point data is configured to include feature point position data and feature point grayscale data;

[0037] Performing a corner feature search on the current frame using a FAST corner detector to determine a second type of feature point of the target to be tracked, and obtaining feature point data of the second type of feature point;

[0038] Obtaining a minimum pixel interval between adjacent feature points in a feature point set consisting of the first type of feature points and the second type of feature points;

[0039] performing 2D feature dedistortion processing on the elements of the feature point set of the target to be tracked based on the minimum pixel interval, and projecting feature point data corresponding to the elements of the feature point set after the processing onto a unit sphere;

[0040] The projected feature point data is subjected to outlier elimination using a random sampling consistency algorithm to obtain final feature point data of the target to be tracked.

[0041] Furthermore, the camera trajectory data is configured to include camera position data, camera velocity data, and camera attitude data, and the inertial measurement unit is configured to include an accelerometer and a gyroscope; and the acquiring of the camera trajectory data and the bias data of each of the inertial measurement units based on the inertial measurement unit data includes:

[0042] acquiring camera trajectory data based on the inertial measurement unit data;

[0043] Acquire a measurement model of the inertial measurement unit, and perform pre-integration processing on the inertial measurement unit data based on the measurement model within a time interval between two consecutive image frames to obtain a relative displacement, relative velocity, and relative rotation angle of the camera within the time interval;

[0044] determining a covariance propagation matrix of the measurement model based on the relative displacement, relative velocity, and relative rotation angle;

[0045] Optimizing the parameters of the measurement model based on the covariance propagation matrix so that the bias data of the inertial measurement assembly converge to respective set threshold ranges, and ending the optimization;

[0046] The bias data of each of the inertial measurement components is obtained based on the measurement model after parameter optimization.

[0047] Furthermore, the method further includes: upon identifying that the bias data of any of the inertial measurement components exceeds the corresponding set threshold, calibrating the relative displacement, the relative velocity, and the relative rotation angle based on the measurement model, thereby calibrating the camera trajectory data; and,

[0048] The performing visual-inertial joint optimization based on all the feature point data, the camera trajectory data, and the bias data and generating point cloud data containing each feature point of the target to be tracked includes:

[0049] For each feature point of the target to be tracked as a current feature point, obtaining state data corresponding to the current feature point in each frame of the multiple frames of images, wherein the state data is configured to include the feature point position data, the feature point grayscale data, the camera position data, the camera speed data, the camera attitude data, and bias data of each of the inertial measurement units;

[0050] Obtaining an inverse depth parameter corresponding to the current feature point in each frame of the multiple frames of the image, and generating a state vector X of the current feature point based on the state data and the inverse depth parameter;

[0051] Inputting the state vector X into a visual reprojection error model to obtain a visual reprojection error;

[0052] Inputting the state vector X into a pre-integration residual model to obtain a pre-integration residual;

[0053] Performing nonlinear optimization on the state vector X based on the visual reprojection error and the pre-integration residual to obtain optimized state data corresponding to the current feature point in each frame of the multiple images;

[0054] The optimized state data corresponding to each feature point of the target to be tracked is obtained, and the point cloud data is generated based on the feature point position data and the feature point grayscale data in the optimized state data.

[0055] Furthermore, generating a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and rendering the 3D Gaussian distribution to obtain a first 3D rendered image of the current scene includes:

[0056] Representing each of the feature points in the point cloud data by the 3D Gaussian distribution;

[0057] Initializing 3D Gaussian points in the 3D Gaussian distribution, wherein the initialization is configured to include: projecting the 3D Gaussian distribution onto a 2D image plane, obtaining a rotation matrix R and a scaling matrix S during the projection process, and parameterizing a covariance matrix Σ of the 3D Gaussian distribution based on the rotation matrix R and the scaling matrix S;

[0058] Optimizing parameters of the 3D Gaussian distribution based on the key frame and the camera position, the camera speed, and the camera pose corresponding to the key frame so as to minimize a projection error of the 3D Gaussian distribution;

[0059] Adaptively adjusting the number N of the 3D Gaussian points in the 3D Gaussian distribution based on a scene complexity index C, wherein the number N is configured as follows:

[0060] N=N0×(1+α×C)

[0061] In the formula, N0 is the basic number, α is the adjustment factor;

[0062] Performing differentiable rasterization processing on the 3D Gaussian distribution, and rendering the processed 3D Gaussian distribution to generate a first 3D rendered image of the current scene.

[0063] A second aspect of the present disclosure provides a 3D scene reconstruction system based on adaptive key frames, comprising:

[0064] a data acquisition unit configured to acquire a plurality of consecutive frames of images of a current scene and inertial measurement unit data, wherein the inertial measurement unit is disposed inside a camera that captures the images and is equipped with a plurality of inertial measurement components;

[0065] A first pre-processing unit is configured to perform target feature extraction and target tracking on each frame of the multiple frames of images to obtain feature point data of the target to be tracked;

[0066] a second pre-processing unit, configured to acquire camera trajectory data and bias data of each of the inertial measurement components based on the inertial measurement unit data;

[0067] A third pre-processing unit is configured to determine a key frame based on a scene complexity index and a system resource utilization index of the current scene; and

[0068] a data optimization unit configured to perform visual-inertial joint optimization based on all of the feature point data, the camera trajectory data, and the bias data and generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate visual reprojection errors of the point cloud data by correcting the feature point data, the camera trajectory data, and the bias data;

[0069] The image generation unit is configured to generate a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and render the 3D Gaussian distribution to obtain a first 3D rendered image of the current scene.

[0070] In the 3D scene reconstruction method provided in the embodiment of the present disclosure, an adaptive key frame selection strategy is adopted. By determining the key frame based on the scene complexity index and system resource utilization index of the current scene, and reconstructing the 3D scene based on the key frame, the purpose of improving the adaptability of the method is achieved, thereby achieving the goal of automatically adjusting the 3D Gaussian distribution according to the scene complexity and hardware conditions, and maintaining efficient operation in different environments. This adaptability enables excellent rendering results in complex and changeable actual application scenarios, thereby solving the technical problem of the related technology that the real-time 3D scene reconstruction effect is limited due to the inability to adaptively adjust the key frame according to the scene complexity and hardware conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0072] Figure 1 A schematic diagram of a method flow chart provided in an embodiment of the present disclosure;

[0073] Figure 2 A software function diagram designed based on the method provided by the embodiment of the present disclosure;

[0074] Figure 3 A schematic diagram of the rendering process provided by an embodiment of the present disclosure;

[0075] Figure 4 A schematic diagram of the process flow of the adaptive key frame selection strategy provided by an embodiment of the present disclosure;

[0076] Figure 5 A system block diagram provided for an embodiment of the present disclosure.

[0077] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0079] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate for the embodiments of the present disclosure described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0080] In this disclosure, terms such as "upper," "lower," "left," "right," "front," and "rear" indicate positions or locations based on those shown in the accompanying drawings. These terms are primarily intended to better describe this disclosure and its embodiments and are not intended to limit the devices, elements, or components indicated to having a specific orientation, or to being constructed or operated in a specific orientation.

[0081] Furthermore, some of the above terms may be used to convey other meanings besides orientation or positional relationships. For example, the term "on" may also be used to convey a dependency or connection relationship in certain circumstances. Those skilled in the art will understand the specific meanings of these terms in this disclosure based on the specific circumstances.

[0082] Furthermore, the terms "disposed," "provided with," "connected," and "connected" should be interpreted broadly. For example, they can refer to fixed connections, removable connections, or integral structures; mechanical connections or electrical connections; direct connections, indirect connections through an intermediary, or internal communication between two devices, elements, or components. Those skilled in the art will understand the specific meanings of these terms in this disclosure based on specific circumstances.

[0083] 3D Gaussian Splatting, or 3DGS (3D Gaussian Splatting), is a rasterization technique for real-time radiance field rendering that describes a 3D Gaussian distribution. It combines high-quality image representation with real-time rendering performance. By leveraging three core elements: applying a 3D Gaussian distribution, optimizing anisotropic covariance for accurate scene description, and developing a fast visibility-aware rendering algorithm, 3DGS has demonstrated superior visual quality and real-time rendering performance across multiple datasets.

[0084] The term "Inertial Measurement Unit," or IMU for short, typically consists of components such as accelerometers and gyroscopes. Some IMUs also integrate magnetometers.

[0085] The term "inertial measurement unit data," or IMU data for short, covers a variety of information including accelerometer data and gyroscope data.

[0086] The term "inertial measurement unit bias," or IMU bias for short, refers to the deviation between the actual IMU output and its theoretical value under specific conditions. This bias is primarily caused by a variety of factors, including manufacturing errors, environmental factors, and wear and tear from long-term use.

[0087] The KLT sparse optical flow algorithm is a classic optical flow estimation method specifically designed for tracking the position of specific points in an image sequence. The algorithm selects appropriate feature points for tracking. It first uses corner detection techniques to obtain a sparse set of feature points from the previous image. These feature points are then matched with the next image to identify sets of points with similar features and infer the optical flow of the sparse point set between the two images.

[0088] The term "FAST corner detector" refers to a feature point detection algorithm widely used in the field of computer vision. The full name of FAST is Features from Accelerated Segment Test. The basic principle of the FAST corner detector is to determine whether a pixel is a corner by comparing the brightness values ​​of a pixel and its surrounding pixels. Specifically, a circle with a radius of 3 is drawn around the target pixel, with 16 pixels on the circumference. If among these 16 pixels, the brightness values ​​of N consecutive pixels are brighter or darker than the brightness value of the center pixel of the circle plus or minus a threshold, then the center pixel of the circle is determined to be a corner.

[0089] The term "Random Sampling Consensus Algorithm," or RANSAC (full name RANdom SAmple Consensus), is an iterative algorithm widely used in computer vision and image processing. It determines the optimal model parameters by repeatedly randomly selecting subsets of data and verifying their consistency.

[0090] The term "texture complexity" describes the complexity of texture features in an image. In technical fields such as image processing and video encoding and decoding, texture complexity is a core concept, relating to aspects such as the regularity, variation, and information content of image textures. Texture complexity can typically be quantified using techniques such as the Gray Level Co-occurrence Matrix (GLCM) to extract texture characteristics such as mean, variance, and entropy.

[0091] SLAM (Simultaneous Localization and Mapping) is a key technology in robotics, autonomous driving, augmented reality, and other fields. It enables machines to navigate autonomously in unknown environments and build maps of their surroundings.

[0092] NeRF (Neural Radiance Fields) is an advanced view synthesis method. Its core concept is to use volume rendering technology to simulate the interaction between light and the scene through a deep neural network, thereby achieving high-quality 3D reconstruction and view synthesis.

[0093] The term "Levenberg-Marquardt algorithm," or LM for short (its full English name), is a widely used nonlinear least squares algorithm. It combines the advantages of the gradient method and the Newton method. When the value of λ in the algorithm is small, the step size is close to that of the Newton method; when the value of λ is large, the step size is close to that of the gradient descent method. The LM algorithm is insensitive to over-parameterization, can effectively handle redundant parameters, and significantly reduces the risk of the cost function falling into a local minimum. Essentially, when solving a nonlinear least squares problem, it uses the sum of squared errors as the cost function and searches for a set of parameters that minimizes this cost function. At each iteration, the parameter values ​​are updated by calculating the first and second derivatives of the cost function. To prevent overfitting, regularization methods are also used to deal with noise and outliers.

[0094] The "Bag of Words" model is a commonly used text representation method in natural language processing and information retrieval. It treats a text as a collection of words, counts the occurrence of each word, and uses this statistics to construct a feature vector for the text. The core idea of ​​this method is to treat words in a text as independent features, ignoring the order and grammatical structure of the words.

[0095] The term "3D Gaussian Splatting" refers to a technique for converting discrete data points or voxels into a continuous surface or volume. In 3D Gaussian Splatting, each data point or voxel is treated as a Gaussian kernel whose values ​​exhibit a Gaussian distribution in space. By superimposing these Gaussian kernels, a continuous surface or volume can be generated. This superposition process is typically achieved by interpolating the values ​​of each data point or voxel to obtain a smooth and continuous distribution in space.

[0096] The term "GPD radix sort algorithm", abbreviated as NeRF (the full name in English is GPD radix sort), its basic idea is to start from the lowest bit, sort according to the value of the current bit to get a new sequence, and re-sort the new sequence according to the value of the higher bit, and so on until the highest bit. The resulting sequence is an ordered sequence. The following example briefly introduces radix sort.

[0097] Pre-integration processing is a key technique used to reduce computational overhead in visual-inertial odometry systems. Its primary purpose is to process IMU data to calculate the relative pose transformation between two keyframes, assuming constant sensor bias. The effects of white noise and bias are considered and stripped away, while the bias is updated via first-order derivatives. This eliminates the need to recalculate the pre-integral even with bias changes, improving optimization efficiency.

[0098] The term "inverse depth parameter," or Inverse Depth Parametrization, is the inverse of depth. In computer vision, depth typically refers to the distance from a camera's lens or sensor to the surface of an object—that is, the Z-axis distance in a point's 3D coordinates. Inverse depth is the inverse of this distance.

[0099] The term "visual reprojection error model", which refers to the computational model of visual reprojection error, is a key concept in computer vision and is mainly used to evaluate and optimize homography matrices and projection matrices. By minimizing the visual reprojection error, the accuracy of pose estimation and structure from motion can be effectively improved. Visual reprojection error refers to the difference between the observed value and the actual value of a point in three-dimensional space after projecting it onto a two-dimensional plane. Specifically, the reprojection error model is the error obtained by comparing the calculated pixel coordinates (the observed projected position) with the position obtained by projecting the three-dimensional point according to the current estimated pose.

[0100] The term "2D feature dedistortion processing" refers to calibrating and dedistorting an image so that the original image can be presented more realistically and accurately.

[0101] The term "pre-integration" refers to using continuous IMU measurements to estimate the camera's motion between two frames. This method reduces reliance on high-frequency measurements, thereby improving the stability of the algorithm.

[0102] The term "pre-integrated residual model" is an important component of visual-inertial odometry (VIO), mainly used to improve positioning accuracy and robustness. By fusing visual and inertial data, the pre-integrated residual model can effectively reduce the dependence on high-frequency measurements, thereby improving overall performance.

[0103] The term "nonlinear optimization" refers to the mathematical theory and method of finding the optimal solution (usually the minimum value) when the objective function or constraints are nonlinear functions.

[0104] The term "global map loop closure" is a key step in simultaneous localization and map construction, particularly in the fields of autonomous driving and robotic navigation. It involves using optimization algorithms to eliminate drift in the poses of keyframes and landmarks within the closed loop, thereby improving the accuracy and robustness of the system.

[0105] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0106] The related technology has a technical problem that the real-time 3D scene reconstruction effect is limited due to the inability to adaptively adjust key frames according to scene complexity and hardware conditions.

[0107] In order to solve the above technical problems, the present disclosure provides a 3D scene reconstruction method based on adaptive key frames, such as Figure 1 As shown, the method includes the following steps S11 to S16.

[0108] Step S11: Acquire multiple consecutive frames of images of the current scene and inertial measurement unit data (IMU data for short), wherein the inertial measurement unit is set inside the camera that captures the image and is equipped with multiple inertial measurement components.

[0109] Step S12: performing target feature extraction and target tracking on each frame of the multiple image frames to obtain feature point data of the target to be tracked.

[0110] Step S13: Acquire camera trajectory data and bias data of each inertial measurement component based on the inertial measurement unit data.

[0111] Step S14: determining a key frame based on the scene complexity index and the system resource utilization index of the current scene.

[0112] Step S15: Perform visual-inertial joint optimization based on all feature point data, camera trajectory data, and bias data to generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate the visual reprojection error of the point cloud data by correcting the feature point data, camera trajectory data, and bias data.

[0113] Step S16: Generate a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and render the 3D Gaussian distribution to obtain a first 3D rendered image of the current scene.

[0114] Among them, step S11 is used to complete the data acquisition function, and different data acquisition methods can be used. Step S12 is used to complete the visual feature extraction and tracking function, and different visual feature extraction and tracking algorithms can be used. Step S13 is used to complete the IMU data processing function, and different IMU data processing algorithms can be used. Step S14 is used to complete the adaptive key frame selection function, and the key frame is determined based on the scene complexity index and system resource utilization index of the current scene. Step S15 is used to complete the visual-inertial joint optimization function, and different visual-inertial joint optimization algorithms can be used. Step S16 is used to complete the 3D Gaussian representation and rendering function, and different rendering algorithms can be used. The embodiments of the present disclosure do not impose any restrictions on this.

[0115] Some embodiments of the present disclosure also provide a 3D scene reconstruction system, a storage medium, and an electronic device corresponding to the above method.

[0116] The 3D scene reconstruction method provided by the embodiments of the present disclosure is applicable to any existing 3D scene reconstruction scenario. For example, in simulating multi-map scenes in a large-scale environment, the 3D scene reconstruction method provided by the embodiments of the present disclosure can maintain high efficiency in various environments compared to existing technologies. This adaptability makes the 3D scene reconstruction method particularly effective in complex and diverse practical application scenarios.

[0117] exist Figure 1 Based on the method, preferably, the camera trajectory data is configured to include camera position data, camera velocity data, and camera attitude data. The inertial measurement component is configured to include an accelerometer and a gyroscope. The feature point data is configured to include feature point position data and feature point grayscale data.

[0118] exist Figure 1 Based on the method shown, preferably, the method further includes the following steps S21 to S23 to complete the function of performing global consistency optimization processing on the current scene based on the historical scene.

[0119] Step S21: Identify similar scenes in historical scenes that match the current scene.

[0120] Step S22: Obtain a camera pose calculation model, and perform global consistency optimization on the camera pose calculation model based on camera trajectory data of similar scenes and camera trajectory data of the current scene.

[0121] Step S23: Obtain optimized camera trajectory data of the current scene based on the optimized camera pose calculation model.

[0122] As a preferred embodiment, step S21 uses the Bag of Words model to perform scene recognition and quickly detect potential closed loops in historical scenes. This method can efficiently identify similar scenes and provide closed loop detection support. The score is calculated as follows:

[0123] score=∑tf(w i )·idf(w i ) (1)

[0124] Where, tf(w i ) is the word frequency, idf(w i ) is the inverse document frequency, w i is the feature weight.

[0125] To improve the accuracy of loop closure detection, the RANSAC algorithm is used for geometric verification, relative pose estimation, and filtering out incorrect loop closure candidates. The estimation formula is as follows:

[0126] [R,t]=argmin∑||x′ i -(Rx i +tt)|| 2 (2) Where R is the rotation matrix, tt is the translation vector, and x′ i is the three-dimensional coordinate of the matching point, || || 2 is the square norm.

[0127] Step S22 enhances the observability of the visual-inertial system by constructing a pose graph and performing a global optimization of the four degrees of freedom (x, y, z, yaw), where (x, y, z) are the three-dimensional coordinates of the feature points and yaw is the yaw angle, thereby improving global consistency. The global consistency optimization is configured to include:

[0128] min{∑||r i,j || 2 +∑ρ(||r i,j || 2 )}(3)

[0129] Where r i,j represents the residual of the relative pose constraint, and ρ(·) is the robust kernel function.

[0130] Step S24: performing visual-inertial joint optimization based on the optimized camera trajectory data, thereby generating a second 3D rendered image of the current scene.

[0131] By combining steps S11-S16 and S21-S24, we implement a global feature point update and Gaussian point redistribution strategy to ensure consistency between the current scene image and the optimized pose. This process improves the long-term operational stability of the system by maintaining its global consistency.

[0132] exist Figure 1 Based on the method shown, preferably, the method further includes the following steps S31 to S34.

[0133] Step S31: Generate a first sub-map based on the point cloud data of the current scene.

[0134] Step S32: Identify a second scene in the historical scene that forms a global map closed loop with the current scene.

[0135] Step S33: Acquire point cloud data of the second scene, and generate a second sub-map based on the point cloud data of the second scene.

[0136] Step S34: Merge the first submap and the second submap to generate a global map.

[0137] By combining steps S11-S16 and S31-S34, we implement a global map update and Gaussian point redistribution strategy to ensure map consistency with the optimized pose. This also implements a map merging function, supporting the fusion of multiple submaps. This allows for handling multi-map systems in large-scale environments and supports advanced application scenarios such as collaborative mapping.

[0138] In some embodiments of the present disclosure, for the above step S12, in order to achieve good visual feature extraction and tracking functions, preferably, step S12 further includes the following sub-steps S121 to S125.

[0139] Sub-step S121: For each frame in the multiple-frame image as the current frame, feature tracking processing is performed on the current frame through the KLT sparse optical flow algorithm to determine the first type of feature points of the target to be tracked, and feature point data of the first type of feature points is obtained, wherein the feature point data is configured to include feature point position data and feature point grayscale data.

[0140] Specifically, for each new image, the KLT sparse optical flow algorithm is used to track existing features. The algorithm is based on minimizing the following objective function E:

[0141] E=∫∫[I(x,y,t)-I(x+Δx,y+Δy,t+Δt)] 2 dxdy (4)

[0142] Where I(x,y,t) is the grayscale value of the image at position (x,y) at time t, and I(x+Δx,y+Δy,t+Δt) is the grayscale value of the image at position (x+Δx,y+Δy) at time t+Δt.

[0143] Sub-step S122: performing a corner feature search on the current frame using the FAST corner detector to determine a second type of feature point of the target to be tracked, and obtaining feature point data of the second type of feature point.

[0144] Specifically, the FAST corner detector is used to detect new corner features to maintain 100-300 feature points per frame. The judgment criteria are: |I p –I c |>th. Among them, I p is the strength of the candidate point, I c is the intensity of the center pixel, and th is the threshold.

[0145] Sub-step S123: Obtaining the minimum pixel interval between adjacent feature points in the feature point set consisting of the first type feature points and the second type feature points.

[0146] Sub-step S124: performing 2D feature dedistortion processing on the elements of the feature point set based on the minimum pixel interval, and projecting the feature point data corresponding to the elements of the feature point set after the processing onto the unit sphere.

[0147] Specifically, a uniform distribution is enforced, a minimum pixel spacing between adjacent feature points is set, and the 2D features are dedistorted and then projected onto the unit sphere:

[0148]

[0149] Where image_area is the image area, desired_feature_count is the desired number of feature points, and d is the distance value, which is configured to be equal to the minimum pixel interval used when performing uniform distribution of feature points.

[0150] Sub-step S125: removing outliers from the projected feature point data using a random sampling consensus algorithm (RANSAC algorithm) to obtain the final feature point data of the target to be tracked.

[0151] Specifically, the RANSAC algorithm is used to remove outliers, and the basic matrix model is adopted:

[0152] x′ T Fx=0 (6)

[0153] Where F is the basic matrix, x and x′ are adjacent feature points.

[0154] In some embodiments of the present disclosure, for step S13, in order to achieve better IMU data processing function, preferably, step S13 further includes the following sub-steps S131 to S135.

[0155] Sub-step S131: Acquire camera trajectory data based on inertial measurement unit data.

[0156] Sub-step S132: Obtain a measurement model of the inertial measurement unit, and pre-integrate the inertial measurement unit data based on the measurement model within the time interval between two consecutive image frames to obtain the relative displacement, relative velocity, and relative rotation angle of the camera within the time interval.

[0157] Specifically, the measurement model is configured to include:

[0158]

[0159] Where, is the actual value of the accelerometer, a t is the accelerometer measurement, is the bias data of the accelerometer, is the rotation matrix, g w is the acceleration due to gravity, n a is the accelerometer noise term, is the actual value of the gyroscope, ω t is the measurement value of the gyroscope, is the bias data of the gyroscope, n w is the noise term of the gyroscope.

[0160] The relative displacement, relative velocity and relative rotation angle are configured as follows:

[0161]

[0162] Where, is the relative displacement, is the relative speed, is the relative rotation angle, is the rotation matrix, Ω is the angular velocity, is the posture in the local coordinate system, t is the time, t k , t k+1 are the moments corresponding to two consecutive image frames.

[0163] Sub-step S134: Determine the covariance propagation matrix of the measurement model based on the relative displacement, relative velocity and relative rotation angle.

[0164] Specifically, the covariance propagation matrix of the measurement model is configured to include:

[0165]

[0166] Where, is the covariance matrix at time t+δt, δt is the time interval, I is the identity matrix, is the covariance matrix at time t, F t is the system matrix, G t is the noise driving matrix, Q t is the noise covariance matrix.

[0167] in, Obtained through iterative calculation, describing the uncertainty of state estimation. t The linearized model describing the state transition is obtained by taking partial derivatives of the motion equations in the measurement model. t It is obtained by taking the partial derivative of the motion equation in the measurement model with respect to the noise term, describing the influence of system noise on the state variables. t Obtained through sensor specifications or experimental calibration, it describes the statistical characteristics of system noise.

[0168] Sub-step S135: Optimizing the parameters of the measurement model based on the covariance propagation matrix so that the bias data of the inertial measurement unit converge to their respective set threshold ranges, and then ending the optimization.

[0169] Sub-step S136: Obtain bias data of each inertial measurement unit based on the measurement model after parameter optimization.

[0170] As a preferred embodiment, the method further includes: identifying that the bias data of any inertial measurement component exceeds a corresponding set threshold, calibrating the relative displacement, relative velocity and relative rotation angle based on the measurement model, and then calibrating the camera trajectory data.

[0171] In some embodiments of the present disclosure, for step S14, in order to achieve a better adaptive key frame selection function, preferably, step S14 further includes the following sub-steps S141 to S145.

[0172] Sub-step S141: For each frame in the multi-frame image as the current frame, obtain the number of feature points N of the current frame f , feature point distribution variance V f , texture complexity T f .

[0173] Sub-step S142: Based on the number of feature points N f , feature point distribution variance V f , texture complexity T f Get the scene complexity index C of the current frame.

[0174] The scene complexity index C can be configured as:

[0175] C=w1×N f +w2×V f +w3×T f (10)

[0176] Where w1, w2, and w3 are weight coefficients.

[0177] The above evaluation method can fully consider the geometric and texture characteristics of the scene.

[0178] Sub-step S143: Determine the nearest key frame, and obtain the relative displacement A, overlap B, and information entropy H of the current frame relative to the nearest key frame.

[0179] Specifically, the most recent key frame is a key frame with the smallest relative displacement from the current frame.

[0180] The information entropy H is configured as:

[0181] H=-∑p(x i )logp(x i ) (11)

[0182] In the formula, p(x i ) is the feature point in the image region x i The probability distribution in .

[0183] The calculation method of the relative displacement A and the overlap degree B is common knowledge in the art, and the embodiments of the present disclosure will not elaborate on or limit this.

[0184] Sub-step S144: Obtain the system resource utilization index R, and dynamically adjust the selection threshold T based on the scenario complexity index C and the system resource utilization index R.

[0185] Specifically, the threshold T is selected as:

[0186] T=T0×(1+α×C)×(1-β×R) (12)

[0187] Where T0 is the base threshold, α is the adjustment coefficient, and β is the correction coefficient. This mechanism can lower the selection criteria in complex scenarios and raise them in simple scenarios, while also taking system load into account.

[0188] Sub-step S145: Determine whether the current frame is a key frame based on the information entropy H, the selection threshold T, the relative displacement A, and the overlap degree B.

[0189] As a preferred embodiment, the key frame is configured to simultaneously satisfy: the information entropy H exceeds the selection threshold T, and the relative displacement A exceeds a preset maximum allowable value, and the overlap degree B is lower than a preset overlap threshold.

[0190] In some embodiments of the present disclosure, for step S15, in order to achieve a better visual-inertial joint optimization function, step S15 may further include the following sub-steps S151 to S15.

[0191] Sub-step S151: For each feature point of the target to be tracked as the current feature point, obtain the status data corresponding to the current feature point in each frame of the multi-frame image, wherein the status data is configured to include feature point position data, feature point grayscale data, camera position data, camera speed data, camera attitude data and bias data of each inertial measurement component.

[0192] Sub-step S152: Obtain the inverse depth parameter corresponding to the current feature point in each frame of the multi-frame image, and generate a state vector X of the current feature point based on the state data and the inverse depth parameter.

[0193] Specifically, the state vector X can be configured as:

[0194]

[0195] Where x i is the status data of the i-th frame, and n is the number of frames in the multi-frame image. is the camera's configuration state vector, which is used to describe the camera's current state, including parameters such as the camera's position, speed, and attitude.m The inverse depth parameter of the current feature point is used to describe the depth information of the feature point in the camera coordinate system. The inverse depth parameter can be used to more efficiently optimize and calculate the projection process of the visual feature point.

[0196] Sub-step S152 constructs a complete state vector containing the camera pose (camera position data, camera velocity data, camera attitude data), IMU bias, and feature point locations for joint optimization of the visual-inertial system. This state vector can be unified to represent all variables in the system, enabling accurate estimation of the camera trajectory and feature point locations.

[0197] Sub-step S153: inputting the state vector X into the visual reprojection error model to obtain the visual reprojection error.

[0198] Specifically, the visual reprojection error in the visual reprojection error model Configured to:

[0199]

[0200] Where π() is the projection function, is the camera pose transformation matrix, p l is the three-dimensional coordinate of the current feature point, It is the two-dimensional coordinate of the current feature point projected on the 2D image plane.

[0201] The camera pose transformation matrix is ​​used to fuse the pre-integrated information from the IMU with the visual information. By establishing a relationship between the pre-integrated measurement and the state vector, a tightly coupled optimization of the visual-inertial data is achieved.

[0202] The visual reprojection error is defined to measure the difference between the projection of a feature point on the image and the actual observation. The visual reprojection error is a core constraint in visual simultaneous localization and mapping (SLAM), used to correct the camera pose and feature point positions.

[0203] Sub-step S154: input the state vector X into the pre-integration residual model to obtain the pre-integration residual.

[0204] Specifically, the pre-integrated residuals in the pre-integrated residual model Configured to:

[0205]

[0206] Where δα is the error term of camera position, δβ is the error term of camera velocity, δθ is the error term of camera attitude, and δb a is the bias error of the accelerometer, δb w is the bias error of the gyroscope, is the parameter vector to be estimated.

[0207] To fully utilize the IMU data, a pre-integration residual is defined. This residual measures the consistency between the IMU pre-integration result and the state estimate. The introduction of this residual helps improve the smoothness of the trajectory estimation and the accuracy of the overall system.

[0208] Sub-step S155: performing nonlinear optimization on the state vector X based on the visual reprojection error and the pre-integration residual to obtain optimized state data corresponding to the current feature point in each frame of the multi-frame image.

[0209] Specifically, the above nonlinear optimization is configured as follows:

[0210]

[0211] Where r p is the prior residual, H p is the Jacobian matrix of the prior information, P is the inverse of the covariance propagation matrix, ρ(·) is the robust kernel function, is the Euclidean norm.

[0212] Sub-step S155 combines the various visual and IMU constraints by constructing a nonlinear optimization problem, with the goal of minimizing the weighted sum of all error terms. This optimization process not only considers the visual reprojection error and pre-integration residual, but also incorporates prior information, thereby achieving global optimization of the camera pose, IMU state, and feature point positions.

[0213] As a preferred embodiment, in order to effectively solve the constructed nonlinear optimization problem, the Levenberg-Marquardt algorithm can be used to solve it. This algorithm combines the fast convergence characteristics of the Gauss-Newton method and the stability of the gradient descent method, and can cope with the nonlinear problems in the SLAM system. The linearization formula of the optimization problem is:

[0214] (J T J+λI)δx=-J T r (17)

[0215] Where J is the Jacobian matrix, r is the residual vector, λ is the damping factor, and δx is the increment of the optimization variable. By iteratively solving this equation, the camera pose, IMU bias, and feature point positions can be gradually optimized.

[0216] Sub-step S156: obtaining optimized state data corresponding to each feature point of the target to be tracked, and generating point cloud data based on the feature point position data and feature point grayscale data in the optimized state data.

[0217] In some embodiments of the present disclosure, for step S16, in order to achieve better 3D Gaussian distribution and rendering effects, preferably, step S16 further includes the following sub-steps S161 to S165.

[0218] Sub-step S161: Represent each feature point in the point cloud data by a 3D Gaussian distribution.

[0219] Specifically, a 3D Gaussian distribution is used to represent each point in the scene. The Gaussian distribution of each point is described by the mean and covariance matrix, which can effectively capture the spatial uncertainty of the point. By using a 3D Gaussian distribution, the position of the point and its uncertainty can be accurately described in scene reconstruction. The definition of a 3D Gaussian distribution is as follows:

[0220]

[0221] Where μ is the mean of the Gaussian distribution and ∑ is the covariance matrix, which is used to describe the spatial distribution characteristics of the points.

[0222] Sub-step S162: Initialize the 3D Gaussian points in the 3D Gaussian distribution, wherein the initialization is configured to include: projecting the 3D Gaussian distribution onto the 2D image plane, obtaining the rotation matrix R and the scaling matrix S during the projection process, and parameterizing the covariance matrix ∑ of the 3D Gaussian distribution based on the rotation matrix R and the scaling matrix S.

[0223] Specifically, the relationship between the rotation matrix R, the scaling matrix S and the covariance matrix ∑ is established by the following formula:

[0224] ∑=RSS T R T (19)

[0225] The 3D Gaussian distribution is projected onto the 2D image plane to effectively analyze the 3D space. During the projection process, the internal and external parameters of the camera and the characteristics of the Gaussian distribution are taken into account, so that the information in the 3D space can be accurately converted to a 2D image. The expression for the 3D Gaussian distribution projection is as follows:

[0226] ∑′=JW∑W T J T (20)

[0227] Where J is the Jacobian matrix of the projection transformation, and W is the transformation matrix from the current scene to the camera.

[0228] In order to ensure the positive definiteness of the covariance matrix and the stability of the optimization process, the covariance matrix ∑ is parameterized using the scaling matrix S and the rotation matrix r, see formula (19). This parameterization method not only ensures the positive definiteness of the covariance matrix, but also provides an intuitive geometric interpretation.

[0229] Sub-step S163: Optimizing the parameters of the 3D Gaussian distribution based on the key frames and the camera positions, camera velocities, and camera postures corresponding to the key frames, so as to minimize the projection error of the 3D Gaussian distribution.

[0230] Specifically, the projection error is minimized by optimizing the parameters of the Gaussian distribution, including the position parameter μ, the scaling parameter s, and the rotation parameter q. The parameters of the Gaussian distribution are adjusted at multiple viewing angles to make the projection more consistent with the observation. The optimization objectives are as follows:

[0231]

[0232] Where π is the projection function, z is the actual observation, and G(μ,s,q) is the Gaussian distribution. The specific form of the Gaussian distribution is not detailed here. Ultimately, the parameters of the optimized 3D Gaussian distribution are obtained, including the position parameter μ, the scale parameter s, and the rotation parameter q.

[0233] Sub-step S164: Adaptively adjust the number N of 3D Gaussian points in the 3D Gaussian distribution based on the scene complexity index C, wherein the number N is configured as:

[0234] N=N0×(1+α×C) (22)

[0235] In the formula, N0 is the basic number and α is the adjustment factor.

[0236] To improve accuracy in complex areas and reduce computational effort in simpler areas, a mechanism is proposed to adaptively adjust the number of Gaussian points based on scene complexity. The number of Gaussian points is adjusted based on scene complexity to balance reconstruction quality and computational efficiency.

[0237] Sub-step S165: performing differentiable rasterization processing on the 3D Gaussian distribution, and rendering the processed 3D Gaussian distribution to generate a first 3D rendered image of the current scene.

[0238] As a preferred embodiment, sub-step S165 may further include the following sub-steps 165.1 to S165.5.

[0239] Sub-step 165.1: Use the 3D Gaussian sputtering technique to project the Gaussian points onto the 2D image plane. This method fully considers the spatial characteristics of the Gaussian distribution, produces smooth rendering results, and ensures accurate projection of the 3D scene information onto the 2D image. The projection formula is as follows:

[0240] p 2D =K[R|t]·p 3D (twenty three)

[0241] Where K is the camera intrinsic parameter matrix, [R|t] is the camera extrinsic parameter matrix, and p 2Dis the 2D coordinate of the feature point, p 3D is the 3D coordinate of the feature point.

[0242] Sub-step 165.2: Take into account the distortion caused by the 3D Gaussian projection process by calculating the covariance of the 2D Gaussian distribution to ensure that the rendering result accurately reflects the appearance of the 3D structure on the 2D plane. The formula for calculating the 2D Gaussian covariance is:

[0243] Σ 2D =J·∑ 3D ·J T (twenty four)

[0244] Where J is the Jacobian matrix of the projection, ∑ 3D is the covariance matrix of the 3D Gaussian projection.

[0245] Sub-step 165.3: Rasterize the Gaussian points using the Elliptical Weighted Average (EWA) filter technique. The EWA filter can produce high-quality anti-aliasing effects, thereby improving the visual quality of the image. The specific filter formula is as follows:

[0246]

[0247] Where r is the pixel radius, ∑ 2D is the 2D covariance matrix, w(x) is the filtered pixel, and x is the original pixel.

[0248] Sub-step 165.4: Use an alpha blending algorithm based on depth order and opacity. This algorithm can correctly handle translucency and generate a realistic rendered image C. The formula is:

[0249] C=∑c i α i ·∏(1-α j ),j

[0250] Where c i is the color of the Gaussian point, α i , α j is opacity.

[0251] Substep 165.5: Use spherical harmonics (SH) to capture the perspective-dependent lighting effects for the directional appearance. The use of spherical harmonics can efficiently simulate complex lighting variations and improve the quality of the rendered image. The formula is as follows:

[0252] L(ω)=∑c i ·Y i (ω) (27)

[0253] Among them, c i is the SH coefficient, Y i ​(ω) is the spherical harmonic basis function, L(ω) is the irradiance, and ω is the direction vector.

[0254] As a preferred embodiment, in order to better render the 3D Gaussian distribution, GPU-accelerated real-time rendering may be used, see the following sub-steps S165.6 to S165.9.

[0255] Sub-step S165.6: Implement a tile-based parallel rendering strategy, dividing the image into multiple small tiles and processing them in parallel on the GPU. This method fully utilizes the parallel computing capabilities of the GPU, significantly improving rendering speed and making it suitable for real-time rendering scenarios. To ensure the correct rendering order of Gaussian points, a GPD radix sort algorithm is used to sort the depths of the Gaussian points. This sorting algorithm is highly efficient and ensures correct depth order processing while maintaining real-time rendering performance.

[0256] Sub-step S165.7: Ensure the correctness of transparency processing through the forward-backward traversal alpha blending strategy. This blending strategy can handle complex occlusion relationships, thereby improving rendering quality. The formula is as follows:

[0257] C=∑c i α i ·∏(1-α j ),j

[0258] Where c i is the color of the Gaussian point, α i , α j is the transparency of the Gaussian point.

[0259] Sub-step S165.8: Dynamically adjust the rendering resolution and number of Gaussian points based on hardware conditions through an adaptive mechanism to balance image quality and frame rate. This maintains a smooth user experience under different hardware conditions. The adjustment formula is as follows:

[0260]

[0261] Where R is the rendering resolution, R0 is the initial rendering resolution, N is the number of Gaussian points, N0 is the number of initial Gaussian points, and f r is the current frame rate, f t is the target frame rate.

[0262] Sub-step S165.9: To optimize the Gaussian parameters, GPU-accelerated gradient calculation is used to significantly improve the efficiency of parameter optimization. The specific gradient calculation formula is as follows:

[0263]

[0264] Where L is the rendering loss, θ is the Gaussian parameter, and p​i is the pixel value.

[0265] Figure 2 This is a functional diagram of software designed based on the methods provided in the embodiments of this disclosure. This software includes input ports for camera images and IMU data, as well as image processing buttons for recovering depth maps, feature extraction, and tracking. After receiving initialization instructions, it predicts the camera pose using IMU data and triangulates feature points. It then optimizes the sliding window based on the predicted camera pose and triangulated feature point data, ultimately outputting camera pose data, keyframes, and a point cloud.

[0266] The rendering process of this software is as follows Figure 3 As shown, the adaptive key frame selection strategy is as follows Figure 4 As shown, the rendering process corresponds to the above sub-step S165, and the adaptive key frame selection strategy corresponds to the above step S14, which will not be repeated here.

[0267] From the above description, it can be seen that the present disclosure achieves the following technical effects:

[0268] 1. This paper provides a real-time 3D scene reconstruction method that can adaptively adjust keyframes, optimize resource allocation, balance rendering quality and real-time performance, and has good robustness.

[0269] 2. Improved keyframe selection adaptability and environmental adaptability. A keyframe selection strategy was designed that dynamically adjusts based on scene complexity and system resource status. Through this adaptive keyframe selection strategy and dynamic Gaussian point adjustment mechanism, the system automatically adjusts to scene complexity and hardware conditions, maintaining efficient operation in diverse environments. This adaptability enables the system to perform well in complex and diverse real-world scenarios.

[0270] 3. Optimized the allocation of computing resources. The resource calculations during the rendering process are fed back to the keyframes. The adaptive keyframe strategy improves the system's computing efficiency and memory utilization.

[0271] 4. Improve rendering quality and real-time performance. By decoupling pose and keyframes and combining them with optimized keyframe strategies, the real-time performance of the system can be improved while ensuring rendering quality.

[0272] 5. Enhance system robustness. Through technologies such as visual-inertial joint optimization and closed-loop detection, the system has strong anti-interference ability and long-term stability, and can maintain reliable operation in various complex environments.

[0273] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0274] The present disclosure also provides a 3D scene reconstruction system for implementing the above method embodiment. Figure 5 As shown, the 3D scene reconstruction system 20 includes a data acquisition unit 201 , a first preprocessing unit 202 , a second preprocessing unit 203 , a third preprocessing unit 204 , a data optimization unit 205 , and an image generation unit 206 .

[0275] The data acquisition unit 201 is configured to acquire a plurality of consecutive frames of images of the current scene and inertial measurement unit data, wherein the inertial measurement unit is arranged inside a camera that captures the images and is equipped with a plurality of inertial measurement components.

[0276] The first pre-processing unit 202 is configured to perform target feature extraction and target tracking on each frame of the multiple frames of images to obtain feature point data of the target to be tracked.

[0277] The second pre-processing unit 203 is configured to obtain camera trajectory data and bias data of each inertial measurement component based on the inertial measurement unit data.

[0278] The third pre-processing unit 204 is configured to determine a key frame based on a scene complexity index and a system resource utilization index of the current scene.

[0279] The data optimization unit 205 is configured to perform visual-inertial joint optimization based on all feature point data, camera trajectory data, and bias data and generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate the visual reprojection error of the point cloud data by correcting the feature point data, camera trajectory data, and bias data.

[0280] The image generation unit 206 is configured to generate a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and render the 3D Gaussian distribution to obtain a first 3D rendered image of the current scene.

[0281] The specific manner in which each unit in the above device embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0282] The present disclosure also provides an electronic device, such as Figure 6 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 6A processor 31 is taken as an example.

[0283] The controller may further include an input device 33 and an output device 34 .

[0284] The processor 31, memory 32, input device 33 and output device 34 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0285] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. The general-purpose processor can be a microprocessor or any conventional processor.

[0286] Memory 32, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the control method in the embodiments of the present disclosure. Processor 31 executes the non-transitory software programs, instructions, and modules stored in memory 32 to execute various server functional applications and data processing, thereby implementing the 3D scene reconstruction method of the above-mentioned method embodiment.

[0287] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the processing device operated by the server, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely located relative to the processor 31, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0288] The input device 33 can receive input digital or character information and generate key signal input related to user settings and function control of the processing device of the server. The output device 34 can include a display device such as a display screen.

[0289] One or more modules are stored in the memory 32 and when executed by one or more processors 31, perform the following operations: Figure 1 The 3D scene reconstruction method shown.

[0290] Those skilled in the art will appreciate that all or part of the processes in the above method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes in the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FM), a hard disk drive (HDD), or a solid-state drive (SSD). The storage medium can also include a combination of the above types of memory.

[0291] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A 3D scene reconstruction method based on adaptive keyframes, characterized in that: include: Acquire a plurality of consecutive frames of images of a current scene and inertial measurement unit data, wherein the inertial measurement unit is disposed inside a camera that captures the images and is equipped with a plurality of inertial measurement components; Performing target feature extraction and target tracking on each frame of the multiple frames to obtain feature point data of the target to be tracked; Acquire camera trajectory data and bias data of each of the inertial measurement components based on the inertial measurement unit data; Determine a key frame based on the scene complexity index and system resource utilization index of the current scene, wherein, for each frame in the multiple frames of the image as the current frame, obtain the number of feature points of the current frame , feature point distribution variance , texture complexity ; Based on the number of feature points , the feature point distribution variance , the texture complexity Get the scene complexity index of the current frame C ; Performing visual-inertial joint optimization based on all of the feature point data, the camera trajectory data, and the bias data to generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate visual reprojection errors of the point cloud data by correcting the feature point data, the camera trajectory data, and the bias data; and A 3D Gaussian distribution of the current scene is generated based on the point cloud data and the key frame, and the 3D Gaussian distribution is rendered to obtain a first 3D rendered image of the current scene.

2. The method according to claim 1, characterized in that Also includes: Identifying similar scenes in historical scenes that match the current scene; Obtaining a camera pose calculation model, and performing global consistency optimization on the camera pose calculation model based on the camera trajectory data of the similar scene and the camera trajectory data of the current scene; Acquire optimized camera trajectory data of the current scene based on the optimized camera pose calculation model; as well as, The visual-inertial joint optimization is performed based on the optimized camera trajectory data to generate a second 3D rendered image of the current scene.

3. The method according to claim 1, characterized in that Also includes: generating a first sub-map based on the point cloud data of the current scene; Identifying a second scene in the historical scene that forms a global map closed loop with the current scene; Acquiring point cloud data of the second scene, and generating a second submap based on the point cloud data of the second scene; as well as, Merge the first submap and the second submap to generate a global map.

4. The method according to claim 1, wherein The determining of the key frame based on the scene complexity index and the system resource utilization index of the current scene includes: Determine the nearest key frame and obtain the relative displacement of the current frame relative to the nearest key frame A , overlap B and information entropy H ; Get system resource utilization metrics R , and based on the scenario complexity index C and the system resource utilization indicators R Dynamically adjust the selection threshold T ; as well as, Based on the information entropy H , the selection threshold T、 The relative displacement A , the degree of overlap B Determine whether the current frame is the key frame.

5. The method according to claim 4, characterized in that The scene complexity index C , the information entropy H , the selection threshold T They are configured as: Where, 、 、 is the weight coefficient, The feature points in the image area The probability distribution in is the basic threshold, is the adjustment coefficient, is the correction factor; and The key frame is configured to simultaneously satisfy: the information entropy H Exceeding the selection threshold T , and the relative displacement A exceeds the preset maximum allowable value, the overlap B Below a pre-set overlap threshold.

6. The method according to claim 5, characterized in that The target feature extraction and target tracking are performed on each frame of the multiple frames to obtain feature point data of the target to be tracked, including: For each frame of the multiple image frames as a current frame, perform feature tracking processing on the current frame using a KLT sparse optical flow algorithm to determine a first type of feature point of the target to be tracked, and obtain feature point data of the first type of feature point, wherein the feature point data is configured to include feature point position data and feature point grayscale data; Performing a corner feature search on the current frame using a FAST corner detector to determine a second type of feature point of the target to be tracked, and obtaining feature point data of the second type of feature point; Obtaining a minimum pixel interval between adjacent feature points in a feature point set consisting of the first type of feature points and the second type of feature points; Performing 2D feature dedistortion processing on the elements of the feature point set based on the minimum pixel interval, and projecting feature point data corresponding to the elements of the feature point set after the processing onto a unit sphere; and The projected feature point data is subjected to outlier elimination using a random sampling consistency algorithm to obtain final feature point data of the target to be tracked.

7. The method according to claim 1, characterized in that The camera trajectory data is configured to include camera position data, camera velocity data, and camera attitude data, and the inertial measurement unit is configured to include an accelerometer and a gyroscope; and the camera trajectory data and the bias data of each of the inertial measurement units are obtained based on the inertial measurement unit data, including: acquiring camera trajectory data based on the inertial measurement unit data; and, Acquire a measurement model of the inertial measurement unit, and perform pre-integration processing on the inertial measurement unit data based on the measurement model within a time interval between two consecutive image frames to obtain a relative displacement, relative velocity, and relative rotation angle of the camera within the time interval; determining a covariance propagation matrix of the measurement model based on the relative displacement, relative velocity, and relative rotation angle; Optimizing the parameters of the measurement model based on the covariance propagation matrix so that the bias data of the inertial measurement assembly converge to respective set threshold ranges, and ending the optimization; The bias data of each of the inertial measurement components is obtained based on the measurement model after parameter optimization.

8. The method according to claim 7, characterized in that Also includes: When it is identified that the bias data of any of the inertial measurement components exceeds the corresponding set threshold, the relative displacement, the relative velocity, and the relative rotation angle are calibrated based on the measurement model, thereby calibrating the camera trajectory data; and The performing visual-inertial joint optimization based on all the feature point data, the camera trajectory data, and the bias data and generating point cloud data containing each feature point of the target to be tracked includes: For each feature point of the target to be tracked as a current feature point, obtaining state data corresponding to the current feature point in each frame of the multiple frames of images, wherein the state data is configured to include the feature point position data, the feature point grayscale data, the camera position data, the camera speed data, the camera attitude data, and bias data of each of the inertial measurement units; Obtain the inverse depth parameter corresponding to the current feature point in each frame of the multiple frames, and generate a state vector of the current feature point based on the state data and the inverse depth parameter X ; The state vector X Input the visual reprojection error model to obtain the visual reprojection error; The state vector X Input the pre-integration residual model to obtain the pre-integration residual; The state vector is calculated based on the visual reprojection error and the pre-integrated residual. X Performing nonlinear optimization to obtain optimized state data corresponding to the current feature point in each frame of the multiple frames of images; and The optimized state data corresponding to each feature point of the target to be tracked is obtained, and the point cloud data is generated based on the feature point position data and the feature point grayscale data in the optimized state data.

9. The method according to any one of claims 1 to 8, characterized in that Generating a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, rendering the 3D Gaussian distribution, and obtaining a first 3D rendered image of the current scene includes: Representing each of the feature points in the point cloud data by the 3D Gaussian distribution; Initializing the 3D Gaussian points in the 3D Gaussian distribution, wherein the initialization is configured to include: projecting the 3D Gaussian distribution onto a 2D image plane, and obtaining a rotation matrix during the projection process , scaling matrix , and, based on the rotation matrix , the scaling matrix Covariance matrix of the 3D Gaussian distribution Parameterize; Optimizing parameters of the 3D Gaussian distribution based on the key frame and the camera position, the camera speed, and the camera pose corresponding to the key frame so as to minimize a projection error of the 3D Gaussian distribution; Based on scenario complexity index C Adaptively adjust the number of the 3D Gaussian points in the 3D Gaussian distribution , wherein the number Configured to: Where, As the basic quantity, is the regulating factor; Performing differentiable rasterization processing on the 3D Gaussian distribution, and rendering the processed 3D Gaussian distribution to generate a first 3D rendered image of the current scene.

10. A 3D scene reconstruction system based on adaptive keyframes, characterized in that: include: a data acquisition unit configured to acquire a plurality of consecutive frames of images of a current scene and inertial measurement unit data, wherein the inertial measurement unit is disposed inside a camera that captures the images and is equipped with a plurality of inertial measurement components; A first pre-processing unit is configured to perform target feature extraction and target tracking on each frame of the multiple frames of images to obtain feature point data of the target to be tracked; a second pre-processing unit, configured to acquire camera trajectory data and bias data of each of the inertial measurement components based on the inertial measurement unit data; The third preprocessing unit is configured to determine a key frame based on the scene complexity index and the system resource utilization index of the current scene, wherein, for each frame in the multiple frames of the image as the current frame, the number of feature points of the current frame is obtained. , feature point distribution variance , texture complexity ; Based on the number of feature points , the feature point distribution variance , the texture complexity Get the scene complexity index of the current frame C ; a data optimization unit configured to perform visual-inertial joint optimization based on all of the feature point data, the camera trajectory data, and the bias data and generate point cloud data containing each feature point of the target to be tracked, wherein the visual-inertial joint optimization is configured to eliminate visual reprojection errors of the point cloud data by correcting the feature point data, the camera trajectory data, and the bias data; and The image generation unit is configured to generate a 3D Gaussian distribution of the current scene based on the point cloud data and the key frame, and render the 3D Gaussian distribution to obtain a first 3D rendered image of the current scene.

Images