Multi-view geometric constraint dynamic scene reconstruction method and device, equipment and medium

Abstract

The invention relates to a multi-view geometric constraint dynamic scene reconstruction method and device, equipment and a medium. The method comprises the steps that after an initial view data set is obtained, all frames of images are preprocessed, key point features and initial depth information are extracted, and an initial geometric corresponding relation is obtained through geometric registration; constructing a geometric dependency network between the views based on the geometric dependency network, taking the geometric dependency network as a framework matched with iterative optimization correction key points, and fusing multi-view data to generate a space structure model; further analyzing the influence of dynamic scene change on view association in the model, extracting object movement tracks and deformation features, and determining a dynamic change feature set; and finally, constructing a trajectory prediction model according to the feature set, dynamically correcting the geometric dependency network, and generating a final simulation result of the dynamic scene. By adopting the method, the problems of data processing, model construction, feature analysis, scene simulation and the like in dynamic scene reconstruction under multi-view geometric constraints can be systematically solved, and the precision and reliability of dynamic scene reconstruction are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a method, apparatus, device and medium for dynamic scene reconstruction with multi-view geometric constraints. Background Technology

[0002] In the field of computer vision, dynamic scene reconstruction technology aims to construct 3D models that accurately reflect the dynamic characteristics of objects in real-world scenes, such as movement and shape changes, using multi-source image data. This is significant for applications such as real-time perception of the surrounding dynamic environment in autonomous driving systems, creating immersive experiences in virtual reality scenes, and achieving realistic special effects in film and television production. Multi-view geometrically constrained dynamic scene reconstruction methods, as a key technological path in this field, mainly rely on the geometric relationships between images from multiple perspectives to drive the dynamic scene reconstruction process. Traditional multi-view reconstruction methods are mostly geared towards static scenes, relying on feature extraction, matching, and triangulation to construct 3D models. For example, matching methods based on SIFT and SURF feature points can achieve good reconstruction results in static scenes. However, in dynamic scenes, the rapid movement and deformation of objects, frequent switching of viewpoints, and occlusion phenomena severely interfere with the accuracy of feature point matching, leading to triangulation errors and making it difficult to accurately reconstruct the true structure and changes of the dynamic scene. In recent years, the rise of deep learning technology has spurred the emergence of many neural network-based dynamic scene reconstruction methods. For example, patent CN115731616A uses neural radiation fields to present dynamic scenes to achieve realistic view synthesis. However, the reconstructed surface geometry suffers from significant noise due to insufficient constraints, making it difficult to accurately reproduce the true shape of objects in complex scenes. Patent CN116210688A focuses on multi-scale 3D reconstruction, employing scalable adaptive chessboard sampling and hierarchical optimized depth estimation strategies. However, it primarily targets static or relatively simple dynamic scenes, and its ability to capture and process the motion trajectories and deformation features of objects in complex dynamic changes is limited. Furthermore, some methods attempt to introduce surface constraints into implicit neural representations for static scene reconstruction, but these are difficult to directly adapt to the complex and ever-changing characteristics of dynamic scenes. Existing technologies generally struggle to accurately extract and effectively utilize complex object motion trajectories and deformation features when processing dynamic scenes, resulting in significant deviations between the reconstruction results and the actual scene. This fails to meet the demand for high-precision reconstruction of dynamic scenes, and when constructing multi-view geometric constraints, it is difficult to establish stable and reliable geometric correspondences between different viewpoints, severely impacting the overall quality and reliability of dynamic scene reconstruction. Summary of the Invention

[0003] Therefore, it is necessary to provide a dynamic scene reconstruction method, apparatus, equipment, and medium with multi-view geometric constraints that can accurately handle the motion and deformation of objects in dynamic scenes and improve the accuracy and reliability of dynamic scene reconstruction, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a dynamic scene reconstruction method based on multi-view geometric constraints, including:

[0005] The initial view data set is obtained, and each frame image is preprocessed to extract key point features and preliminary depth information to obtain the initial geometric correspondence.

[0006] Based on the initial geometric correspondence, a geometric dependency network between views is constructed, which serves as a framework for iteratively optimizing and correcting key point matching. Multi-view data is then integrated to generate a spatial structure model.

[0007] Based on the spatial structure model, the impact of dynamic scene changes on view association is analyzed, the motion trajectory and deformation characteristics of objects are extracted, and the set of dynamic change characteristics is determined.

[0008] A trajectory prediction model is constructed based on a dynamically changing feature set, the geometric dependency network is dynamically corrected, and the final simulation results of the dynamic scene are generated.

[0009] In one embodiment, each frame of the image is preprocessed to extract key point features and preliminary depth information to obtain an initial geometric correspondence, including:

[0010] Preprocess each frame image in the view data set to obtain the first image set after denoising.

[0011] Based on an improved two-stream convolutional neural network architecture, key point features are extracted from the first image set to obtain a key point feature set.

[0012] The key point feature set is input into a depth estimation model based on generative adversarial networks to predict depth information, thus obtaining a preliminary depth information set.

[0013] Geometric registration is performed on the key point feature set and the preliminary depth information set to obtain the initial geometric correspondence between views.

[0014] A stability evaluation system based on topological graph analysis is constructed to quantitatively evaluate the initial geometric correspondence. If the evaluation value is lower than the threshold, the key point feature set is optimized through an attention mechanism to form a second key point feature set.

[0015] The depth information is re-estimated based on the second keypoint feature set, and then the depth information is associated with the feature set through geometric registration to generate a stable geometric correspondence.

[0016] In one embodiment, a geometric dependency network between views is constructed based on an initial geometric correspondence, which serves as a framework for iteratively optimizing and correcting keypoint matching. This network then fuses multi-view data to generate a spatial structure model, including:

[0017] An iterative optimization algorithm is used to correct the key point matching set in the initial geometric correspondence and generate an optimized matching relationship.

[0018] Construct a geometric dependency network between views and determine the spatial constraints between views through topology analysis.

[0019] By integrating and optimizing matching relationships and spatial constraints, a preliminary spatial structure model is generated.

[0020] Based on the preliminary spatial structure model, the relative poses between views are calculated to obtain pose estimation results.

[0021] Deep learning algorithms are used to optimize the pose estimation results, and point cloud datasets are obtained based on the optimized pose data.

[0022] The point cloud dataset is meshed, and the surface texture mapping accuracy and geometric details of the model are optimized to obtain the final dynamic scene spatial structure model.

[0023] In one embodiment, the relative poses between views are calculated using the following formula:

[0024]

[0025] Where R represents the rotation matrix, Represents the rotation vector. express Let ξ be a skew-symmetric matrix, where exp(·) denotes the exponential mapping from the Lie algebra to the Lie group, and ξ denotes the elements of the Lie algebra. Let p represent the Lie algebra matrix. i p represents the coordinates (pixel coordinate system) of a two-dimensional feature point in the first view. ' i Let π(·) represent the coordinates of the corresponding 3D point in the second view (camera coordinate system), K represent the camera intrinsic parameter matrix, Ω(ξ) represent the regularization term, λ represent the regularization weight parameter, and f i f' i Let ξ represent the d-dimensional geometric perception features extracted by deep learning, T(ξ) represent the feature transformation function based on pose ξ, and α represent the feature matching error weight parameter.

[0026] In one embodiment, the impact of dynamic scene changes on view association is analyzed based on a spatial structure model, extracting object motion trajectories and deformation features to determine a set of dynamic change features, including:

[0027] Dynamic scene change data is extracted based on the spatial structure model; the dynamic scene change data includes the object's motion trajectory and deformation characteristics.

[0028] Based on dynamic scene change data analysis, the view correlation changes are obtained, resulting in a set of view correlation features that include object motion trajectories and deformation characteristics.

[0029] Spatiotemporal convolutional networks are used to process motion trajectories and determine the temporal continuity characteristics of the trajectories.

[0030] The deformation characteristics are used to calculate the geometric changes on the object's surface, thus obtaining the deformation amplitude characteristics.

[0031] Based on the characteristics of temporal continuity and deformation amplitude, a dynamic change feature set is generated; the dynamic change feature set is used to characterize the changing patterns of the scene.

[0032] In one embodiment, a trajectory prediction model is constructed based on a set of dynamically changing features to generate the final simulation result of the dynamic scene, including:

[0033] A model for predicting the trajectory of an object is constructed based on a dynamically changing set of features to obtain predicted trajectory data.

[0034] Based on the predicted trajectory data, a geometric dependency network is constructed using a graph attention mechanism to determine the dynamic relationships between nodes.

[0035] Based on dynamic relationships, an adaptive weight update algorithm is used to adjust the parameters of the geometric dependency network, generating preliminary simulation results of dynamic scene changes; the preliminary simulation results include trajectory deviation information.

[0036] Multi-scale deviation distribution features are extracted from trajectory deviation information to obtain deviation correction parameters.

[0037] The deviation correction parameter is used to adjust the object motion trajectory prediction model, and the optimized trajectory prediction data is output.

[0038] By combining trajectory prediction data with neural radiation fields and differentiable rendering technology, the final simulation results of dynamic scene changes are generated.

[0039] Secondly, this application also provides a dynamic scene reconstruction apparatus with multi-view geometric constraints, the apparatus comprising:

[0040] The data processing and extraction module is used to obtain the initial view data set, preprocess each frame image and extract key point features and preliminary depth information to obtain the initial geometric correspondence.

[0041] The spatial model building module is used to construct a geometric dependency network between views based on the initial geometric correspondence, and uses it as a framework for iterative optimization and correction of key point matching, and integrates multi-view data to generate a spatial structure model.

[0042] The dynamic feature analysis module is used to analyze the impact of dynamic scene changes on view association based on the spatial structure model, extract the motion trajectory and deformation features of objects, and determine the set of dynamic change features.

[0043] The scene simulation generation module is used to build a trajectory prediction model based on a dynamically changing feature set, dynamically correct the geometric dependency network, and generate the final simulation result of the dynamic scene.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method.

[0046] The aforementioned method, apparatus, computer equipment, and storage medium for dynamic scene reconstruction based on multi-view geometric constraints acquire an initial set of view data, preprocesses each frame image to extract key point features and preliminary depth information, and obtains initial geometric correspondences through geometric registration. Based on this, a geometric dependency network between views is constructed, which serves as a framework for iterative optimization and correction of key point matching. Multi-view data is then fused to generate a spatial structure model. The impact of dynamic scene changes on view associations in this model is analyzed, extracting object motion trajectories and deformation features to determine a dynamic change feature set. Finally, a trajectory prediction model is constructed based on the feature set, dynamically correcting the geometric dependency network to generate the final simulation result of the dynamic scene. Dynamic scene reconstruction is achieved through multi-stage processing. In the data preprocessing and feature extraction stages, the quality and feature accuracy of the input data are ensured; the geometric dependency network construction and spatial structure modeling effectively utilize multi-view geometric constraints; dynamic feature analysis and trajectory prediction optimization model the dynamic characteristics of the scene. This approach can more accurately handle object motion and deformation in dynamic scenes, establish stable view geometric correspondences, and improve the accuracy and reliability of dynamic scene reconstruction, making it suitable for applications in autonomous driving, virtual reality, and other scenarios. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a dynamic scene reconstruction method with multi-view geometric constraints provided in an embodiment of the present invention;

[0049] Figure 2 This is a structural block diagram of a dynamic scene reconstruction device with multi-view geometric constraints provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, such as Figure 1 As shown, this application provides a dynamic scene reconstruction method based on multi-view geometric constraints, which may include the following steps:

[0052] Step S101: Obtain the initial view data set, preprocess each frame image and extract key point features and preliminary depth information to obtain the initial geometric correspondence.

[0053] Specifically, the process begins by acquiring a set of view data collected from multiple perspectives, containing image frames from different times and angles. Preprocessing operations such as denoising and enhancement are performed on each frame to remove noise interference and improve image quality. Using an improved two-stream convolutional neural network architecture, keypoint features are extracted from the preprocessed images to form a keypoint feature set. Simultaneously, this keypoint feature set is input into a depth estimation model based on a generative adversarial network to predict a preliminary depth information set. Finally, a geometric registration algorithm is used to match and align the keypoint features with the preliminary depth information, establishing an initial geometric correspondence between the views.

[0054] Step S102: Construct a geometric dependency network between views based on the initial geometric correspondence, use it as a framework for iterative optimization and correction of key point matching, and fuse multi-view data to generate a spatial structure model.

[0055] Based on the initial geometric correspondences, a geometric dependency network between views is constructed. This network uses image views as nodes and geometric constraints between views as edges. Topological analysis clarifies the spatial constraints between views. An iterative optimization algorithm is used to correct the keypoint matching set in the initial geometric correspondences. By continuously adjusting the matching parameters, mismatches are reduced, generating optimized matching relationships. The optimized matching relationships are then fused with spatial constraints. Utilizing the geometric consistency between multi-view data, algorithms such as triangulation are used to calculate the spatial positions of 3D points, gradually constructing a preliminary spatial structure model. Multi-view data includes images, point clouds, depth information, and corresponding camera parameters acquired at different locations, angles, or times of the same scene. Based on this, the relative poses between views are further calculated to clarify the position and orientation of each view in space.

[0056] Step S103: Analyze the impact of dynamic scene changes on view association based on the spatial structure model, extract the motion trajectory and deformation features of objects, and determine the set of dynamic change features.

[0057] Specifically, dynamic scene changes are analyzed based on the constructed preliminary spatial structure model. By comparing the model's state at different times, the motion trajectories and shape deformation characteristics of objects in space are captured, forming dynamic scene change data. The impact of dynamic scene change data on view relevance is analyzed in depth, clarifying how object motion and deformation change the geometric relationships between views, and extracting a set of view relevance features containing object motion trajectories and deformation characteristics. A spatiotemporal convolutional network is used to process the motion trajectory data, mining the continuity features of the trajectory in the time dimension; by calculating the geometric changes of object surfaces, features such as deformation amplitude are obtained. Finally, these features are integrated to determine a dynamic change feature set, which comprehensively describes the dynamic change patterns of objects in the scene.

[0058] Step S104: Construct a trajectory prediction model based on the dynamically changing feature set, dynamically correct the geometric dependency network, and generate the final simulation result of the dynamic scene.

[0059] Specifically, based on a defined set of dynamic change features, an object motion trajectory prediction model is constructed. This model uses deep learning algorithms to predict the future motion trajectory of the object, obtaining predicted trajectory data. Based on this predicted trajectory data, a graph attention mechanism is used to reconstruct or adjust the geometric dependency network. By focusing on the impact of the object's dynamic changes on geometric relationships, the dynamic relationships between network nodes are determined. Utilizing these dynamic relationships, an adaptive weight update algorithm is employed to adjust the network parameters, generating preliminary simulation results of dynamic scene changes that include trajectory deviation information. From the trajectory deviation information in the preliminary simulation results, multi-scale deviation distribution features are extracted, and deviation correction parameters are calculated. These deviation correction parameters are fed back into the object motion trajectory prediction model to adjust and optimize the model, outputting more accurate trajectory prediction data. Finally, combining neural radiation field and differentiable rendering techniques, based on the optimized trajectory prediction data, a final simulation result of a dynamic scene with rich detail and conforming to actual dynamic changes is generated.

[0060] The aforementioned dynamic scene reconstruction method based on multi-view geometric constraints obtains an initial set of view data, preprocesses each frame image to extract key point features and preliminary depth information, and obtains initial geometric correspondences through geometric registration. Based on this, a geometric dependency network between views is constructed, serving as a framework for iterative optimization and correction of key point matching. Multi-view data is then fused to generate a spatial structure model. The impact of dynamic scene changes on view associations is analyzed, extracting object motion trajectories and deformation features to determine a dynamic change feature set. Finally, a trajectory prediction model is constructed based on the feature set, dynamically correcting the geometric dependency network to generate the final simulation result of the dynamic scene. Dynamic scene reconstruction is achieved through multi-stage processing. In the data preprocessing and feature extraction stages, the quality and accuracy of the input data are ensured; the geometric dependency network construction and spatial structure modeling effectively utilize multi-view geometric constraints; dynamic feature analysis and trajectory prediction optimization model the dynamic characteristics of the scene. This method can more accurately handle object motion and deformation in dynamic scenes, establish stable view geometric correspondences, and improve the accuracy and reliability of dynamic scene reconstruction, making it suitable for applications in autonomous driving, virtual reality, and other scenarios.

[0061] In one embodiment, preprocessing each frame of the image and extracting key point features and preliminary depth information to obtain an initial geometric correspondence may include the following steps:

[0062] Step S201: Preprocess each frame image in the view data set to obtain the first image set after denoising.

[0063] Step S202: Extract key point features from the first image set based on the improved dual-stream convolutional neural network architecture to obtain a key point feature set.

[0064] Step S203: Input the key point feature set into the depth estimation model based on generative adversarial network to predict the depth information and obtain a preliminary depth information set.

[0065] Preferably, the keypoint feature set extracted from the improved two-stream convolutional neural network is used as input to a depth estimation model based on a generative adversarial network (GAN) to obtain a preliminary depth information set. The GAN consists of a generator G and a discriminator D. The goal of the generator G is to generate the corresponding depth image based on the input keypoint feature set. Its input-output relationship can be expressed as:

[0066]

[0067] Among them, F k Represents the key feature set, This represents the generated depth image.

[0068] Discriminator D is responsible for distinguishing the generated depth images. Compared with the true depth image I d Its output is the judgment result. and D(I) d The value range is usually [0,1], and it is used to represent the probability that the input image is a true depth image.

[0069] During training, the generator G and discriminator D are optimized in an adversarial manner, with the objective function being:

[0070]

[0071] Through continuous iterative training, the generator G gradually becomes able to generate predictions that approximate the true depth of the image. Finally, the keypoint feature set is input into the trained generator G to obtain a preliminary depth information set. Each element in this set corresponds to the depth estimation result of one frame of the image.

[0072] Step S204: Perform geometric registration on the key point feature set and the preliminary depth information set to obtain the initial geometric correspondence between views.

[0073] Step S205: Construct a stability evaluation system based on topological graph analysis, quantify the initial geometric correspondence, and if the evaluation value is lower than the threshold, optimize the key point feature set through an attention mechanism to form a second key point feature set.

[0074] Step S206: Re-estimate the depth information based on the second keypoint feature set, and then associate the depth information with the feature set through geometric registration to generate a stable geometric correspondence.

[0075] First, the images in each frame of the view dataset are preprocessed to remove noise interference, resulting in a denoised first image set. Using an improved two-stream convolutional neural network architecture, keypoint features are extracted from the first image set to form a keypoint feature set. This keypoint feature set is then input into a depth estimation model based on a generative adversarial network to predict a preliminary depth information set. A geometric registration algorithm is used to match the keypoint feature set with the preliminary depth information set, establishing an initial geometric correspondence between the views. Subsequently, a stability evaluation system based on topological graph analysis is constructed to quantitatively evaluate the initial geometric correspondence. If the evaluation value is lower than a set threshold, an attention mechanism is used to optimize the keypoint feature set, resulting in a second keypoint feature set. Its depth information is then re-estimated, and registration is performed again, ultimately generating a stable and reliable geometric correspondence.

[0076] This embodiment employs a multi-step data processing and optimization mechanism to ensure the quality of view data and the reliability of geometric correspondences. The preprocessing stage removes image noise, improving the fundamental quality of the data; the combination of a two-stream convolutional neural network and a generative adversarial network enables efficient extraction of key point features and depth information; geometric registration establishes initial correspondences, while topological graph analysis and attention mechanism optimization further filter and correct errors, dynamically adjusting the feature set and depth information. This allows for more accurate acquisition of geometric relationships between views, reducing mismatches and data errors, and effectively improving the accuracy and reliability of the reconstruction model.

[0077] In one embodiment, constructing a geometric dependency network between views based on initial geometric correspondences and using it as a framework for iteratively optimizing and correcting keypoint matching, and fusing multi-view data to generate a spatial structure model, may include the following steps:

[0078] Step S301: Use an iterative optimization algorithm to correct the key point matching set in the initial geometric correspondence and generate an optimized matching relationship.

[0079] Step S302: Construct a geometric dependency network between views and determine the spatial constraints between views through topology analysis.

[0080] Step S303: Integrate and optimize the matching relationship and spatial constraints to generate a preliminary spatial structure model.

[0081] Step S304: Based on the preliminary spatial structure model, calculate the relative poses between views to obtain pose estimation results.

[0082] Step S305: The pose estimation results are optimized using a deep learning algorithm, and a point cloud dataset is obtained based on the optimized pose data.

[0083] Step S306: Mesh the point cloud dataset and optimize the surface texture mapping accuracy and geometric details of the model to obtain the final dynamic scene spatial structure model.

[0084] Specifically, firstly, an iterative optimization algorithm is used to correct the keypoint matching set in the initial geometric correspondence. Through multiple iterations, matching parameters are adjusted to reduce mismatches and generate optimized matching relationships. Based on these optimized matching relationships, a geometric dependency network between views is constructed, with each view as a node and the geometric connections between views as edges. Topological analysis clarifies the spatial constraints between views. The optimized matching relationships and spatial constraints are then integrated, and a preliminary spatial structure model is generated using the principle of multi-view geometric consistency. Based on this preliminary spatial structure model, the relative poses between views are calculated using multi-view geometric formulas to obtain pose estimation results. Next, a deep learning algorithm is used to further refine the pose estimation results. The optimized pose data is extracted through a neural network to form a point cloud dataset. Finally, the point cloud dataset is meshed, and the model's surface texture mapping and geometric details are optimized to obtain the final dynamic scene spatial structure model.

[0085] This embodiment achieves accurate construction of a spatial structure model through multi-stage processing. Iterative optimization algorithms effectively improve the accuracy of key point matching and reduce data errors; geometric dependency networks and topology analysis clarify view space constraints, providing a geometric basis for model construction; pose calculation and deep learning refinement ensure the accuracy of relative view positions; point cloud processing and meshing operations improve model surface details. By systematically integrating geometric constraints, data optimization, and deep learning techniques, it can more accurately handle complex relationships between multiple views, reduce accumulated errors, and improve the integrity and accuracy of dynamic scene spatial structure models.

[0086] In one embodiment, the relative pose between views can be calculated using the following formula:

[0087]

[0088] Where R represents the rotation matrix, Represents the rotation vector. express Let ξ be a skew-symmetric matrix, where exp(·) denotes the exponential mapping from the Lie algebra to the Lie group, and ξ denotes the elements of the Lie algebra. Let p represent the Lie algebra matrix. i p represents the coordinates (pixel coordinate system) of a two-dimensional feature point in the first view. ' i Let π(·) represent the coordinates of the corresponding 3D point in the second view (camera coordinate system), K represent the camera intrinsic parameter matrix, Ω(ξ) represent the regularization term, λ represent the regularization weight parameter, and f i f' i Let ξ represent the d-dimensional geometric perception features extracted by deep learning, T(ξ) represent the feature transformation function based on pose ξ, and α represent the feature matching error weight parameter.

[0089] Preferably, Where f x f y For focal length, (c x ,c y ) is the optical center.

[0090] When calculating the relative pose between views, the transformation relationship between rotation matrices and rotation vectors is utilized. Through the exponential mapping from Lie algebras to Lie groups, Lie algebra elements are converted into rotation matrices. Specifically, the skew-symmetric matrix corresponding to the rotation vector is obtained through the exponential mapping. During the calculation, the geometric relationship between views is established by combining the coordinates of the two-dimensional feature points of the first view (in pixel coordinates) with the coordinates of the corresponding three-dimensional points of the second view (in camera coordinates) through a projection function and the camera intrinsic parameter matrix. To optimize the calculation results, a regularization term is introduced, with its influence controlled by regularization weight parameters. Simultaneously, the d-dimensional geometric perception features extracted by deep learning are processed through a pose-based feature transformation function, and adjusted using feature matching error weight parameters. Finally, the relative pose between views is obtained.

[0091] This embodiment integrates geometric constraints and deep learning features, offering advantages over traditional methods. Through mathematical transformations using Lie algebras and Lie groups, it accurately describes the rotational relationships between views. Combining the 2D-3D point correspondence in multi-view geometry, it constructs a basic computational framework using projection functions and camera intrinsic parameters. Introducing regularization terms avoids overfitting during computation, improving result stability. The integration of deep learning features enables the computation to capture complex geometric information that is difficult to obtain using traditional methods. Adjusting the feature transformation function and error weight parameters further optimizes the accuracy of pose calculation. This method can more accurately determine the relative poses between views, providing a precise spatial relationship basis for dynamic scene reconstruction under multi-view geometric constraints, and improving the overall accuracy and reliability of the reconstruction model.

[0092] In one embodiment, analyzing the impact of dynamic scene changes on view association based on a spatial structure model, extracting object motion trajectories and deformation features, and determining a set of dynamic change features may include the following steps:

[0093] Step S401: Extract dynamic scene change data based on the spatial structure model; the dynamic scene change data includes the object's motion trajectory and deformation features.

[0094] Step S402: Analyze the changes in view correlation based on dynamic scene change data to obtain a set of view correlation features that includes object motion trajectories and deformation features.

[0095] Step S403: Use a spatiotemporal convolutional network to process the motion trajectory and determine the temporal continuity characteristics of the trajectory.

[0096] Step S404: Calculate the geometric changes on the object surface based on the deformation characteristics to obtain the deformation amplitude characteristics.

[0097] Step S405: Generate a dynamic change feature set based on the time continuity feature and the deformation amplitude feature; the dynamic change feature set is used to characterize the scene change pattern.

[0098] Specifically, firstly, dynamic scene change data is extracted from the constructed spatial structure model, encompassing object motion trajectories and deformation features. Next, the dynamic scene change data is analyzed to explore its impact on view relevance, thereby obtaining a set of view relevance features containing object motion trajectories and deformation characteristics. Subsequently, a spatiotemporal convolutional network is used to process the motion trajectories, mining their continuity features in the time dimension; simultaneously, deformation amplitude features are obtained by calculating geometric changes on the object surface. Finally, the temporal continuity features and deformation amplitude features are integrated to generate a dynamic change feature set, which comprehensively characterizes the scene's change patterns.

[0099] This embodiment achieves accurate capture of dynamic scene change patterns through systematic feature extraction and analysis. Motion trajectories and deformation features are directly extracted from the spatial structure model, ensuring the reliability of the data source. Analysis of view correlation changes further mines dynamic information from multiple perspectives. The spatiotemporal convolutional network effectively extracts dynamic features in the time dimension for motion trajectory processing, compensating for the shortcomings of traditional methods in temporal analysis. Geometric calculations of object deformation features precisely quantify changes in object shape. The resulting dynamic change feature set comprehensively and accurately reflects the dynamic change patterns of the scene, providing rich and accurate feature data for subsequent trajectory prediction and scene simulation, thus improving the ability and accuracy of dynamic scene reconstruction under multi-view geometric constraints to handle complex dynamic changes.

[0100] In one embodiment, constructing a trajectory prediction model based on a set of dynamically changing features to generate the final simulation result of the dynamic scene may include the following steps:

[0101] Step S501: Construct an object motion trajectory prediction model based on the dynamically changing feature set and obtain predicted trajectory data.

[0102] Step S502: Construct a geometric dependency network based on the predicted trajectory data using a graph attention mechanism to determine the dynamic relationships between nodes.

[0103] Step S503: Based on the dynamic relationship, an adaptive weight update algorithm is used to adjust the parameters of the geometric dependency network and generate preliminary simulation results of dynamic scene changes; the preliminary simulation results include trajectory deviation information.

[0104] Step S504: Extract multi-scale deviation distribution features from trajectory deviation information to obtain deviation correction parameters.

[0105] Step S505: Adjust the object motion trajectory prediction model using deviation correction parameters and output optimized trajectory prediction data.

[0106] Step S506: By combining trajectory prediction data with neural radiation field and differentiable rendering technology, the final simulation result of dynamic scene changes is generated.

[0107] Specifically, firstly, a trajectory prediction model for object motion is constructed based on a dynamically changing feature set, and predicted trajectory data is obtained through this model. Based on the predicted trajectory data, a geometrically dependent network is constructed using a graph attention mechanism to clarify the dynamic relationships between network nodes. Utilizing these dynamic relationships, an adaptive weight update algorithm is employed to adjust the network parameters, generating preliminary simulation results of dynamic scene changes that include trajectory deviation information. Multi-scale deviation distribution features are extracted from the trajectory deviation information, and deviation correction parameters are derived. These deviation correction parameters are fed back to the object motion trajectory prediction model for adjustment, outputting optimized trajectory prediction data. Finally, combining neural radiation field and differentiable rendering techniques, the final simulation results of dynamic scene changes are generated based on the optimized trajectory prediction data.

[0108] This embodiment achieves high-precision simulation of dynamic scenes through multi-stage iterative optimization. The trajectory prediction model, combined with a set of dynamically changing features, effectively captures the motion patterns of objects. The geometrically dependent network constructed using a graph attention mechanism can adaptively learn dynamic relationships between nodes, enhancing the network's ability to represent complex scenes. The adaptive weight update algorithm optimizes network parameters based on dynamic relationships, and the deviation information contained in the initial simulation results provides direction for subsequent optimization. By extracting multi-scale deviation distribution features, correction parameters are obtained, and the trajectory prediction model is specifically adjusted to reduce prediction errors. Finally, by combining neural radiation fields and differentiable rendering technology, the optimized trajectory data is transformed into high-quality dynamic scene simulation results. Compared to traditional methods, this process, through the combination of data-driven and algorithmic optimization, significantly improves the accuracy and realism of dynamic scene reconstruction, making it suitable for applications with high requirements for reproducing dynamic scene changes.

[0109] In one embodiment, such as Figure 2 As shown, this application also provides a dynamic scene reconstruction apparatus with multi-view geometric constraints, the apparatus may include:

[0110] The data processing and extraction module 601 is used to acquire the initial view data set, preprocess each frame image and extract key point features and preliminary depth information to obtain the initial geometric correspondence.

[0111] The spatial model construction module 602 is used to construct a geometric dependency network between views based on the initial geometric correspondence, and use it as a framework for iterative optimization and correction of key point matching, and to generate a spatial structure model by fusing multi-view data.

[0112] The dynamic feature analysis module 603 is used to analyze the impact of dynamic scene changes on view association based on the spatial structure model, extract the motion trajectory and deformation features of objects, and determine the set of dynamic change features.

[0113] The scene simulation generation module 604 is used to construct a trajectory prediction model based on a dynamically changing feature set, dynamically correct the geometric dependency network, and generate the final simulation result of the dynamic scene.

[0114] The aforementioned multi-view geometrically constrained dynamic scene reconstruction device comprises a data processing and extraction module responsible for acquiring an initial view data set. Through image preprocessing, key point feature extraction, depth information prediction, and geometric registration, an initial geometric correspondence is obtained. Based on this correspondence, a spatial model construction module builds a geometric dependency network, which serves as a framework for iteratively optimizing and correcting key point matching, and integrates multi-view data to generate a preliminary spatial structure model. A dynamic feature analysis module analyzes the spatial structure model, extracting the motion trajectory and deformation features of the objects to determine the set of dynamic change features. A scene simulation generation module constructs a trajectory prediction model based on the dynamic change feature set, dynamically corrects the geometric dependency network, and combines neural radiation field and differentiable rendering techniques to finally generate a dynamic scene simulation result. This device's data processing and extraction module ensures the quality of the original data and the accuracy of basic feature extraction; the spatial model construction module utilizes geometric constraints and iterative optimization to improve the accuracy of 3D model construction; the dynamic feature analysis module specifically mines dynamic scene information; and the scene simulation generation module achieves high-precision simulation of the dynamic scene through trajectory prediction and network correction. It can systematically solve problems such as data processing, model building, feature analysis and scene simulation in dynamic scene reconstruction under multi-view geometric constraints, improve reconstruction efficiency and result accuracy, and meet the needs of dynamic scene modeling in fields such as autonomous driving and virtual reality.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the dynamic scene reconstruction method, apparatus, device and medium with multi-view geometric constraints as described above.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0118] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0119] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A dynamic scene reconstruction method with multi-view geometric constraints, characterized in that, The method includes: Obtain the initial view data set, preprocess each frame image and extract key point features and preliminary depth information to obtain the initial geometric correspondence; Based on the initial geometric correspondence, a geometric dependency network between views is constructed, which is used as a framework for iterative optimization and correction of key point matching, and multi-view data is fused to generate a spatial structure model. Based on the spatial structure model, analyze the impact of dynamic scene changes on view association, extract the motion trajectory and deformation characteristics of objects, and determine the set of dynamic change characteristics. A trajectory prediction model is constructed based on the set of dynamic change features to generate the final simulation results of the dynamic scene.

2. The method according to claim 1, characterized in that, The process of preprocessing each frame of the image and extracting key point features and preliminary depth information to obtain the initial geometric correspondence includes: Preprocessing is performed on each frame image in the view data set to obtain a first image set after denoising; Based on an improved two-stream convolutional neural network architecture, key point features are extracted from the first image set to obtain a key point feature set; The key point feature set is input into a depth estimation model based on a generative adversarial network to predict depth information, thereby obtaining a preliminary depth information set. Geometric registration is performed on the key point feature set and the preliminary depth information set to obtain the initial geometric correspondence between views; A stability evaluation system based on topological graph analysis is constructed to quantitatively evaluate the initial geometric correspondence. If the evaluation value is lower than the threshold, the key point feature set is optimized through an attention mechanism to form a second key point feature set. The depth information is re-estimated based on the second keypoint feature set, and then the depth information is associated with the feature set through geometric registration to generate a stable geometric correspondence.

3. The method according to claim 1, characterized in that, The step of constructing a geometric dependency network between views based on the initial geometric correspondence, using it as a framework for iteratively optimizing and correcting keypoint matching, and fusing multi-view data to generate a spatial structure model includes: An iterative optimization algorithm is used to correct the key point matching set in the initial geometric correspondence to generate an optimized matching relationship; Construct a geometric dependency network between views and determine the spatial constraints between views through topology analysis; By integrating the optimized matching relationship and spatial constraints, a preliminary spatial structure model is generated; Based on the preliminary spatial structure model, the relative poses between the views are calculated to obtain the pose estimation results; The pose estimation results are optimized using a deep learning algorithm, and a point cloud dataset is obtained by extracting 3D point cloud data based on the optimized pose data. The point cloud dataset is meshed, and the surface texture mapping accuracy and geometric details of the model are optimized to obtain the final dynamic scene spatial structure model.

4. The method according to claim 3, characterized in that, The relative poses between the views are calculated using the following formula: Where R represents the rotation matrix, Represents the rotation vector. express Let ξ be a skew-symmetric matrix, where exp(·) denotes the exponential mapping from the Lie algebra to the Lie group, and ξ denotes the elements of the Lie algebra. Let p represent the Lie algebra matrix. i p' represents the coordinates (pixel coordinates) of a two-dimensional feature point in the first view. i Let π(·) represent the coordinates of the corresponding 3D point in the second view (camera coordinate system), K represent the camera intrinsic parameter matrix, Ω(ξ) represent the regularization term, λ represent the regularization weight parameter, and f i f' i Let ξ represent the d-dimensional geometric perception features extracted by deep learning, T(ξ) represent the feature transformation function based on pose ξ, and α represent the feature matching error weight parameter.

5. The method according to claim 1, characterized in that, The step involves analyzing the impact of dynamic scene changes on view association based on the spatial structure model, extracting object motion trajectories and deformation features, and determining a set of dynamic change features, including: Dynamic scene change data is extracted based on the spatial structure model; the dynamic scene change data includes object motion trajectories and deformation features. Based on the dynamic scene change data analysis, the view correlation changes are obtained, resulting in a view correlation feature set that includes object motion trajectories and deformation features; The motion trajectory is processed using a spatiotemporal convolutional network to determine the temporal continuity characteristics of the trajectory; The deformation characteristics are used to calculate the geometric changes on the object surface to obtain the deformation amplitude characteristics; Based on the time continuity feature and deformation amplitude feature, a dynamic change feature set is generated; the dynamic change feature set is used to characterize the scene change pattern.

6. The method according to claim 1, characterized in that, The step of constructing a trajectory prediction model based on the dynamically changing feature set and generating the final simulation result of the dynamic scene includes: Based on the dynamically changing feature set, an object motion trajectory prediction model is constructed to obtain predicted trajectory data; Based on the predicted trajectory data, a geometric dependency network is constructed using a graph attention mechanism to determine the dynamic relationships between nodes; Based on the dynamic relationship, an adaptive weight update algorithm is used to adjust the parameters of the geometric dependency network to generate preliminary simulation results of dynamic scene changes; the preliminary simulation results include trajectory deviation information. Multi-scale deviation distribution features are extracted from the trajectory deviation information to obtain deviation correction parameters; The deviation correction parameters are used to adjust the object motion trajectory prediction model, and the optimized trajectory prediction data is output. By combining the trajectory prediction data with neural radiation fields and differentiable rendering technology, the final simulation result of dynamic scene changes is generated.

7. A dynamic scene reconstruction device with multi-view geometric constraints, characterized in that, The device includes: The data processing and extraction module is used to obtain the initial view data set, preprocess each frame image and extract key point features and preliminary depth information to obtain the initial geometric correspondence. The spatial model construction module is used to construct a geometric dependency network between views based on the initial geometric correspondence, and use it as a framework for iterative optimization and correction of key point matching, and to fuse multi-view data to generate a spatial structure model. The dynamic feature analysis module is used to analyze the impact of dynamic scene changes on view association based on the spatial structure model, extract the motion trajectory and deformation features of objects, and determine the set of dynamic change features. The scene simulation generation module is used to construct a trajectory prediction model based on the set of dynamically changing features, dynamically correct the geometric dependency network, and generate the final simulation result of the dynamic scene.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images