Adaptive generation method of 3D model based on video and point cloud data

Abstract

The embodiment of the present application provides a method for adaptively generating a three-dimensional model based on video and point cloud data, which collects scene data through a multi-view camera array and a depth scanning array, and uses a hierarchical feature extraction network and an attention mechanism to extract and screen key features. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional grids are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method for adaptively generating a three-dimensional model based on video and point cloud data. Background Art

[0002] With the rapid development of 3D reconstruction technology, 3D model generation methods based on video and point cloud data have been widely used in virtual reality, smart cities and other fields. The current mainstream 3D reconstruction solutions mainly use a single data source or a simple multi-source data fusion strategy, which has obvious shortcomings when dealing with large-scale and complex scenes.

[0003] From the perspective of data acquisition and processing, traditional methods often rely on a single type of sensor data, which makes it difficult to fully capture the geometric and texture information of the scene. Even if a multi-sensor solution is adopted, there is a lack of effective data fusion mechanism, resulting in missing details or distortion in the reconstructed model. At the same time, the existing feature extraction algorithm has poor adaptability to scene changes and occlusion, and it is difficult to accurately establish the correspondence between multi-source data.

[0004] From the perspective of reconstruction efficiency, existing systems generally use unified reconstruction parameters and fixed processing procedures, and are unable to dynamically adjust reconstruction strategies based on scene characteristics. This rigid processing method not only wastes computing resources, but also affects the quality of reconstruction results. Especially when dealing with large-scale scenes, it is difficult for the system to balance reconstruction accuracy and computing efficiency, and problems such as long processing time or unstable reconstruction quality often occur.

[0005] From the perspective of model quality control, existing methods lack a systematic quality assessment mechanism, making it difficult to timely discover and correct errors in the reconstruction process. At the same time, in the model optimization stage, there is a lack of intelligent parameter adjustment strategies, and parameter settings often rely on manual experience, which greatly limits the system's adaptive ability and reconstruction effect.

[0006] Therefore, how to achieve deep fusion of multi-source data, build intelligent reconstruction strategies, and improve the system's computational efficiency and adaptability are key issues that need to be addressed in the current development of 3D reconstruction technology. This is not only related to the quality of the reconstructed model, but also an important basis for achieving efficient and stable 3D scene reconstruction. Summary of the invention

[0007] In response to the problems in the prior art, the present application provides a method for adaptively generating three-dimensional models based on video and point cloud data, which can effectively solve the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes and improve the quality of model generation.

[0008] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a method for adaptively generating a three-dimensional model based on video and point cloud data, comprising:

[0010] Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to a confidence screening threshold;

[0011] Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional meshes into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the scores into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional meshes into a virtual space using a hierarchical compression method;

[0012] Construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to a computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0013] Furthermore, the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array are written into a data buffer matrix according to timestamps, and a hierarchical feature extraction network is constructed to extract feature vectors from the image data and the point cloud data respectively, including:

[0014] The image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array are transmitted to a data timing unification module, the data timing unification module generates a global timing stamp number, performs timing alignment on the image data and the point cloud data according to the global timing stamp number, establishes a multi-resolution data buffer, writes the timing-aligned data into the multi-resolution data buffer according to the acquisition time, and constructs a cyclic data writing queue to maintain the space capacity of the multi-resolution data buffer;

[0015] A hierarchical feature extraction network including a convolution layer and a pooling layer is constructed, wherein the convolution layer sets a plurality of convolution kernels of different scales to extract edge and texture features of the image data, and the pooling layer performs dimensionality reduction compression on the feature map to generate an image feature vector. Meanwhile, a voxelization processing unit is constructed to divide the point cloud data into regular voxels, and the spatial distribution features of the points in each voxel are calculated to generate a point cloud feature vector, and the image feature vector and the point cloud feature vector are concatenated to form a multimodal feature vector.

[0016] Furthermore, the attention weight calculation module is constructed by combining the spatial geometric constraints, the feature vector is input into the attention weight calculation module to obtain the feature weight, the deep mapping model is trained based on the feature weight, and the corresponding relationship of the feature points is determined according to the confidence screening threshold, including:

[0017] Constructing a geometric constraint matrix based on spatial position and projective geometric relationship, inputting the multimodal feature vector into a self-attention calculation unit, the self-attention calculation unit generating a query matrix, a key matrix and a value matrix, multiplying the geometric constraint matrix with the query matrix and the key matrix to obtain an attention weight coefficient, and weighting the value matrix with the attention weight coefficient to obtain a feature weight vector fused with geometric constraints;

[0018] A deep mapping neural network model is constructed, and the feature weight vector is input into the deep mapping neural network. The deep mapping neural network includes an encoder and a decoder structure. The encoder maps the feature weight vector to a latent space, and the decoder restores the latent space features to a depth prediction map. A loss function is constructed based on the depth truth map to train the deep mapping neural network. The depth prediction map output by the trained deep mapping neural network is matched with the feature point position information, and a feature point correspondence set is obtained by screening according to a preset confidence threshold.

[0019] Furthermore, the dividing the image data and the point cloud data into local data blocks, constructing a union-find-based hierarchical index tree to store the local data blocks, and reconstructing the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm includes:

[0020] Constructing a data block processing unit, wherein the data block processing unit divides the image data and the point cloud data into local data blocks of equal size according to the spatial adjacency relationship, establishes a spatial coordinate index for the local data blocks, aggregates the local data blocks with spatial connectivity into a data block set by using a union-find data structure, and constructs a hierarchical index tree to store the hierarchical relationship of the data block set;

[0021] The data block set is input into a surface reconstruction processing unit, which constructs an implicit surface function based on Poisson's equation, calculates normal vectors and position parameters of data points in the data block set, solves the implicit surface function to obtain an isosurface, and triangulates the isosurface to generate a local three-dimensional mesh model.

[0022] Further, the model evaluation index calculation module is established, the local three-dimensional mesh is input into the evaluation index calculation module to obtain a completeness score and an accuracy score, the score is input into a parameter adjustment controller to generate a parameter optimization strategy, and the local three-dimensional mesh is written into the virtual space using a hierarchical compression method, including:

[0023] Establishing a three-dimensional mesh evaluation module, wherein the three-dimensional mesh evaluation module calculates the vertex distribution density and surface smoothness of the local three-dimensional mesh to obtain a completeness score, calculates the Euclidean distance deviation between the local three-dimensional mesh and the original point cloud data to obtain an accuracy score, inputs the completeness score and the accuracy score into a parameter adjustment controller, and the parameter adjustment controller calculates the gradient direction of the surface reconstruction parameter according to the score difference, and generates a parameter iterative optimization sequence;

[0024] A mesh compression encoder is constructed, which divides the local three-dimensional mesh into different resolution levels, performs entropy coding on the mesh vertex coordinates and connection relationships of each level, generates a progressively compressed data stream, establishes a virtual space data management unit, writes the compressed data stream into a virtual space storage area, and constructs an index table to record the storage location and decompression parameters of the compressed data.

[0025] Furthermore, the constructing of the parallel computing channel, calculating the reconstruction deviation value of the local three-dimensional grid in the computing channel, and transferring the deviation value to the computing resource scheduler includes:

[0026] Constructing a multi-threaded computing framework, wherein the multi-threaded computing framework sets a working thread pool and a task queue, and distributes the local three-dimensional mesh to different working threads according to the spatial position, each working thread calculates the vertex distance and the normal vector angle between the assigned three-dimensional mesh and the reference model, and combines the calculated distance values ​​and angle values ​​to generate a reconstruction deviation value matrix;

[0027] A task scheduling system is established, which receives the reconstructed deviation value matrix, prioritizes the working threads according to the deviation value, allocates computing tasks in high deviation value areas to high-performance computing nodes, builds a load balancing module to monitor the resource occupancy rate of each computing node, and dynamically adjusts the allocation strategy of computing tasks.

[0028] Furthermore, the allocating calculation nodes based on the deviation value, inputting the local three-dimensional grid into the calculation nodes for material mapping and lighting calculation, and updating the calculation results of the calculation nodes to the virtual scene according to the structure of the hierarchical index tree, includes:

[0029] Constructing a computing node resource pool, wherein the computing node resource pool manages computing nodes in groups according to the deviation value, and assigns local three-dimensional grids with larger deviation values ​​to a high-computing-power computing node group, wherein the computing node group constructs a texture mapping processing unit and a ray tracing processing unit, wherein the texture mapping processing unit projects texture information in image data onto a three-dimensional grid surface, and the ray tracing processing unit calculates ambient lighting and shadow effects;

[0030] A scene update controller is established, which organizes the rendering results output by the computing nodes according to the hierarchical structure of the hierarchical index tree, constructs a scene buffer to store rendering data at different levels, the scene buffer maintains the data version number to ensure the synchronization of the rendering results, and writes the rendering results into the virtual scene display buffer for display update.

[0031] In a second aspect, the present application provides a device for adaptively generating a three-dimensional model based on video and point cloud data, comprising:

[0032] A correspondence determination module is used to arrange a multi-view camera array and a depth scanning array in a scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to timestamps, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the correspondence of feature points according to a confidence screening threshold;

[0033] A data evaluation module is used to divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on a union-find set to store the local data blocks, reconstruct the local data blocks into local three-dimensional grids using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional grid into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional grid into a virtual space using a hierarchical compression method;

[0034] An adaptive adjustment module is used to construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional grid in the computing channel, transfer the deviation value to a computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional grid into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0035] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data are implemented.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data.

[0037] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data.

[0038] It can be seen from the above technical solutions that the present application provides a method for adaptively generating three-dimensional models based on video and point cloud data, which collects scene data through a multi-view camera array and a depth scanning array, and uses a hierarchical feature extraction network and an attention mechanism to extract and screen key features. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional meshes are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is one of the flow charts of the method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0041] Figure 2The second flowchart of the method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0042] Figure 3 The third flowchart of the method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0043] Figure 4 This is a fourth flow chart of a method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0044] Figure 5 This is a fifth flow chart of a method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0045] Figure 6 This is a sixth flow chart of a method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0046] Figure 7 FIG7 is a flowchart of a method for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0047] Figure 8 A structural diagram of a device for adaptively generating a three-dimensional model based on video and point cloud data in an embodiment of the present application;

[0048] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0049] Reference numerals:

[0050] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0053] Taking into account the problems existing in the prior art, the present application provides a method for adaptively generating a three-dimensional model based on video and point cloud data. Scene data is collected through a multi-view camera array and a depth scanning array, and key features are extracted and screened using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional meshes are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0054] In order to effectively solve the accuracy and efficiency problems of traditional 3D reconstruction solutions when processing large-scale scenes and improve the quality of model generation, this application provides an embodiment of a method for adaptively generating a 3D model based on video and point cloud data, see Figure 1 The method for adaptively generating a three-dimensional model based on video and point cloud data specifically includes the following contents:

[0055] Step S101: Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to the confidence screening threshold;

[0056] Optionally, in this embodiment, a multi-view camera array and a depth scanning array are first arranged in the scene space to be reconstructed. The multi-view camera array includes multiple high-resolution cameras, which maintain a fixed relative position and angle between the cameras to form a circular or hemispherical acquisition array to ensure all-round image acquisition of the scene without blind spots. The depth scanning array uses depth sensors such as laser radar to obtain the three-dimensional spatial position information of objects in the scene by emitting laser beams and receiving reflected signals.

[0057] The raw data generated by the acquisition device includes image data stream and point cloud data stream. In order to ensure the timing consistency of the data, this embodiment designs a data synchronization mechanism based on timestamp. Each acquisition device is equipped with a high-precision clock, and a timestamp mark is added to each frame of data during data acquisition. After transmission, the data enters the data buffer matrix, which has multiple cache units, each of which stores an image and point cloud data pair at a specific moment. The buffer matrix adopts a circular queue structure. When new data arrives, the earliest data will be overwritten, thereby achieving efficient data storage and access.

[0058] This embodiment constructs a hierarchical feature extraction network, which includes multiple convolutional layers and pooling layers. For image data, multi-scale features, such as edges, textures and other visual features, are extracted through convolution kernels of different scales. For point cloud data, voxelization is first performed to divide the space into regular grids, and the density distribution, normal vector and other geometric features of the points in each voxel are calculated. The two features are normalized and spliced ​​to form a multimodal feature vector.

[0059] In order to accurately establish the correspondence between the image feature points and the spatial positions of the point cloud, this embodiment designs a feature matching method based on the attention mechanism. First, the constraint matrix is ​​constructed using the camera calibration parameters and the spatial geometric relationship. The matrix describes the projection relationship of the feature points under different viewing angles. The feature vector is input into the self-attention calculation unit to generate the query matrix, key matrix and value matrix, and the attention weight considering the geometric constraints is calculated by matrix multiplication.

[0060] Based on the obtained feature weights, this embodiment trains a depth mapping model. The model uses an encoder-decoder structure to map two-dimensional image features to three-dimensional space depth values. During the training process, the real depth value provided by the point cloud data is used as a supervisory signal to optimize the model parameters by minimizing the error between the predicted depth and the real depth. After the training is completed, the confidence threshold is set to filter the feature point matching pairs to ensure the accuracy of the matching results.

[0061] Through the above processing, this embodiment achieves high-precision registration of image data and point cloud data, laying the foundation for subsequent 3D reconstruction. This method can effectively handle data differences between different acquisition devices, improve the robustness of feature extraction, and reduce computing resource consumption while ensuring reconstruction accuracy.

[0062] Step S102: Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional mesh into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional mesh into a virtual space using a hierarchical compression method;

[0063] Optionally, this embodiment performs local processing on the acquired image data and point cloud data. First, the data is divided into local data blocks of equal size according to the spatial adjacency relationship, and each data block contains image texture information and point cloud geometry information within a specific spatial range. The size of the data block needs to balance the computational efficiency and reconstruction accuracy. Too large a data block will increase the computational burden, and too small a data block may result in insufficient local feature information.

[0064] Based on data partitioning, this embodiment constructs a hierarchical index tree structure based on union-find. Union-find maintains the connectivity between data blocks in the form of a parent pointer tree, with search and merge operations that are approximately constant time. The index tree adopts a top-down hierarchical structure, where the top-level node represents the entire scene, and each layer down represents a finer-grained spatial division. Adjacent data blocks are dynamically merged through a union-find operation to form a set of data blocks with spatial continuity.

[0065] This embodiment uses a surface reconstruction algorithm based on the Poisson equation to process local data blocks. First, the normal vector information of the point cloud data is calculated, and the implicit surface function is constructed by combining the edge features in the image data. The isosurface is obtained by solving the Poisson equation, and then the isosurface is adaptively triangulated to generate a local three-dimensional mesh model with a reasonable topological structure. In order to improve the reconstruction effect, factors such as point cloud density changes and surface curvature are considered in the mesh generation process.

[0066] In order to evaluate the reconstruction quality, this embodiment establishes a model evaluation index calculation module. This module evaluates from two dimensions: completeness and accuracy: the completeness score measures the completeness of the model by calculating the spatial distribution density of mesh vertices and the smoothness of local surfaces; the accuracy score measures the reconstruction accuracy by calculating the Euclidean distance deviation between the reconstructed mesh and the original point cloud.

[0067] Based on the scores obtained from the evaluation, this embodiment designs a parameter adjustment controller. The controller analyzes the difference between the integrity score and the accuracy score and calculates the optimization direction of the reconstruction parameters. When the integrity score is low, the degree of mesh simplification is appropriately reduced; when the accuracy score is low, the degree of retention of local details is increased. Through iterative optimization, the reconstruction effect is gradually improved.

[0068] Considering the storage requirements of three-dimensional mesh data, this embodiment adopts a hierarchical compression method. The mesh model is divided into multiple resolution levels, and the geometric information and topological information of each level are compressed and encoded respectively. During data transmission, data of different resolutions can be selected for loading according to actual needs to achieve progressive display. The compressed data is written into the storage area of ​​the virtual space, and an index table is established to record the data location and decompression parameters.

[0069] Through the above processing, this embodiment realizes efficient reconstruction and storage of large-scale three-dimensional scenes. The data organization method based on union-find provides flexible space management capabilities, the localized processing strategy reduces the computational complexity, the adaptive parameter optimization ensures the reconstruction quality, and the hierarchical compression realizes efficient storage and transmission of data.

[0070] Step S103: construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to the computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0071] Optionally, this embodiment constructs an efficient parallel computing channel in the process of 3D scene reconstruction. Each computing channel contains an independent computing unit, which can simultaneously process the reconstruction tasks of multiple local 3D grids. The computing channel adopts a pipeline architecture, decomposing the complex reconstruction process into multiple computing stages, and asynchronous transmission is achieved between the stages through the data buffer, thereby improving the computing efficiency.

[0072] In order to evaluate the reconstruction quality, this embodiment calculates the reconstruction deviation value of the local 3D mesh in the operation channel. The deviation value calculation takes into account multiple dimensions: the geometric deviation is obtained by comparing the distance between the reconstructed mesh and the original point cloud; the texture deviation is obtained by comparing the difference between the texture mapping of the reconstructed model and the original image; the topological deviation is measured by analyzing the mesh connectivity and boundary integrity. These deviation values ​​comprehensively reflect the quality of the reconstruction result.

[0073] This embodiment designs an intelligent computing resource scheduler to receive the deviation value information from each computing channel. The scheduler maintains a dynamic priority queue, and areas with larger deviation values ​​are given higher processing priority. Based on the priority and the status of available computing resources, the scheduler dynamically allocates computing nodes. When it is detected that the reconstruction quality of a certain area does not meet the standard, the computing resource quota of the area will be automatically increased.

[0074] In the material mapping stage, this embodiment adopts a multi-level texture mapping strategy. First, texture information is extracted from multi-view images, and the seams caused by perspective conversion are eliminated through image stitching technology. Then the texture coordinates are mapped to the three-dimensional mesh surface, and the UV coordinates are stored at the mesh vertices. In order to deal with occlusion and lighting changes, the embodiment also introduces a texture weight mechanism to assign weights to textures of different perspectives according to the perspective quality and lighting conditions.

[0075] The lighting calculation adopts a physically based rendering method. This embodiment builds a material attribute database to store parameters such as reflectivity and roughness of different surface materials. By solving the bidirectional reflectance distribution function, the rendering effect of each surface point in the scene under different lighting conditions is calculated. In order to enhance the sense of reality, the global illumination effect is also considered, including diffuse reflection, specular reflection and ambient occlusion.

[0076] The update of the calculation results follows the organizational structure of the hierarchical index tree. This embodiment designs an incremental update mechanism that only updates the changed nodes and their parent nodes, avoiding the performance overhead caused by global updates. A version control strategy is used during the update process to ensure data consistency when multiple computing nodes are updated in parallel. When a node is updated, its status will be immediately reflected in the virtual scene, achieving real-time visual feedback.

[0077] Through the above processing, this embodiment realizes efficient parallel computing of three-dimensional scene reconstruction. Resource scheduling based on deviation values ​​ensures the reasonable allocation of computing resources, multi-level material mapping and physically based lighting calculation enhance the realism of the scene, and the incremental update mechanism ensures the real-time performance of the virtual scene. This method is particularly suitable for high-quality reconstruction of large-scale indoor and outdoor scenes, and can effectively balance the requirements of computing efficiency and reconstruction quality.

[0078] From the above description, it can be seen that the adaptive generation method of three-dimensional models based on video and point cloud data provided by the embodiment of the present application can collect scene data through a multi-view camera array and a depth scanning array, and extract and screen key features using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional grids are achieved through parallel computing channels and dynamic computing resource scheduling mechanisms. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0079] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 2 , and can also include the following:

[0080] Step S201: the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array are transmitted to the data timing unification module, the data timing unification module generates a global timing stamp number, performs timing alignment on the image data and the point cloud data according to the global timing stamp number, establishes a multi-resolution data buffer, writes the timing-aligned data into the multi-resolution data buffer according to the acquisition time, and constructs a cyclic data writing queue to maintain the space capacity of the multi-resolution data buffer;

[0081] Step S202: construct a hierarchical feature extraction network including a convolution layer and a pooling layer, wherein the convolution layer sets a plurality of convolution kernels of different scales to extract edge and texture features of the image data, and the pooling layer performs dimensionality reduction compression on the feature map to generate an image feature vector. Meanwhile, a voxelization processing unit is constructed to divide the point cloud data into regular voxels, and the spatial distribution features of the points in each voxel are calculated to generate a point cloud feature vector, and the image feature vector and the point cloud feature vector are concatenated to form a multimodal feature vector.

[0082] Optionally, this embodiment first addresses the synchronization problem of multi-source heterogeneous data. The multi-view camera array and the depth scanning array collect image data and point cloud data respectively. Due to the differences in sampling frequency and data transmission delay of different sensors, precise timing alignment is required. The data timing unification module generates a global time stamp number through a high-precision clock source. The number has a nanosecond time accuracy, ensuring the accuracy of data synchronization.

[0083] In the temporal registration process, this embodiment adopts a sliding window mechanism. For each temporal window, the timestamp difference between the image data and the point cloud data is calculated and aligned to a unified time axis through a linear interpolation method. Considering the sampling jitter of the sensor, a time tolerance parameter is also introduced, and data within the tolerance range is regarded as synchronous data.

[0084] In order to efficiently manage the synchronized data, this embodiment establishes a multi-resolution data buffer. The buffer adopts a pyramid structure, and different levels store data of different resolutions. The top layer stores the original resolution data, and the resolution is reduced by half each time it goes down. This structure facilitates the subsequent feature extraction network to perform multi-scale feature analysis. When writing data, it is stored in the order of the acquisition time to ensure the temporal continuity of the data.

[0085] This embodiment manages the buffer capacity by writing data into the queue in a loop. When new data arrives, the queue adopts a first-in-first-out strategy to automatically clear the oldest data to free up space. The queue length is dynamically adjusted according to the real-time requirements of the application scenario, ensuring data continuity while avoiding memory overflow.

[0086] In the feature extraction stage, this embodiment constructs a deep neural network containing multiple convolutional layers and pooling layers. The convolutional layers use convolution kernels of different sizes. Smaller convolution kernels (such as 3×3) are used to extract local texture features, and larger convolution kernels (such as 7×7) are used to capture large-scale structural features. In order to enhance the expressive power of features, the network also uses the dilated convolution technology to expand the receptive field by adjusting the expansion rate of the convolution kernel.

[0087] The pooling layer reduces the spatial dimension of the feature map by combining the maximum pooling and the mean pooling. This embodiment designs an adaptive pooling strategy, dynamically adjusts the pooling parameters according to the information entropy of the feature map, retains important feature information and realizes data dimensionality reduction. The final image feature vector contains the visual semantic information of the scene.

[0088] For point cloud data, this embodiment first performs voxel processing. The three-dimensional space is divided into a regular voxel grid according to the spatial scale of the scene, and the size of each voxel is adaptively determined by the scene complexity. For each non-empty voxel, the density distribution, normal vector statistics, geometric curvature and other features of the internal points are calculated to generate a high-dimensional point cloud feature vector.

[0089] Finally, this embodiment uses a feature fusion module to concatenate the image feature vector and the point cloud feature vector. In order to balance the contribution of different modal features, a feature normalization layer is introduced to map the two features to the same numerical range. The concatenated multimodal feature vector integrates the visual and geometric information of the scene, providing rich feature expressions for subsequent scene reconstruction.

[0090] Through the above processing, this embodiment achieves accurate synchronization of multi-source data and efficient feature extraction, laying a solid data foundation for 3D scene reconstruction. This method shows strong robustness when processing large-scale dynamic scenes and can effectively cope with the influence of sensor noise and environmental changes.

[0091] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 3 , and can also include the following:

[0092] Step S301: construct a geometric constraint matrix based on spatial position and projective geometric relationship, input the multimodal feature vector into a self-attention calculation unit, the self-attention calculation unit generates a query matrix, a key matrix and a value matrix, multiplies the geometric constraint matrix with the query matrix and the key matrix to obtain an attention weight coefficient, and weights the value matrix using the attention weight coefficient to obtain a feature weight vector fused with geometric constraints;

[0093] Step S302: construct a deep mapping neural network model, input the feature weight vector into the deep mapping neural network, the deep mapping neural network includes an encoder and a decoder structure, the encoder maps the feature weight vector to the latent space, and the decoder restores the latent space features to a depth prediction map, constructs a loss function based on the depth truth map to train the deep mapping neural network, matches the depth prediction map output by the trained deep mapping neural network with the feature point position information, and obtains a feature point correspondence set by screening according to a preset confidence threshold.

[0094] Optionally, this embodiment first constructs a geometric constraint matrix based on the spatial geometric characteristics of the scene. By analyzing the projective transformation relationship between multi-view images, a basic matrix and an essential matrix are established to describe the epipolar geometric constraints between image feature points. At the same time, the three-dimensional spatial information provided by the point cloud data is used to construct the spatial position constraints of the feature points to form a complete geometric constraint matrix.

[0095] In order to make full use of multimodal feature information, this embodiment designs a feature fusion module based on the self-attention mechanism. The self-attention calculation unit receives the multimodal feature vector as input and generates a query matrix, a key matrix, and a value matrix through linear transformation. The query matrix represents the features that need to be paid attention to at present, the key matrix stores the feature information that may be related, and the value matrix contains the actual feature content.

[0096] This embodiment innovatively introduces geometric constraint information into the attention calculation process. The geometric constraint matrix is ​​multiplied with the query matrix and the key matrix to obtain the attention weight coefficient that takes into account the spatial geometric relationship. This design ensures the effective use of spatial position constraints in the feature fusion process and improves the accuracy of feature matching. The value matrix is ​​weighted by the attention weight coefficient to generate a feature weight vector that incorporates geometric constraints.

[0097] In the depth mapping stage, this embodiment constructs a deep neural network based on an encoder-decoder structure. The encoder uses a residual connection structure to map the feature weight vector to a low-dimensional latent space through multi-layer convolution and downsampling operations. This process can be understood as extracting high-level semantic representations of features. The decoder gradually restores spatial details through deconvolution and upsampling operations to generate a high-precision depth prediction map.

[0098] In order to improve the accuracy of depth prediction, a multi-scale loss function is designed in this embodiment. The loss function contains a depth value error term, a gradient consistency term, and a smoothing regularization term. The depth value error ensures that the prediction result is consistent with the true value as a whole, the gradient consistency maintains the clarity of the depth boundary, and the smoothing regularization suppresses the noise fluctuations in the depth map. Through this multi-objective optimization, the network can learn a more accurate depth mapping relationship.

[0099] In the feature matching stage, this embodiment combines the depth prediction map with the feature point position information. First, the two-dimensional feature points are back-projected into three-dimensional space through the depth information, and then the geometric consistency score between the feature point pairs is calculated. According to the preset confidence threshold, reliable feature point correspondences are screened out. These correspondences provide key registration information for subsequent scene reconstruction.

[0100] Through the above processing flow, this embodiment realizes feature fusion and depth mapping based on geometric constraints. This method shows good robustness when processing complex scenes, especially when dealing with challenging situations such as occlusion and illumination changes, and can maintain stable performance. The feature matching strategy integrating geometric constraints significantly improves the accuracy of feature correspondence and provides a reliable data basis for subsequent 3D reconstruction.

[0101] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 4 , and can also include the following:

[0102] Step S401: constructing a data block processing unit, wherein the data block processing unit divides the image data and point cloud data into local data blocks of equal size according to spatial adjacency relationships, establishes spatial coordinate indexes for the local data blocks, aggregates local data blocks with spatial connectivity into a data block set using a union-find data structure, and constructs a hierarchical index tree to store the hierarchical relationship of the data block set;

[0103] Step S402: Input the data block set into the surface reconstruction processing unit, the surface reconstruction processing unit constructs an implicit surface function based on Poisson's equation, calculates the normal vectors and position parameters of the data points in the data block set, solves the implicit surface function to obtain an isosurface, and triangulates the isosurface to generate a local three-dimensional mesh model.

[0104] Optionally, in order to efficiently process large-scale scene data, this embodiment first constructs a data block processing unit to organize the data. The unit adopts an octree space partitioning strategy to divide the image data and point cloud data into equal-sized local data blocks according to the spatial distribution characteristics of the scene. The partitioning process takes into account the density distribution of the data to ensure that each data block contains an appropriate number of data points to avoid uneven data distribution.

[0105] This embodiment establishes a spatial coordinate index for each local data block, and uses a multidimensional spatial index structure to record the location information of the data block. The index structure contains information such as the center coordinates, boundary range, and adjacency of the data block, supporting fast spatial query and neighborhood search operations. This index mechanism significantly improves the data access efficiency in the subsequent processing process.

[0106] In order to process the continuous structure in the scene, this embodiment innovatively introduces a union-find data structure to manage the association relationship between data blocks. The union-find structure efficiently maintains the connectivity information between data blocks through path compression and rank-based merging optimization. When sufficient feature overlap or point cloud continuity is detected between adjacent data blocks, these data blocks are merged into the same set to form a larger data block set.

[0107] This embodiment constructs a hierarchical index tree to manage the hierarchical relationship of the data block set. The index tree is constructed in a bottom-up manner, with the bottom-level nodes corresponding to the original data blocks and the upper-level nodes representing the merged data block set. Each node stores the data summary information of the current level, including statistical features such as point cloud density and feature distribution, which facilitates the rapid assessment of reconstruction difficulty and resource requirements.

[0108] In the surface reconstruction stage, this embodiment constructs an implicit surface representation based on the Poisson equation. First, the normal vector of each data point is calculated, and the principal component analysis method is used to estimate the direction of the local surface. In order to improve the accuracy of the normal vector estimation, a confidence-based weighting strategy is introduced to suppress the influence of noise points and outliers.

[0109] This embodiment innovatively introduces an adaptive basis function when constructing an implicit surface function. The support range of the basis function is dynamically adjusted according to the local point cloud density. A smaller support range is used in areas with rich details to maintain the details, and a larger support range is used in flat areas to achieve a smooth transition. This adaptive strategy effectively balances the accuracy and smoothness of the reconstruction.

[0110] When solving the implicit surface function, this embodiment adopts a multi-resolution solution strategy. First, the initial solution is obtained at a coarse resolution, and then the grid resolution is gradually refined, and each refinement uses the solution of the previous layer as the initial value. This strategy not only speeds up the solution convergence speed, but also helps to avoid local optimal solutions.

[0111] When performing triangular meshing on the obtained isosurface, this embodiment uses an adaptive mesh generation algorithm. The algorithm dynamically adjusts the size of the triangle according to the curvature distribution of the surface, using denser meshing in areas with larger curvatures and sparser meshing in flat areas. At the same time, the algorithm ensures the quality of the generated mesh, controls the minimum angle of the triangle, and avoids the generation of pathological triangles.

[0112] Through the above processing flow, this embodiment achieves efficient local 3D reconstruction. The design of data segmentation and union-find enables the system to effectively process large-scale scene data, and the adaptive surface reconstruction strategy ensures the accuracy and efficiency of the reconstruction results. This method is particularly suitable for processing indoor and outdoor scenes with complex topological structures, and can accurately reconstruct the geometric details of the scene.

[0113] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 5 , and can also include the following:

[0114] Step S501: establishing a three-dimensional mesh evaluation module, wherein the three-dimensional mesh evaluation module calculates the vertex distribution density and surface smoothness of the local three-dimensional mesh to obtain a completeness score, calculates the Euclidean distance deviation between the local three-dimensional mesh and the original point cloud data to obtain an accuracy score, and inputs the completeness score and the accuracy score into a parameter adjustment controller, wherein the parameter adjustment controller calculates the gradient direction of the surface reconstruction parameter according to the score difference, and generates a parameter iterative optimization sequence;

[0115] Step S502: construct a mesh compression encoder, which divides the local three-dimensional mesh into different resolution levels, performs entropy encoding on the mesh vertex coordinates and connection relationships of each level, generates a progressively compressed data stream, establishes a virtual space data management unit, writes the compressed data stream into the virtual space storage area, and constructs an index table to record the storage location and decompression parameters of the compressed data.

[0116] Optionally, this embodiment first constructs a three-dimensional mesh evaluation module to comprehensively evaluate the quality of the reconstruction results. When evaluating the vertex distribution density, an adaptive mesh sampling strategy is used to calculate the spatial distribution characteristics of vertices in the local area. By analyzing the statistical distribution of vertex spacing, sparse or over-dense areas that may exist in the mesh are identified to ensure the geometric integrity of the reconstructed model.

[0117] This embodiment introduces a multi-scale curvature analysis method when calculating the surface smoothness. First, the Gaussian curvature and mean curvature of each vertex are calculated, and then the curvature change trend is evaluated at different scales. By setting a reasonable smoothness threshold, the local area that needs to be optimized is identified to avoid over-smoothing resulting in loss of details or under-smoothing resulting in residual noise.

[0118] In order to evaluate the reconstruction accuracy, this embodiment calculates the Euclidean distance deviation between the local 3D grid and the original point cloud data. The nearest neighbor search algorithm is used to find the corresponding point cloud data point for each grid vertex, and the distance between them is calculated as the local error metric. At the same time, considering that the point cloud data may be noisy, a robust statistical method is introduced to reduce the impact of outliers.

[0119] The parameter adjustment controller receives the completeness score and the accuracy score as input and optimizes the surface reconstruction parameters by gradient descent. The controller adopts an adaptive learning rate strategy, using a larger learning rate to converge quickly in the early stage of optimization, and gradually reducing the learning rate as the optimization process progresses to achieve fine adjustment. This optimization strategy can effectively balance the accuracy and efficiency of reconstruction.

[0120] This embodiment adopts a multi-resolution encoding strategy in the mesh compression stage. First, a hierarchical structure of the mesh is constructed, with the base layer storing complete geometric details and the higher layer storing simplified mesh structures. At each level, vertex coordinates and connection relationships are encoded separately, and predictive coding technology is used to reduce data redundancy.

[0121] In order to improve the compression efficiency, this embodiment designs an adaptive quantization scheme. According to the complexity of local geometric features, the quantization parameters are dynamically adjusted. A finer quantization is used in areas with rich details, and a coarser quantization is used in flat areas, which not only ensures the retention of important details, but also improves the overall compression ratio.

[0122] This embodiment innovatively introduces the concept of progressively compressed data streams. Compressed data is organized hierarchically according to importance, and supports gradual decompression and display according to demand. This design is particularly suitable for application in network transmission and online display scenarios, where users can quickly obtain a rough model preview and then gradually load the details.

[0123] The virtual space data management unit adopts a distributed storage architecture to evenly distribute compressed data streams to multiple storage nodes. By building a multi-level index table, the storage location, compression parameters and dependencies of each data block are recorded. The index table adopts a lightweight key-value storage structure to support fast data retrieval and on-demand loading.

[0124] Through the above processing flow, this embodiment achieves high-quality 3D model reconstruction and efficient data management. Mesh evaluation and parameter optimization ensure the accuracy and integrity of the reconstruction results, and the multi-resolution compression strategy achieves efficient storage and transmission of data. This method is particularly suitable for processing large-scale 3D scene reconstruction tasks, and can significantly reduce storage and transmission overhead while ensuring reconstruction quality.

[0125] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 6 , and can also include the following:

[0126] Step S601: construct a multi-threaded computing framework, wherein the multi-threaded computing framework sets a working thread pool and a task queue, and distributes the local three-dimensional mesh to different working threads according to the spatial position, wherein each working thread calculates the vertex distance and the normal vector angle between the assigned three-dimensional mesh and the reference model, and combines the calculated distance values ​​and angle values ​​to generate a reconstruction deviation value matrix;

[0127] Step S602: Establish a task scheduling system, which receives the reconstructed deviation value matrix, prioritizes the working threads according to the deviation value, allocates computing tasks in high deviation value areas to high-performance computing nodes, builds a load balancing module to monitor the resource occupancy rate of each computing node, and dynamically adjusts the allocation strategy of computing tasks.

[0128] Optionally, this embodiment first constructs a multi-threaded computing framework and uses thread pool technology to efficiently manage computing resources. The size of the working thread pool is dynamically adjusted according to the number of processor cores available in the system, and a moderate number of threads is maintained to avoid excessive context switching overhead. The task queue adopts a double-ended queue structure to support task priority scheduling and work retrieval mechanism.

[0129] In the task allocation stage, this embodiment divides tasks according to the spatial position relationship of local three-dimensional grids. A spatial hash algorithm is used to allocate adjacent grid blocks to the same working thread, reducing the data interaction overhead between threads. Each working thread maintains an independent computing context, including necessary reference model data and computing cache, to improve the locality of data access.

[0130] This embodiment adopts a multi-level precision control strategy when calculating geometric errors. First, the Euclidean distance between the mesh vertex and the reference model is calculated, and the spatial index structure is used to accelerate the nearest point search. At the same time, the angle between the normal vectors of the corresponding points is calculated to evaluate the directional consistency of the local surface. The combination of these two indicators can fully reflect the reconstruction quality.

[0131] In order to improve the computational efficiency, this embodiment designs an adaptive sampling strategy. The sampling point density is increased in areas with large curvature, and the sampling density is appropriately reduced in flat areas. The selection of sampling points takes into account the complexity of local geometric features to ensure that important detail areas are fully evaluated.

[0132] This embodiment innovatively proposes the concept of reconstructing the deviation value matrix, and forms a unified evaluation index by weighted combination of the distance value and the angle value. The weight coefficient is dynamically adjusted according to the specific needs of the application scenario. For example, in a scenario where shape details need to be maintained, the weight of the normal vector angle is increased.

[0133] The task scheduling system adopts a dynamic priority scheduling strategy. According to the analysis results of the reconstructed deviation value matrix, the areas that need to be optimized are identified. The computing tasks in the high deviation value area are given a higher execution priority and are preferentially allocated to the computing nodes with better performance. This scheduling strategy ensures the efficient use of computing resources.

[0134] This embodiment implements a multi-dimensional resource monitoring mechanism in the load balancing module. In addition to traditional CPU usage monitoring, it also includes real-time tracking of performance indicators such as memory usage and cache hit rate. Based on these monitoring data, the system can promptly discover performance bottlenecks and dynamically adjust task allocation strategies.

[0135] In order to handle dynamic load changes, this embodiment implements a task migration mechanism. When it is detected that a computing node is overloaded, the system will automatically migrate some tasks to nodes with lighter loads. The migration process adopts an incremental transmission strategy to transmit only necessary data and minimize migration overhead.

[0136] Through the above processing flow, this embodiment realizes an efficient parallel computing framework. Multi-threaded collaborative processing and dynamic load balancing ensure the optimal utilization of computing resources, and the priority scheduling strategy ensures that important areas can be processed in a timely manner. This method is particularly suitable for processing large-scale three-dimensional reconstruction tasks, which can significantly improve the reconstruction efficiency while ensuring the uniformity of reconstruction quality. In practical applications, this parallel processing framework can effectively adapt to reconstruction tasks of different scales and adaptively adjust the processing strategy according to changes in hardware resources.

[0137] In one embodiment of the method for adaptively generating a three-dimensional model based on video and point cloud data of the present application, see Figure 7 , and can also include the following:

[0138] Step S701: constructing a computing node resource pool, wherein the computing node resource pool manages computing nodes in groups according to the deviation value, and allocates local three-dimensional grids with larger deviation values ​​to a high-computing-power computing node group, wherein the computing node group constructs a texture mapping processing unit and a ray tracing processing unit, wherein the texture mapping processing unit projects texture information in image data onto a three-dimensional grid surface, and the ray tracing processing unit calculates ambient lighting and shadow effects;

[0139] Step S702: Establish a scene update controller, which organizes the rendering results output by the computing nodes according to the hierarchical structure of the hierarchical index tree, builds a scene buffer to store rendering data at different levels, and the scene buffer maintains the data version number to ensure the synchronization of the rendering results, and writes the rendering results to the virtual scene display buffer for display update.

[0140] Optionally, this embodiment first constructs a computing node resource pool and uses a hierarchical management strategy to optimize the configuration of computing resources. According to the reconstruction deviation value of the local three-dimensional grid, the computing nodes are divided into node groups with different performance levels. The high-computing power computing node group is equipped with a high-performance GPU and a large-capacity memory, which is specially used to process areas with large deviation values ​​and require fine processing.

[0141] In the texture mapping process, this embodiment adopts multi-view texture fusion technology. First, the view coverage relationship in the image data is analyzed, and the best texture mapping source is selected for each 3D mesh patch. When there are multiple available view angles, a fusion strategy based on view angle weight is adopted to ensure the smoothness of texture transition.

[0142] This embodiment introduces an intelligent completion mechanism when processing occluded areas. By analyzing the texture features of the surrounding area, an image restoration algorithm is used to fill in the texture missing due to occlusion. The completion process takes into account the continuity and structural features of the material to ensure the naturalness of the restoration result.

[0143] The ray tracing processing unit uses real-time rendering technology to achieve high-quality lighting effects. By building a spatial acceleration structure, the intersection operation between light and the scene is optimized. When calculating ambient lighting, spherical harmonics are used to represent the ambient lighting distribution, which not only ensures rendering quality but also reduces computing overhead.

[0144] This embodiment innovatively designs an adaptive shadow calculation method. The resolution and update frequency of the shadow map are dynamically adjusted according to the characteristics of the light source in the scene and the observation position. For areas with high visual importance, more sophisticated shadow calculations are used to ensure the rendering quality of key areas.

[0145] The scene update controller uses a multi-level cache mechanism to manage rendering data. The rendering results are organized according to the structure of a hierarchical index tree, supporting fast switching of different levels of detail. The cache strategy takes into account the temporal and spatial locality of data and optimizes memory usage efficiency.

[0146] This embodiment implements a version control mechanism in the scene buffer. Each rendering update generates a new version number to ensure that the rendering results output by multiple computing nodes can be correctly synchronized. Version control also supports rollback operations to quickly restore to a stable state when rendering problems occur.

[0147] In order to improve display performance, this embodiment adopts double buffering technology in the virtual scene display buffer. The foreground buffer is responsible for displaying the current frame, and the background buffer prepares the data of the next frame, and a synchronous exchange mechanism is used to achieve smooth screen update.

[0148] This embodiment also implements a dynamic scheduling mechanism for rendering tasks. According to the changes in the user's viewpoint and the complexity of the scene, the rendering priority and resource allocation are adaptively adjusted. For the area near the viewpoint, high-quality rendering is given priority, and the rendering accuracy is appropriately reduced for the distant area, so as to achieve a balance between performance and quality.

[0149] Through the above processing flow, this embodiment achieves high-quality scene rendering and efficient display update. The reasonable allocation of computing resources and the multi-level cache mechanism ensure the rendering performance, and the version control and dynamic scheduling ensure the correctness and continuity of the rendering results. This method is particularly suitable for processing the real-time rendering requirements of large-scale virtual scenes, and can provide a smooth interactive experience while ensuring visual quality. In practical applications, this rendering framework can effectively support various virtual reality and augmented reality application scenarios.

[0150] In order to effectively solve the accuracy and efficiency problems of traditional 3D reconstruction solutions when processing large-scale scenes and improve the quality of model generation, the present application provides an embodiment of a 3D model adaptive generation device based on video and point cloud data for implementing all or part of the content of the 3D model adaptive generation method based on video and point cloud data, see Figure 8 The three-dimensional model adaptive generation device based on video and point cloud data specifically includes the following contents:

[0151] A correspondence determination module 10 is used to arrange a multi-view camera array and a depth scanning array in a scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to timestamps, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the correspondence of feature points according to a confidence screening threshold;

[0152] A data evaluation module 20 is used to divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on a union-find set to store the local data blocks, reconstruct the local data blocks into local three-dimensional grids using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional grid into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional grid into a virtual space using a hierarchical compression method;

[0153] The adaptive adjustment module 30 is used to construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, transmit the deviation value to the computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0154] From the above description, it can be seen that the three-dimensional model adaptive generation device based on video and point cloud data provided by the embodiment of the present application can collect scene data through a multi-view camera array and a depth scanning array, and extract and screen key features using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional grids are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0155] From the hardware level, in order to effectively solve the accuracy and efficiency problems of traditional 3D reconstruction solutions when processing large-scale scenes and improve the quality of model generation, the present application provides an embodiment of an electronic device for implementing all or part of the content of the 3D model adaptive generation method based on video and point cloud data, and the electronic device specifically includes the following content:

[0156] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the three-dimensional model adaptive generation device based on video and point cloud data and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the three-dimensional model adaptive generation method based on video and point cloud data and the embodiment of the three-dimensional model adaptive generation device based on video and point cloud data in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0157] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0158] In practical applications, part of the method for adaptively generating a three-dimensional model based on video and point cloud data can be executed on the electronic device side as described above, or all operations can be completed in the client device. The selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0159] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0160] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0161] In one embodiment, the function of the method for adaptively generating a 3D model based on video and point cloud data may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:

[0162] Step S101: Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to the confidence screening threshold;

[0163] Step S102: Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional mesh into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional mesh into a virtual space using a hierarchical compression method;

[0164] Step S103: construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to the computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0165] From the above description, it can be seen that the electronic device provided by the embodiment of the present application collects scene data through a multi-view camera array and a depth scanning array, and extracts and screens key features using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional meshes are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0166] In another embodiment, the three-dimensional model adaptive generation device based on video and point cloud data can be configured separately from the central processing unit 9100. For example, the three-dimensional model adaptive generation device based on video and point cloud data can be configured as a chip connected to the central processing unit 9100, and the function of the three-dimensional model adaptive generation method based on video and point cloud data can be implemented through the control of the central processing unit.

[0167] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0168] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0169] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0170] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0171] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0172] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0173] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0174] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0175] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0176] Step S101: Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to the confidence screening threshold;

[0177] Step S102: Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional mesh into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional mesh into a virtual space using a hierarchical compression method;

[0178] Step S103: construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to the computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0179] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application collects scene data through a multi-view camera array and a depth scanning array, and extracts and screens key features using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional grids are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0180] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data are implemented. For example, the computer program / instruction implements the following steps:

[0181] Step S101: Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to the confidence screening threshold;

[0182] Step S102: Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional mesh into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional mesh into a virtual space using a hierarchical compression method;

[0183] Step S103: construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to the computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

[0184] From the above description, it can be seen that the computer program product provided by the embodiment of the present application collects scene data through a multi-view camera array and a depth scanning array, and extracts and screens key features using a hierarchical feature extraction network and an attention mechanism. A hierarchical index tree based on a union-find set is used to organize local data blocks, and adaptive parameter optimization is achieved by combining integrity and accuracy scores. At the same time, efficient reconstruction and material mapping of local three-dimensional grids are achieved through parallel computing channels and a dynamic computing resource scheduling mechanism. This method effectively solves the accuracy and efficiency problems of traditional three-dimensional reconstruction solutions when processing large-scale scenes, and improves the quality of model generation.

[0185] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for adaptively generating a three-dimensional model based on video and point cloud data, characterized in that: The method comprises: Arrange a multi-view camera array and a depth scanning array in the scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to the timestamp, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the corresponding relationship of feature points according to a confidence screening threshold; Divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on union-find to store the local data blocks, reconstruct the local data blocks into local three-dimensional meshes using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional meshes into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the scores into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional meshes into a virtual space using a hierarchical compression method; Construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional mesh in the computing channel, pass the deviation value to a computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

2. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 1, characterized in that: The step of writing the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to timestamps, and constructing a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, comprises: The image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array are transmitted to a data timing unification module, the data timing unification module generates a global timing stamp number, performs timing alignment on the image data and the point cloud data according to the global timing stamp number, establishes a multi-resolution data buffer, writes the timing-aligned data into the multi-resolution data buffer according to the acquisition time, and constructs a cyclic data writing queue to maintain the space capacity of the multi-resolution data buffer; A hierarchical feature extraction network including a convolution layer and a pooling layer is constructed, wherein the convolution layer sets a plurality of convolution kernels of different scales to extract edge and texture features of the image data, and the pooling layer performs dimensionality reduction compression on the feature map to generate an image feature vector. Meanwhile, a voxelization processing unit is constructed to divide the point cloud data into regular voxels, and the spatial distribution features of the points in each voxel are calculated to generate a point cloud feature vector, and the image feature vector and the point cloud feature vector are concatenated to form a multimodal feature vector.

3. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 2, characterized in that: The method of constructing an attention weight calculation module in combination with spatial geometric constraints, inputting the feature vector into the attention weight calculation module to obtain feature weights, training a deep mapping model based on the feature weights, and determining feature point correspondences according to a confidence screening threshold comprises: Constructing a geometric constraint matrix based on spatial position and projective geometric relationship, inputting the multimodal feature vector into a self-attention calculation unit, the self-attention calculation unit generating a query matrix, a key matrix and a value matrix, multiplying the geometric constraint matrix with the query matrix and the key matrix to obtain an attention weight coefficient, and weighting the value matrix with the attention weight coefficient to obtain a feature weight vector fused with geometric constraints; A deep mapping neural network model is constructed, and the feature weight vector is input into the deep mapping neural network. The deep mapping neural network includes an encoder and a decoder structure. The encoder maps the feature weight vector to a latent space, and the decoder restores the latent space features to a depth prediction map. A loss function is constructed based on the depth truth map to train the deep mapping neural network. The depth prediction map output by the trained deep mapping neural network is matched with the feature point position information, and a feature point correspondence set is obtained by screening according to a preset confidence threshold.

4. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 1, characterized in that: The step of dividing the image data and the point cloud data into local data blocks, constructing a union-find-based hierarchical index tree to store the local data blocks, and reconstructing the local data blocks into local three-dimensional grids using a surface reconstruction algorithm includes: Constructing a data block processing unit, wherein the data block processing unit divides the image data and the point cloud data into local data blocks of equal size according to the spatial adjacency relationship, establishes a spatial coordinate index for the local data blocks, aggregates the local data blocks with spatial connectivity into a data block set by using a union-find data structure, and constructs a hierarchical index tree to store the hierarchical relationship of the data block set; The data block set is input into a surface reconstruction processing unit, which constructs an implicit surface function based on Poisson's equation, calculates normal vectors and position parameters of data points in the data block set, solves the implicit surface function to obtain an isosurface, and triangulates the isosurface to generate a local three-dimensional mesh model.

5. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 1, characterized in that: The method of establishing a model evaluation index calculation module, inputting the local three-dimensional mesh into the evaluation index calculation module to obtain a completeness score and an accuracy score, inputting the score into a parameter adjustment controller to generate a parameter optimization strategy, and writing the local three-dimensional mesh into a virtual space using a hierarchical compression method includes: Establishing a three-dimensional mesh evaluation module, wherein the three-dimensional mesh evaluation module calculates the vertex distribution density and surface smoothness of the local three-dimensional mesh to obtain a completeness score, calculates the Euclidean distance deviation between the local three-dimensional mesh and the original point cloud data to obtain an accuracy score, inputs the completeness score and the accuracy score into a parameter adjustment controller, and the parameter adjustment controller calculates the gradient direction of the surface reconstruction parameter according to the score difference, and generates a parameter iterative optimization sequence; A mesh compression encoder is constructed, which divides the local three-dimensional mesh into different resolution levels, performs entropy coding on the mesh vertex coordinates and connection relationships of each level, generates a progressively compressed data stream, establishes a virtual space data management unit, writes the compressed data stream into a virtual space storage area, and constructs an index table to record the storage location and decompression parameters of the compressed data.

6. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 1, characterized in that: The step of constructing a parallel computing channel, calculating a reconstruction deviation value of the local three-dimensional grid in the computing channel, and transferring the deviation value to a computing resource scheduler includes: Constructing a multi-threaded computing framework, wherein the multi-threaded computing framework sets a working thread pool and a task queue, and distributes the local three-dimensional mesh to different working threads according to the spatial position, each working thread calculates the vertex distance and the normal vector angle between the assigned three-dimensional mesh and the reference model, and combines the calculated distance values ​​and angle values ​​to generate a reconstruction deviation value matrix; A task scheduling system is established, which receives the reconstructed deviation value matrix, prioritizes the working threads according to the deviation value, allocates computing tasks in high deviation value areas to high-performance computing nodes, builds a load balancing module to monitor the resource occupancy rate of each computing node, and dynamically adjusts the allocation strategy of computing tasks.

7. The method for adaptively generating a three-dimensional model based on video and point cloud data according to claim 1, characterized in that: The method of allocating a computing node based on the deviation value, inputting the local three-dimensional mesh into the computing node for material mapping and lighting calculation, and updating the calculation result of the computing node to the virtual scene according to the structure of the hierarchical index tree includes: Constructing a computing node resource pool, wherein the computing node resource pool manages computing nodes in groups according to the deviation value, and assigns local three-dimensional grids with larger deviation values ​​to a high-computing-power computing node group, wherein the computing node group constructs a texture mapping processing unit and a ray tracing processing unit, wherein the texture mapping processing unit projects texture information in image data onto a three-dimensional grid surface, and the ray tracing processing unit calculates ambient lighting and shadow effects; A scene update controller is established, which organizes the rendering results output by the computing nodes according to the hierarchical structure of the hierarchical index tree, constructs a scene buffer to store rendering data at different levels, the scene buffer maintains the data version number to ensure the synchronization of the rendering results, and writes the rendering results into the virtual scene display buffer for display update.

8. A device for adaptively generating a three-dimensional model based on video and point cloud data, characterized in that: The device comprises: A correspondence determination module is used to arrange a multi-view camera array and a depth scanning array in a scene space, write the image data collected by the multi-view camera array and the point cloud data collected by the depth scanning array into a data buffer matrix according to timestamps, construct a hierarchical feature extraction network to extract feature vectors from the image data and the point cloud data respectively, construct an attention weight calculation module in combination with spatial geometric constraints, input the feature vector into the attention weight calculation module to obtain feature weights, train a deep mapping model based on the feature weights, and determine the correspondence of feature points according to a confidence screening threshold; A data evaluation module is used to divide the image data and point cloud data into local data blocks, construct a hierarchical index tree based on a union-find set to store the local data blocks, reconstruct the local data blocks into local three-dimensional grids using a surface reconstruction algorithm, establish a model evaluation index calculation module, input the local three-dimensional grid into the evaluation index calculation module to obtain a completeness score and an accuracy score, input the score into a parameter adjustment controller to generate a parameter optimization strategy, and write the local three-dimensional grid into a virtual space using a hierarchical compression method; An adaptive adjustment module is used to construct a parallel computing channel, calculate the reconstruction deviation value of the local three-dimensional grid in the computing channel, transfer the deviation value to a computing resource scheduler, allocate computing nodes based on the deviation value, input the local three-dimensional grid into the computing node for material mapping and lighting calculation, and update the calculation results of the computing node to the virtual scene according to the structure of the hierarchical index tree.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for adaptively generating a three-dimensional model based on video and point cloud data as described in any one of claims 1 to 7 are implemented.

Images