Real-time three-dimensional reconstruction method based on online evolutionary learning

Abstract

The invention provides a real-time three-dimensional reconstruction method based on online evolutionary learning, and relates to the technical field of three-dimensional reconstruction methods. The real-time three-dimensional reconstruction method based on online evolutionary learning specifically comprises the following steps: S1, online evolutionary learning, S101, pseudo-label generation: adopting a pseudo-label generation algorithm to generate reliable labels for unlabeled data; s102, updating a self-adaptive model, and dynamically adjusting model parameters according to real-time feedback; s103, sensing and optimizing the environmental resources; s104, carrying out continuous optimization and evolution; s105, terminal output and real-time three-dimensional reconstruction output are carried out; s2, self-adaptive three-dimensional reconstruction; and S3, dynamic and static cooperation and heterogeneous model fusion. According to the method, self-adaptive reconstruction can be carried out under an unsupervised learning and reinforcement learning framework, so that a reconstruction model can be automatically optimized in a dynamic environment, dependence on labeled data is reduced, and complexity and precision of a reconstruction algorithm are adjusted in real time according to computing power and network conditions so as to ensure efficient operation of a system under different resource conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction methods, and in particular to a real-time three-dimensional reconstruction method based on online evolutionary learning. Background Art

[0002] Currently, 3D reconstruction technology is widely used in the field of computer vision for scenarios such as virtual reality, industrial inspection, and intelligent manufacturing. Traditional 3D reconstruction methods, such as point cloud registration or structured light-based technologies, usually rely on a large amount of labeled data and training models in static environments to ensure reconstruction accuracy. However, these methods lack adaptability in dynamic environments or new scenarios, requiring retraining of the model, which increases labor costs and computational overhead.

[0003] The main defects of existing technologies include: dependence on labeled data. Traditional methods require a large amount of labeled data for training, which has high labor costs. Lack of adaptability. Inability to quickly adapt to dynamic environments or new scenarios, requiring retraining of models. Low resource utilization efficiency. Under conditions of limited computing power or network conditions, reconstruction accuracy is difficult to adjust dynamically, affecting real-time performance and stability.

[0004] In contrast, some studies have attempted to perform adaptive reconstruction under the framework of unsupervised learning and reinforcement learning, but the stability in dealing with changing environments and limited resources is still insufficient. Therefore, the existing technology urgently needs a 3D reconstruction method that can continuously optimize in a dynamic environment, does not require a large amount of labeled data, and can adaptively adjust the reconstruction accuracy. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a real-time three-dimensional reconstruction method based on online evolutionary learning, which can automatically optimize the reconstruction model in a dynamic environment, reduce the dependence on labeled data, and adjust the complexity and accuracy of the reconstruction algorithm in real time according to computing power and network conditions to ensure the efficient operation of the system under different resource conditions.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a real-time three-dimensional reconstruction method based on online evolutionary learning, specifically comprising the following steps: S1. Online Evolutionary Learning S101. Pseudo label generation, using a pseudo label generation algorithm to generate reliable labels for unlabeled data; S102. Adaptive model update, dynamically adjusting model parameters according to real-time feedback; S103. Environmental resource perception and optimization; S104.Continuous optimization and evolution; S105. Terminal output, real-time 3D reconstruction output; S2. Adaptive 3D reconstruction S201. Design of system resource monitoring module, which includes computing power monitoring and network monitoring; S202. Computing resource and network status evaluation: Based on the aforementioned monitoring data, the system evaluates the current computing resources and network status to determine how to adjust the complexity and accuracy of the 3D reconstruction algorithm; S203. Dynamically adjust the complexity and accuracy of the 3D reconstruction algorithm. According to the current computing resources and network conditions, the system will dynamically adjust each aspect of the 3D reconstruction to ensure optimal real-time performance and accuracy. S204. Adaptive adjustment and feedback; S3. Dynamic and static coordination and heterogeneous model fusion S301. Fusion mechanism of dynamic and static data; S302. Combination of dynamic and static synergy and evolutionary learning; S303. Fusion of heterogeneous models; S304. Feedback and real-time optimization.

[0007] Preferably, the pseudo-label generation includes model confidence calculation, multimodal consistency verification and pseudo-label update, and the adaptive model update includes error feedback, complexity adjustment and feature weight adjustment.

[0008] Preferably, the environmental resource perception and optimization include computing power monitoring and dynamic decision-making, and the continuous optimization and evolution strategy include fitness function design, online evolution strategy and model pool management.

[0009] Preferably, the computing power monitoring includes CPU load monitoring, GPU load monitoring and memory usage monitoring, the network monitoring includes bandwidth monitoring and delay monitoring, and the computing resource and network status evaluation includes computing power evaluation and network evaluation.

[0010] Preferably, the dynamic adjustment of the complexity and accuracy of the three-dimensional reconstruction algorithm includes image and depth data preprocessing, dynamic adjustment of the model reasoning process and reconstruction accuracy control, and the adaptive adjustment and feedback include a real-time feedback mechanism and dynamic optimization.

[0011] Preferably, the fusion mechanism of the dynamic and static data includes the registration of dynamic and static data and weighted fusion of dynamic and static information, the registration of dynamic and static data includes feature matching and spatial alignment, and the weighted fusion of dynamic and static information includes weight calculation and weighted fusion of dynamic information.

[0012] Preferably, the combination of dynamic-static collaboration and evolutionary learning includes dynamic-static collaboration mechanism, online evolutionary learning combined with dynamic-static collaboration and dynamic-static collaboration accelerated convergence, the dynamic-static collaboration mechanism includes dynamic data accelerated convergence and static data stabilization model, the online evolutionary learning combined with dynamic-static collaboration includes the application of evolutionary learning and evolutionary strategy, the dynamic-static collaboration accelerated convergence includes physical constraints and continuity and collaborative feedback.

[0013] Preferably, the fusion of heterogeneous models includes heterogeneous model definition, fusion of heterogeneous models and three-dimensional reconstruction after fusion, the heterogeneous model definition includes convolutional neural network, PointNet, LSTM / RNN and graph neural network, the fusion of heterogeneous models includes multimodal fusion network, layered fusion and weighted fusion, and the three-dimensional reconstruction after fusion includes global optimization and local update.

[0014] Preferably, the feedback and real-time optimization include a real-time feedback mechanism and a dynamic-static coordinated adaptive optimization, the real-time feedback mechanism includes error feedback and weight adjustment, and the dynamic-static coordinated adaptive optimization includes an optimization strategy and real-time adjustment.

[0015] The present invention provides a real-time 3D reconstruction method based on online evolutionary learning. It has the following beneficial effects: 1. The present invention provides a real-time three-dimensional reconstruction method based on online evolutionary learning. When the method is used, it can perform adaptive reconstruction under the framework of unsupervised learning and reinforcement learning, so that it can automatically optimize the reconstruction model in a dynamic environment, reduce dependence on labeled data, and adjust the complexity and accuracy of the reconstruction algorithm in real time according to computing power and network conditions to ensure the efficient operation of the system under different resource conditions. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The embodiment of the present invention provides a real-time 3D reconstruction method based on online evolutionary learning, which specifically includes the following steps: S1. Online Evolutionary Learning S101. Pseudo label generation, using a pseudo label generation algorithm to generate reliable labels for unlabeled data; S102. Adaptive model update, dynamically adjusting model parameters according to real-time feedback; S103. Environmental resource perception and optimization; S104.Continuous optimization and evolution; S105. Terminal output, real-time 3D reconstruction output.

[0018] Specifically, the 3D reconstruction system uses online evolutionary learning (OEL) to dynamically adjust and optimize the model. The system includes the following modules: Input data module: multi-view and multi-modal data input, including RGB images, depth information, etc. Feature extraction and fusion: Multi-modal data is fused through a deep neural network to extract features that are helpful for reconstruction. 3D reconstruction model: The model generates a 3D reconstruction structure based on the input feature data and continuously optimizes it to adapt to different scene changes and resource conditions. The core of the OEL mechanism is to analyze and adaptively update the model output through real-time feedback. The following are its key steps: Multi-level feedback optimization: During the 3D reconstruction process, the system monitors the reconstruction accuracy, system computing power, and network conditions in real time. Each reconstruction output is compared with the actual data of the current scene to calculate the error. The feedback mechanism adaptively adjusts the model parameters according to environmental changes. For example, when computing resources are limited, the system will automatically reduce the algorithm complexity and accuracy requirements. Online learning of unlabeled data: The system uses unlabeled data for self-supervised learning and continuously optimizes the model by generating pseudo labels. Pseudo-label generation is based on the confident prediction of the current model and the correlation of multimodal features. Continuous optimization and adjustment: When environmental changes or increased reconstruction errors are detected, the system will start the adaptive update process. Model structure adjustment: Dynamically enable or disable certain neural network layers to adapt to different computing power conditions. Feature weight adjustment: Through the self-supervised learning process, the fusion weights of multimodal features are readjusted to make the model more sensitive to key features.

[0019] S2. Adaptive 3D reconstruction S201. Design of system resource monitoring module, which includes computing power monitoring and network monitoring; S202. Computing resource and network status evaluation: Based on the aforementioned monitoring data, the system evaluates the current computing resources and network status to determine how to adjust the complexity and accuracy of the 3D reconstruction algorithm; S203. Dynamically adjust the complexity and accuracy of the 3D reconstruction algorithm. According to the current computing resources and network conditions, the system will dynamically adjust each aspect of the 3D reconstruction to ensure optimal real-time performance and accuracy. S204. Adaptive adjustment and feedback.

[0020] Specifically, the system computing power and network status are monitored in real time, and the complexity and accuracy of the 3D reconstruction algorithm are dynamically adjusted. According to the computing resource status detected in real time, the reconstruction accuracy is automatically reduced or improved to ensure stability and real-time performance under different computing resource conditions. Through the monitoring module embedded in the system, the system computing power information (such as CPU / GPU load) and network transmission status (such as bandwidth delay) are collected in real time. According to the current computing resource status, the complexity and accuracy of the 3D reconstruction algorithm are dynamically adjusted. When the system detects that the computing power is sufficient, the accuracy of the model is improved to generate high-resolution 3D reconstruction results. When the computing power is limited or the network bandwidth is low, the reconstruction accuracy is reduced to ensure the real-time and continuity of the reconstruction process. Construction of the system monitoring module, monitoring computing power: evaluate the current system computing power by periodically collecting parameters such as CPU and GPU utilization and temperature. These data are unified and summarized to quantify the current resource status. Monitoring network status: evaluate the network status by detecting the system's network bandwidth and delay. This can be achieved using tools such as ping and network bandwidth measurement API. Data format and update frequency: store the monitored data in the status table regularly, for example, refresh it once per second to ensure that the model can respond quickly to resource changes. Resource evaluation model: Based on the data from the monitoring module, a resource evaluation model is established. Resource conditions can be divided by setting thresholds, such as "sufficient computing power", "medium resources", and "resources are limited". Decision tree of the adjustment mechanism: Build a simple decision tree model to dynamically adjust the complexity and accuracy of 3D reconstruction according to the output of the resource evaluation model. Sufficient computing power: Select high-precision mode, use high-resolution grids and multi-level textures to obtain detailed reconstruction results. Medium resources: Switch to medium-precision mode, control the resolution of grids and textures, and ensure a certain reconstruction effect without affecting real-time performance. Limited resources: Select low-precision mode to speed up processing by reducing the number of grid points and simplifying texture data to ensure smooth output of results. Algorithm implementation: You can use multiple pre-trained models or a single model with different parameter configurations to meet the needs of different complexity and accuracy, and switch when computing resources change.

[0021] S3. Dynamic and static coordination and heterogeneous model fusion S301. Fusion mechanism of dynamic and static data; S302. Combination of dynamic and static synergy and evolutionary learning; S303. Fusion of heterogeneous models; S304. Feedback and real-time optimization.

[0022] Specifically, in the 3D reconstruction process, dynamic data (such as real-time capture data of moving objects) and static prior information (such as fixed structure data of the scene) are combined to improve the reconstruction accuracy by fusing data from different sources. The dynamic-static synergy mechanism can accelerate the convergence speed of online evolutionary learning and improve the overall performance of reconstruction. Dynamic data capture is conducive to obtaining real-time changing environmental information, while static prior information (such as the structure of fixed objects in the room) helps the model provide a stable reference when dynamic data is missing. Multimodal data fusion technology includes using convolutional neural networks (CNNs) to process visual data, combining point cloud data for spatial alignment and fusion, and generating more accurate 3D models. In the online evolutionary learning process, dynamic data is used to optimize the short-term responsiveness of the model, while static data is used to optimize the long-term reconstruction effect. Physical rules (such as object continuity and deformation constraints) are used to accelerate the convergence of the learning process and improve the stability of the reconstructed model.

[0023] Establishment of dynamic and static coordination mechanism Dynamic-static collaboration combines dynamic data (change information captured in real time) and static priors (such as fixed scene structure) to provide a comprehensive source of information for the 3D reconstruction process, thereby improving the system's response speed and accuracy.

[0024] Implementation steps Dynamic data capture: Use cameras, laser scanners or other sensors to collect real-time dynamic data, such as the position, speed, direction, etc. of moving objects. This data helps the model adapt to environmental changes in real time.

[0025] Keyframe extraction technology is used to filter out representative data points to reduce data redundancy and improve the real-time performance of the system.

[0026] Acquisition of static prior information: Obtain static prior data through scanning or existing scene modeling, such as fixed structures in the room (walls, tables and chairs, etc.).

[0027] Static data is stored as a priori maps or point clouds as a reference frame for the model to provide a stable reconstruction benchmark when dynamic data is missing.

[0028] Dynamic and static data fusion strategy: During 3D reconstruction, the system dynamically calls real-time data and superimposes static prior data to form a dynamic and static coordinated fusion effect.

[0029] Adjust the weights of dynamic and static data as needed. Dynamic data takes up the majority of the weight, but when the environment changes suddenly or data is missing, static priors can provide a reference for the model to ensure the continuity and stability of reconstruction.

[0030] Multimodal data fusion refers to the fusion of data from different sensors (such as visual data and point cloud data) to create a high-precision 3D model. Common multimodal data fusion methods include CNN visual processing, point cloud alignment and spatial fusion.

[0031] Visual data processing: A convolutional neural network (CNN) is used to process the video stream from the camera and extract deep features to reconstruct the shape and surface details of the object.

[0032] A multi-layer convolutional network structure is designed in the model, and visual features are used to extract information such as the edge and shape of the target object to provide basic data for three-dimensional reconstruction.

[0033] Point cloud data processing and spatial alignment: Use sensors such as laser scanners and LiDAR to obtain point cloud data, and perform preprocessing (such as noise reduction and downsampling) to improve data quality.

[0034] Alignment of point cloud and visual data: Use feature matching algorithms (such as ICP algorithm, Iterative Closest Point) to spatially align the 3D points generated by visual data with the point cloud data.

[0035] Heterogeneous data fusion: Establish a data fusion model to merge visual features with point cloud information. A multi-layer network structure can be used, with the front layer responsible for visual feature extraction and the back layer responsible for point cloud alignment and deep fusion.

[0036] The fusion model uses an adaptive weighting method to dynamically adjust the weights based on the reliability of the data sources to produce the most accurate 3D model.

[0037] In online evolutionary learning, dynamic and static data optimize the short-term and long-term effects of the model respectively, thereby improving the system's adaptability to new scenarios. By combining physical rules with data fusion, the convergence of the learning process can be further accelerated.

[0038] Dynamic data optimizes short-term responsiveness: Based on dynamic data captured in real time (such as the motion trajectory of an object), a fitness function is designed to evaluate the short-term performance of the model, such as accuracy and real-time performance.

[0039] Through online evolution strategy, model parameters are dynamically updated to adapt to current scene changes.

[0040] Static data optimizes long-term effects: Incorporate static prior information into long-term fitness evaluation to optimize the long-term stability of the model.

[0041] Static data helps the model form a solid three-dimensional structural foundation. When dynamic data is temporarily missing, static priors can prevent model degradation and ensure the continuity of reconstruction accuracy.

[0042] Introducing physical constraints to accelerate convergence: Combined with physical rules (such as object continuity and deformation constraints), the model parameters are screened and evolved to prevent instability caused by parameter randomness.

[0043] During the online evolution process, position continuity constraints are added to make the reconstruction results generated by the model conform to the physical reality and accelerate learning convergence.

[0044] Pseudo-label generation includes model confidence calculation, multimodal consistency verification and pseudo-label update. Adaptive model update includes error feedback, complexity adjustment and feature weight adjustment. Environmental resource perception and optimization include computing power monitoring and dynamic decision-making. Continuous optimization and evolutionary strategies include fitness function design, online evolutionary strategies and model pool management.

[0045] Specifically, model confidence calculation: filter high confidence areas based on the prediction confidence of the current model (such as Softmax probability distribution). Multimodal consistency verification: verify the feature consistency of RGB images and depth data to enhance the accuracy of pseudo labels. Pseudo label update: dynamically update the pseudo label library, remove low-confidence labels, and retain high-quality pseudo labels. Dynamically adjust model parameters based on real-time feedback: Error feedback: calculate the error between real-time scene data and reconstruction results , analyze the model performance. Complexity adjustment: If the error , reduce the number of network layers and the number of convolution kernels. And the computing power is sufficient, increase the model complexity to optimize the accuracy. Feature weight adjustment: Adaptively adjust the modal weight according to the pseudo-label and environmental changes. Computing power monitoring: Regularly monitor the CPU / GPU utilization and evaluate the computing power status. Store resource data as a time series and provide it to the optimization module in real time. Dynamic decision-making: Build an optimization strategy based on a decision tree and select the corresponding model configuration according to the computing power status: Sufficient computing power: Enable high-precision mode. Limited computing power: Switch to low-complexity mode to ensure real-time performance. Fitness function design: The fitness function is used to measure model performance and includes the following indicators: Accuracy (reconstruction error ). Real-time (model inference time). Stability (quality and quantity of pseudo labels). Online evolution strategy: Use genetic algorithm (or other evolutionary algorithm) to evolve the model: Population initialization: Initialize several model individuals, each corresponding to a different network configuration. Fitness evaluation: Select the best performing model individuals according to the fitness function. Crossover and mutation: Use new samples generated by pseudo labels to cross-train the model. Dynamically adjust network parameters to cope with scene changes. Individual update: save the best performing model individuals and use them to replace low-fitness individuals. Model pool management: build a model pool, call it in real time according to environmental requirements, save the newly generated model to the pool, and retain the best performing previous models. The optimized model Generate high-precision 3D reconstruction results. They can be output as point cloud data, 3D mesh or surface model for subsequent applications (such as VR or industrial inspection).

[0046] Computing power monitoring includes CPU load monitoring, GPU load monitoring and memory usage monitoring; network monitoring includes bandwidth monitoring and latency monitoring; computing resource and network status assessment includes computing power assessment and network assessment; dynamic adjustment of the complexity and accuracy of the 3D reconstruction algorithm includes image and depth data preprocessing, dynamic adjustment of the model inference process and reconstruction accuracy control; adaptive adjustment and feedback include real-time feedback mechanism and dynamic optimization.

[0047] Specifically, CPU load monitoring: regularly obtain CPU load information, usually through the system API (such as psutil) to obtain the current CPU usage. High load (>90%): means that the system computing resources are limited and the complexity of the algorithm must be reduced. Medium load (70% - 90%): medium-precision algorithms can be used. Low load (<70%): high-precision reconstruction algorithms can be enabled.

[0048] GPU load monitoring: Get GPU usage in real time through APIs (such as nvidia-smi). High load (>90%): Indicates that GPU computing resources are saturated. Reduce the amount of computing and the depth of the model. Medium load (70% - 90%): Suitable for models with medium precision. Low load (<70%): When resources are sufficient, complex deep learning models can be used for high-precision 3D reconstruction.

[0049] Memory usage monitoring: Monitor the system's memory usage to ensure that the system has enough memory to process 3D data. High memory usage (>80%): When memory pressure is high, the reconstruction accuracy should be reduced to avoid memory overflow.

[0050] Bandwidth: Check the current network bandwidth to ensure that it can meet the needs of real-time data flow when transmitting multi-view images. Good bandwidth (>50Mbps): It can transmit high-resolution images and a large amount of point cloud data to ensure fine calculations during the reconstruction process. Medium bandwidth (20-50Mbps): It is suitable for medium-precision data transmission and processing. Low bandwidth (<20Mbps): When the network is limited, it is necessary to reduce the amount of data transmission, such as using low-resolution images and simplifying point cloud data.

[0051] Latency: Network latency affects the speed of data exchange, which is particularly important for real-time 3D reconstruction. Low latency (<50ms): Data can be exchanged in real time, maintaining high-precision calculations. Medium latency (50ms - 100ms): The latency is moderate, and a medium-precision reconstruction method can be selected. High latency (>100ms): When the latency is high, a more simplified reconstruction method must be used to ensure real-time performance.

[0052] Computing power assessment: Sufficient resources: If the CPU load and GPU load are both less than 70%, it means that the computing resources are sufficient to handle high-precision 3D reconstruction tasks, and the system can use complex algorithms such as deep neural networks. Medium resources: When the CPU and GPU loads are between 70% and 90%, it means that the system is in a medium load state. At this time, the system can appropriately simplify the algorithm and reduce the amount of calculation (such as reducing the number of network layers and using smaller convolution kernels) to ensure the real-time performance of the algorithm. Resource-limited: If the CPU or GPU load exceeds 90%, the system needs to greatly simplify the 3D reconstruction algorithm (such as reducing the image resolution and reducing the number of feature extraction layers) to avoid overload.

[0053] Network evaluation: Good bandwidth and low latency: When the bandwidth is greater than 50Mbps and the latency is less than 50ms, it means that the data transmission is good and can support high-precision 3D reconstruction algorithms using high-resolution images and point cloud data. Medium bandwidth and latency: If the bandwidth is 20-50Mbps and the latency is 50-100ms, the system can choose medium-precision reconstruction to reduce the amount of data transmission and reduce computing pressure. Low bandwidth and high latency: When the bandwidth is less than 20Mbps and the latency exceeds 100ms, the system must reduce the frequency and accuracy of data transmission to ensure the real-time and stability of reconstruction.

[0054] Image and depth data preprocessing: Image resolution: Adjust the image resolution based on the computing power and network bandwidth. Use high-resolution images when resources are sufficient, and reduce the image resolution when resources are limited, such as from 1080p to 720p. Depth data accuracy: For low-bandwidth environments, the accuracy of the depth map can be reduced by reducing the sampling rate of each depth point. For example, reduce the point density of the depth image to reduce the amount of data.

[0055] Dynamic adjustment of the model inference process: Network depth adjustment: When resources are sufficient, use deep neural networks (such as ResNet, DenseNet) to reconstruct three-dimensional structures with high precision. When resources are limited, use lightweight networks (such as MobileNet or lightweight ResNet) to reduce the amount of calculation. Convolution kernel size: When computing power is limited, use smaller convolution kernels (such as 3x3) instead of larger convolution kernels (such as 5x5) to reduce the amount of calculation for each convolution operation. Network layer adjustment: By dynamically adjusting the number of layers or depth of the network, the number of layers can be reduced when resources are limited, shortening the inference time and ensuring real-time performance.

[0056] Reconstruction accuracy control: High-precision mode: When system resources are sufficient, use high-resolution mesh generation, texture refinement, and multi-level reconstruction algorithms to obtain a detailed 3D model. Low-precision mode: When system resources are limited, reduce mesh resolution and calculation details to ensure smoothness and real-time reconstruction. Resolution selection: In a low-bandwidth environment, reduce the amount of data transmitted, such as using lower-resolution images and simpler point cloud data.

[0057] Real-time feedback mechanism: Reconstruction accuracy feedback: Feedback is provided based on the accuracy (such as error, clarity and other indicators) and real-time performance (such as processing time) of the reconstruction results. If the reconstruction accuracy is low, the system will automatically adjust the complexity of the reconstruction algorithm and increase computing resources. Resource usage feedback: When the computing resource pressure is high, the system will reduce the accuracy requirements and reduce the network complexity. Otherwise, it will return to high-precision mode.

[0058] Dynamic optimization: Model optimization: Based on real-time data feedback, continuously optimize the parameter settings of the 3D reconstruction model to gradually adapt to different computing environments and data changes. Algorithm self-adjustment: By analyzing the performance under different loads, the system can automatically adjust the algorithm configuration to gradually improve the stability and accuracy under different conditions.

[0059] The fusion mechanism of dynamic and static data includes the registration of dynamic and static data and the weighted fusion of dynamic and static information. The registration of dynamic and static data includes feature matching and spatial alignment. The weighted fusion of dynamic and static information includes weight calculation and weighted fusion of dynamic information. The combination of dynamic and static collaboration and evolutionary learning includes dynamic and static collaboration mechanism, online evolutionary learning combined with dynamic and static collaboration and dynamic and static collaboration accelerated convergence. The dynamic and static collaboration mechanism includes dynamic data accelerated convergence and static data stable model. The combination of online evolutionary learning and dynamic and static collaboration includes the application of evolutionary learning and evolutionary strategy. Dynamic and static collaboration accelerated convergence includes physical constraints and continuity and collaborative feedback. The fusion of heterogeneous models includes heterogeneous model definition, fusion of heterogeneous models and 3D reconstruction after fusion. The definition of heterogeneous models includes convolutional neural network, PointNet, LSTM / RNN and graph neural network. The fusion of heterogeneous models includes multimodal fusion network, layered fusion and weighted fusion. The 3D reconstruction after fusion includes global optimization and local update. Feedback and real-time optimization include real-time feedback mechanism and adaptive optimization of dynamic and static collaboration. The real-time feedback mechanism includes error feedback and weight adjustment. The adaptive optimization of dynamic and static collaboration includes optimization strategy and real-time adjustment.

[0060] Specifically, in order to achieve effective fusion of dynamic and static data, the spatial alignment between dynamic data and static data must be ensured, which is usually achieved through a registration algorithm.

[0061] Feature matching: Use feature matching algorithms (such as SIFT, SURF, ORB, etc.) to extract feature points from the image and align these feature points with the static point cloud data.

[0062] Spatial alignment: The dynamic point cloud and the static point cloud are spatially aligned through methods such as the ICP (Iterative Closest Point) algorithm, combining the motion trajectory of the dynamic object with the fixed structure of the scene.

[0063] Weighted Fusion of Dynamic and Static Information Weight calculation of dynamic information: Dynamic data (such as the coordinates and speed of moving objects) and static data (such as walls, tables and chairs) should be weighted according to their credibility, stability and other factors when they are fused. The weight of dynamic information can be adjusted according to the speed and direction of the object's movement, while the weight of static information is provided by the fixed structure of the scene.

[0064] Weighted fusion: The dynamic and static data are fused according to the weights, and a comprehensive data model is obtained by weighted average or weighted summation. This step can help balance the amount of information between dynamic objects and static backgrounds and reduce the error impact of dynamic objects.

[0065] Dynamic and static coordination mechanism The key to dynamic and static coordination is to use an effective fusion mechanism to enable dynamic data to play a complementary role in static scenes and improve the accuracy of three-dimensional reconstruction.

[0066] Dynamic data accelerates convergence: Dynamic data can provide real-time feedback to help the model quickly adapt to environmental changes. Especially in 3D reconstruction in a dynamic environment, the motion information of dynamic objects can help the model accelerate convergence.

[0067] Static data stabilizes the model: Static data provides a stable benchmark for 3D reconstruction, ensuring that the model can maintain high accuracy and stability in a dynamically changing environment. The fusion of static prior data can help mitigate the overfitting phenomenon that may occur during the model convergence process.

[0068] Online evolutionary learning combined with dynamic and static synergy In the 3D reconstruction process, the Online Evolutionary Learning (OEL) mechanism can be used to improve the accuracy and stability of reconstruction by continuously optimizing model parameters.

[0069] Application of evolutionary learning: Optimizing model parameters through online evolutionary algorithms (such as genetic algorithms and particle swarm optimization) enables the model to adapt to changes in dynamic data while maintaining robust processing of static data.

[0070] Evolutionary strategy: During each reconstruction process, the system adjusts the evolutionary strategy based on the dynamic information of the current scene (such as the trajectory changes of moving objects) and static data (such as the changes in structural information). For example, when the system detects the speed change of a dynamic object, online evolutionary learning can adaptively adjust the influence weight of the moving object in the reconstruction model.

[0071] Dynamic and static collaboration accelerates convergence Physical constraints and continuity: Accelerate the convergence of dynamic data by introducing physical constraints (such as object continuity, rigid body transformation, deformation constraints, etc.). For example, when dynamic objects in the model move, physical constraints can prevent unreasonable jumps or discontinuities.

[0072] Collaborative feedback: By monitoring the reconstruction errors of dynamic and static data in real time, the system can achieve adaptive adjustment of dynamic data. When the system detects that the error of a dynamic object is large, it will automatically increase the weight of static data, thereby balancing the contribution of dynamic and static data to the 3D reconstruction model.

[0073] Heterogeneous model definition In the process of dynamic and static collaboration, different types of data need to be processed by different models. The fusion of heterogeneous models refers to combining multiple different types of networks or algorithms to process different types of data (for example, images, point clouds, motion data, etc.).

[0074] Convolutional Neural Network (CNN): used to process image data and extract deep features and semantic information.

[0075] PointNet: used to process point cloud data and extract spatial structure information.

[0076] LSTM / RNN: used to process time series data (such as motion trajectories) and capture the time-varying information of dynamic objects.

[0077] Graph Neural Network (GNN): used to fuse multimodal data and process global and local information in the scene.

[0078] Fusion of heterogeneous models Multimodal fusion network: Design a multimodal network (e.g., a multi-input convolutional neural network) in which each input module processes different types of data (images, point clouds, motion data, etc.) and integrates the information in subsequent fusion layers.

[0079] Hierarchical fusion: Heterogeneous data is fused at different levels of the model. For example, the low-level layer uses a convolutional network to extract image features, the middle-level layer fuses dynamic information and static point cloud data, and the high-level layer fuses global information to generate the final 3D reconstruction model.

[0080] Weighted fusion: Assign different weights to the output of each model based on the importance of different data sources (e.g. image data is more stable than point cloud data, dynamic data is more sensitive than static data). The weight of dynamic data can be adjusted according to the speed of the object, while the weight of static data remains stable.

[0081] 3D reconstruction after fusion Global optimization: By globally optimizing the outputs of multiple heterogeneous models, a high-precision 3D reconstruction result is finally generated. In this process, by integrating the advantages of dynamic data and static data, the reconstruction effect can achieve a higher accuracy.

[0082] Local Update: For specific areas or areas with large dynamic changes, dynamic data provides local fine updates to maintain high accuracy of the entire reconstructed model.

[0083] Real-time feedback mechanism Error feedback: After each reconstruction, the error between the reconstruction result and the actual scene is calculated, and the model parameters are adjusted according to the error size. Areas with large errors can be optimized using more dynamic data or more physical constraints.

[0084] Weight adjustment: Automatically adjust the fusion weight of dynamic data and static data according to the feedback results. For example, when a large reconstruction error of a dynamic object is detected, the weight of static data is increased to reduce the impact of fluctuations in dynamic data.

[0085] Adaptive optimization of dynamic and static collaboration Optimization strategy: In different scenarios, the system automatically selects the appropriate optimization strategy based on the changes in current data. For example, in a relatively static scenario, static data has a greater weight; in a dynamic scenario, dynamic data has a greater weight.

[0086] Real-time adjustment: As the scene changes and objects move, the system will adjust the model parameters in real time to optimize the dynamic and static coordination process.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time 3D reconstruction method based on online evolutionary learning, characterized in that: The specific steps include: S1. Online Evolutionary Learning S101. Pseudo label generation, using a pseudo label generation algorithm to generate reliable labels for unlabeled data; S102. Adaptive model update, dynamically adjusting model parameters according to real-time feedback; S103. Environmental resource perception and optimization; S104.Continuous optimization and evolution; S105. Terminal output, real-time 3D reconstruction output; S2. Adaptive 3D reconstruction S201. Design of system resource monitoring module, which includes computing power monitoring and network monitoring; S202. Computing resource and network status evaluation: Based on the aforementioned monitoring data, the system evaluates the current computing resources and network status to determine how to adjust the complexity and accuracy of the 3D reconstruction algorithm; S203. Dynamically adjust the complexity and accuracy of the 3D reconstruction algorithm. According to the current computing resources and network conditions, the system will dynamically adjust each aspect of the 3D reconstruction to ensure optimal real-time performance and accuracy. S204. Adaptive adjustment and feedback; S3. Dynamic and static coordination and heterogeneous model fusion S301. Fusion mechanism of dynamic and static data; S302. Combination of dynamic and static synergy and evolutionary learning; S303. Fusion of heterogeneous models; S304. Feedback and real-time optimization.

2. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The pseudo-label generation includes model confidence calculation, multimodal consistency verification and pseudo-label update, and the adaptive model update includes error feedback, complexity adjustment and feature weight adjustment.

3. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The environmental resource perception and optimization include computing power monitoring and dynamic decision-making, and the continuous optimization and evolution strategy include fitness function design, online evolution strategy and model pool management.

4. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The computing power monitoring includes CPU load monitoring, GPU load monitoring and memory usage monitoring, the network monitoring includes bandwidth monitoring and latency monitoring, and the computing resource and network status evaluation includes computing power evaluation and network evaluation.

5. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The dynamic adjustment of the complexity and accuracy of the three-dimensional reconstruction algorithm includes image and depth data preprocessing, dynamic adjustment of the model reasoning process and reconstruction accuracy control, and the adaptive adjustment and feedback include real-time feedback mechanism and dynamic optimization.

6. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The fusion mechanism of dynamic and static data includes the registration of dynamic and static data and weighted fusion of dynamic and static information. The registration of dynamic and static data includes feature matching and space alignment. The weighted fusion of dynamic and static information includes weight calculation and weighted fusion of dynamic information.

7. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The combination of dynamic-static collaboration and evolutionary learning includes dynamic-static collaboration mechanism, online evolutionary learning combined with dynamic-static collaboration and dynamic-static collaboration accelerated convergence. The dynamic-static collaboration mechanism includes dynamic data accelerated convergence and static data stabilization model. The online evolutionary learning combined with dynamic-static collaboration includes the application of evolutionary learning and evolutionary strategy. The dynamic-static collaboration accelerated convergence includes physical constraints and continuity and collaborative feedback.

8. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The fusion of heterogeneous models includes heterogeneous model definition, fusion of heterogeneous models and three-dimensional reconstruction after fusion. The heterogeneous model definition includes convolutional neural network, PointNet, LSTM / RNN and graph neural network. The fusion of heterogeneous models includes multimodal fusion network, layered fusion and weighted fusion. The three-dimensional reconstruction after fusion includes global optimization and local update.

9. The real-time 3D reconstruction method based on online evolutionary learning according to claim 1, characterized in that: The feedback and real-time optimization include a real-time feedback mechanism and a dynamic-static coordinated adaptive optimization. The real-time feedback mechanism includes error feedback and weight adjustment. The dynamic-static coordinated adaptive optimization includes an optimization strategy and real-time adjustment.