Real-time three-dimensional greening maintenance system, method and equipment based on multi-sensor fusion and medium

Abstract

The invention provides a real-time three-dimensional greening maintenance system, method and device based on multi-sensor fusion and a medium, and the method comprises the following steps: S1, collecting multi-modal data of a target area through multiple sensors, including point cloud, environmental parameters and multi-view images; s2, performing space-time registration, noise filtering and feature fusion on the multi-modal data to generate a vegetation three-dimensional data set; s3, performing multi-scale modeling on the data set based on a Gaussian mixture model (GMM) to generate a dynamically updated vegetation three-dimensional model; s4, adopting a hierarchical data structure and a GPU parallel rendering technology to realize real-time visualization of more than 30FPS in a dynamic scene; and S5, generating a vegetation health assessment report and an automatic maintenance decision in combination with the environmental parameters and the model data, thereby solving the defects of the existing method in frame rate, illumination simulation and detail modeling, reducing the manual intervention cost, and meeting multiple requirements for real-time performance and authenticity in the actual greening maintenance process.

Description

Technical Field

[0001] The present invention relates to the technical field of greening maintenance and management, and more specifically, to a real-time three-dimensional greening maintenance system, method, equipment and medium based on multi-sensor fusion. Background Art

[0002] Traditional greening maintenance relies on manual inspections or 2D image analysis, which cannot accurately capture the 3D morphology, density, and microscopic health of vegetation. Existing 3D vegetation modeling technologies primarily rely on LiDAR scanning or Multi-View Stereo (MVS) matching, generating 3D point cloud data from 2D image sequences. However, these methods suffer from the following issues:

[0003] (1) High computational complexity: Traditional algorithms (such as COLMAP and OpenMVS) require a lot of computing resources when dealing with complex forms such as vegetation, resulting in low modeling efficiency.

[0004] (2) Insufficient detail restoration: The modeling resolution of vegetation microstructure (such as leaf vein texture and bark cracks) is limited, making it difficult to meet the needs of precise maintenance.

[0005] (3) Poor real-time performance: In dynamic scenes (such as swaying branches and leaves), the rendering frame rate of traditional methods is usually lower than 20FPS, which cannot support real-time monitoring and decision-making.

[0006] The processing method closest to the existing technology, application publication number: CN118447047A, discloses a plant image processing method, device and equipment based on three-dimensional phenotypic modeling, wherein the method includes: extracting feature points from the initial image sequence of the plant under multiple perspectives, determining its camera position, and obtaining sparse point cloud data based on the camera position using triangulation; using a tracking network improved by combining the SAM model to perform foreground segmentation on the initial image sequence under each perspective to obtain initial segmented plant foreground data, refine the edges, spatially align each perspective, and then use the tracking network to remove incomplete features to obtain accurate segmented plant foreground results for each perspective; the segmented plant foreground results for each perspective are processed by Gaussian rendering to obtain a three-dimensional plant rendering result, which is gridded into three-dimensional surface processing to obtain a three-dimensional grid surface image of the plant under each perspective, so that plant morphology analysis can be better performed based on the three-dimensional grid surface image of the plant.

[0007] This method optimizes plant phenotyping accuracy through multi-view image segmentation and Gaussian rendering. However, it relies on a complex image segmentation (SAM model) and Gaussian rendering pipeline, and lacks explicit real-time processing capabilities in dynamic scenarios. The multi-step computational process (feature extraction, segmentation, registration, rendering, etc.) results in a low frame rate, making it difficult to meet the real-time requirements of dynamic vegetation changes (such as swaying branches and leaves, and growth monitoring). Consequently, in smart agriculture or urban greening scenarios, it is impossible to update vegetation status in real time, hindering the timeliness of pest and disease warnings and automated maintenance decisions.

[0008] In view of this, the present invention proposes a real-time, scene-adaptable, and real-time three-dimensional greening maintenance system, method, equipment, and medium based on multi-sensor fusion. Summary of the Invention

[0009] In view of this, in order to solve the above problems, the present invention proposes a real-time, scene-adaptable, real-time three-dimensional greening maintenance system, method, equipment and medium based on multi-sensor fusion.

[0010] To achieve the above objectives, the present invention provides a real-time three-dimensional greening maintenance system based on multi-sensor fusion, which is characterized by comprising:

[0011] A data acquisition module is used to acquire multimodal data of the target area through at least two of a laser scanner, a depth sensor, a multi-camera, and an environmental sensor, wherein the multimodal data includes point cloud data, environmental parameters, and multi-view images;

[0012] A data fusion and preprocessing module is used to perform noise filtering, spatiotemporal registration and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset with a unified spatiotemporal reference;

[0013] A dynamic modeling module performs multi-scale modeling on the vegetation 3D dataset based on a Gaussian mixture model (GMM) to generate a dynamic 3D vegetation model that includes both macroscopic morphology and microscopic details, such as leaf vein texture, bark cracks, and pest and disease characteristics.

[0014] The real-time rendering optimization module uses hierarchical data structures, adaptive level of detail (LOD) technology, and GPU parallel computing to achieve real-time rendering at over 30 FPS in dynamic scenes and support real-time visualization of vegetation morphological changes.

[0015] The decision support module integrates three-dimensional measurement tools and environmental parameter analysis algorithms to generate vegetation health assessments, irrigation recommendations, and pest and disease warning information, which are then output through an interactive interface.

[0016] In some embodiments, the data fusion and preprocessing module includes:

[0017] Multi-sensor data registration unit based on ICP algorithm, used to align spatiotemporal data acquired by different sensors;

[0018] The noise filtering unit based on deep learning uses the convolutional neural network (CNN) to identify and remove abnormal point clouds and image noise;

[0019] The dynamic feature extraction unit uses time series analysis technology to capture vegetation growth trends and correlations with environmental parameters.

[0020] In some embodiments, the dynamic modeling module further includes:

[0021] The coarse-grained modeling unit constructs the overall skeleton model of vegetation through low-resolution Gaussian distribution;

[0022] Fine-grained modeling unit extracts leaf microtexture and canopy light transmittance characteristics, and integrates environmental sensor data to dynamically update model parameters;

[0023] The scalability enhancement unit embeds geometric coordinates, size, and color information into the model, supporting direct measurement of vegetation height, volume, and leaf area index.

[0024] In some implementations, the real-time rendering optimization module achieves efficient resource management by:

[0025] Use octree or bounding volume hierarchy (BVH) structure to perform spatial partitioning and fast cropping of large-scale point clouds;

[0026] Dynamically adjust the LOD level based on the user's viewing distance, using high-resolution rendering for close-up shots and simplified models for distant shots;

[0027] Shader programming is used to optimize the lighting model and simulate the shadow interaction and complex light transmission effects of vegetation canopies.

[0028] In addition, to achieve the above-mentioned purpose, the present invention also provides a real-time three-dimensional greening maintenance method based on multi-sensor fusion, which is characterized by comprising the following steps:

[0029] S1, collects multimodal data of the target area through multiple sensors, including point clouds, environmental parameters and multi-view images;

[0030] S2, performing spatiotemporal registration, noise filtering, and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset;

[0031] S3, based on the Gaussian mixture model GMM, multi-scale modeling of the dataset is performed to generate a dynamically updated three-dimensional vegetation model;

[0032] S4, uses hierarchical data structure and GPU parallel rendering technology to achieve real-time visualization of more than 30FPS in dynamic scenes;

[0033] S5, combines environmental parameters with model data to generate vegetation health assessment reports and automated maintenance decisions.

[0034] In some embodiments, S3 further includes: the multi-scale modeling comprising:

[0035] Perform cluster analysis on point cloud data and decompose it into multiple local feature clusters;

[0036] Macroscopic morphology modeling and microscopic detail extraction are performed on each feature cluster, and the model accuracy is dynamically corrected through environmental parameters;

[0037] Fusion of multi-scale models into a unified 3D digital twin, supporting cross-scale data interaction and visualization.

[0038] In some embodiments, S4 further includes: the real-time visualization further comprising:

[0039] Dynamically adjust rendering detail levels based on user interaction, with a response time of less than 50ms;

[0040] Achieve cross-platform visualization on mobile terminals or embedded devices through WebGL or lightweight rendering engines.

[0041] In some embodiments, S5 further includes: the automated maintenance decision-making comprising:

[0042] Predict vegetation growth trends and pest and disease risk levels based on comparative analysis of historical data and real-time models;

[0043] Combined with geographic information system (GIS) data, spatial maintenance plans are generated to guide irrigation, pruning and fertilization operations.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also provides an electronic device, including a memory, a processor and a computer program stored in the memory, characterized in that when the processor executes the program, it implements the steps of a real-time three-dimensional greening maintenance method based on multi-sensor fusion.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium storing computer instructions, characterized in that when the instructions are executed by a processor, the steps of a real-time three-dimensional greening maintenance method based on multi-sensor fusion are implemented.

[0046] Beneficial effects of the present invention:

[0047] (1) High-precision 3D modeling and detail restoration address the shortcomings of traditional technologies. Through multi-scale Gaussian point cloud modeling, the macroscopic morphology (such as trunk and canopy structure) and microscopic details (such as leaf vein texture and bark cracks) of vegetation are simultaneously captured. The modeling resolution is greatly improved compared to traditional methods, addressing the shortcomings of existing technologies in microstructure modeling. Accurately restoring vegetation morphology provides reliable data support for pest and disease identification, leaf area index calculation, etc., improving detection accuracy. It also has scalability, allowing various ecological parameters to be directly extracted from the model, providing data support for maintenance decisions.

[0048] (2) Real-time dynamic rendering and efficient resource management. By establishing a tree-like data structure and adaptive detail control, the system can significantly reduce the usage of computing and memory resources while ensuring the realism of rendering.

[0049] (3) Multi-sensor fusion and scene adaptability, improved data robustness, integrated laser scanners, depth sensors and environmental sensors, reducing dependence on a single data source (such as multi-view images), and still being able to generate complete three-dimensional models in scenes with occlusion or missing perspectives, thereby improving data coverage; dynamic environmental adaptation: combining dynamic correction models of environmental parameters such as temperature, humidity, and wind speed to support vegetation growth trend prediction and disaster warning (such as drought and frost damage), with response time shortened to less than 10 seconds.

[0050] (4) Low cost and automated maintenance decision-making. Compared with the large amount of manual intervention required by traditional manual modeling, this invention realizes automatic modeling based on Gaussian point cloud, which greatly reduces the modeling cost and improves the applicability of the system, meeting the dual requirements of realism and automation in daily greening maintenance scenarios. 5.

[0051] (5) Cross-platform deployment and scalability, lightweight application, support for running on mobile terminals and embedded devices through WebGL or lightweight rendering engines, covering multiple scenarios such as smart agriculture, urban greening, and forestry monitoring.

[0052] (6) Digital twin integration: The generated three-dimensional model can be seamlessly connected to the GIS system and the Internet of Things platform to build a digital twin of vegetation growth, providing a data basis for long-term ecological research. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of a method for generating greening maintenance plots based on road network data.

[0054] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION Example 1:

[0055] The present invention provides a real-time three-dimensional greening maintenance system based on multi-sensor fusion, which is characterized by comprising:

[0056] A data acquisition module is used to acquire multimodal data of the target area through at least two of a laser scanner, a depth sensor, a multi-camera, and an environmental sensor, wherein the multimodal data includes point cloud data, environmental parameters, and multi-view images;

[0057] A data fusion and preprocessing module is used to perform noise filtering, spatiotemporal registration and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset with a unified spatiotemporal reference;

[0058] A dynamic modeling module performs multi-scale modeling on the vegetation 3D dataset based on a Gaussian mixture model (GMM) to generate a dynamic 3D vegetation model that includes both macroscopic morphology and microscopic details, such as leaf vein texture, bark cracks, and pest and disease characteristics.

[0059] The real-time rendering optimization module uses hierarchical data structures, adaptive level of detail (LOD) technology, and GPU parallel computing to achieve real-time rendering at over 30 FPS in dynamic scenes and support real-time visualization of vegetation morphological changes.

[0060] The decision support module integrates three-dimensional measurement tools and environmental parameter analysis algorithms to generate vegetation health assessments, irrigation recommendations, and pest and disease warning information, which are then output through an interactive interface.

[0061] In some embodiments, the data fusion and preprocessing module includes:

[0062] Multi-sensor data registration unit based on ICP algorithm, used to align spatiotemporal data acquired by different sensors;

[0063] The noise filtering unit based on deep learning uses the convolutional neural network (CNN) to identify and remove abnormal point clouds and image noise;

[0064] The dynamic feature extraction unit uses time series analysis technology to capture vegetation growth trends and correlations with environmental parameters.

[0065] The dynamic modeling module further includes:

[0066] The coarse-grained modeling unit constructs the overall skeleton model of vegetation through low-resolution Gaussian distribution;

[0067] Fine-grained modeling unit extracts leaf microtexture and canopy light transmittance characteristics, and integrates environmental sensor data to dynamically update model parameters;

[0068] The scalability enhancement unit embeds geometric coordinates, size, and color information into the model, supporting direct measurement of vegetation height, volume, and leaf area index.

[0069] The real-time rendering optimization module achieves efficient resource management through the following methods:

[0070] Use octree or bounding volume hierarchy (BVH) structure to perform spatial partitioning and fast cropping of large-scale point clouds;

[0071] Dynamically adjust the LOD level based on the user's viewing distance, using high-resolution rendering for close-up shots and simplified models for distant shots;

[0072] Shader programming is used to optimize the lighting model and simulate the shadow interaction and complex light transmission effects of vegetation canopies.

[0073] In addition, an embodiment of the present invention further proposes a real-time three-dimensional greening maintenance method based on multi-sensor fusion, which is characterized by comprising the following steps:

[0074] S1, collects multimodal data of the target area through multiple sensors, including point clouds, environmental parameters and multi-view images;

[0075] S2, performing spatiotemporal registration, noise filtering, and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset;

[0076] S3, based on the Gaussian mixture model GMM, multi-scale modeling of the dataset is performed to generate a dynamically updated three-dimensional vegetation model;

[0077] S4, uses hierarchical data structure and GPU parallel rendering technology to achieve real-time visualization of more than 30FPS in dynamic scenes;

[0078] S5, combines environmental parameters with model data to generate vegetation health assessment reports and automated maintenance decisions.

[0079] Also included in S3, the multi-scale modeling includes:

[0080] Perform cluster analysis on point cloud data and decompose it into multiple local feature clusters;

[0081] Macroscopic morphology modeling and microscopic detail extraction are performed on each feature cluster, and the model accuracy is dynamically corrected through environmental parameters;

[0082] Fusion of multi-scale models into a unified 3D digital twin, supporting cross-scale data interaction and visualization.

[0083] Also included in S4, the real-time visualization further includes:

[0084] Dynamically adjust rendering detail levels based on user interaction, with a response time of less than 50ms;

[0085] Achieve cross-platform visualization on mobile terminals or embedded devices through WebGL or lightweight rendering engines.

[0086] Also included in S5 is that the automated maintenance decision includes:

[0087] Predict vegetation growth trends and pest and disease risk levels based on comparative analysis of historical data and real-time models;

[0088] Combined with geographic information system (GIS) data, spatial maintenance plans are generated to guide irrigation, pruning and fertilization operations.

[0089] In addition, an embodiment of the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the program, it implements the steps of a real-time three-dimensional greening maintenance method based on multi-sensor fusion.

[0090] In addition, an embodiment of the present invention further proposes a computer-readable storage medium storing computer instructions, wherein when the instructions are executed by a processor, the steps of a real-time three-dimensional greening maintenance method based on multi-sensor fusion are implemented.

[0091] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any actual relationship or order between these entities / operations / objects; the terms "include", "comprise", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or system that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "includes a ..." does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.

[0092] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a storage device (or server) to execute the methods described in each embodiment of the present invention.

[0094] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time three-dimensional greening maintenance system based on multi-sensor fusion, characterized by: include: A data acquisition module is used to acquire multimodal data of the target area through at least two of a laser scanner, a depth sensor, a multi-camera, and an environmental sensor, wherein the multimodal data includes point cloud data, environmental parameters, and multi-view images; A data fusion and preprocessing module is used to perform noise filtering, spatiotemporal registration and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset with a unified spatiotemporal reference; A dynamic modeling module performs multi-scale modeling on the vegetation 3D dataset based on a Gaussian mixture model (GMM) to generate a dynamic 3D vegetation model that includes both macroscopic morphology and microscopic details, such as leaf vein texture, bark cracks, and pest and disease characteristics. The real-time rendering optimization module uses hierarchical data structures, adaptive level of detail (LOD) technology, and GPU parallel computing to achieve real-time rendering at over 30 FPS in dynamic scenes and support real-time visualization of vegetation morphological changes. The decision support module integrates three-dimensional measurement tools and environmental parameter analysis algorithms to generate vegetation health assessments, irrigation recommendations, and pest and disease warning information, which are then output through an interactive interface.

2. The real-time three-dimensional greening maintenance system based on multi-sensor fusion according to claim 1 is characterized in that: The data fusion and preprocessing module includes: Multi-sensor data registration unit based on ICP algorithm, used to align spatiotemporal data acquired by different sensors; The noise filtering unit based on deep learning uses the convolutional neural network (CNN) to identify and remove abnormal point clouds and image noise; The dynamic feature extraction unit uses time series analysis technology to capture vegetation growth trends and correlations with environmental parameters.

3. The real-time three-dimensional greening maintenance system based on multi-sensor fusion according to claim 1 is characterized in that: The dynamic modeling module further includes: The coarse-grained modeling unit constructs the overall skeleton model of vegetation through low-resolution Gaussian distribution; Fine-grained modeling unit extracts leaf microtexture and canopy light transmittance characteristics, and integrates environmental sensor data to dynamically update model parameters; The scalability enhancement unit embeds geometric coordinates, size, and color information into the model, supporting direct measurement of vegetation height, volume, and leaf area index.

4. The real-time three-dimensional greening maintenance system based on multi-sensor fusion according to claim 1 is characterized in that: The real-time rendering optimization module achieves efficient resource management through the following methods: Use octree or bounding volume hierarchy (BVH) structure to perform spatial partitioning and fast cropping of large-scale point clouds; Dynamically adjust the LOD level based on the user's viewing distance, using high-resolution rendering for close-up shots and simplified models for distant shots; Shader programming is used to optimize the lighting model and simulate the shadow interaction and complex light transmission effects of vegetation canopies.

5. A real-time three-dimensional greening maintenance method based on multi-sensor fusion, characterized in that: The following steps are involved: S1, collects multimodal data of the target area through multiple sensors, including point clouds, environmental parameters and multi-view images; S2, performing spatiotemporal registration, noise filtering, and feature fusion on the multimodal data to generate a three-dimensional vegetation dataset; S3, based on the Gaussian mixture model GMM, multi-scale modeling of the dataset is performed to generate a dynamically updated three-dimensional vegetation model; S4, uses hierarchical data structure and GPU parallel rendering technology to achieve real-time visualization of more than 30FPS in dynamic scenes; S5, combines environmental parameters with model data to generate vegetation health assessment reports and automated maintenance decisions.

6. The real-time three-dimensional greening maintenance method based on multi-sensor fusion according to claim 5 is characterized in that: Also included in S3, the multi-scale modeling includes: Perform cluster analysis on point cloud data and decompose it into multiple local feature clusters; Macroscopic morphology modeling and microscopic detail extraction are performed on each feature cluster, and the model accuracy is dynamically corrected through environmental parameters; Fusion of multi-scale models into a unified 3D digital twin, supporting cross-scale data interaction and visualization.

7. The real-time three-dimensional greening maintenance method based on multi-sensor fusion according to claim 5 is characterized in that: Also included in S4, the real-time visualization further includes: Dynamically adjust rendering detail levels based on user interaction, with a response time of less than 50ms; Achieve cross-platform visualization on mobile terminals or embedded devices through WebGL or lightweight rendering engines.

8. The real-time three-dimensional greening maintenance method based on multi-sensor fusion according to claim 5 is characterized in that: Also included in S5 is that the automated maintenance decision includes: Predict vegetation growth trends and pest and disease risk levels based on comparative analysis of historical data and real-time models; Combined with geographic information system (GIS) data, spatial maintenance plans are generated to guide irrigation, pruning and fertilization operations.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 5 to 8 is implemented.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 5 to 8 is implemented.

Images