Multi-temporal data fusion compensation method and system under water surface refraction condition

Abstract

The application provides a multi-time-phase data fusion compensation method and system under water surface refraction conditions, and relates to the technical field of data processing. The method comprises the following steps: obtaining environment time sequence prediction data in a loading and unloading operation window period, performing feature analysis, and positioning a plurality of sampling time points; performing multi-modal data acquisition to obtain a plurality of data packets; performing single-time-phase refraction compensation reconstruction to obtain a plurality of single-time-phase compensation point clouds, then performing multi-time-phase weighted fusion, and outputting a high-precision reference point cloud model; calculating a coal pile volume change amount; using a same-period coal transportation account record to perform inventory verification to obtain an inventory deviation, and performing abnormal alarm. The application solves the technical problem that in the prior art, single-time-phase point cloud data is usually directly used for coal pile reconstruction, only a single scanning result is relied on to complete point cloud completion, a large-area refraction missing area formed under high-humidity environment or shielding conditions is difficult to accurately recover, and then the coal pile volume calculation error is increased, thereby affecting the inventory monitoring precision.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for multi-temporal data fusion and compensation under water surface refraction conditions. Background Technology

[0002] With the increasing demand for digital management of coal stockpiles, using lidar to perform 3D point cloud scanning of coal piles and combining the point cloud model for coal pile volume measurement and inventory monitoring has gradually become an important technical means for intelligent coal yard management. Existing technologies typically collect point cloud data from coal piles according to a preset time period, and based on the point cloud data obtained from a single collection, construct a 3D model of the coal pile through multi-view point cloud stitching, spatial registration, or point cloud fusion to achieve coal pile volume calculation and inventory analysis.

[0003] However, in actual open-pit coal yard environments, the surface condition of coal piles is easily affected by environmental factors and undergoes dynamic changes. This is especially true in areas near water ditches, during open-pit rainfall, or when the surface moisture content of the coal pile is high. These conditions can easily lead to localized water accumulation, water film reflection areas, and high humidity, causing the laser scanning process to be affected by water surface refraction, specular reflection, and changes in ambient humidity. This results in abnormal laser echo signals, reduced point cloud density, and missing data in localized areas. Since current technologies primarily rely on single-phase scanning results to reconstruct the 3D model of the coal pile, when areas with refraction loss, occlusion loss, or abnormal reflection occur during a single acquisition, they are typically handled using only local interpolation, geometric fitting, or simple point cloud completion. This can easily cause inconsistencies between the compensated area and the actual coal pile shape, leading to voids, edge distortion, and localized geometric distortion in the 3D model. This results in cumulative errors in the coal pile volume calculation results, affecting the accuracy of inventory monitoring. Summary of the Invention

[0004] This application provides a multi-temporal data fusion compensation method and system under water surface refraction conditions, aiming to solve the technical problem that the existing technology usually uses single-temporal point cloud data to directly reconstruct coal piles, relying only on the results of a single scan to complete the point cloud, which makes it difficult to accurately restore large-area refraction loss areas formed under high humidity or shading conditions, thus increasing the error in coal pile volume calculation and affecting the accuracy of inventory monitoring.

[0005] The first aspect disclosed in this application provides a multi-temporal data fusion compensation method under water surface refraction conditions. The method includes: interacting with a production scheduling terminal to obtain environmental time-series prediction data for the Kth coal pile loading / unloading operation window; performing feature analysis based on environmental humidity fluctuations to locate multiple sampling time points; triggering a multi-modal data acquisition device to perform multi-modal data acquisition on the target coal pile at the multiple sampling time points to obtain multiple data packets, wherein the data packets include fused laser point clouds, panoramic visible light images, and environmental humidity parameters; performing single-temporal refraction compensation reconstruction based on multiple fused laser point clouds and multiple panoramic visible light images to obtain multiple single-temporal compensated point clouds; combining multiple environmental humidity parameters to perform multi-temporal weighted fusion to output a high-precision benchmark point cloud model; retrieving the high-precision benchmark point cloud model for the K-1th coal pile loading / unloading operation window; comparing the Kth high-precision benchmark point cloud model to calculate the coal pile volume change; and using the same-period coal transportation ledger records to perform inventory verification of the coal pile volume change, obtaining inventory deviations, and executing an anomaly alarm.

[0006] The second aspect of this application discloses a multi-temporal data fusion compensation system under water surface refraction conditions. The system is used in the aforementioned multi-temporal data fusion compensation method under water surface refraction conditions. The system includes: a feature analysis module for interacting with a production scheduling terminal to obtain environmental time-series prediction data for the Kth coal pile loading and unloading operation window, performing feature analysis based on environmental humidity fluctuations, and locating multiple sampling time points; and a data acquisition module for triggering a multi-modal data acquisition device to perform multi-modal data acquisition on the target coal pile at the multiple sampling time points, obtaining multiple data packets, wherein the data packets include fused laser point clouds and panoramic visibility data. The system includes: a light image and environmental humidity parameters; a weighted fusion module for performing single-temporal refraction compensation reconstruction based on multiple fused laser point clouds and multiple panoramic visible light images, obtaining multiple single-temporal compensated point clouds, and then performing multi-temporal weighted fusion with multiple environmental humidity parameters to output a high-precision reference point cloud model; a change comparison module for retrieving the high-precision reference point cloud model for the K-1th coal pile loading and unloading operation window, comparing it with the Kth high-precision reference point cloud model, and calculating the change in coal pile volume; and an anomaly alarm module for verifying the stock of the coal pile volume change using the same period's coal transportation ledger records, obtaining the stock deviation, and executing an anomaly alarm.

[0007] One or more technical solutions provided in this application have at least the following beneficial effects: By analyzing environmental time-series prediction data during the coal pile loading and unloading operation window and dynamically locating multiple sampling points based on environmental humidity fluctuations, data acquisition is prioritized during periods of stable humidity or low refraction influence. Simultaneously, the sampling frequency is increased during periods of drastic humidity changes, thereby reducing the impact of laser refraction and specular reflection on point cloud accuracy under high humidity conditions and improving the reliability of subsequent 3D reconstruction data. A multimodal data acquisition device simultaneously collects and fuses laser point clouds, panoramic visible light images, and environmental humidity parameters at multiple sampling points, achieving coordinated acquisition of coal pile spatial structure information, surface texture information, and environmental state information. This improves data coverage under complex water surface refraction environments and reduces problems such as local occlusion, voids, and reflection distortion caused by single-sensor acquisition. Single-phase refraction compensation reconstruction is performed on the fused laser point cloud and panoramic visible light images, combined with historical stable points. Cloud, visual depth estimation, and anomaly region classification processing enable differentiated repair of refraction-deficient, shadowed, and noisy regions. Furthermore, weighted fusion of multiple single-temporal compensation point clouds is performed using environmental humidity parameters to reduce the impact of low-reliability data in high-humidity environments on the overall model, thereby improving the integrity, stability, and spatial accuracy of the coal pile 3D model. By retrieving high-precision benchmark point cloud models from adjacent loading and unloading operation windows for temporal comparison analysis, automatic calculation of coal pile volume changes is achieved, accurately reflecting changes in coal inventory during loading and unloading, improving coal pile inventory efficiency and dynamic inventory monitoring capabilities. By jointly verifying coal pile volume changes with concurrent coal transportation ledger records, deviations between actual coal inventory and transportation records can be identified, and anomaly alarms are triggered when deviations exceed thresholds, thereby improving coal yard inventory supervision capabilities and reducing measurement errors, abnormal losses, and management risks.

[0008] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0009] Figure 1 A schematic diagram of the multi-temporal data fusion compensation method under water surface refraction conditions provided in the embodiments of this application.

[0010] Figure 2 A schematic diagram of the structure of a multi-temporal data fusion compensation system under water surface refraction conditions provided in an embodiment of this application.

[0011] Figure labeling: Feature analysis module 10, data acquisition module 20, weighted fusion module 30, change comparison module 40, anomaly alarm module 50. Detailed Implementation

[0012] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0013] Example 1, as Figure 1 As shown in the embodiments of this application, a multi-temporal data fusion compensation method under water surface refraction conditions is provided, the method comprising: The interactive production scheduling terminal obtains environmental time-series prediction data for the Kth coal pile loading and unloading operation window, performs feature analysis based on environmental humidity fluctuations, and locates multiple sampling time points.

[0014] The interactive production scheduling terminal acquires environmental time-series prediction data for the Kth coal pile loading and unloading operation window. This data includes environmental parameters such as temperature, humidity, wind speed, and precipitation probability. A humidity time-series curve is constructed based on humidity changes, and fluctuation trend analysis is performed in conjunction with the temperature time-series curve to identify intervals of rapid humidity change, stable humidity intervals, and high-humidity risk intervals. Joint time-series matching is performed on the temperature and humidity segments, and the sampling density is dynamically adjusted according to the degree of refraction influence under different environmental conditions. Sampling priority is increased during periods of low humidity and environmental stability, and compensatory sampling frequency is increased during periods of drastic humidity fluctuations. This identifies multiple sampling points suitable for 3D reconstruction of the coal pile and generates sampling scheduling instructions.

[0015] At the multiple sampling points, the multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, resulting in multiple data packets, which include fused laser point clouds, panoramic visible light images, and environmental humidity parameters.

[0016] Upon arrival at each sampling time point, the multimodal data acquisition device synchronously performs coal pile data acquisition. This device includes a lidar, a panoramic visible light camera, and an environmental sensing unit. The lidar acquires three-dimensional point cloud data of the coal pile surface, the panoramic visible light camera acquires images of the coal pile surface texture and shadow distribution, and the environmental sensing unit synchronously records the current humidity parameters. Spatial registration and fusion are performed on the raw point clouds acquired from multiple observation stations to form a fused lidar point cloud; panoramic stitching is performed on multiple visible light images to form a panoramic visible light image; and spatial interpolation and fusion are performed on the environmental monitoring data to obtain the environmental humidity parameters, ultimately forming a data packet corresponding to the sampling time point.

[0017] Based on multiple fused laser point clouds and multiple panoramic visible light images, single-temporal refraction compensation reconstruction is performed to obtain multiple single-temporal compensated point clouds. Then, multi-temporal weighted fusion is performed by combining multiple environmental humidity parameters to output a high-precision benchmark point cloud model.

[0018] For each sampling point in the fused laser point cloud, density anomaly detection, reflection intensity anomaly detection, and geometric rationality detection are used to identify void regions, specular reflection regions, and geometrically abrupt regions, forming comprehensive suspicious regions. These comprehensive suspicious regions are projected onto a panoramic visible light image, and visual information is used to determine the authenticity of the abnormal regions, distinguishing between different anomaly types such as refraction loss, true shadows, and sensor noise. For refraction loss regions, local geometric deformation is extrapolated using historical stable point clouds, and point cloud compensation is completed by fusing monocular depth estimation results. For true shadow regions, region replacement is performed using historical stable point clouds. For noisy regions, filtering and denoising are performed. After compensation, the corresponding single-phase compensated point cloud is output.

[0019] Confidence weights are constructed based on the environmental humidity parameters corresponding to each sampling point, and weighted fusion is performed on multiple single-phase compensated point clouds. Higher humidity corresponds to lower point cloud weights to reduce the impact of high-humidity refractive environments on model accuracy. Spatiotemporal consistency optimization is then performed on the fused point cloud sequence to obtain a high-precision benchmark point cloud model.

[0020] Retrieve the high-precision reference point cloud model for the K-1 coal pile loading and unloading operation window, compare it with the Kth high-precision reference point cloud model, and calculate the change in coal pile volume.

[0021] The historical high-precision benchmark point cloud model generated during the K-1 coal pile loading and unloading operation window is retrieved and spatially aligned with the high-precision benchmark point cloud model generated during the current K coal pile loading and unloading operation window. Difference analysis is performed on the coal pile surface between the two time periods. By calculating the height changes and spatial increases / decreases in corresponding areas, the volume change of the coal pile is obtained. Integral calculations are performed based on the coal pile boundary constraints to obtain the volume change of the coal pile between the two loading and unloading operation windows, reflecting the changes in coal inventory.

[0022] The inventory of the coal pile volume change is verified using the coal transportation ledger records from the same period, the inventory deviation is obtained, and an abnormal alarm is triggered.

[0023] Obtain the coal transportation ledger record corresponding to the Kth coal pile loading and unloading operation window. The transportation ledger record includes data such as coal inbound volume, outbound volume, and allocation volume. Compare the theoretical inventory change result obtained from the transportation ledger with the actual inventory change result corresponding to the coal pile volume change to calculate the inventory deviation value. When the inventory deviation exceeds a preset threshold, an abnormal situation is determined, and an abnormal alarm message is generated and sent to the interactive production scheduling terminal to indicate possible coal loss, metering error, or abnormal transportation situation.

[0024] Furthermore, based on multiple fused laser point clouds and multiple panoramic visible light images, single-temporal refraction compensation reconstruction is performed to obtain multiple single-temporal compensated point clouds. The method includes: The first fused laser point cloud is input to a multi-task parallel detection architecture, driving a density anomaly detection head, a reflection intensity anomaly detection head, and a geometric rationality detection head to perform anomaly feature detection in parallel, locating candidate regions for holes, specular reflection, and geometric abrupt changes. These candidate regions are then logically fused to locate multiple comprehensive suspicious regions. These multiple comprehensive suspicious regions are projected onto a first panoramic visible light image to segment multiple visual verification regions. Anomaly type discrimination is performed on these multiple visual verification regions to generate multiple anomaly type labels. Based on these multiple anomaly type labels, differentiated point cloud completion is performed on the multiple visual verification regions to obtain multiple repaired point clouds. Point cloud replacement and fusion are then performed on the first fused laser point cloud in the multiple comprehensive suspicious regions to output a first single-temporal-compensated point cloud.

[0025] The first fused laser point cloud is input into a pre-constructed multi-task parallel detection architecture, where multiple detection heads simultaneously perform anomaly detection. Specifically, the density anomaly detection head analyzes the local point cloud distribution density to identify areas with abnormally increased point spacing, thus locating data voids caused by water refraction or occlusion; the reflection intensity anomaly detection head analyzes changes in laser echo reflection intensity to identify areas with abnormally high or low reflection values, thus locating specular reflection candidate areas; and the geometric rationality detection head analyzes the continuity and slope variation characteristics of the coal pile surface to identify geometrically abrupt changes that do not conform to the natural accumulation patterns of the coal pile. Finally, it outputs candidate areas for voids, specular reflections, and geometrically abrupt changes.

[0026] Spatial correlation analysis was performed on candidate regions for voids, specular reflections, and geometrical aberrations. Logical fusion was performed based on regional overlap, neighborhood correlation, and anomaly confidence. Regions that simultaneously met the characteristics of density anomalies and reflection anomalies were given higher anomaly priority. Regions adjacent to geometrical aberrations and voids were identified as high-risk refraction anomaly regions. Through joint constraints of multiple anomaly features, isolated noise regions and false detection regions were filtered out, ultimately forming multiple comprehensive suspicious regions for subsequent visual verification and compensation processing.

[0027] Based on the spatial calibration parameters between the LiDAR and the visible light camera, multiple suspicious regions are mapped from the 3D point cloud coordinate system to the 2D image coordinate system corresponding to the first panoramic visible light image to determine the corresponding positions of the abnormal regions in the image. Image region segmentation is performed with the projected area as the center, and the corresponding texture, shadow, brightness, and edge features are extracted to form multiple visual verification areas, providing visual information support for subsequent anomaly type identification.

[0028] Image feature analysis was performed on multiple visual verification regions. Combining brightness distribution, texture continuity, edge integrity, and reflectivity, the causes of abnormal regions were classified and identified. Regions with significant high-brightness reflections and missing textures were tagged as "refractional loss"; regions with reduced brightness but continuous textures were tagged as "true shadows"; and regions with random scattered points and irregular edge fluctuations were tagged as "sensor noise." Corresponding anomaly type labels were generated for each visual verification region to guide subsequent differential point cloud completion.

[0029] Differentiated point cloud completion processing is performed for different anomaly type labels. For refraction-deficient regions, the missing regions are reconstructed by combining historical stable point clouds and visual depth estimation results to generate compensated point clouds. For true shadow regions, historical stable point clouds are used for region mapping restoration. For sensor noise regions, effective point clouds are retained through noise filtering and outlier removal. The obtained multiple repaired point clouds are replaced in the comprehensive suspicious region corresponding to the first fused laser point cloud, and boundary smoothing fusion and local continuity optimization are performed to finally output the first single-temporal compensated point cloud.

[0030] Furthermore, the method also includes: Retrieve the historical stable point cloud of the K-1 coal pile loading and unloading operation window; perform local geometric deformation inference on the historical stable point cloud and the first fused laser point cloud based on the reliable area of ​​the coal pile to generate a first set of geometric inference candidate point clouds; extract H geometric inference candidate point sets from the first set of geometric inference candidate point clouds, which are marked with refraction loss labels for H refraction loss regions; perform monocular depth estimation on the H visual verification regions corresponding to the H refraction loss regions to recover H visual depth candidate point sets; and weightedly fuse the H geometric inference candidate point sets and the H visual depth candidate point sets to generate H first compensation point clouds.

[0031] The historical stable point cloud generated during the K-1 coal pile loading and unloading operation window is retrieved, and coordinate system one and spatial alignment processing are performed between the historical stable point cloud and the current first fused laser point cloud. Based on the boundary range of the coal pile and the distribution of stable areas, effective regions are filtered in the historical stable point cloud to obtain historical reference data that can reflect the normal stacking morphology of the coal pile, providing geometric prior constraints for subsequent deduction of missing regions.

[0032] A reliable region is extracted from the first fused laser point cloud as the true geometric constraint region of the current coal pile, and a local matching analysis is performed with the corresponding region in the historical stable point cloud. Based on the natural accumulation change law during the coal pile loading and unloading process, local geometric deformation inference is performed on the historical stable point cloud to adaptively transform the historical point cloud shape to the current coal pile shape, so as to restore the spatial structure that may correspond to the refraction missing region, and thus generate the first set of geometric inference candidate point clouds.

[0033] Based on the refractive error labels generated during the visual verification phase, the corresponding refractive error regions are located in the first set of geometric deduction candidate point clouds, and local candidate point cloud data corresponding to the spatial location of each refractive error region are extracted. H sets of geometric deduction candidate points are formed according to the region correspondence, serving as candidate results for structural restoration of the refractive error regions.

[0034] For the visual verification areas corresponding to H refraction-deficient regions, monocular depth estimation is performed based on panoramic visible light images. By analyzing image texture changes, edge structures, shadow relationships, and perspective features, the depth distribution information of the refraction-deficient regions is recovered. The depth estimation results are mapped to a three-dimensional spatial coordinate system to form a corresponding set of H visual depth candidate points, which are used to supplement the local geometric details of the refraction regions.

[0035] Spatially align H geometric inference candidate point sets with H visual depth candidate point sets, and assign fusion weights based on point cloud continuity, boundary fit, and visual depth reliability. For structurally stable regions, increase the weight of the geometric inference candidate point sets; for regions with rich detail and texture, increase the weight of the visual depth candidate point sets. Perform weighted fusion and boundary smoothing optimization on the two types of candidate point sets to generate H first compensation point clouds to achieve point cloud recovery of refraction-deficient regions.

[0036] Furthermore, the method also includes: P visual verification regions marked as true shadow labels are projected onto the historical stable point cloud to segment P first compensation point clouds; noise filtering is performed on W visual verification regions marked as sensor noise labels to retain W first effective point clouds; the H first compensation point clouds, P first compensation point clouds and W first effective point clouds are used to perform multi-source point cloud replacement fusion of the first fused laser point cloud to output the first single-temporal compensation point cloud.

[0037] For the P visual verification areas labeled as true shadows, the spatial mapping relationship between laser point clouds and visible light images is used to project the corresponding areas onto historical stable point clouds to locate the corresponding spatial positions of the shadow areas within the historical stable point clouds. Since true shadow areas are usually only affected by illumination occlusion and their actual coal pile geometry does not change significantly, local point cloud data of the corresponding areas are extracted from the historical stable point clouds. Local clipping and edge alignment are then performed based on the current coal pile boundaries to form P first compensation point clouds, which are used to restore the point cloud structure of the shadowed areas.

[0038] For the W visual verification regions labeled as sensor noise, noise filtering is performed on the corresponding point clouds. By analyzing the local neighborhood distribution, inter-point distance, and normal vector continuity of the point clouds, isolated outliers, high-frequency scattered points, and randomly drifting points are identified and removed. Simultaneously, point cloud data that meet spatial continuity constraints and conform to the geometric patterns of the coal pile surface are retained, thus obtaining W first effective point clouds to reduce the impact of sensor noise on the accuracy of subsequent point cloud fusion.

[0039] H first-compensation point clouds, P first-compensation point clouds, and W first-effective point clouds are respectively replaced in the refraction-deficient region, the true shadow region, and the sensor noise region of the first fused laser point cloud. Boundary continuity constraints are applied to the replaced regions, and abrupt changes in splicing between point clouds from different sources are eliminated through local surface smoothing, density equalization, and geometric transition optimization. Finally, unified spatial optimization is performed on the overall point cloud after replacement and fusion, outputting the first single-temporal-compensation point cloud to improve the integrity and accuracy of the coal pile 3D model under complex humidity and refraction environments.

[0040] Furthermore, the interactive production scheduling terminal obtains environmental time-series prediction data for the Kth coal pile loading and unloading operation window, performs feature analysis based on environmental humidity fluctuations, and locates multiple sampling time points. The method includes: Based on the environmental time-series prediction data, after constructing temperature and humidity time-series curves, gradient identification of the rate of change is performed to obtain temperature state segment sequences and humidity state segment sequences. The temperature and humidity state segment sequences are time-aligned to construct a joint state segment sequence, and then two-dimensional state sampling density matching is performed to obtain a sampling density configuration sequence. Sampling timing scheduling is performed according to the sampling density configuration sequence to locate the multiple sampling timing points.

[0041] Environmental time-series forecast data for the coal pile loading and unloading operation window were acquired, and temperature and humidity parameters were extracted to construct temperature and humidity time-series curves. Rate-of-change gradient analysis was performed on the temperature and humidity time-series curves. By calculating the magnitude and trend of changes between adjacent time points, the stable, slowly changing, and drastically fluctuating states of environmental parameters in different time periods were identified. Based on the rate-of-change gradient threshold, the environmental change process was divided into intervals, forming temperature and humidity state segment sequences to characterize the environmental stability and humidity fluctuation characteristics in different time periods.

[0042] Time axis alignment is performed on the temperature and humidity state segment sequences to establish a correspondence between temperature and humidity states within the same time interval. The temperature and humidity states within corresponding time periods are combined to construct a joint state segment sequence, and sampling density is matched based on the impact of different joint environmental conditions on laser refraction, reflection, and point cloud stability. When humidity is high or environmental fluctuations are severe, the sampling frequency is increased to enhance data compensation capabilities under abnormal conditions; when the environment is stable, the sampling frequency is decreased to reduce redundant acquisition. This ultimately forms the sampling density configuration sequence for each time interval.

[0043] Based on the sampling density configuration sequence, sampling tasks are dynamically scheduled within the coal pile loading and unloading operation window. In time intervals with high sampling density, the sampling interval is shortened and the number of samples is increased to improve data coverage under complex humidity conditions; in time intervals with low sampling density, the sampling interval is appropriately extended to reduce system resource consumption. A sampling scheduling plan is generated based on the sampling interval parameters corresponding to each time interval, and sampling trigger commands are sent to the multimodal data acquisition device to ultimately locate multiple sampling timing points.

[0044] Furthermore, the method also includes: Based on the historical 3D point cloud data of the target coal pile, a spatial envelope of the coal pile is constructed, and sampling line-of-sight coverage analysis is performed to locate the coordinates of M observation stations. By integrating lidar, visible light camera and environmental sensing unit at the coordinates of the M observation stations, M fixed acquisition units are obtained, which constitute a multimodal data acquisition device.

[0045] Historical 3D point cloud data of the target coal pile was acquired, and a spatial envelope of the coal pile was constructed based on the spatial distribution range of the historical point cloud to characterize the overall spatial boundary and height variation range of the coal pile during long-term loading and unloading. Combining the surface slope distribution, location of obstructed areas, and historically high-refractive-rate areas of the coal pile, a line-of-sight coverage analysis was performed on the coal pile to calculate the effective coverage range and obstruction of the laser scanning line of sight under different observation directions. By optimizing the observation angle, observation distance, and regional coverage overlap, the risk of data loss caused by concave areas, edge obstructed areas, and water surface reflection areas of the coal pile was reduced. Finally, the coordinates of M observation stations that can cover the main observation area of ​​the target coal pile were determined.

[0046] LiDAR, visible light cameras, and environmental sensing units are deployed at the coordinates of the M observation stations, and time synchronization and spatial calibration are performed between the devices to form corresponding fixed acquisition units. The LiDAR is used to acquire 3D point cloud data of the coal pile surface, the visible light camera is used to acquire surface texture and shadow images of the coal pile, and the environmental sensing unit is used to monitor environmental parameters such as humidity and temperature in real time. Unified communication and collaborative control are implemented for each fixed acquisition unit, enabling multiple fixed acquisition units to jointly constitute a multimodal data acquisition device to achieve multi-angle, multimodal, synchronous data acquisition of the coal pile area.

[0047] Furthermore, at the multiple sampling points, a multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile to obtain multiple data packets. The method includes: At the first sampling point, the multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, obtaining M synchronous raw data packets; M original fused laser point clouds, M original panoramic visible light images, and M original environmental monitoring data are extracted from the M synchronous raw data packets; the M original fused laser point clouds are spatially registered and fused to obtain a first fused laser point cloud; the M original panoramic visible light images are spatially stitched and aligned to obtain a first panoramic visible light image; and the M original environmental monitoring data are spatially interpolated and fused to obtain a first environmental humidity parameter.

[0048] Upon arrival of the first sampling time, the multimodal data acquisition device synchronously activates each fixed acquisition unit to perform multimodal data acquisition on the target coal pile. Each fixed acquisition unit acquires laser scanning data, visible light image data, and environmental monitoring data from different observation angles on the coal pile surface, and a unified time synchronization mechanism ensures that all types of data are acquired at the same sampling time. Each fixed acquisition unit generates a corresponding raw data packet, resulting in M ​​synchronized raw data packets to achieve multi-angle synchronous sensing of the coal pile area.

[0049] Data parsing and classification are performed on M synchronous raw data packets. Raw point cloud data acquired by lidar, raw image data acquired by a visible light camera, and environmental monitoring data acquired by an environmental sensing unit are extracted from each raw data packet. The raw point cloud data is used to characterize the spatial structure information of the coal pile surface, the raw image data is used to characterize the texture and shadow features of the coal pile surface, and the environmental monitoring data reflects environmental parameters such as humidity and temperature at the time of sampling. This results in M ​​raw fused lidar point clouds, M raw panoramic visible light images, and M raw environmental monitoring data.

[0050] Spatial coordinate unification and point cloud registration are performed on M original fused laser point clouds. By extracting the overlapping areas between different observation perspectives, spatial matching relationships between corresponding point clouds are established, and coordinate transformation and error correction are performed on each original fused laser point cloud. The multiple point clouds that have completed spatial registration are fused to eliminate local occlusion and point cloud missing problems caused by multi-view scanning, and finally generate the first fused laser point cloud covering the entire area of ​​the target coal pile.

[0051] Image spatial stitching and viewpoint alignment are performed on M original panoramic visible light images. Image matching relationships are established by extracting overlapping texture regions in adjacent images, and image geometric correction is performed based on camera calibration parameters. Images from multiple observation directions are uniformly stitched and edge-fused to form a first panoramic visible light image covering the entire coal pile area, thus providing complete image information for subsequent visual verification of anomalies.

[0052] The M raw environmental monitoring data were time-synchronized and spatially correlated. Based on the spatial location of each fixed acquisition unit, spatial interpolation fusion was performed on the environmental humidity data collected from different observation areas. By comprehensively analyzing the humidity distribution at different observation locations, a first environmental humidity parameter reflecting the overall environmental state of the target coal pile was obtained, which was used for confidence weight calculation in subsequent multi-temporal point cloud fusion.

[0053] Furthermore, the method also includes: A pre-constructed weight mapping function is used to perform negative correlation mapping calculations on the multiple environmental humidity parameters to obtain multiple initial confidence weights. Based on the compensation data source, local quality weights are allocated to the multiple single-phase compensation point clouds, and then confidence weighted fusion is performed in combination with the multiple initial confidence weights to output a weighted point cloud sequence. The weighted point cloud sequence is then subjected to spatiotemporal consistency optimization fusion to obtain the high-precision benchmark point cloud model.

[0054] A weighted mapping function is pre-constructed based on the influence of humidity changes on laser refraction and reflection in a coal pile scenario. Environmental humidity parameters corresponding to multiple sampling points are then input into this function to perform negative correlation mapping calculations. Since increased environmental humidity increases water surface reflection, laser refraction, and the probability of point cloud distortion, the confidence weight of the corresponding sampling point gradually decreases as the environmental humidity parameter increases. Multiple initial confidence weights are generated based on the mapping results corresponding to each sampling point to characterize the data reliability of each single-phase compensated point cloud under different environmental conditions.

[0055] Local quality assessments are performed on multiple single-temporal compensated point clouds, and corresponding local quality weights are assigned to different regions based on the data source. Specifically, original reliable laser points, directly derived from stable laser scan results, are assigned higher quality weights. Compensated points obtained through geometric deduction from historical stable point clouds are assigned secondary quality weights based on their matching degree with the current coal pile structure. For compensated points recovered based on visual depth estimation, the quality weights are dynamically adjusted based on image texture clarity, edge integrity, and depth estimation stability. The local quality weights are jointly weighted with the initial confidence weights at the corresponding sampling points, and fusion processing is performed on multiple single-temporal compensated point clouds to reduce the impact of high humidity environments and low-confidence compensated areas on the overall model accuracy. The final output is a weighted point cloud sequence.

[0056] Spatiotemporal consistency optimization was performed on the weighted point cloud sequence. By analyzing the spatial overlap and temporal continuity between different sampling points, unified constraints were applied to local point cloud density, boundary continuity, and surface smoothness. For regions with significant geometric conflicts or local jumps between multiple temporal phases, high-confidence point cloud data was prioritized for retention, while low-confidence regions underwent smooth transition optimization to reduce spatial breaks and local drift issues during multi-temporal phase fusion. The optimized point cloud data was then fused and reconstructed to obtain a high-precision benchmark point cloud model that reflects the true spatial structure of the target coal pile.

[0057] Furthermore, the weight mapping function is as follows: ; Where Wi is the initial confidence weight corresponding to the i-th sampling time point, Hi is the environmental humidity parameter of the i-th sampling time point, and β is the attenuation coefficient preset according to the reflection characteristics of the coal pile material.

[0058] A pre-constructed weight mapping function is used to characterize the impact of environmental humidity changes on point cloud reliability. Because coal pile surfaces are prone to water film reflection, laser refraction, and localized specular reflection in high-humidity environments, the reliability of point cloud data gradually decreases as environmental humidity increases. Based on this, a negative correlation mapping calculation is performed on the environmental humidity parameters corresponding to each sampling point to obtain the corresponding initial confidence weights.

[0059] Specifically, the weight mapping function is as follows: ; Through the aforementioned weight mapping function, when the environmental humidity parameter is low, the initial confidence weight of the corresponding sampling time point is higher, so as to increase the contribution ratio of point cloud data in subsequent fusion under stable environmental conditions; when the environmental humidity parameter increases, the corresponding initial confidence weight decays exponentially, so as to reduce the impact of distorted point clouds caused by refraction, reflection and other factors on the overall model accuracy under high humidity environment, thereby improving the stability and reliability of multi-temporal point cloud fusion results.

[0060] Example 2, based on the same inventive concept as the multi-temporal data fusion compensation method under water surface refraction conditions in the foregoing examples, such as... Figure 2 As shown in the embodiment of this application, a multi-temporal data fusion compensation system under water surface refraction conditions is provided. The system includes: The feature analysis module 10 is used to interact with the production scheduling terminal to obtain environmental time-series prediction data for the Kth coal pile loading and unloading operation window, perform feature analysis based on environmental humidity fluctuations, and locate multiple sampling time points; the data acquisition module 20 is used to trigger the multimodal data acquisition device to perform multimodal data acquisition on the target coal pile at the multiple sampling time points, and obtain multiple data packets, wherein the data packets include fused laser point clouds, panoramic visible light images, and environmental humidity parameters; the weighted fusion module 30 is used to perform single-phase refraction compensation reconstruction based on multiple fused laser point clouds and multiple panoramic visible light images, obtain multiple single-phase compensated point clouds, and then perform multi-phase weighted fusion with multiple environmental humidity parameters to output a high-precision reference point cloud model; the change comparison module 40 is used to retrieve the high-precision reference point cloud model for the K-1th coal pile loading and unloading operation window, compare it with the Kth high-precision reference point cloud model, and calculate the change in coal pile volume; the anomaly alarm module 50 is used to perform inventory verification of the change in coal pile volume using the same period's coal transportation ledger records, obtain inventory deviations, and execute an anomaly alarm.

[0061] Furthermore, the weighted fusion module 30 is used to perform the following operation steps: The first fused laser point cloud is input to a multi-task parallel detection architecture, driving a density anomaly detection head, a reflection intensity anomaly detection head, and a geometric rationality detection head to perform anomaly feature detection in parallel, locating candidate regions for holes, specular reflection, and geometric abrupt changes. These candidate regions are then logically fused to locate multiple comprehensive suspicious regions. These multiple comprehensive suspicious regions are projected onto a first panoramic visible light image to segment multiple visual verification regions. Anomaly type discrimination is performed on these multiple visual verification regions to generate multiple anomaly type labels. Based on these multiple anomaly type labels, differentiated point cloud completion is performed on the multiple visual verification regions to obtain multiple repaired point clouds. Point cloud replacement and fusion are then performed on the first fused laser point cloud in the multiple comprehensive suspicious regions to output a first single-temporal-compensated point cloud.

[0062] Furthermore, the weighted fusion module 30 is used to perform the following operation steps: Retrieve the historical stable point cloud of the K-1 coal pile loading and unloading operation window; perform local geometric deformation inference on the historical stable point cloud and the first fused laser point cloud based on the reliable area of ​​the coal pile to generate a first set of geometric inference candidate point clouds; extract H geometric inference candidate point sets from the first set of geometric inference candidate point clouds, which are marked with refraction loss labels for H refraction loss regions; perform monocular depth estimation on the H visual verification regions corresponding to the H refraction loss regions to recover H visual depth candidate point sets; and weightedly fuse the H geometric inference candidate point sets and the H visual depth candidate point sets to generate H first compensation point clouds.

[0063] Furthermore, the weighted fusion module 30 is used to perform the following operation steps: P visual verification regions marked as true shadow labels are projected onto the historical stable point cloud to segment P first compensation point clouds; noise filtering is performed on W visual verification regions marked as sensor noise labels to retain W first effective point clouds; the H first compensation point clouds, P first compensation point clouds and W first effective point clouds are used to perform multi-source point cloud replacement fusion of the first fused laser point cloud to output the first single-temporal compensation point cloud.

[0064] Furthermore, the feature analysis module 10 is used to perform the following operation steps: Based on the environmental time-series prediction data, after constructing temperature and humidity time-series curves, gradient identification of the rate of change is performed to obtain temperature state segment sequences and humidity state segment sequences. The temperature and humidity state segment sequences are time-aligned to construct a joint state segment sequence, and then two-dimensional state sampling density matching is performed to obtain a sampling density configuration sequence. Sampling timing scheduling is performed according to the sampling density configuration sequence to locate the multiple sampling timing points.

[0065] Furthermore, the data acquisition module 20 is used to perform the following operation steps: Based on the historical 3D point cloud data of the target coal pile, a spatial envelope of the coal pile is constructed, and sampling line-of-sight coverage analysis is performed to locate the coordinates of M observation stations. By integrating lidar, visible light camera and environmental sensing unit at the coordinates of the M observation stations, M fixed acquisition units are obtained, which constitute a multimodal data acquisition device.

[0066] Furthermore, the data acquisition module 20 is used to perform the following operation steps: At the first sampling point, the multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, obtaining M synchronous raw data packets; M original fused laser point clouds, M original panoramic visible light images, and M original environmental monitoring data are extracted from the M synchronous raw data packets; the M original fused laser point clouds are spatially registered and fused to obtain a first fused laser point cloud; the M original panoramic visible light images are spatially stitched and aligned to obtain a first panoramic visible light image; and the M original environmental monitoring data are spatially interpolated and fused to obtain a first environmental humidity parameter.

[0067] Furthermore, the weighted fusion module 30 is used to perform the following operation steps: A pre-constructed weight mapping function is used to perform negative correlation mapping calculations on the multiple environmental humidity parameters to obtain multiple initial confidence weights. Based on the compensation data source, local quality weights are allocated to the multiple single-phase compensation point clouds, and then confidence weighted fusion is performed in combination with the multiple initial confidence weights to output a weighted point cloud sequence. The weighted point cloud sequence is then subjected to spatiotemporal consistency optimization fusion to obtain the high-precision benchmark point cloud model.

[0068] Furthermore, the weight mapping function is as follows: ; Where Wi is the initial confidence weight corresponding to the i-th sampling time point, Hi is the environmental humidity parameter of the i-th sampling time point, and β is the attenuation coefficient preset according to the reflection characteristics of the coal pile material.

[0069] Through the foregoing detailed description of the multi-temporal data fusion compensation method under water surface refraction conditions, those skilled in the art can clearly understand the multi-temporal data fusion compensation system under water surface refraction conditions in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be referred to the method section.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-temporal data fusion compensation method under water surface refraction conditions, characterized in that, The method includes: The interactive production scheduling terminal obtains environmental time-series prediction data for the Kth coal pile loading and unloading operation window, performs feature analysis based on environmental humidity fluctuations, and locates multiple sampling time points. At the multiple sampling points, the multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, and multiple data packets are obtained, wherein the data packets include fused laser point clouds, panoramic visible light images and environmental humidity parameters; Based on multiple fused laser point clouds and multiple panoramic visible light images, single-temporal refraction compensation reconstruction is performed to obtain multiple single-temporal compensated point clouds. Then, multi-temporal weighted fusion is performed by combining multiple environmental humidity parameters to output a high-precision benchmark point cloud model. Retrieve the high-precision reference point cloud model for the K-1 coal pile loading and unloading operation window, compare it with the Kth high-precision reference point cloud model, and calculate the change in coal pile volume. The inventory of the coal pile volume change is verified using the coal transportation ledger records from the same period, the inventory deviation is obtained, and an abnormal alarm is triggered.

2. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 1, characterized in that, Based on multiple fused laser point clouds and multiple panoramic visible light images, single-temporal refraction compensation reconstruction is performed to obtain multiple single-temporal compensated point clouds. The method includes: Input the first fused laser point cloud to the multi-task parallel detection architecture, drive the density anomaly detection head, reflection intensity anomaly detection head and geometric rationality detection head to perform anomaly feature detection in parallel, and locate the hole candidate region, specular reflection candidate region and geometric change candidate region; Logically fuse the candidate regions for holes, specular reflections, and geometric mutations to locate multiple comprehensive suspicious regions; The multiple suspected areas are projected onto the first panoramic visible light image to segment multiple visual verification areas; Anomaly type determination is performed on the multiple visual verification regions to generate multiple anomaly type labels; Based on the multiple anomaly type labels, differential point cloud completion is performed on the multiple visual verification regions to obtain multiple repair point clouds. Point cloud replacement and fusion are performed on the first fused laser point cloud in the multiple comprehensive suspicious regions to output the first single-phase compensation point cloud.

3. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 2, characterized in that, The method further includes: Retrieve the historical stable point cloud for the K-1 coal pile loading and unloading operation window; Based on the reliable region of the coal pile, the historical stable point cloud and the first fused laser point cloud are subjected to local geometric deformation inference to generate the first set of geometric inference candidate point clouds. Extract H candidate points for geometric inference from the first set of geometric inference candidate point clouds, which are labeled with refraction missing tags for H refraction missing regions; For the H visual verification regions corresponding to the H refractive missing regions, monocular depth estimation is performed to recover the H visual depth candidate point set; The H geometric inference candidate point sets and the H visual depth candidate point sets are weighted and fused to generate H first compensation point clouds.

4. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 3, characterized in that, The method further includes: P visual verification regions marked as true shadow labels are projected onto the historical stable point cloud to segment the P first compensation point cloud. For the W visual verification regions marked as sensor noise labels, noise filtering is performed to retain the W first valid point clouds; The first fused laser point cloud is replaced and fused using the H first compensation point clouds, P first compensation point clouds and W first effective point clouds to output the first single-temporal compensation point cloud.

5. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 1, characterized in that, The interactive production scheduling terminal obtains environmental time-series prediction data for the Kth coal pile loading and unloading operation window, performs feature analysis based on environmental humidity fluctuations, and locates multiple sampling time points. The method includes: Based on the environmental time-series prediction data, after constructing temperature time-series curves and humidity time-series curves, the rate of change gradient is identified to obtain temperature state segment sequences and humidity state segment sequences. After aligning the temperature state segment sequence and humidity state segment sequence in time and constructing a joint state segment sequence, two-dimensional state sampling density matching is performed to obtain a sampling density configuration sequence. Sampling timing is scheduled based on the sampling density configuration sequence to locate the multiple sampling timing points.

6. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 1, characterized in that, The method further includes: Based on the historical 3D point cloud data of the target coal pile, a spatial envelope of the coal pile is constructed, and a sampling line-of-sight coverage analysis is performed to locate the coordinates of M observation stations. By integrating lidar, visible light cameras, and environmental sensing units at the coordinates of the M observation stations, M fixed acquisition units are obtained, forming a multimodal data acquisition device.

7. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 6, characterized in that, At the multiple sampling points, a multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, obtaining multiple data packets. The method includes: At the first sampling point, the multimodal data acquisition device is triggered to perform multimodal data acquisition on the target coal pile, obtaining M synchronous raw data packets; M original fused laser point clouds, M original panoramic visible light images, and M original environmental monitoring data are extracted from the M original synchronous data packets. Spatial registration and fusion of the M original fused laser point clouds yields the first fused laser point cloud; The M original panoramic visible light images are spatially stitched and aligned to obtain the first panoramic visible light image; Spatial interpolation is used to fuse the M original environmental monitoring data to obtain the first environmental humidity parameter.

8. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 1, characterized in that, The method further includes: A pre-constructed weight mapping function is used to perform negative correlation mapping calculations on the multiple environmental humidity parameters to obtain multiple initial confidence weights; After allocating local quality weights to the multiple single-phase compensated point clouds based on the source of the compensated data, the multiple initial confidence weights are combined to perform confidence-weighted fusion, and a weighted point cloud sequence is output. The weighted point cloud sequence is subjected to spatiotemporal consistency optimization and fusion to obtain the high-precision reference point cloud model.

9. The multi-temporal data fusion compensation method under water surface refraction conditions as described in claim 8, characterized in that, The weight mapping function is as follows: ； Where Wi is the initial confidence weight corresponding to the i-th sampling time point, Hi is the environmental humidity parameter of the i-th sampling time point, and β is the attenuation coefficient preset according to the reflection characteristics of the coal pile material.

10. A multi-temporal data fusion compensation system under water surface refraction conditions, characterized in that, The system is used to implement the multi-temporal data fusion compensation method under water surface refraction conditions as described in any one of claims 1-9, the system comprising: The feature analysis module is used to interact with the production scheduling terminal to obtain environmental time series prediction data for the Kth coal pile loading and unloading operation window, perform feature analysis based on environmental humidity fluctuations, and locate multiple sampling time points. The data acquisition module is used to trigger the multimodal data acquisition device to perform multimodal data acquisition on the target coal pile at the multiple sampling time points to obtain multiple data packets, wherein the data packets include fused laser point clouds, panoramic visible light images and environmental humidity parameters; The weighted fusion module is used to perform single-temporal refraction compensation reconstruction based on multiple fused laser point clouds and multiple panoramic visible light images. After obtaining multiple single-temporal compensated point clouds, it combines multiple environmental humidity parameters to perform multi-temporal weighted fusion and output a high-precision benchmark point cloud model. The change comparison module is used to retrieve the high-precision benchmark point cloud model of the K-1 coal pile loading and unloading operation window, compare it with the Kth high-precision benchmark point cloud model, and calculate the change in coal pile volume. The abnormal alarm module is used to verify the inventory of changes in the volume of the coal pile by using the coal transportation ledger records of the same period, obtain the inventory deviation, and execute an abnormal alarm.

Images