Level monitoring method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a kind of material level monitoring method, device, equipment and medium.The method comprises: first, the image data and three-dimensional point cloud data of target area are acquired;Afterwards, image data and three-dimensional point cloud data are respectively preprocessed, and the fusion data of preprocessed image data and three-dimensional point cloud data is carried out fusion processing;Then, the three-dimensional data of target area is obtained based on the semantic segmentation of fusion data to pre-trained deep learning model;Further, the material volume of current target area is calculated according to the three-dimensional data of target area, and the material level percentage is determined according to full load reference volume;Finally, according to material level percentage, double filtering is judged, when the judgment result satisfies the preset low material level trigger condition, generation and output replenishment prompt information are carried out.Through the above method, the precise monitoring of material can be realized, and the continuity and efficiency of loader operation are improved.

Description

Technical Field

[0001] This application relates to the field of intelligent sensing and material level monitoring technology, and in particular to a material level monitoring method, device, equipment and medium. Background Technology

[0002] Articulated loaders are widely used in material loading operations in industries such as mining, construction, ports, and metallurgy. In typical operations, loaders frequently need to unload material from the feed inlets of crushers, conveyors, or silos. For example, in mining, loaders load ore from stockpiles into crushers or silos, and their unloading efficiency directly affects the continuity of subsequent crushing and screening processes. In port bulk cargo handling scenarios, loaders need to load materials such as coal and ore into transport ships or silos, and the unloading speed and material level directly affect the overall efficiency of the loading and unloading operation. However, in actual operations, the material level at the feed inlet (i.e., the material level height) is a key parameter determining the timing of material replenishment. Therefore, there is an urgent need for a material level sensing technology that can adapt to complex environments, achieve continuous material level monitoring, and support automated decision-making to improve the accuracy, efficiency, and safety of loader operations.

[0003] In existing technologies, commonly used material level monitoring methods generally rely on sensors to determine the level. Specifically, radar level gauges can be used to emit electromagnetic waves and receive reflected signals. The distance to the material surface is calculated based on the time of flight or frequency modulated continuous wave (FMCW) principle to determine the material level height.

[0004] However, existing material level monitoring methods cannot perceive the complete three-dimensional shape of the material pile, resulting in a large difference between the measured volume and the actual volume. Summary of the Invention

[0005] This application provides a material level monitoring method, device, equipment, and medium to solve the problem of large discrepancies between measured volume and actual volume in the prior art.

[0006] In a first aspect, embodiments of this application provide a material level monitoring method, including:

[0007] Acquire image data and 3D point cloud data of the target area;

[0008] The image data and the 3D point cloud data are preprocessed separately, and the preprocessed image data and the 3D point cloud data are fused together to obtain fused data.

[0009] The fused data is semantically segmented based on a pre-trained deep learning model to obtain the three-dimensional data of the target region.

[0010] The material volume of the current target area is calculated based on the three-dimensional data of the target area, and the material level percentage is determined based on the full-load reference volume.

[0011] The material level percentage is used for dual filtering and judgment. When the judgment result meets the preset low material level trigger condition, a material replenishment prompt message is generated and output.

[0012] In one possible implementation, before acquiring the image data and the 3D point cloud data, the method further includes:

[0013] The intrinsic parameters of the visual camera that acquires the image data are calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients.

[0014] The lidar and the vision camera that acquire the 3D point cloud data are jointly calibrated to obtain the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system.

[0015] The mapping relationship between the pixel coordinate system and the three-dimensional spatial coordinate system of the image data and the three-dimensional point cloud data is established by using the camera intrinsic parameter matrix, distortion coefficient, rotation matrix and translation vector.

[0016] The baseline point clouds of the target area are obtained in the empty and full-load states, respectively, to obtain the empty warehouse baseline point cloud and the full-load baseline point cloud.

[0017] The full-load reference volume is calculated based on the empty warehouse reference point cloud and the full-load reference point cloud.

[0018] In one possible implementation, the preprocessing of the image data and the 3D point cloud data, and the fusion processing of the preprocessed image data and the 3D point cloud data to obtain fused data, includes:

[0019] The image data is subjected to distortion correction processing to obtain preprocessed image data;

[0020] The three-dimensional point cloud data is subjected to statistical filtering and voxel filtering to obtain the processed three-dimensional point cloud data.

[0021] According to the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and each valid point cloud data is assigned corresponding pixel color information to obtain the fused data.

[0022] In one possible implementation, the pre-trained deep learning model performs semantic segmentation on the fused data to obtain three-dimensional data of the target region, including:

[0023] The fused data is input into a pre-trained deep learning model, and the probability value of each point cloud point in the fused data output by the model belonging to the target region is obtained. The trained deep learning model is trained based on historical fused data with semantic annotations. The semantic annotations are used to identify point cloud points in the historical fused data that belong to the material pile region.

[0024] The probability value is compared with a preset probability threshold, and a segmentation mask is generated based on the comparison result.

[0025] The point cloud set belonging to the material region is extracted from the fused data using the segmentation mask, and used as the three-dimensional data of the target region.

[0026] In one possible implementation, the step of calculating the material volume of the current target area based on the three-dimensional data of the target area, and determining the material level percentage based on the full-load reference volume, includes:

[0027] The three-dimensional data of the target area is registered with the empty warehouse reference point cloud to obtain aligned point cloud data;

[0028] The aligned point cloud data is compared with the empty warehouse reference point cloud by differential calculation to obtain a differential point cloud representing the material distribution.

[0029] The material volume of the current target region is obtained by performing three-dimensional reconstruction on the differential point cloud;

[0030] The material level percentage is determined based on the ratio of the material volume in the current target area to the full-load reference volume.

[0031] In one possible implementation, after determining the material level percentage, the method further includes:

[0032] The full-load reference volume is updated using the exponentially weighted moving average method based on the measured stockpile volume obtained after each unloading or replenishment.

[0033] In one possible implementation, the step of performing dual filtering based on the material level percentage, and generating and outputting a replenishment prompt message when the judgment result meets a preset low material level trigger condition, includes:

[0034] Obtain the material level percentage of a consecutive preset number of frames, and determine whether the material level percentage of the consecutive preset number of frames is lower than a preset low material level percentage threshold.

[0035] If so, the material volume threshold is calculated based on the product of the preset low material level percentage threshold and the full-load reference volume.

[0036] Based on the material volume, material volume threshold, and historical average material consumption rate of the current target area, the remaining time required for the material to drop to the low material level threshold from the current moment is calculated, and it is determined whether the remaining time is lower than the preset warning time.

[0037] If so, a replenishment prompt message is generated and output. The replenishment prompt message includes at least one of the following: displaying alarm information through a human-machine interface terminal, driving an audible and visual alarm to issue a prompt, or sending a replenishment request to the loader control system.

[0038] Secondly, embodiments of this application provide a material level monitoring device, comprising:

[0039] The acquisition module is used to acquire image data and 3D point cloud data of the target area;

[0040] The first processing module is used to preprocess the image data and the three-dimensional point cloud data respectively, and to fuse the preprocessed image data and the three-dimensional point cloud data to obtain fused data.

[0041] The second processing module is used to perform semantic segmentation on the fused data based on a pre-trained deep learning model to obtain three-dimensional data of the target region.

[0042] The determination module is used to calculate the material volume of the current target area based on the three-dimensional data of the target area, and to determine the material level percentage based on the full-load reference volume;

[0043] The judgment module is used to perform dual filtering judgment based on the material level percentage. When the judgment result meets the preset low material level trigger condition, it generates and outputs material replenishment prompt information.

[0044] In one possible implementation, the device further includes a third processing module, which, before acquiring the image data and the 3D point cloud data, is configured to:

[0045] The intrinsic parameters of the visual camera that acquires the image data are calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients.

[0046] The lidar and the vision camera that acquire the 3D point cloud data are jointly calibrated to obtain the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system.

[0047] The mapping relationship between the pixel coordinate system and the three-dimensional spatial coordinate system of the image data and the three-dimensional point cloud data is established by using the camera intrinsic parameter matrix, distortion coefficient, rotation matrix and translation vector.

[0048] The baseline point clouds of the target area are obtained in the empty and full-load states, respectively, to obtain the empty warehouse baseline point cloud and the full-load baseline point cloud.

[0049] The full-load reference volume is calculated based on the empty warehouse reference point cloud and the full-load reference point cloud.

[0050] In one possible implementation, the first processing module is specifically used for:

[0051] The image data is subjected to distortion correction processing to obtain preprocessed image data;

[0052] The three-dimensional point cloud data is subjected to statistical filtering and voxel filtering to obtain the processed three-dimensional point cloud data.

[0053] According to the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and each valid point cloud data is assigned corresponding pixel color information to obtain the fused data.

[0054] In one possible implementation, the second processing module is specifically used for:

[0055] The fused data is input into a pre-trained deep learning model, and the probability value of each point cloud point in the fused data output by the model belonging to the target region is obtained. The trained deep learning model is trained based on historical fused data with semantic annotations. The semantic annotations are used to identify point cloud points in the historical fused data that belong to the material pile region.

[0056] The probability value is compared with a preset probability threshold, and a segmentation mask is generated based on the comparison result.

[0057] The point cloud set belonging to the material region is extracted from the fused data using the segmentation mask, and used as the three-dimensional data of the target region.

[0058] In one possible implementation, the determining module is specifically used for:

[0059] The three-dimensional data of the target area is registered with the empty warehouse reference point cloud to obtain aligned point cloud data;

[0060] The aligned point cloud data is compared with the empty warehouse reference point cloud by differential calculation to obtain a differential point cloud representing the material distribution.

[0061] The material volume of the current target region is obtained by performing three-dimensional reconstruction on the differential point cloud;

[0062] The material level percentage is determined based on the ratio of the material volume in the current target area to the full-load reference volume.

[0063] In one possible implementation, the device further includes an update module, which, after determining the material level percentage, is configured to:

[0064] The full-load reference volume is updated using the exponentially weighted moving average method based on the measured stockpile volume obtained after each unloading or replenishment.

[0065] In one possible implementation, the determining module is specifically used for:

[0066] Obtain the material level percentage of a consecutive preset number of frames, and determine whether the material level percentage of the consecutive preset number of frames is lower than a preset low material level percentage threshold.

[0067] If so, the material volume threshold is calculated based on the product of the preset low material level percentage threshold and the full-load reference volume.

[0068] Based on the material volume, material volume threshold, and historical average material consumption rate of the current target area, the remaining time required for the material to drop to the low material level threshold from the current moment is calculated, and it is determined whether the remaining time is lower than the preset warning time.

[0069] If so, a replenishment prompt message is generated and output. The replenishment prompt message includes at least one of the following: displaying alarm information through a human-machine interface terminal, driving an audible and visual alarm to issue a prompt, or sending a replenishment request to the loader control system.

[0070] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0071] The memory stores computer-executed instructions;

[0072] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0073] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0074] The material level monitoring method, apparatus, equipment, and medium provided in this application first acquire image data and three-dimensional point cloud data of the target area; then, the image data and three-dimensional point cloud data are preprocessed to improve the accuracy and consistency of the point cloud data; based on this, the preprocessed image data and three-dimensional point cloud data are fused to compensate for the insufficient sensing capability of a single sensor under complex working conditions; then, semantic segmentation is performed on the fused data based on a pre-trained deep learning model; by extracting and discriminating multi-dimensional features from the fused features, the segmentation result of the target area is output, and the corresponding three-dimensional point cloud data is extracted as the three-dimensional data of the target area, thereby realizing the monitoring of material level. Accurate identification of the target area improves robustness in complex environments such as dust and changing lighting. Furthermore, volume calculation is performed based on the 3D data of the target area. By registering the current point cloud with the reference point cloud and constructing a 3D geometric model, the material volume of the current target area is calculated. This is combined with a pre-acquired full-load reference volume to determine the material level percentage, thus achieving a quantitative description of the material level. Finally, a dual-filter judgment is performed based on the material level percentage to improve the stability and reliability of the material level determination. When the filtering result meets the preset low material level trigger condition, a replenishment prompt is generated and output, thereby achieving intelligent monitoring and timely intervention of the loader's operation process, improving operational efficiency and reducing manual intervention costs. Attached Figure Description

[0075] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0076] Figure 1 Flowchart of the material level monitoring method provided in the embodiments of this application Figure 1 ;

[0077] Figure 2 Flowchart of the material level monitoring method provided in the embodiments of this application Figure 2 ;

[0078] Figure 3 This is a schematic diagram of the material level monitoring device provided in the embodiments of this application;

[0079] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0080] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0082] Articulated loaders are widely used in material loading operations in industries such as mining, construction, ports, and metallurgy. In typical operations, loaders frequently need to unload material from the feed inlets of crushers, conveyors, or silos. For example, in mining, loaders load ore from stockpiles into crushers or silos, and their unloading efficiency directly affects the continuity of subsequent crushing and screening processes. In port bulk cargo handling scenarios, loaders need to load materials such as coal and ore into transport ships or silos, and the unloading speed and material level directly affect the overall efficiency of the loading and unloading operation. However, in actual operations, the material level at the feed inlet (i.e., the material level height) is a key parameter determining the timing of material replenishment. Therefore, there is an urgent need for a material level sensing technology that can adapt to complex environments, achieve continuous material level monitoring, and support automated decision-making to improve the accuracy, efficiency, and safety of loader operations.

[0083] In existing technologies, commonly used material level monitoring methods generally rely on sensors to determine the level. Specifically, radar level gauges can be used to emit electromagnetic waves and receive reflected signals. The distance to the material surface is calculated based on the time of flight or frequency modulated continuous wave (FMCW) principle to determine the material level height.

[0084] However, while existing material level monitoring methods can penetrate dust, their beam angles are large and they are easily affected by false echoes generated by structural components such as the inner wall of the silo and crossbeams. Furthermore, they can only obtain height information at a single point or a few points and cannot perceive the complete three-dimensional shape of the material pile. When the shape of the material pile is irregular, the single-point measurement value deviates significantly from the actual volume.

[0085] Based on this, this application proposes a material level monitoring method. The inventors, addressing the aforementioned problems, considered that for complex environments, if global three-dimensional perception of the material pile could be achieved, and semantic information could be combined to accurately separate the target area, the accuracy and stability of material level monitoring could be improved. Specifically, by introducing a visual sensor to acquire image data of the target area, and combining it with a lidar to acquire high-precision three-dimensional point cloud data, utilizing the complementarity of the two in terms of information dimensions, a mapping relationship between image coordinates and spatial coordinates is first constructed to achieve spatial alignment and fusion of multi-source data, thereby obtaining fused data that combines semantic and geometric structural information. On this basis, further processing is performed using pre-trained deep learning... The Xi model performs semantic segmentation on the fused data, accurately extracts the 3D point cloud data corresponding to the material pile area, and then performs 3D modeling and volume calculation based on the point cloud data. It also determines the material level percentage by combining the preset full-load reference volume. At the same time, in order to improve the stability and reliability of material level judgment, the material level percentage is also subjected to dual filtering processing in the time and spatial dimensions to suppress the influence of instantaneous fluctuations and abnormal noise. When the material level is lower than the preset threshold, a material replenishment prompt message is output. Through the above technical solution, the transformation from single-point height measurement to global 3D volume perception is realized, effectively avoiding measurement errors caused by false echo interference, and significantly improving the accuracy and robustness of material level monitoring under complex working conditions.

[0086] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0087] Figure 1 Flowchart of the material level monitoring method provided in the embodiments of this application Figure 1 ;like Figure 1 As shown, the method includes:

[0088] S101. Acquire image data and 3D point cloud data of the target area.

[0089] It should be understood that this step is used to establish a multi-source sensing foundation for the material status at the feed inlet. The target area can be the crusher feed inlet corresponding to the loader, the conveyor belt receiving hopper, the silo opening, or other material accumulation areas requiring continuous feeding. Image data is used to characterize the surface texture, color distribution, edge contours, and occlusion relationships of the target area at the current moment. 3D point cloud data is used to characterize the 3D coordinate distribution of each sampling point within the target area, reflecting the surface undulations, height variations, and spatial shape of the material pile. 3D point cloud data typically consists of multiple discrete spatial points, each including at least spatial coordinate information; in some implementations, it may also include reflection intensity, color information, or timestamp information. The purpose of acquiring image data and 3D point cloud data is to avoid the problem that a single sensing method can only provide local information and cannot accurately describe the overall shape of an irregular material pile, thereby improving the reliability of subsequent material level identification, volume calculation, and low-level warning from the source.

[0090] In specific implementation, the execution entity of this application embodiment can be an embedded processing unit installed in the loader body, material inlet fixing bracket, or field control cabinet. This embedded processing unit is communicatively connected to a vision camera and a 3D sensing device. The vision camera can be an industrial RGB (Red Green Blue) camera, a binocular camera, a structured light camera, or a composite camera with visible light and infrared imaging capabilities, used to acquire image frames of the target area within the current sampling period. The 3D sensing device can be a lidar, a line laser scanner, an area array depth camera, or other sensors capable of outputting 3D point clouds, used to acquire spatial point clouds corresponding to the time of the image frames. To ensure the consistency of the two types of data in the time dimension, the embedded processing unit can simultaneously send acquisition commands to the camera and lidar via a hardware trigger line, or it can separately receive timestamps uploaded by the two types of sensors and perform frame-level alignment. When using the timestamp alignment method, a maximum allowable time deviation can be set, for example, within tens of milliseconds, binding image frames and point cloud frames that meet the time difference condition into the same set of sensing data to reduce cross-time mismatch caused by loader vibration, material falling, and rapid changes in pile shape.

[0091] In one possible embodiment, to improve data effectiveness under complex operating conditions, the executing entity can also receive environmental parameters before or during data acquisition. These environmental parameters include at least one of dust concentration, ambient light intensity, backlighting, rain / fog conditions, and equipment vibration intensity. Based on these environmental parameters, the entity can adjust the exposure time, gain, shutter mode, supplementary lighting, or infrared band switching of the vision camera, while simultaneously adjusting the emission frequency, scanning angular resolution, or echo intensity threshold of the lidar. For example, in nighttime conditions with high dust concentration and weak ambient light, the vision camera can switch to low-light mode and shorten the exposure time to suppress motion blur, while the 3D sensing device increases the echo filtering threshold to suppress false near points formed by suspended dust, thereby making the acquired image data and point cloud data closer to the actual material pile surface. Alternatively, a main sensor and a backup sensor can be deployed in parallel. When the image clarity of the main vision camera is detected to be below a set threshold, the effective number of points in the main lidar point cloud is below a threshold, or a communication link malfunctions, the system automatically switches to the redundant camera or redundant lidar to continue data acquisition, ensuring the continuity of the monitoring process.

[0092] In terms of data organization, after receiving image data and 3D point cloud data, the embedded processing unit can cache and index the data according to the sampling time, device number, and region number, and establish a one-to-one corresponding multimodal data frame. Each multimodal data frame includes at least an image matrix, a point cloud set, a sampling time, device attitude information, and a synchronization identifier. In the loader's operating environment, the equipment may experience slight shaking or attitude changes. Therefore, it can also synchronously receive attitude angle information output by the inertial measurement unit, vehicle mounting reference information, or the initial pose parameters of the fixed mounting frame for subsequent unified transformation of the point cloud coordinates. Based on the above analysis, by synchronously acquiring image data and 3D point cloud data of the target area, this embodiment of the application retains both texture features and geometric features at the perception layer, providing complete input for subsequent fusion processing, thereby reducing the accumulation of errors caused by dust, occlusion, and surface disturbances in single-point ranging.

[0093] In one possible implementation, before acquiring image data and 3D point cloud data, it is also necessary to calculate the full-load reference volume, specifically:

[0094] First, the intrinsic parameters of the visual camera acquiring image data are calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients. Then, the LiDAR and visual camera acquiring 3D point cloud data are jointly calibrated to obtain the rotation matrix and translation vector from the LiDAR coordinate system to the camera coordinate system. Next, using the camera intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector, a mapping relationship is established between the pixel coordinate system and the 3D spatial coordinate system of the image data and the 3D point cloud data. Further, reference point clouds of the target area are acquired under empty and full-load conditions, resulting in empty and full-load reference point clouds. Finally, the full-load reference volume is calculated based on the empty and full-load reference point clouds.

[0095] The intrinsic parameter calibration of the vision camera is used to determine the focal length, principal point position, and radial and tangential distortion parameters in the camera imaging model. The resulting camera intrinsic parameter matrix and distortion coefficients are used for subsequent image distortion correction and spatial back-projection calculations. The joint calibration of the LiDAR and vision camera is used to solve the extrinsic parameter relationship between their installation attitudes, ensuring consistency between the point cloud coordinate system and the camera coordinate system. After the mapping relationship is established, spatial points in the point cloud can be accurately projected onto the image plane, or image pixels can be inversely calculated to their three-dimensional spatial positions, thus providing a unified coordinate basis for image and point cloud fusion. The empty silo reference point cloud represents the structural contour of the target area in a material-free state, while the full-load reference point cloud represents the geometric shape of the target area when it reaches full load. The full-load reference volume is calculated based on the spatial difference between the two sets of reference point clouds and is used as the volume normalization benchmark for subsequent material level percentage calculations.

[0096] Specifically, the intrinsic parameters of the visual camera are first calibrated to obtain the camera intrinsic parameter matrix K and distortion coefficients d:

[0097]

[0098]

[0099] in, , Focal length , Principal point coordinates, i.e., the intersection of the optical axis and the image plane; For radial distortion parameters; This refers to the tangential distortion parameter.

[0100] Furthermore, the lidar and camera are jointly calibrated to obtain the rotation matrix R and translation vector t from the lidar coordinate system to the camera coordinate system.

[0101]

[0102] in For points in the lidar point cloud, These are the coordinates of the corresponding point in the camera coordinate system.

[0103] The projection relationship from the camera coordinate system to the image pixel coordinate system is as follows:

[0104]

[0105] Where s is the scale factor and (u, v) are the pixel coordinates.

[0106] In one feasible approach, the empty and full-load reference point clouds can be acquired across multiple frames under empty and full-load conditions, respectively, and then averaged over time and outlier removal can be performed to reduce the impact of single-frame noise. The full-load reference volume can be obtained by integrating the voxelized difference between the full-load and empty reference point clouds, or by calculating the volume after mesh reconstruction, facilitating comparison between the current volume and the full-load state in subsequent monitoring.

[0107] Taking the process of obtaining the full-load reference volume by integrating after voxelization difference as an example:

[0108] First, with the material inlet empty, a reference point cloud of the empty hopper is obtained by scanning with LiDAR. When the material inlet is fully loaded, a baseline cloud of the full warehouse is obtained through LiDAR scanning. Then, calculate the full-load reference volume according to the following calculation formula. :

[0109]

[0110] Where S is the material region obtained by differentiating the full warehouse point cloud from the empty warehouse baseline. Let Voxel volume be the discretized volume. In actual 3D point cloud processing, the shape of the material pile is an irregular continuous surface, which cannot be directly integrated using integral formulas. Accurate volume calculation is required. Therefore, it is necessary to divide the continuous space into many extremely small cubic grids (like cutting three-dimensional space into countless small squares).

[0111] It is understandable that this implementation method can enhance the spatial correspondence between images and point clouds in complex dust and lighting environments, reducing misalignment fusion and volume estimation deviations. At the same time, the pre-establishment of the benchmark point cloud and the full-load reference volume is conducive to forming a stable quantitative reference, making material level status identification and replenishment judgment more reliable, thereby improving the continuity and engineering applicability of material level monitoring at the loader's loading port.

[0112] S102. Preprocess the image data and the 3D point cloud data respectively, and then fuse the preprocessed image data and the 3D point cloud data to obtain fused data.

[0113] In one possible implementation, the image data is first subjected to distortion correction to obtain preprocessed image data; then, the 3D point cloud data is subjected to statistical filtering and voxel filtering to obtain processed 3D point cloud data; then, according to the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and each valid point cloud data is assigned corresponding pixel color information to obtain fused data.

[0114] It should be understood that image data can be acquired by a vision camera mounted on a loader. Distortion correction processing is used to eliminate the influence of radial and tangential distortion of the lens on the geometric relationship of the image, so as to ensure the coordinate consistency of subsequent projection mapping. 3D point cloud data can be acquired by LiDAR. Statistical filtering is used to remove outliers formed by dust, reflection interference, or isolated echoes. Voxel filtering is used to discretize the spatial point cloud according to a preset voxel size and then aggregate it to reduce the point cloud density and retain the main geometric contours of the target area.

[0115] In the actual processing, the image data of the material inlet area is first subjected to distortion correction, specifically as follows:

[0116]

[0117] in, is the actual pixel position in the distorted image, and d is the camera's distortion coefficient, which is usually a vector containing multiple parameters. and For raw pixels The corrected pixel coordinates obtained after applying the distortion correction algorithm.

[0118] Subsequently, statistical filtering and voxel filtering processes were performed on the 3D point cloud data, specifically;

[0119] First, outliers are removed using statistical filtering, i.e., each point is calculated... To its The average distance of the nearest neighbors ,like Exceeding the global mean add Double standard deviation Then remove:

[0120]

[0121] Voxel filtering: Downsampling the point cloud and dividing the space into segments with sides of length 1. A voxel grid, preserving the centroid point within each voxel:

[0122]

[0123] in, The centroid of all points within the current voxel (i.e., the points retained after filtering) is typically a three-dimensional coordinate vector. This represents the total number of points in the original point cloud contained within the current voxel. Let be the coordinate vector of the j-th point within the current voxel.

[0124] Based on the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and color values ​​are read from the corresponding pixel positions to assign red, green, and blue color attributes to the point cloud, forming fused data that simultaneously contains spatial coordinates and texture information. This fused data can be directly input into the subsequent semantic segmentation model to improve the recognition accuracy of material pile boundaries and spatial distribution.

[0125] Projecting the LiDAR point cloud onto the image plane yields the pixel coordinates of each point:

[0126]

[0127] in This represents the camera projection function (including intrinsic parameters and distortion).

[0128] Understandably, through the above processing, the geometric distortion of the image is corrected, the noise and redundant data in the point cloud are suppressed, and the fused data can more accurately reflect the real spatial shape and surface color characteristics of the target area, thereby improving the stability and robustness of material level identification and reducing the probability of misjudgment in complex dust environments.

[0129] S103. Based on a pre-trained deep learning model, semantic segmentation is performed on the fused data to obtain the three-dimensional data of the target region.

[0130] In one possible implementation, the fused data is first input into a pre-trained deep learning model to obtain the probability value of each point cloud in the fused data output by the model belonging to the target region; then the probability value is compared with a preset probability threshold, and a segmentation mask is generated based on the comparison result; finally, the point cloud set belonging to the material region is extracted from the fused data using the segmentation mask as the three-dimensional data of the target region.

[0131] The trained deep learning model is trained on historical fusion data with semantic annotations, which are used to identify point cloud points in the historical fusion data that belong to the material pile area.

[0132] It should be noted that the fused data is multimodal data formed by coordinate mapping and feature alignment of image information and 3D point cloud information. It preserves both the texture differences on the material pile surface and its spatial geometric contours. The deep learning model can be constructed using a point cloud segmentation network, an image-point cloud joint encoding network, or a multi-scale semantic segmentation network. The probability value output by the model represents the confidence level that each point in the point cloud belongs to the material pile region. Historical fused data can be obtained through manual or semi-automatic annotation. Semantic annotation is used to distinguish between the material pile, background, silo walls, and occlusion structures to improve the model's adaptability to complex environments. In practical applications, other network models can also be selected for this model; this embodiment does not limit this selection.

[0133] In practical implementation, after the fused data is input into the model, the model first extracts local geometric features and neighborhood context features for each point cloud point, and then combines color and depth features to output the corresponding probability value. The preset probability threshold can be set as a fixed value based on the training set validation results, or it can be dynamically adjusted based on on-site dust concentration, light intensity, and model confidence statistics to reduce false positives and false negatives. The segmentation mask generated by the comparison results is used to mark point cloud points above the threshold as target points and filter them out from the fused data to form a continuous point cloud set of material regions. This point cloud set can be directly used as the three-dimensional data of the target region to input into the subsequent volume calculation module, thereby achieving an accurate representation of the spatial distribution of the material pile.

[0134] Understandably, this implementation method uses a deep learning model trained on semantically labeled historical data to perform point-level segmentation on the fused data. This effectively eliminates interference from silo walls, background, and obstructions while maintaining the ability to identify irregular material pile boundaries. Because a probability threshold is used to generate the segmentation mask, the model output can be converted into a stable binary judgment result, resulting in higher purity and completeness of the extracted 3D point cloud. The resulting 3D data of the target area more realistically reflects the material accumulation morphology, providing a reliable basis for subsequent volume conversion and material level determination, thereby improving the accuracy, robustness, and continuity of material level monitoring at the feed inlet.

[0135] S104. Calculate the material volume of the current target area based on the three-dimensional data of the target area, and determine the material level percentage based on the full-load reference volume.

[0136] In one possible implementation, the three-dimensional data of the target area is first registered with the empty warehouse reference point cloud to obtain aligned point cloud data; then, the aligned point cloud data and the empty warehouse reference point cloud are differentially calculated to obtain a differential point cloud representing the material distribution; then, the differential point cloud is reconstructed in three dimensions to obtain the material volume of the current target area; finally, the material level percentage is determined based on the ratio of the material volume of the current target area to the full-load reference volume.

[0137] It should be understood that, in practical implementations, the 3D data of the target area is typically acquired by LiDAR or a depth camera and unified with the empty warehouse reference point cloud in the same coordinate system before being used for volume calculation. The registration process can employ an iterative nearest-point algorithm to iteratively match the currently acquired point cloud with the empty warehouse reference point cloud, ensuring that they overlap as much as possible in spatial position and orientation, thereby reducing volume deviations caused by equipment installation errors. To improve registration stability, the point cloud can undergo voxel downsampling and outlier removal before entering iterative matching to reduce interference from noise points on the matching results. In practical applications, other registration algorithms can also be selected, and this application embodiment does not limit this choice.

[0138] After registration is completed, the aligned point cloud data is compared with the empty warehouse baseline point cloud using differential calculation. Specifically:

[0139]

[0140] in, It is a point set with the same resolution as the stockpile point cloud obtained by interpolating the empty silo baseline.

[0141] Subsequently, 3D reconstruction of the difference point cloud can be performed using triangulation (such as Delaunay triangulation) or voxel integration. Taking triangulation as an example, the specific calculation process is as follows:

[0142] Construct a triangular mesh, each triangle Together with a reference plane (such as the material inlet plane), it forms a triangular prism with the following volume:

[0143]

[0144] Finally, after calculating the volume, the ratio of this volume to the full-load reference volume is calculated to obtain the material level percentage, which can intuitively represent the current degree of material occupancy.

[0145] It is understandable that, because the volume calculation in this embodiment is based on the overall geometry of the target area rather than a single-point distance measurement, it can more accurately reflect the actual inventory of irregular material piles. After calculating the ratio with the full-load reference volume, the material level percentage can be directly used for material replenishment judgment and status monitoring, thereby improving the stability, continuity, and anti-interference ability of material level identification in complex environments.

[0146] S105. Perform dual filtering judgment based on the material level percentage. When the judgment result meets the preset low material level trigger condition, generate and output material replenishment prompt information.

[0147] This step is used to suppress misjudgments caused by instantaneous fluctuations based on the already obtained quantitative material level results, and to achieve stable and timely low material level early warnings by combining historical trends. The dual filtering judgment can be understood as applying two different types of stabilization processing to the material level percentage sequentially, including at least time-dimensional filtering and trend-dimensional or spatial correlation-dimensional filtering. The low material level trigger condition is a set of rules used to determine whether replenishment is needed in the current state, and may include one or more of the following: material level percentage threshold, duration threshold, consecutive frame count threshold, descent rate threshold, and predicted remaining time threshold. The replenishment prompt information is a status signal output to the operator, vehicle controller, or remote dispatch system, which can be presented in the form of displayed text, graphical interface color changes, voice broadcast, audible and visual alarms, bus messages, or wireless messages.

[0148] It should be noted that the specific implementation process of this embodiment is as follows: Figure 2 The embodiments are described in detail, and will not be repeated in this embodiment.

[0149] The material level monitoring method provided in this application first acquires image data and 3D point cloud data of the target area; then, it preprocesses the image data and 3D point cloud data respectively to improve the accuracy and consistency of the point cloud data; based on this, it fuses the preprocessed image data and 3D point cloud data to compensate for the insufficient sensing capability of a single sensor under complex working conditions; then, it performs semantic segmentation on the fused data based on a pre-trained deep learning model, extracts and discriminates multi-dimensional features from the fused features, outputs the segmentation result of the target area, and extracts the corresponding 3D point cloud data as the 3D data of the target area, thereby achieving accurate identification of the material area. Furthermore, it enhances robustness in complex environments such as dust and changing lighting conditions. It further calculates volume based on the 3D data of the target area by registering the current point cloud with a reference point cloud and constructing a 3D geometric model. This yields the material volume of the current target area, which is then combined with a pre-acquired full-load reference volume to determine the material level percentage, thus achieving a quantitative description of the material level. Finally, it performs dual filtering based on the material level percentage to improve the stability and reliability of the material level determination. When the filtering result meets the preset low material level trigger condition, it generates and outputs a material replenishment prompt, thereby achieving intelligent monitoring and timely intervention of the loader's operation process, improving operational efficiency and reducing manual intervention costs.

[0150] In one possible implementation, after determining the material level percentage, an exponentially weighted moving average method can be used to update the full-load reference volume based on the measured stockpile volume obtained after each unloading or replenishment.

[0151] Among them, the exponentially weighted moving average method refers to assigning a high weight to the newly obtained measured stockpile volume, while assigning a gradually decreasing weight to the historical full-load reference volume, so that the update results take into account both real-time performance and smoothness.

[0152] It should be understood that, in actual implementation, after calculating the material level percentage, it can be determined whether a complete unloading or replenishment has occurred based on the current operational status. Once the material morphology is confirmed to be stable, the measured stockpile volume under this stable state is sampled. This volume is then input into an exponentially weighted moving average update model, and the current measured value is weighted and fused with the original full-load reference volume according to a preset smoothing coefficient to obtain a new full-load reference volume. This smoothing coefficient can be set to a value between 0 and 1 based on the degree of material fluctuation on site. When the coefficient is larger, the new measurement value has a stronger impact on the update result, reflecting changes in the full-load state more quickly; when the coefficient is smaller, the historical reference value maintains stronger stability. To avoid short-term abnormal fluctuations affecting the reference volume, the stability of the measured stockpile volume can be verified before updating. The reference volume update is only performed when multiple consecutive measurement results meet the deviation threshold requirement. The volume calculation results can be derived from the 3D reconstructed data after differential analysis with the empty silo reference point cloud. The updated full-load reference volume can be written to the storage module for repeated use in subsequent material level percentage calculations.

[0153] The specific update formula is as follows:

[0154]

[0155] in, This is the current updated full-load reference volume; The full-load reference volume from the previous moment; This is the current measured full-load volume; This is the historical weighting coefficient (smoothing factor).

[0156] Understandably, through the above methods, the system can continuously correct the full-load reference volume after each unloading or replenishment, making the reference value closer to the current material state. This reduces the cumulative error caused by changes in material density, angle of repose, and environmental disturbances, and improves the consistency and reliability of material level percentage calculation. Because the full-load reference volume can adaptively update according to on-site operating conditions, the triggering basis for replenishment prompts is also more stable, thereby reducing the probability of false alarms and missed alarms, and improving the long-term accuracy and engineering adaptability of material level monitoring at the feed inlet.

[0157] Figure 2 Flowchart of the material level monitoring method provided in the embodiments of this application Figure 2 ;like Figure 2 As shown, in this embodiment... Figure 1Based on the examples, the process of performing dual filtering is described in detail, and the method includes:

[0158] S201. Obtain the material level percentage of a consecutive preset number of frames, and determine whether the material level percentage of the consecutive preset number of frames is lower than the preset low material level percentage threshold. If yes, execute S202; otherwise, execute S206.

[0159] The preset number of consecutive frames represents multiple material level detection results acquired continuously at a fixed sampling period. By continuously determining the material level percentage at multiple moments, false alarms caused by single-frame noise, momentary occlusion, and dust interference can be suppressed. The preset low material level percentage threshold defines the critical percentage that requires triggering a replenishment warning.

[0160] The specific judgment formula is as follows:

[0161]

[0162] in, The final output value is 1, which indicates that a low material level alarm is triggered, and 0 indicates that it is not triggered. The set low material level threshold; For at any time The calculated material level percentage, where N represents the threshold number of frames required to determine a continuous low material level.

[0163] Understandably, by consistently determining the percentage of material level across multiple consecutive frames, instantaneous fluctuations and sensor noise can be effectively suppressed, thereby avoiding misjudgments caused by single-frame anomalies (such as dust obstruction, missing point clouds, or sudden changes in illumination). This step essentially constructs a robust constraint mechanism in the time dimension, elevating the "low material level determination" from a single-point decision to a sequential decision, significantly improving the stability and reliability of the system under complex operating conditions.

[0164] S202. The material volume threshold is calculated based on the product of the preset low material level percentage threshold and the full-load reference volume.

[0165] The specific calculation formula is as follows:

[0166]

[0167] in, This is the material volume threshold. This is the full-load reference volume; This is a preset low material level percentage threshold.

[0168] Understandably, by mapping the low material level percentage threshold to the actual material volume threshold, the conversion from relative quantity (percentage) to absolute quantity (volume) is achieved, giving the system a unified physical criterion. This step eliminates scale differences between different silo sizes and under different calibration conditions, enabling subsequent calculations to be performed under a unified dimension for reasoning and prediction, thereby improving the system's portability and engineering adaptability.

[0169] S203. Based on the current material volume in the target area, the material volume threshold, and the historical average material consumption rate, calculate the remaining time required for the material to drop to the low material level threshold from the current moment.

[0170] Specifically, the formula for calculating the remaining time is:

[0171]

[0172] in, The remaining time required for the material level to drop to the low threshold at the current moment. This is the material volume threshold. This represents the historical average material consumption rate. This represents the current material volume.

[0173] Understandably, by combining the current material volume with the historical average consumption rate, the material decline trend can be modeled, thereby expanding the capability from state perception to trend prediction. This allows for the estimation of the time window when the low material level threshold is reached, transforming the system from passive detection to proactive prediction. This provides a forward-looking basis for scheduling and replenishment decisions, significantly improving operational continuity.

[0174] S204. Determine whether the remaining time is lower than the preset warning time. If yes, proceed to S205; otherwise, proceed to S206.

[0175] Understandably, by comparing the predicted remaining time with the preset warning time, a controllable warning triggering mechanism can be established, enabling fine-tuning of the warning timing. This allows the system to adaptively configure itself based on actual production rhythms (such as loader response time and transportation cycles), thereby avoiding resource waste due to premature warnings or material shortages due to late warnings, and improving overall operational coordination and efficiency.

[0176] S205. Generate and output material replenishment prompt information.

[0177] The replenishment prompt information includes at least one of the following: displaying alarm information through a human-machine interface terminal, triggering an audible and visual alarm to issue a prompt, or sending a replenishment request to the loader control system.

[0178] In one possible embodiment, to prevent repeated alarms from interfering with operation, the executing entity can enter an alarm hold state after the replenishment prompt is output, and set a release condition, such as releasing the alarm after the material level percentage rises above the recovery threshold and remains above it for several cycles; alternatively, an alarm suppression time window can be set to prevent high-priority alarms from being repeatedly sent before the same low material level event is resolved, and only update the status data. Based on the above analysis, it can be seen that this step, by implementing dual filtering judgment on the material level percentage, transforms the original continuous measurement value into a stable and reliable basis for replenishment decisions, which can effectively solve the problems of false alarms and missed alarms coexisting and inaccurate replenishment timing in continuous on-site monitoring, thereby improving the continuity of loader loading operations and the stability of production line operation.

[0179] Understandably, by combining continuous multi-frame judgment with time prediction based on consumption rate, the system can first filter instantaneous fluctuations and then determine whether the low material level is persistent and urgent, thereby improving the reliability and foresight of low material level warnings. This method can avoid false alarms caused by relying solely on a single material level drop, and can also issue replenishment prompts in advance when the material is indeed close to being insufficient, giving operators sufficient time to arrange replenishment operations, reducing downtime and waiting time, and lowering the risk of excessive accumulation.

[0180] S206. Conduct the next round of material level monitoring.

[0181] Figure 3 This is a schematic diagram of the material level monitoring device provided in the embodiments of this application; as shown. Figure 3 As shown, the device includes:

[0182] The acquisition module 301 is used to acquire image data and 3D point cloud data of the target area;

[0183] The first processing module 302 is used to preprocess the image data and the three-dimensional point cloud data respectively, and to fuse the preprocessed image data and the three-dimensional point cloud data to obtain fused data.

[0184] The second processing module 303 is used to perform semantic segmentation on the fused data based on a pre-trained deep learning model to obtain the three-dimensional data of the target region.

[0185] The determination module 304 is used to calculate the material volume of the current target area based on the three-dimensional data of the target area, and determine the material level percentage based on the full-load reference volume;

[0186] The judgment module 305 is used to perform dual filtering judgment based on the material level percentage. When the judgment result meets the preset low material level trigger condition, it generates and outputs material replenishment prompt information.

[0187] In one possible implementation, the apparatus further includes a third processing module, which, before acquiring the image data and the 3D point cloud data, is configured to:

[0188] Perform intrinsic parameter calibration on the visual camera that acquires image data, and obtain the camera intrinsic parameter matrix and distortion coefficients;

[0189] The lidar and vision camera that acquire 3D point cloud data are jointly calibrated to obtain the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system.

[0190] By using the camera intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector, a mapping relationship between the pixel coordinate system and the three-dimensional spatial coordinate system of image data and 3D point cloud data is established.

[0191] The baseline point clouds of the target area are obtained in the empty and full-load states, respectively, to obtain the empty warehouse baseline point cloud and the full-load baseline point cloud.

[0192] The full-load reference volume is calculated based on the empty warehouse reference point cloud and the full-load reference point cloud.

[0193] In one possible implementation, the first processing module 302 is specifically used for:

[0194] The image data is subjected to distortion correction to obtain preprocessed image data;

[0195] Statistical filtering and voxel filtering are performed on the 3D point cloud data to obtain the processed 3D point cloud data.

[0196] Based on the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and each valid point cloud data is assigned corresponding pixel color information to obtain fused data.

[0197] In one possible implementation, the second processing module 303 is specifically used for:

[0198] The fused data is input into a pre-trained deep learning model, and the probability value of each point cloud in the fused data output by the model is obtained to belong to the target region. The trained deep learning model is trained based on historical fused data with semantic annotations. The semantic annotations are used to identify point cloud points in the historical fused data that belong to the material pile region.

[0199] The probability value is compared with a preset probability threshold, and a segmentation mask is generated based on the comparison result.

[0200] The point cloud set belonging to the material region is extracted from the fused data using a segmentation mask, and used as the three-dimensional data of the target region.

[0201] In one possible implementation, the determining module 304 is specifically used for:

[0202] The 3D data of the target area is registered with the empty warehouse reference point cloud to obtain aligned point cloud data;

[0203] The aligned point cloud data is compared with the empty warehouse baseline point cloud by differential calculation to obtain the differential point cloud representing the material distribution.

[0204] The material volume of the current target area is obtained by performing 3D reconstruction of the differential point cloud;

[0205] The material level percentage is determined based on the ratio of the current material volume in the target area to the full-load reference volume.

[0206] In one possible implementation, the apparatus further includes an update module, which, after determining the material level percentage, is used to:

[0207] The full-load reference volume is updated using the exponentially weighted moving average method based on the measured stockpile volume obtained after each unloading or replenishment.

[0208] In one possible implementation, the determination module 305 is specifically used for:

[0209] Obtain the material level percentage for a consecutive preset number of frames, and determine whether the material level percentage for the consecutive preset number of frames is lower than a preset low material level percentage threshold.

[0210] If so, the material volume threshold is calculated based on the product of the preset low material level percentage threshold and the full-load reference volume.

[0211] Based on the current material volume in the target area, the material volume threshold, and the historical average material consumption rate, the remaining time required for the material to drop to the low material level threshold from the current moment is calculated, and it is determined whether the remaining time is lower than the preset warning time.

[0212] If so, a replenishment prompt message is generated and output. The replenishment prompt message includes at least one of the following: displaying alarm information through the human-machine interface terminal, driving the audible and visual alarm to issue a prompt, or sending a replenishment request to the loader control system.

[0213] The material level monitoring device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0214] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0215] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0216] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0217] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0218] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0219] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0220] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0221] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0222] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0223] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0226] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0228] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A material level monitoring method, characterized in that, include: Acquire image data and 3D point cloud data of the target area; The image data and the 3D point cloud data are preprocessed separately, and the preprocessed image data and the 3D point cloud data are fused together to obtain fused data. The fused data is semantically segmented based on a pre-trained deep learning model to obtain the three-dimensional data of the target region. The material volume of the current target area is calculated based on the three-dimensional data of the target area, and the material level percentage is determined based on the full-load reference volume. The material level percentage is used for dual filtering and judgment. When the judgment result meets the preset low material level trigger condition, a material replenishment prompt message is generated and output.

2. The method according to claim 1, characterized in that, Before acquiring image data and 3D point cloud data, the method further includes: The intrinsic parameters of the visual camera that acquires the image data are calibrated to obtain the camera intrinsic parameter matrix and distortion coefficients. The lidar and the vision camera that acquire the 3D point cloud data are jointly calibrated to obtain the rotation matrix and translation vector from the lidar coordinate system to the camera coordinate system. The mapping relationship between the pixel coordinate system and the three-dimensional spatial coordinate system of the image data and the three-dimensional point cloud data is established by using the camera intrinsic parameter matrix, distortion coefficient, rotation matrix and translation vector. The baseline point clouds of the target area are obtained in the empty and full-load states, respectively, to obtain the empty warehouse baseline point cloud and the full-load baseline point cloud. The full-load reference volume is calculated based on the empty warehouse reference point cloud and the full-load reference point cloud.

3. The method according to claim 2, characterized in that, The image data and 3D point cloud data are preprocessed separately, and the preprocessed image data and 3D point cloud data are fused to obtain fused data, including: The image data is subjected to distortion correction processing to obtain preprocessed image data; The three-dimensional point cloud data is subjected to statistical filtering and voxel filtering to obtain the processed three-dimensional point cloud data. According to the mapping relationship, the preprocessed 3D point cloud data is projected onto the image plane of the preprocessed image data, and each valid point cloud data is assigned corresponding pixel color information to obtain the fused data.

4. The method according to claim 3, characterized in that, The pre-trained deep learning model performs semantic segmentation on the fused data to obtain three-dimensional data of the target region, including: The fused data is input into a pre-trained deep learning model, and the probability value of each point cloud point in the fused data output by the model belonging to the target region is obtained. The trained deep learning model is trained based on historical fused data with semantic annotations. The semantic annotations are used to identify point cloud points in the historical fused data that belong to the material pile region. The probability value is compared with a preset probability threshold, and a segmentation mask is generated based on the comparison result. The point cloud set belonging to the material region is extracted from the fused data using the segmentation mask, and used as the three-dimensional data of the target region.

5. The method according to claim 2, characterized in that, The step of calculating the material volume of the current target area based on the three-dimensional data of the target area, and determining the material level percentage based on the full-load reference volume, includes: The three-dimensional data of the target area is registered with the empty warehouse reference point cloud to obtain aligned point cloud data; The aligned point cloud data is compared with the empty warehouse reference point cloud by differential calculation to obtain a differential point cloud representing the material distribution. The material volume of the current target region is obtained by performing three-dimensional reconstruction on the differential point cloud; The material level percentage is determined based on the ratio of the material volume in the current target area to the full-load reference volume.

6. The method according to claim 5, characterized in that, After determining the material level percentage, the method further includes: The full-load reference volume is updated using the exponentially weighted moving average method based on the measured stockpile volume obtained after each unloading or replenishment.

7. The method according to claim 2, characterized in that, The process involves dual filtering based on the material level percentage. When the judgment result meets the preset low material level trigger condition, a replenishment prompt message is generated and output, including: Obtain the material level percentage of a consecutive preset number of frames, and determine whether the material level percentage of the consecutive preset number of frames is lower than a preset low material level percentage threshold. If so, the material volume threshold is calculated based on the product of the preset low material level percentage threshold and the full-load reference volume. Based on the material volume, material volume threshold, and historical average material consumption rate of the current target area, the remaining time required for the material to drop to the low material level threshold from the current moment is calculated, and it is determined whether the remaining time is lower than the preset warning time. If so, a replenishment prompt message is generated and output. The replenishment prompt message includes at least one of the following: displaying alarm information through a human-machine interface terminal, driving an audible and visual alarm to issue a prompt, or sending a replenishment request to the loader control system.

8. A material level monitoring device for an articulated loader, characterized in that, include: The acquisition module is used to acquire image data and 3D point cloud data of the target area; The first processing module is used to preprocess the image data and the three-dimensional point cloud data respectively, and to fuse the preprocessed image data and the three-dimensional point cloud data to obtain fused data. The second processing module is used to perform semantic segmentation on the fused data based on a pre-trained deep learning model to obtain three-dimensional data of the target region. The determination module is used to calculate the material volume of the current target area based on the three-dimensional data of the target area, and to determine the material level percentage based on the full-load reference volume; The judgment module is used to perform dual filtering judgment based on the material level percentage. When the judgment result meets the preset low material level trigger condition, it generates and outputs material replenishment prompt information.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images