Material detection method based on fusion of laser radar and millimeter wave radar

Abstract

The invention discloses a material detection method based on fusion of a laser radar and a millimeter wave radar, and relates to the field of material monitoring, and the method comprises the following steps: obtaining three-dimensional point cloud data of the surface of a material in a stock bin through the laser radar, obtaining distance measurement data of the material in the stock bin through the millimeter wave radar, and inputting the two data into a fusion processing module; and a preset time synchronization mechanism is adopted, weight coefficients of the two types of data are dynamically adjusted based on environmental parameter changes, fused material level information is obtained, and the fused material level information is transmitted to a monitoring terminal to be displayed and recorded. The method provided by the invention effectively solves the problems of inaccurate measurement and slow response of a traditional detection system in severe environments such as dust and moisture, and has the advantages of high precision, strong real-time performance and wide adaptability.

Description

Technical Field

[0001] The present invention relates to the field of material monitoring, and more specifically, to a material detection method based on the fusion of lidar and millimeter-wave radar. Background Art

[0002] At present, in industrial production, the requirements for real-time monitoring of the material state in the silo are increasing day by day. Especially in the field of large-scale bulk material storage, it is crucial to obtain the material surface height in a timely and accurate manner for production scheduling and safety management. Existing monitoring systems usually use a single sensor such as lidar, ultrasonic wave or millimeter-wave radar to obtain material surface data. However, a single-sensor system is prone to measurement errors or data loss when facing strong dust, heavy moisture and other environmental interferences. At present, some systems lack the ability to automatically identify abnormal material surface phenomena (such as material pile collapse, wall hanging or cavities), resulting in deviation or missed reporting of monitoring results, which in turn affects production safety. The above problems indicate that there is still room for improvement in the existing technology in aspects such as sensor data integration, dynamic adjustment and abnormal identification. Therefore, there is an urgent need for a new method that can achieve high-precision measurement in a harsh environment, automatically optimize the data fusion strategy according to the changes in the silo state, and give an early warning of abnormal material surfaces in a timely manner. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a material detection method based on the fusion of lidar and millimeter-wave radar to solve the problems mentioned in the background art.

[0004] To achieve the above object, the present invention adopts the following technical solutions: A material detection method based on the fusion of lidar and millimeter-wave radar, comprising the following steps: Obtaining three-dimensional point cloud data of the material surface in the silo by using lidar; Obtaining distance measurement data of the material in the silo by using millimeter-wave radar; Inputting the three-dimensional point cloud data and the distance measurement data into a fusion processing module, and the fusion processing module performs time alignment on the three-dimensional point cloud data and the distance measurement data based on a preset time synchronization mechanism, and dynamically adjusts the weight coefficients of the two types of data according to the changes in environmental parameters to obtain the fused material level information; Transmitting the fused material level information to a monitoring terminal, and performing information display and recording on the monitoring terminal.

[0005] In an optional embodiment, the method further includes: Collect the target area image by using a camera set inside the silo, perform haze analysis on the target area image to obtain the image haze score at the current moment, determine the confidence adjustment factor of the lidar data based on the image haze score, and then input the confidence adjustment factor as the weight parameter of the lidar data during fusion processing into the fusion algorithm.

[0006] In an alternative embodiment, the method further performs abnormal stock surface recognition on the fused three-dimensional point cloud data, including the following steps: Perform surface reconstruction on the three-dimensional point cloud data to obtain a continuous stock surface grid structure; Calculate the curvature change value of each area in the stock surface grid structure; Judge the area where the curvature change value exceeds the set threshold as an abnormal area; According to the spatial position, inclination angle characteristics and curvature continuity of the abnormal area, identify whether there is any one or more of the following abnormalities: overhead, cavity, stockpile collapse, residue hanging on the wall; and output the corresponding abnormal type information.

[0007] In an alternative embodiment, the dynamic weight adjustment adopts the following fusion formula: ; Where: F represents the fused material level information; P represents the measurement value corresponding to the three-dimensional point cloud data obtained by the lidar; D represents the distance measurement data obtained by the millimeter wave radar; represents the weight coefficient of the lidar data; represents the weight coefficient of the millimeter wave radar data; And satisfy .

[0008] In an alternative embodiment, the image haze score is used to determine the confidence adjustment factor of the lidar data , and the calculation formula is as follows: ; Where: represents the confidence adjustment factor of the lidar data; H represents the image haze score; K is a preset positive coefficient used to adjust the influence of the haze score on the adjustment factor.

[0009] In an alternative embodiment, the difference between the normal vectors of adjacent grid cells is used as the curvature change value, and its calculation formula is: ; Wherein: represents the unit normal vector of region i; represents the unit normal vector of region j; represents the Euclidean norm; K represents the curvature change value.

[0010] In an alternative embodiment, the surface reconstruction of the three-dimensional point cloud data is implemented by using the Poisson surface reconstruction algorithm.

[0011] In an alternative embodiment, the abnormal region recognition step further includes comparing the three-dimensional point cloud data after fusion of multiple consecutive frames to determine the persistence of the abnormal region in time, so as to exclude misjudgments caused by instantaneous noise or environmental changes.

[0012] In an alternative embodiment, the monitoring terminal includes a display unit for three-dimensional visualization display. The display unit generates a three-dimensional image according to the fused stock surface information, and marks the detected abnormal region with a preset color or symbol.

[0013] In an alternative embodiment, the method further includes the following steps: Detect the feeding and discharging states of the silo by using a state detection device arranged at the feeding or discharging port of the silo; Judge whether the silo is in the feeding state, discharging state or static state according to the state detection result; When the silo is in the feeding or discharging state, adaptively switch the sampling frequency, weight coefficient and data fusion strategy of the lidar and millimeter wave radar; When the silo is in the static state, restore the preset conventional radar working mode.

[0014] The advantages of the present invention over the prior art are as follows. By adopting the method of fusing lidar and millimeter-wave radar data, the present invention realizes high-precision and real-time monitoring of the surface state of materials in the silo, overcoming the defects of traditional single sensors being vulnerable to interference and inaccurate measurement in harsh environments such as dust and moisture. By introducing a preset time synchronization mechanism and a dynamic weight adjustment strategy, effective complementarity of the data of the two types of sensors is achieved, significantly improving the stability and accuracy of the fused level information. At the same time, by using the camera installed inside the silo to collect images and perform haze analysis, the system can automatically adjust the confidence of the lidar data according to environmental changes, thereby reducing the weight of the lidar data when the environment deteriorates, further enhancing the robustness of the overall measurement. Coupled with surface reconstruction and local curvature analysis of the fused three-dimensional point cloud data, the system can automatically identify abnormal phenomena such as overhead, voids, stockpile collapses, and residue wall hanging, providing an effective basis for production safety early warning. This method not only has the advantages of high precision, real-time performance, and wide adaptability, but also greatly reduces false alarms and missed alarms caused by environmental changes, and has significant application value for ensuring production safety and optimizing silo management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is the overall flowchart of the method of the present invention; Figure 2 is the schematic diagram of the haze analysis of the present invention; Figure 3 is the schematic diagram of the surface modeling of the present invention; Figure 4 is the schematic flowchart of the dynamic adjustment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following describes the specific embodiments of the present invention with reference to the drawings.

[0017] As Figure 1 shown is the overall flowchart of the present invention. The overall system of the present invention is used to detect the state of materials in the silo. First, the system obtains the three-dimensional point cloud data of the surface of the materials inside the silo through the lidar, and at the same time uses the millimeter-wave radar to collect the distance measurement data of the materials.

[0018] Before these two types of data are input into the fusion processing module, the system will first assign a unified timestamp to the lidar and millimeter-wave radar data frames to achieve strict time synchronization. In implementation, the acquisition time of the lidar data frame is denoted as , and the acquisition time of the millimeter-wave radar data frame is denoted as , and the system requires that: ; where Δt is the preset maximum allowable time difference, in seconds (s), to ensure that the two types of data are fused under the same time reference, thus avoiding measurement errors caused by time differences. After receiving the time-aligned data, the system dynamically adjusts the weights of the lidar and millimeter-wave radar data using the preset environmental parameters and real-time working conditions, and calculates the fused material level information using the following fusion formula.

[0019] Specifically, let the measurement value corresponding to the three-dimensional point cloud data obtained by the lidar be P, the distance measurement data obtained by the millimeter-wave radar be D, the weight coefficient of the lidar data be and the weight coefficient of the millimeter-wave radar data be , then the fused material level information F can be expressed as: ; where the unit of F is usually meters (m) or other applicable distance units, and it can represent the numerical value of the overall height or stacking level of the material obtained after data fusion processing.

[0020] It is required that the weight coefficients satisfy ; to ensure the normalization of the overall measurement.

[0021] The adjustment of the dynamic weight coefficient depends on the real-time environmental state. For example, in the case of high dust concentration or other large interferences, the system can reduce the weight of the lidar data according to the supplementary environmental information, and correspondingly increase the proportion of the millimeter-wave radar data to ensure the stability and accuracy of the output result.

[0022] In another embodiment, the formula for evaluating the overall measurement error is as follows: ; where E represents the overall measurement error (the unit is the same as the material level information, such as meters), and are the weight coefficients (dimensionless) of the lidar and millimeter-wave radar data D respectively, represents the standard deviation error of the lidar data P, represents the standard deviation error of the millimeter-wave radar data.

[0023] To further optimize the weight coefficient of the lidar data, as shown in Figure 2, the present invention also sets a camera inside the silo to collect image data of the target area, and obtains the image haze score H at the current moment by analyzing the haze of the image. Using this image haze score, the system calculates the confidence adjustment factor of the lidar data, and its calculation formula is ; Where H represents the image haze score, which can be a dimensionless value or measured in a preset unit. K is a preset positive coefficient with the unit of 1 / (unit of H), used to adjust the influence of the haze score on the weight of lidar data. The calculated value can be used to correct the original lidar data weight , so that when the haze is high is smaller, thereby reducing the proportion of lidar data in the fusion process and ensuring that the fusion result is more in line with the actual situation.

[0024] Among them, the process of the camera obtaining the image haze score is to capture the real-time image inside the silo and use image processing algorithms to quantitatively evaluate the visual blurring and contrast reduction caused by fog, dust, water vapor, etc. in the image.

[0025] Specifically, first, the image captured by the camera undergoes preprocessing steps such as noise suppression and color normalization, and then one or more haze estimation algorithms are applied, such as the method based on dark channel prior or the lightweight model trained using a convolutional neural network. These algorithms analyze information such as the statistical characteristics of darker regions in the image, the overall contrast of the image, and the brightness distribution, etc., so as to generate a numerical score H reflecting the current environmental haze level. For example, the dark channel prior method first calculates the dark channel value of each local region of the image, and then through the processes of atmospheric light estimation and transmittance recovery, finally derives a global haze score. This score H can be normalized to a fixed range (such as 0 to 1), where the higher the value, the more serious the interference of fog or dust. Furthermore, in the data fusion process, the system will dynamically adjust the confidence of lidar data according to the H value, thereby reducing the weight of lidar data in a high-haze environment.

[0026] The fused material level information is processed and then transmitted to the monitoring terminal. This terminal uses a three-dimensional visualization display module to generate an intuitive three-dimensional image of the fused data and mark the detected abnormal areas with preset colors or symbols, facilitating the operator to observe and judge the production safety status in real time.

[0027] As shown in Figure 3, in order to further enhance the intelligent detection and safety warning functions, the system performs abnormal material surface recognition on the fused three-dimensional point cloud data.

[0028] Specifically, first, the received point cloud data is subjected to surface reconstruction, and methods such as Poisson surface reconstruction are used to generate a continuous material surface grid structure, which is convenient for subsequent geometric analysis. Subsequently, the system calculates the curvature change value of each region in the constructed grid.

[0029] In a specific embodiment, the normal vectors of adjacent grid cells can be used for comparison. That is, the unit normal vectors of adjacent regions i and j are respectively denoted as and , and the calculation formula for the local curvature change K is: ; where represents the Euclidean norm, and this value reflects the degree of change in the surface continuity of adjacent regions. In actual operation, when the local curvature change K exceeds a preset threshold, this region is determined to be an abnormal region.

[0030] The formula for the average curvature of the region is as follows: ; where, represents the average curvature (dimensionless) within a certain abnormal region, N represents the number of grid cells segmented or clustered within this region, represents the local curvature value (dimensionless) of the i-th grid cell. Introducing this formula can more accurately describe the overall curvature characteristics of the abnormal region, which helps the subsequent judgment of abnormal phenomena (such as collapses, cavities, etc.).

[0031] The abnormal region can correspond to various phenomena. For example, in a silo, an overhead or cavity phenomenon is detected (that is, there is a hollow at the top of the silo but there is floating material on the surface), a sudden inclination region in the stockpile may indicate the risk of collapse, or there is a phenomenon of residue hanging on the wall in some regions. To avoid misjudgment caused by environmental noise or instantaneous data anomalies, the system also conducts time comparison analysis on the three-dimensional point cloud data after continuous multi-frame fusion. Only when the abnormal region persists in time, the abnormality of this region is finally confirmed, and the corresponding abnormal type and risk level information are output to the monitoring terminal.

[0032] Multiple methods can be used to identify abnormal regions. It can be determined based on the rules of a fixed threshold or can be automatically classified by combining artificial intelligence algorithms. First, the rule-based abnormal detection method is determined by presetting an empirical threshold. When the K value of a certain region is continuously higher than, it is considered that there is an abnormality in this region. In practical applications, the point cloud data can be processed in blocks, the average K value of each small region is calculated, and regional clustering technology is used to aggregate adjacent high-K regions together, thereby filtering out isolated high values caused by noise. By further calculating the area, average curvature, and local inclination change of these aggregated regions, the system can initially determine the type of abnormality. For example, if a large inclination change or height mutation is detected in a local region, it may indicate the risk of overhead, cavity, or impending collapse in the stockpile.

[0033] On the other hand, in order to improve the robustness of anomaly detection and adapt to complex working conditions, the system can also introduce artificial intelligence technology. Specifically, a large amount of point cloud data labeled with anomaly types can be collected, and the geometric features of each anomaly area (such as the average K, curvature variance, area size, local inclination angle, etc. within the area) are used as input features, and machine learning models such as support vector machines, decision trees, or deep neural networks are used for training. After training, the model can automatically learn the distribution laws of different anomaly forms (such as material heap wall hanging, voids, and collapses, etc.) in the feature space, so as to perform real-time classification and early warning on new point cloud data. In addition, an end-to-end deep learning method can also be adopted. For example, the reconstructed grid data or direct point cloud data is input into a convolutional neural network or a graph neural network, allowing the model to automatically extract geometric and topological features to achieve fine recognition of anomaly areas. Such an AI recognition method can not only improve the detection accuracy, but also adapt to new scenarios through continuous learning when the environment and material properties change.

[0034] In the specific implementation process, the present invention can select industrial-grade lidar and millimeter-wave radar, and their installation positions are designed according to the actual silo structure. Usually, the lidar is installed in the center of the silo top for a vertical top view of the entire material surface, while the millimeter-wave radar can be installed at an appropriate position to assist in making up for the data loss that may occur when the lidar is in a high dust concentration or high fog situation. The camera is usually also installed at a position with a good view inside the silo, and the haze score is calculated from the images it captures through an embedded image processing unit, providing a basis for subsequent dynamic weight adjustment. When the system is actually applied, each module can be integrated into an edge computing device to achieve real-time data processing and dynamic adjustment, so that the entire material detection system can maintain a high-precision and high-reliability operating state under various working conditions.

[0035] In addition, in the data fusion and anomaly detection algorithms, each parameter adopted (such as , the K value, and the curvature change threshold) can be preset and calibrated according to the silo size, material properties, and on-site environmental conditions. For example, for a medium-sized cement silo, may be set to 0.05 s, the K value can be determined by fitting experimental data to be 0.8, and the curvature threshold needs to be adjusted in combination with actual measurement data to ensure that the system neither misses alarms nor triggers alarms frequently due to excessive sensitivity.

[0036] In another embodiment, as shown in Figure 4, state detection devices such as vibration sensors, switch quantity detectors, or infrared detectors can be arranged at the inlet or outlet of the silo to obtain real-time change information on the material inlet and outlet states in the silo.

[0037] When the system is installed, the status detection device is connected to the on-site control unit. The control unit determines whether the current silo is in the material feeding state, discharging state, or static state by continuously monitoring the signals output by the sensors. For example, when the vibration sensor detects continuous vibration signals or the digital input sensor detects material flow, the control unit determines it as the feeding state or discharging state; while when the sensor signals are stable and there is no material movement, it is determined as the static state.

[0038] After detecting that the silo is in the feeding or discharging state, the system triggers the radar adaptive mode switching function. In this state, to more accurately capture the dynamic changes of the material, the sampling frequencies of the lidar and millimeter-wave radar will be automatically increased. For example, it will be increased from 5 Hz in the normal mode to 10 Hz or higher. At the same time, the weight coefficients of the data of the two sensors in the fusion algorithm are adjusted. Specifically, the system can reduce the weight of the lidar by a certain proportion (such as reducing by 20% - 30%), and correspondingly increase the weight of the millimeter-wave radar to obtain more stable measurement data in the case of intense material flow and increased environmental interference. In addition, in the feeding and discharging state, the data fusion strategy will also switch from the default static mode to the dynamic mode, adopting a faster filtering algorithm and data compensation method to respond in a timely manner to sudden changes in the material state.

[0039] When it is detected that the silo returns to the static state, the system automatically switches back to the normal radar working mode. At this time, the sampling frequency returns to the preset value (such as 5 Hz), and the weight coefficients of the data of the two sensors also return to the default state, thus ensuring the stability and data accuracy of long-term monitoring.

[0040] The formula for adjusting the dynamic sampling frequency is as follows: ; where represents the new sampling frequency after adaptive switching (unit: Hz), f represents the default sampling frequency (unit: Hz), S represents the normalized value of the status detection signal (dimensionless, 0 represents the static state, 1 represents the maximum material flow state), is the proportionality factor for adjusting the sampling frequency (dimensionless), which is used to control the impact of state changes on the sampling frequency. This formula can help the system automatically increase the sampling rate when the silo is in the dynamic state to more accurately capture the changes in the material state.

[0041] The adaptive switching process can be implemented by an embedded control unit, which dynamically adjusts the working mode of the radar module according to the status detection signal, historical data, and environmental parameters. All parameters can be determined through preset or on-site calibration. For example, for a medium-sized silo, during actual testing, the sampling frequency in the feeding and discharging state can be set to 10 Hz, the weight of the lidar is reduced from the default 0.6 to 0.4, and the weight of the millimeter-wave radar is increased from 0.4 to 0.6. While in the stationary state, the ratio of the lidar weight of 0.6 to the millimeter-wave radar weight of 0.4 is maintained. The system can also adjust various parameters in combination with the actual working conditions on site to achieve the best measurement effect.

[0042] As mentioned above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A material detection method based on the fusion of laser radar and millimeter wave radar, characterized in that: The steps include: Use laser radar to obtain three-dimensional point cloud data of the material surface in the silo; Use millimeter wave radar to obtain distance measurement data of materials in the silo; Inputting the three-dimensional point cloud data and the distance measurement data into a fusion processing module; The fusion processing module performs time alignment on the three-dimensional point cloud data and the distance measurement data based on a preset time synchronization mechanism, and dynamically adjusts the weight coefficients of the two types of data based on changes in environmental parameters to obtain fused material level information; The fused material level information is transmitted to the monitoring terminal, and the information is displayed and recorded on the monitoring terminal.

2. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 1 is characterized in that: The method further comprises: Using a camera disposed inside the silo to collect an image of the target area, and performing haze analysis on the image of the target area; Obtaining a current image haze score, and determining a confidence adjustment factor of the lidar data based on the image haze score; The confidence adjustment factor is input into the fusion algorithm as a weight parameter of the laser radar data during fusion processing.

3. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 1 is characterized in that: The method also performs abnormal material surface recognition on the fused three-dimensional point cloud data, including the following steps: Reconstruct the surface of 3D point cloud data to obtain a continuous material surface grid structure; Calculating the curvature change value of each area in the material surface grid structure; Determine that the area where the curvature change value exceeds the set threshold is an abnormal area; According to the spatial position, inclination characteristics and curvature continuity of the abnormal area, identify whether there is any one or more of the following anomalies: overhead, void, pile collapse, residue hanging on the wall; and output the corresponding anomaly type information.

4. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 1 is characterized in that: The dynamic weight adjustment adopts the following fusion formula: ； in: F represents the material level information after fusion; P represents the measurement value corresponding to the three-dimensional point cloud data acquired by the lidar; D represents the distance measurement data obtained by the millimeter wave radar; Represents the weight coefficient of the lidar data; Represents the weight coefficient of millimeter wave radar data; And meet .

5. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 2 is characterized in that: The image haze score is used to determine the confidence adjustment factor of the lidar data , and the calculation formula is as follows: ； in: Represents the confidence adjustment factor of the lidar data; H represents the image haze score; K is a preset positive coefficient used to adjust the effect of the haze score on the adjustment factor.

6. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 3 is characterized in that: The difference between the normal vectors of adjacent grid cells is used as the curvature change value, and its calculation formula is: ； in: Indicates area i The unit normal vector of represents the unit normal vector of region j; represents the Euclidean norm; K represents the curvature change value.

7. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 3 is characterized in that: The surface reconstruction of the three-dimensional point cloud data is achieved by using a Poisson surface reconstruction algorithm.

8. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 3 is characterized in that: The abnormal area identification step also includes comparing the three-dimensional point cloud data after continuous multi-frame fusion to determine the persistence of the abnormal area in time to eliminate misjudgment caused by instantaneous noise or environmental changes.

9. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 3 is characterized in that: The monitoring terminal includes a display unit for three-dimensional visual display, and the display unit generates a three-dimensional image according to the fused material surface information, and marks the detected abnormal area with a preset color or symbol.

10. The material detection method based on the fusion of laser radar and millimeter wave radar according to claim 1 is characterized in that: The method further comprises the steps of: The state detection device arranged at the feeding port or the discharging port of the silo is used to detect the feeding and discharging state of the silo; According to the state detection results, it is determined whether the silo is in the feeding state, the unloading state or the static state; When the silo is in the feeding or unloading state, the sampling frequency, weight coefficient and data fusion strategy of the laser radar and millimeter wave radar are adaptively switched; When the silo is at rest, the preset normal radar working mode is restored.

Images