A mine belt way carrying capacity identification method based on video volume measurement

Abstract

The application discloses a mine beltway transport capacity identification method based on video volume measurement, relates to the technical field of mine intelligent monitoring, and comprises the following steps: synchronously collecting continuous frame laser stripe image data and radar intermediate frequency signal data of a material section, dividing the laser stripe image data into grid windows, calculating window displacement of the corresponding windows to generate an initial two-dimensional velocity field, performing spectrum processing on the radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar distance and velocity, converting spatial point coordinates of each window in the initial two-dimensional velocity field to a radar coordinate system to obtain radial distance and radial velocity, indexing and extracting corresponding power spectrum values in the two-dimensional power spectrum matrix of distance and velocity based on visual radial distance and visual radial velocity, comparing the corresponding power spectrum values with a preset threshold value, judging the effectiveness of the visual velocity, and generating a spatial binary mask matrix.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology in mines, and in particular to a method for identifying the transport volume of mine conveyor belts based on video volume measurement. Background Technology

[0002] With the continuous improvement of the level of intelligence and digitalization in mines, belt conveyor systems, as the core link in mine material transportation, have their conveying volume monitoring accuracy directly related to production scheduling and energy consumption management. Non-contact measurement technology based on vision and laser scanning has become a research hotspot. Under the harsh working conditions of high dust and high-speed belt operation in actual mines, the existing technology still has the following specific logical defects and technical bottlenecks.

[0003] First, the high-speed operation of the belt will cause severe motion blur in the laser stripe image captured by the camera, which will cause the position of the center point of the directly extracted stripe subpixel to shift, ultimately causing the cross-sectional area calculation to fail. Although existing technologies attempt to compensate for motion by calculating the visual displacement field between image frames, they often fall into a technical dead loop: when the image is partially obscured by dust or lacks features, the initial velocity field extracted based on pure vision is itself severely distorted or even sparse and broken. If the distorted velocity field is directly used for reverse mapping and resampling, not only will motion blur not be eliminated, but severe image tearing and error accumulation will be introduced. Second, in high-concentration dust environments, laser stripes can be severely scattered or completely obscured, leading to the complete failure of local visual images. In such cases, existing vision solutions can usually only perform simple linear interpolation or discard the image directly, without being able to obtain the true physical height of the material in the obscured area. This crude processing method will cause the material cross-sectional profile height function obtained by fusion to have local collapse, ultimately resulting in a severely undersized integral result of the instantaneous cross-sectional area. Summary of the Invention

[0004] This invention provides a method for identifying the transport capacity of mine conveyor belts based on video volume measurement, comprising: Simultaneously acquire continuous frame laser stripe image data and radar intermediate frequency signal data of material cross section, divide the laser stripe image data into grid windows, calculate the window displacement of the corresponding window to generate an initial two-dimensional velocity field, and perform spectrum processing on radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar range and velocity. Transform the spatial coordinates of each window point in the initial two-dimensional velocity field to the radar coordinate system to obtain radial distance and radial velocity. In the two-dimensional power spectrum matrix of distance and velocity, extract the corresponding power spectrum value based on the visual radial distance and visual radial velocity index, and compare it with the preset threshold to determine the validity of the visual velocity and generate a spatial binary mask matrix. The initial two-dimensional velocity field is selected based on the spatial binary mask matrix to generate a dense displacement field. Motion compensation is performed on the laser stripe image, and the sub-pixel center points of the laser stripes in the compensated image are extracted. Combined with the spatial binary mask matrix, the material height is calculated based on the visual stripe center in the effective area, and the radial distance is extracted based on the radar power spectrum in the invalid area to complete the height. The fusion is then used to obtain the material cross-sectional contour height function. The instantaneous cross-sectional area is obtained by integrating the contour height function. The velocity component of the visual velocity of the effective area of ​​the mask in the direction of belt movement is weighted and averaged to obtain the cross-sectional weighted average velocity. The instantaneous cross-sectional area is multiplied by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux, and the cumulative transport volume is obtained by summing them.

[0005] As a preferred embodiment of the mine conveyor belt transport capacity identification method based on video volume measurement described in this invention, the step of dividing the laser stripe image data into grid windows, calculating the initial displacement field of the corresponding window to generate an initial two-dimensional velocity field, and performing spectral processing on the radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar range and velocity includes: Use a checkerboard calibration board to calibrate the camera intrinsic parameters and output the camera intrinsic parameter matrix and distortion coefficient vector. The rectangular calibration block is passed through the laser plane at a constant speed, and multiple laser stripe images are acquired. The center points of the laser stripes on the top and bottom surfaces of the calibration block are extracted, and the plane equation of the laser plane in the camera coordinate system is obtained by fitting. Two frames of laser stripe images are acquired at two consecutive acquisition times using a high-speed area array camera, referred to as the first frame and the second frame. The first and second frames are converted to grayscale and smoothed using a Gaussian filter. For each frame of the filtered image, the grayscale mean and grayscale standard deviation are calculated. The grayscale threshold is set by adding twice the standard deviation to the mean. Then, pixel regions with grayscale values ​​greater than the grayscale threshold are selected and marked as regions of interest. The region of interest is divided into grid windows. The percentage of effective pixels in the laser stripes within each window is counted, and the effective windows are filtered. For each valid window, a frequency domain cross-correlation algorithm based on fast Fourier transform is used to calculate the sub-pixel displacement between two frames of images, and the displacement is divided by the time interval to obtain the initial velocity, forming an initial two-dimensional velocity field matrix. The velocity of the invalid window is recorded as zero. The range-dimensional spectrum is obtained by performing a one-dimensional fast Fourier transform on the radar intermediate frequency signal along the fast time dimension. The intermediate frequency signals of multiple pulse periods are arranged into a two-dimensional data matrix, and a two-dimensional discrete Fourier transform is performed and the square of the modulus is calculated to obtain the distance and Doppler two-dimensional power spectrum matrix. Based on the radar carrier frequency, the Doppler dimension is converted to the velocity dimension, resulting in a two-dimensional power spectrum matrix of radar range and velocity.

[0006] As a preferred embodiment of the mine conveyor belt transport capacity identification method based on video volume measurement described in this invention, the following steps are included: transforming the coordinates of each window spatial point in the initial two-dimensional velocity field to the radar coordinate system to obtain radial distance and radial velocity; extracting the corresponding power spectrum values ​​based on the visual radial distance and visual radial velocity index in the distance and velocity two-dimensional power spectrum matrix, comparing them with a preset threshold to determine the validity of the visual velocity, and generating a spatial binary mask matrix. For each effective velocity sample in the initial two-dimensional velocity field matrix, obtain the image pixel coordinates of the center of the corresponding query window. Based on the camera intrinsic parameter matrix, distortion coefficient vector and laser plane equation, calculate the three-dimensional spatial coordinates of the image pixel coordinates in the camera coordinate system. Transform the three-dimensional coordinates to the radar coordinate system and calculate the radial distance and radial unit direction vector of the three-dimensional spatial coordinates relative to the radar. Based on the coordinates of three-dimensional spatial points and the time interval in two frames of images, the three-dimensional velocity vector in the camera coordinate system is calculated, transformed to the radar coordinate system, and projected onto the radial unit direction vector to obtain the radial velocity inferred from the visual data. Construct a coordinate pair between the radial distance and the inferred radial velocity, and read the corresponding power spectrum value from the two-dimensional power spectrum matrix of radar range and velocity; Set a noise decision threshold; Compare the power spectral density value with the noise decision threshold: If the power spectrum value is greater than or equal to the noise decision threshold, the visual velocity data is deemed valid. If the power spectrum value is less than the noise decision threshold, the visual velocity data is deemed invalid. Based on the determination results of all valid velocity samples, a spatial binary mask matrix with the same size as the initial two-dimensional velocity field matrix is ​​generated, where valid positions are assigned a value of 1 and invalid positions are assigned a value of 0.

[0007] As a preferred embodiment of the mine conveyor belt transport capacity identification method based on video volume measurement described in this invention, the step of filtering the initial two-dimensional velocity field based on a spatial binary mask matrix, generating a dense displacement field, and performing motion compensation on the laser stripe image includes: Based on the spatial binary mask matrix, the window displacements are filtered, and the displacements of invalid windows are spatially interpolated and repaired to generate a connected effective displacement field, which is then fused to obtain the connected effective displacement field. The repaired connected effective displacement field is upsampled into a pixel-level dense displacement field. Based on the pixel-level dense displacement field, the current frame image is reverse-mapped and resampled to obtain the motion-compensated image.

[0008] As a preferred embodiment of the mine conveyor belt transport capacity identification method based on video volume measurement described in this invention, the method involves: extracting the sub-pixel center points of laser stripes in the compensated image, combining this with a spatial binary mask matrix, calculating the material height in the effective region based on the visual stripe center, and performing height completion for the invalid region by extracting the radial distance based on the radar power spectrum, thus fusing to obtain the material cross-sectional contour height function, including: Based on the spatial binary mask matrix, the effective window with a mask value of 1 is mapped to the motion-compensated image and merged to form a pixel-level effective region mask. The sub-pixel center point of the laser stripe is extracted only within the pixel-level effective region. The plane containing the unloaded surface of the conveyor belt is designated as the reference plane, and discrete sampling points are constructed along the transverse direction of the belt. For each sampling point, if it is located within the pixel-level effective area mask, the center coordinates of the stripe are obtained by interpolation of the center points of adjacent sub-pixels. The three-dimensional coordinates are then calculated in reverse by combining the camera parameters and the laser plane equation, and the distance to the reference plane is calculated as the visual height value. If it is located outside the pixel-level effective area mask, radar height completion is performed. The radar height completion includes: determining the spatial search direction corresponding to the invalid sampling position based on the lateral spatial coordinates of the invalid sampling position, and constructing the corresponding radar radial direction; Based on the statistical values ​​of visual radial velocity within the effective area mask, estimate the target radial velocity search interval; Within the velocity range of the two-dimensional power spectrum matrix of radar range and velocity, search for the radial distance corresponding to the maximum power spectrum value; The spatial completion point is determined by combining the radial distance and radial direction, and its distance to the reference plane is calculated in the camera coordinate system as the radar completion height value; The visual height values ​​of the effective area and the radar-completed height values ​​of the invalid area are merged to form a discrete height sequence, and the material cross-section profile height function is generated by spline interpolation.

[0009] As a preferred embodiment of the mine conveyor belt transport capacity identification method based on video volume measurement described in this invention, the method includes: integrating the contour height function to obtain the instantaneous cross-sectional area; performing weighted averaging on the visual radial velocity of the effective mask area to obtain the cross-sectional weighted average velocity; multiplying the instantaneous cross-sectional area by the cross-sectional weighted average velocity to obtain the instantaneous volume throughput; and accumulating these components to obtain the cumulative transport capacity. The material cross-sectional profile height function is preprocessed, negative height values ​​are set to zero, and the composite trapezoidal method is used to numerically integrate the preprocessed height to calculate the instantaneous cross-sectional area. Traverse the spatial binary mask matrix, and for each valid window with a mask value of 1, obtain the velocity component of the visual velocity in the unit vector of the belt running direction. Based on the window center pixel coordinates, camera intrinsic parameter matrix and laser plane equation, the belt lateral spatial coordinates are calculated inversely, and the height value is obtained by interpolation from the contour height function. The cross-sectional weighted average velocity is calculated by taking a weighted average of the velocity components of all valid velocity samples. Multiply the instantaneous cross-sectional area by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux; The instantaneous volumetric flux is multiplied by the image sampling time interval and then summed to obtain the cumulative flux.

[0010] The beneficial effects of this invention are as follows: This invention uses the two-dimensional power spectrum matrix extracted from the radar intermediate frequency signal, combined with the initial two-dimensional velocity field calculated from the visual image window features, and takes advantage of the high penetration of radar electromagnetic waves to use the radial Doppler velocity as the absolute physical benchmark to perform threshold comparison and validity judgment on the visual velocity. This eliminates false visual displacement caused by dust obstruction or uneven reflection, and improves the authenticity and accuracy of the weighted average velocity calculation of the effective area section of the mask. By combining the high-resolution visual stripe center height of the effective mask area with the radial distance extracted from the radar power spectrum of the ineffective mask area, the height of the dust-covered area is physically completed, achieving the complementary advantages of optical surface mapping and microwave penetration ranging. This improves the continuity and completeness of the material cross-sectional profile height function, fundamentally enhancing the final accuracy of instantaneous volumetric throughput and cumulative transport volume integral calculation, and ensuring highly reliable all-weather monitoring of the mine transportation system in extremely harsh environments. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of the mine conveyor belt transport capacity identification method based on video volume measurement in Example 1; Figure 2 This is a structural diagram of the mine conveyor belt transport volume identification method based on video volume measurement in Example 1. Detailed Implementation

[0013] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0014] Example 1, referring to Figure 1 and Figure 2This is the first embodiment of the present invention, which provides a method for identifying the transport capacity of mine conveyor belts based on video volume measurement, including the following steps: S1. Synchronously acquire continuous frame laser stripe image data and radar intermediate frequency signal data of material cross section, divide the laser stripe image data into grid windows, calculate the initial displacement field of the corresponding window to generate the initial two-dimensional velocity field, and perform spectrum processing on radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar range and velocity. Specifically, the laser stripe image data is divided into grid windows, the initial displacement field of the corresponding window is calculated to generate the initial two-dimensional velocity field, and the radar intermediate frequency signal data is subjected to spectral processing to generate a two-dimensional power spectrum matrix of radar range and velocity, including: Intelligent sensors are deployed at the measurement cross-section directly above the belt conveyor and perpendicular to the belt running plane to simultaneously collect multi-source data at the cross-section of the material being measured. The aforementioned multi-source data includes laser stripe image data emitted by a line laser and projected onto the surface of the object under test, modulated by the object surface, and collected by a high-speed area array camera, as well as intermediate frequency signal data collected synchronously by radar. The measurement cross section is preferably fixed at the same spatial position in the direction of the conveyor belt to ensure that continuous frame data corresponds to the same measurement cross section. The aforementioned smart sensors include high-speed area array cameras, line lasers, and radar; All data acquisition processes must ensure time synchronization between the high-speed area array camera and the radar; Preferably, the camera exposure time and the radar sampling time are aligned using a unified hardware trigger signal, and corresponding timestamps are added to the acquired image data and intermediate frequency signal data respectively; Camera intrinsic parameters are calibrated using a checkerboard calibration plate with known precise dimensions, wherein the side length of the checkerboard unit is preferably 20.00 mm ± 0.01 mm; Place the checkerboard calibration board within the camera's field of view and take pictures in at least 10 different poses, including at least translation, tilting about the horizontal axis, tilting about the vertical axis, and rotation about the camera's optical axis. During the acquisition process, it should be ensured that the checkerboard calibration board is clearly imaged and the corner points are fully visible in each image, and that the calibration board in each image covers different areas of the camera's field of view, so as to improve the stability and accuracy of the camera parameter solution. Based on multiple images of the chessboard calibration board obtained by shooting, the Zhang Zhengyou calibration method is used to solve the imaging model parameters of the camera, and output the camera intrinsic parameter matrix and distortion coefficient vector. The camera intrinsic parameter matrix is ​​used to describe the mapping relationship between the pixel coordinate system and the camera normalized coordinate system, and the distortion coefficient vector is used to describe the radial distortion and tangential distortion of the lens. Preferably, the root mean square error of reprojection of all calibration images is no greater than 0.3 pixels as the criterion for the validity of the camera intrinsic parameter calibration result. If the root mean square error of reprojection is greater than 0.3 pixels, the calibration board image is re-acquired and calibrated. Place a rectangular calibration block with a known height H on a translation stage, where H is preferably 50.0 mm; use a level and angle gauge to adjust the attitude of the rectangular calibration block to ensure that the height direction of the rectangular calibration block is parallel to the coordinate axis in the camera coordinate system, preferably parallel to the Y-axis of the camera coordinate system, and the attitude error is controlled within 0.1°. The translation stage is controlled to move the rectangular calibration block through the laser plane at a constant speed in a direction perpendicular to the laser plane, preferably at a speed of 5 mm / s; During this process, the camera is synchronously controlled by hardware trigger signals to acquire laser stripe images when the rectangular calibration block is in different positions, so as to obtain multiple laser stripe images covering different areas of the laser plane. In each laser stripe image, the image is first converted to grayscale and then smoothed and denoised using Gaussian filtering, wherein the standard deviation of the Gaussian filter is preferably 0.8 to 1.5 pixels. The Steger algorithm is used to extract the sub-pixel coordinates of the center line of the laser stripe projected on the side of the rectangular calibration block. For each extracted center line of the laser stripe, the center line point that intersects with or is closest to the top boundary of the rectangular calibration block is selected as the top surface feature point, and the center line point that intersects with or is closest to the bottom boundary of the rectangular calibration block is selected as the bottom surface feature point. The pixel coordinates of the top surface feature point and the bottom surface feature point are recorded and denoted as the top surface point and the bottom surface point, respectively. For each pair of top and bottom points, distortion correction is performed using the calibrated camera intrinsic matrix and distortion coefficients. Based on the distorted pixel coordinates and camera intrinsic matrix, the top and bottom points are projected back onto the camera's normalized coordinate system, thus obtaining two imaging ray directions corresponding to the top and bottom points respectively. Both imaging rays pass through the camera's optical center and point to the imaging directions of the top and bottom points in the actual space. The two imaging ray directions are then normalized to obtain the unit direction vector corresponding to the top point and the unit direction vector corresponding to the bottom point. Based on the known geometric structure of the rectangular calibration block, establish geometric constraint relationships for the spatial positions corresponding to the top and bottom points; Specifically, the top and bottom points are located on the spatial straight line corresponding to the same edge of the rectangular calibration block. Their distance along the height direction of the rectangular calibration block in the camera coordinate system is a known height H, and their coordinates are the same in the other two coordinate directions perpendicular to this height direction. Based on the unit direction vectors of the top and bottom points, the top and bottom points are located on their respective imaging rays in the camera coordinate system. Therefore, the three-dimensional coordinates of the top and bottom points in the camera coordinate system can be expressed as follows: ; ; in, These are the three-dimensional coordinates of the top surface point in the camera coordinate system. These are the three-dimensional coordinates of the base point in the camera coordinate system. Let be the distance scale factor from the top surface point along the unit direction vector to the camera optical center. Let be the distance scale factor from the base point along the unit direction vector to the camera optical center. Let be the unit direction vector corresponding to the top surface point. This is the unit direction vector corresponding to the bottom point; Since the top and bottom points in the camera coordinate system satisfy the condition that they differ by a known height H along the height direction of the rectangular calibration block, and have the same coordinates in the other two coordinate directions, they satisfy the following constraint relationship, as shown in the formula: ; Where H is the known height of the rectangular calibration block. The top point relative to the bottom point in the camera coordinate system has a known displacement only along the height direction, while the displacement in the other two directions is zero; Expanding the constraints into component form, we obtain the following system of equations: ; ; ; in, , as well as Unit direction vectors The components in the X, Y, and Z directions of the camera coordinate system. , as well as Unit direction vectors The components in the X, Y, and Z directions of the camera coordinate system; The scaling factor can be obtained by solving the above system of equations using the least squares method or analytical method. This allows us to obtain the three-dimensional coordinates of the bottom point in the camera coordinate system. ; Collect no fewer than 20 sets of 3D point coordinates, with the spatial distribution of these points covering different regions of the laser plane. Use the least squares method to fit the laser plane and obtain its plane equation in the camera coordinate system: ; in, , as well as These are the three components of the laser plane normal vector. For plane constants, X, Y, and Z are the coordinate components of a point in space in the camera coordinate system; Normalize the plane normal vector; For any three-dimensional coordinate of a bottom point, its distance to the fitted laser plane is defined as: ; in, Let be the distance from the i-th 3D point to the fitted laser plane. , as well as Let be the coordinate components of the i-th 3D point in the camera coordinate system; The root mean square residual of the distances from all sampling points to the fitted plane is no greater than 0.5 mm, which is used as the criterion for the validity of this laser plane calibration result. After completing the camera intrinsic parameter calibration and laser plane calibration, a joint calibration between the radar and the camera is performed to establish the spatial correspondence between the camera coordinate system and the radar coordinate system. This joint calibration is performed under the condition that the installation positions of the camera, laser and radar remain unchanged. Within the shared field of view of the camera and the line laser, the corner reflector is fixed on a two-dimensional translation stage equipped with X- and Y-axis scales and a horizontal adjustment device. The corner reflector is adjusted using the horizontal adjustment device to ensure that the geometric center of the corner reflector is located in a horizontal plane parallel to the radar scanning plane. At least six non-collinear and uniformly distributed spatial positions are selected in this horizontal plane as calibration sampling positions. Each sampling position should have sufficient spatial separation to improve the robustness of the joint calibration solution. For each selected corner reflector location, laser stripe image data containing the corner reflector and the points where the laser stripes fall on its surface are acquired using a high-speed area array camera; Preferably, at least 3 frames of images are continuously acquired at each location, and the frame with the highest clarity or the most stable average gray level is selected as the valid image of the current calibration location to reduce the impact of random noise on the laser landing point positioning accuracy. The pixel coordinates of the laser impact point on the corner reflector are extracted by using threshold segmentation combined with the centroid method. Specifically, the Otsu method is used to calculate the segmentation threshold for the image grayscale histogram. That is, all candidate thresholds in the range of 0 to 255 are traversed, the inter-class variance of the corresponding foreground and background is calculated, and the threshold that maximizes the inter-class variance is selected as the optimal threshold. Pixels with grayscale values ​​higher than the optimal threshold are marked as foreground candidate points, and pixels with grayscale values ​​lower than or equal to the optimal threshold are marked as background. Connected components are marked for the foreground candidate points, and connected regions with an area greater than a preset minimum area threshold are used as potential laser landing point regions. The minimum area threshold is preferably 3 to 20 pixels. If there are multiple potential connected regions, the connected region with the largest average grayscale or the largest area is selected as the target laser landing point region. Using the camera intrinsic parameter matrix and distortion coefficients, the pixel coordinates of the laser landing point are distorted and then projected back onto the camera normalized coordinate system to obtain the corresponding direction vector. This vector is then normalized to obtain the unit direction vector e. Since the laser's point of impact is the intersection of the spatial target surface and the line laser plane, its spatial position lies both on the aforementioned imaging ray and on the laser plane. This spatial point can be represented in the camera coordinate system as: ; in, The coordinates of the laser impact point in the camera coordinate system are: The distance scale factor of this spatial point along the imaging ray relative to the camera optical center; The scale factor is obtained by solving the equation of the plane at the spatial point using the following formula: ; in, , as well as The components of the unit imaging direction vector along the three coordinate axes of the camera coordinate system; ; in, , as well as These are the three components of the laser plane normal vector. Based on the scale factor, the three-dimensional coordinates of the laser impact point in the camera coordinate system are calculated, and the three-dimensional coordinates are used as the camera coordinate system spatial point corresponding to the current calibration position. The intermediate frequency signal acquired by the radar is processed to resolve the radial distance and azimuth angle of the corner reflector relative to the radar; Specifically, a fast Fourier transform is performed on the intermediate frequency signal along the fast time dimension, and the peak frequency corresponding to the target echo is detected in the frequency domain. In the case of a linear frequency modulated continuous wave radar, the radial distance corresponding to the target echo is analyzed using the following formula: ; Where r is the radial distance from the radar to the corner reflector, and c is the speed of light. B is the peak frequency of the intermediate frequency signal, and B is the radar frequency modulation bandwidth. The frequency modulation period; When the radar has at least two spatially separated receiving channels or equivalent angle measurement capabilities, the azimuth angle of the corner reflector relative to the radar is calculated based on the phase difference between the target echo and the receiving channels. For a single-plane scanning radar, the azimuth angle is defined within the radar scanning plane and is determined according to the installation direction of the scanning plane in the radar coordinate system. If the corner reflector deviates from the scanning plane, the height of the corner reflector is readjusted before data acquisition to satisfy the plane assumption that Z is 0. For multiple sets of corresponding points (coordinates in the camera coordinate system and coordinates in the radar coordinate system), the absolute orientation problem is solved. This absolute orientation problem is to find the optimal rotation matrix and translation vector to minimize the alignment error between the two sets of points. This process can be implemented using the classic singular value decomposition algorithm: first, calculate the centroids of the two sets of corresponding points and perform centering processing, and then solve the rotation matrix and translation vector through SVD. Two frames of laser stripe images are acquired at two consecutive acquisition times using a high-speed area array camera, and are denoted as the first frame image and the second frame image. The acquisition time interval between these two frames is equal to the camera frame period, preferably 1.0 ms. The first frame image and the second frame image correspond to the same measurement section and are used for subsequent visual displacement estimation and velocity field calculation. The first and second frames are converted to grayscale and smoothed using a Gaussian filter to suppress image noise. The standard deviation of the Gaussian filter is preferably 1.0 pixel. For each frame of filtered image, the overall grayscale mean and grayscale standard deviation are calculated. The grayscale threshold is set by adding twice the standard deviation to the mean. Pixel regions with grayscale values ​​greater than the grayscale threshold are marked as regions of interest. The region of interest is divided into multiple grid windows, the size of which is preferably 32×32 pixels or 64×64 pixels. The overlap rate of adjacent windows in both the horizontal and vertical directions is preferably 50%. For each grid window, the proportion of effective pixels of the laser stripes inside it to the total number of pixels in the window is counted. When the proportion is lower than the proportion threshold, the window is determined to be an invalid window; otherwise, the window is determined to be a valid window. The proportion threshold is preferably 10%. For each valid window, a rectangular local image region of a fixed size is cropped from the first frame image, centered on the center pixel position of the first frame image, as the query window. The fixed size means that the length and width of the query window remain consistent throughout the processing of all valid windows. Preferably, the size of the query window is 32×32 pixels. When the image texture is sparse, the noise is strong, or the expected displacement between two adjacent frames is large, the size of the query window is preferably 64×64 pixels. When the query window cropped according to the center pixel position exceeds the image boundary, the position of the query window is corrected to ensure that it is completely located inside the first frame image. Using the same center position in the second frame image as the initial center, the corresponding search window is extracted. The size of the search window is preferably 1.5 to 2 times the size of the query window to cover the possible displacement range of the target between adjacent frames. The query window and the search window are normalized respectively. The normalization process includes: subtracting the mean gray value of the window from the gray value of the pixels in the window and dividing by the standard deviation of the gray value of the window to reduce the impact of the overall brightness change on the relevant matching results. The frequency domain cross-correlation algorithm based on fast Fourier transform calculates the two-dimensional cross-correlation function between the normalized query window and the search window. The cross-correlation function can be obtained through the inverse frequency domain product transform. In the two-dimensional cross-correlation function, the location of the discrete peak is determined, and a local region of a certain size, preferably a 3×3 or 5×5 region, is selected in its neighborhood. The local correlation value is then fitted with a two-dimensional Gaussian surface. By solving for the extreme point positions of the fitted surface, subpixel-level displacement is obtained, which represents the planar displacement of the query window relative to the search window. Remove the planar bits of each valid window by the time interval between the first and second frames to obtain the initial velocity corresponding to that valid window; All initial velocities are arranged according to the grid position of each valid window in the image to form an initial two-dimensional velocity field matrix. For invalid windows, the initial velocity at the corresponding position is recorded as the zero vector, so as to facilitate subsequent filtering processing in combination with the mask matrix. The intermediate frequency signal acquired by the radar is subjected to a one-dimensional fast Fourier transform along the fast time dimension to obtain the range dimension spectrum. The range resolution of the radar is determined by the bandwidth of the transmitted signal, which is obtained by dividing the speed of light by twice the bandwidth. A coherent processing interval is set, preferably containing 64 or 128 consecutive radar pulse cycles. The intermediate frequency signals collected within the coherent processing interval are arranged in the sampling order to form a two-dimensional data matrix. The row index of the matrix corresponds to the fast time sampling point, representing the range dimension information, and the column index of the matrix corresponds to the slow time pulse number, representing the Doppler dimension information. To reduce spectral leakage, a window function is applied to the row and column directions of the two-dimensional data matrix, preferably a Hanning window. A two-dimensional discrete Fourier transform is performed on the windowed two-dimensional data matrix to obtain a two-dimensional spectrum matrix. The square of the modulus of the two-dimensional spectrum matrix is ​​then used to obtain the distance and Doppler two-dimensional power spectrum matrix.

[0015] S2. Transform the spatial coordinates of each window point in the initial two-dimensional velocity field to the radar coordinate system to obtain the radial distance and radial velocity. In the two-dimensional power spectrum matrix of distance and velocity, extract the corresponding power spectrum value based on the visual radial distance and visual radial velocity index, and compare it with the preset threshold to determine the validity of the visual velocity and generate a spatial binary mask matrix. Specifically, the coordinates of each window point in the initial two-dimensional velocity field are transformed to the radar coordinate system to obtain radial range and radial velocity. In the two-dimensional power spectrum matrix of range and velocity, the corresponding power spectrum values ​​are extracted based on the visual radial range and visual radial velocity indices, and compared with a preset threshold to determine the validity of the visual velocity. A spatial binary mask matrix is ​​then generated, including: Based on the radar carrier frequency, the Doppler frequency corresponding to the target echo, and the speed of light, the Doppler element is converted into the corresponding radial velocity using the following formula: ; in, The radial velocity corresponding to the m-th Doppler element is... Let m be the Doppler frequency corresponding to the m-th Doppler element. For radar carrier frequency; Map the distance and Doppler two-dimensional power spectrum matrix to a distance and velocity two-dimensional power spectrum matrix; Specifically, in the range dimension of the two-dimensional power spectrum matrix, each element still corresponds to the radial range of the radar target. In the Doppler dimension, each element originally corresponds to the Doppler frequency of the target echo. Based on the radar carrier frequency, the speed of light, and the correspondence between the Doppler frequency and the radial velocity of the target, each frequency element in the Doppler dimension is converted into the corresponding radial velocity element. After the conversion, the power values ​​of each element in the original matrix remain unchanged, and only the physical meaning of the Doppler dimension is replaced by velocity instead of frequency, thus obtaining the range and velocity two-dimensional power spectrum matrix. For each effective velocity sample in the initial two-dimensional velocity field matrix, obtain the image pixel coordinates of the center of the corresponding query window. Based on the camera intrinsic parameter matrix and distortion coefficients, perform distortion removal processing on the pixel coordinates and back-project to obtain the imaging ray direction vector in the camera coordinate system. Combined with the calibrated laser plane equation, by solving the intersection of the imaging ray and the laser plane, obtain the three-dimensional spatial coordinates of the center of the window in the camera coordinate system. Based on the rotation matrix and translation vector between the radar and the camera, the coordinates of three-dimensional points in the camera coordinate system are transformed to the radar coordinate system, resulting in the corresponding three-dimensional coordinates in the radar coordinate system; the transformation relationship is as follows: ; in, These are the three-dimensional coordinates of the spatial point corresponding to the pixel in the camera coordinate system. These are the three-dimensional coordinates of the same spatial point in the radar coordinate system. This is the rotation matrix for transforming from the camera coordinate system to the radar coordinate system. This is the translation vector used to transform the camera coordinate system to the radar coordinate system; Based on the three-dimensional spatial coordinates in the radar coordinate system, with the radar origin as the reference point, the straight-line distance between the spatial point and the radar origin is calculated, and the straight-line distance is taken as the radial distance of the spatial point relative to the radar. The coordinate direction of the spatial point in the radar coordinate system is taken as the radial direction of the radar pointing to the spatial point. Furthermore, based on the geometric relationship between the horizontal and vertical coordinates of the spatial point in the radar coordinate system, the azimuth angle of the spatial point relative to the radar reference direction is calculated. For each valid window, using the center pixel coordinates of the window in the first frame and the corresponding pixel coordinates in the second frame after displacement, the 3D spatial point of the window center in the camera coordinate system at the two acquisition times is calculated according to the camera imaging model, distortion correction results, and laser plane equation. Based on the time interval between the two frames, the 3D velocity vector in the camera coordinate system is calculated using the following formula: ; in, This is the 3D velocity vector in the camera coordinate system corresponding to the effective window. For time intervals, and The effective window center is the three-dimensional point in the camera coordinate system corresponding to the two acquisition times; It should be noted that: the camera imaging model, the camera intrinsic parameter matrix obtained by indexing, and the pinhole projection relationship represented by it are used to describe the mapping relationship between spatial points and image pixel coordinates in the camera coordinate system; The distortion correction result refers to the lens distortion coefficient obtained by indexing, the corrected pixel coordinates or the corrected imaging direction obtained after distortion removal processing of the original pixel coordinates, which is used to eliminate the influence of lens radial distortion and tangential distortion on the spatial reconstruction result. The laser plane equation refers to the plane equation obtained by fitting multiple sets of laser stripe spatial points to represent the spatial position of the laser sheet light emitted by the line laser in the camera coordinate system. Based on the rotation matrix between the radar and the camera, the three-dimensional velocity in the camera coordinate system is rotated to the radar coordinate system to obtain the velocity vector. The projection of the velocity vector onto the unit direction vector in the radar coordinate system is then calculated to obtain the radar radial velocity inferred from the visual data. The formula is as follows: ; ; in, This is a three-dimensional velocity vector in the radar coordinate system. The radar radial velocity is inferred from visual data. The unit radial direction vector of the radar; For each valid pixel window, the radial distance of its corresponding spatial point relative to the radar and the radial velocity inferred from the visual data are combined into a distance-velocity coordinate pair. The corresponding two-dimensional power spectrum value is then queried in the two-dimensional power spectrum matrix of the radar distance and velocity to determine whether the visual velocity is consistent with the physical motion information measured by the radar. Based on the range resolution and velocity resolution of the two-dimensional power spectrum matrix of radar range and velocity, coordinate pairs are mapped to continuous index coordinates. When a coordinate pair corresponds exactly to the center of a matrix cell, the two-dimensional power spectrum value of that matrix cell is directly read. When the coordinate pair is located within a local area enclosed by four adjacent matrix cells, the corresponding interpolated two-dimensional power spectrum value is calculated using bilinear interpolation. When the coordinate pair exceeds the effective index range of the matrix, the corresponding two-dimensional power spectrum value is recorded as zero. When the range and velocity coordinate pair does not fall at the center of a range and velocity matrix cell, but is located between adjacent matrix cells, the two-dimensional power spectrum values ​​of the adjacent matrix cells are interpolated based on the relative positional relationship between the coordinate pair and the centers of the surrounding adjacent matrix cells, thereby obtaining the two-dimensional power spectrum value corresponding to the range and velocity coordinate pair. The interpolation result is used to represent the strength of the target echo at the coordinate position. Set a noise decision threshold; Specifically, in a targetless empty scene, multiple sets of two-dimensional matrices of radar range and velocity are collected, the mean and standard deviation of background noise of all matrix elements are calculated, and the mean of background noise plus 3 to 5 times the standard deviation is set as the noise decision threshold. Compare the interpolated energy value corresponding to each valid window with the noise decision threshold; When the interpolated energy value is greater than or equal to the system noise decision threshold, it is determined that the motion at the current visual velocity window is coherent with the Doppler information detected by the radar, indicating that the visual signal originates from the motion of real materials, and thus the visual velocity data is determined to be valid. When the interpolated energy value is less than the system noise decision threshold, it is determined that the motion at the current visual velocity window is incoherent with the Doppler information detected by the radar, indicating that the visual signal is more likely to come from dust disturbance or noise interference, and thus the visual velocity data is determined to be invalid. Based on the above binary decision result of visual velocity validity, a spatial binary mask matrix with the same size as the visual velocity field grid is generated. For visual velocity data that is determined to be valid, the corresponding element in the mask matrix is ​​assigned a value of 1 to indicate that the visual data at that location has passed the physical authenticity verification. For visual velocity data that is determined to be invalid, the corresponding element in the mask matrix is ​​assigned a value of 0 to indicate that the visual data at that location has been rejected due to dust or noise interference.

[0016] S3. Based on the spatial binary mask matrix, the initial two-dimensional velocity field is screened to generate a dense displacement field. Motion compensation is performed on the laser stripe image, and the sub-pixel center points of the laser stripes in the compensated image are extracted. Combined with the spatial binary mask matrix, the material height is calculated for the effective area according to the visual stripe center, and the radial distance is extracted for the invalid area according to the radar power spectrum to complete the height. The material cross-section contour height function is obtained by fusion. Specifically, based on a spatial binary mask matrix, an initial two-dimensional velocity field is selected to generate a dense displacement field, and motion compensation is performed on the laser stripe image, including: The initial displacement field obtained from visual measurement is acquired and then filtered using a spatial binary mask matrix. Specifically, the displacements corresponding to windows with an element value of 1 in the mask matrix are retained, while the displacements corresponding to windows with an element value of 0 in the mask matrix are set to zero or marked as null values, thereby obtaining a sparse effective displacement field. Bilinear interpolation is used to upsample the sparse effective displacement field into a pixel-level dense displacement field. The pixels in the current frame image are reversed spatially based on the dense displacement field to achieve image motion compensation and obtain the motion-compensated image. In the reverse spatial mapping, the position of each pixel in the output image is reversed by the dense displacement field to the corresponding sampling position in the input image, and its gray value is obtained by interpolation.

[0017] Furthermore, the role of the initial displacement field in this invention is to represent the overall motion trend of the grayscale texture of the material surface between adjacent frames, which is a rough prior for motion compensation. The extraction of the sub-pixel center point of the laser stripe is a geometrically precise positioning process performed after motion compensation is completed and within the effective area. The displacement field calculation and stripe center extraction are distinguished from each other in terms of processing target, processing time and data dependency. The former does not take the latter as a premise, and the latter only uses the compensation result completed by the former, thereby avoiding the implementation paradox caused by the logical closed loop of "a high-quality stripe center must be obtained before a high-quality velocity field can be obtained, and a high-quality velocity field must be obtained before high-quality motion compensation can be completed". The initial displacement is filtered based on the spatial binary mask matrix, and the effective displacement corresponding to the window with an element value of 1 in the mask matrix is ​​retained; For invalid windows with element values ​​of 0 in the mask matrix, visual features are distorted due to severe dust occlusion. Directly setting them to zero will cause severe image tearing in subsequent motion compensation. Therefore, this invention uses a spatial interpolation algorithm based on inverse distance weighting to repair them: with the invalid window as the center, search for the nearest (preferably 4-8) valid windows in its neighborhood. Based on the spatial distance from each valid window to the invalid window, assign inverse weights and calculate the repair displacement vector of the invalid window, thereby obtaining a hole-free connected valid displacement field. Upsampling the connected effective displacement field into a pixel-level dense displacement field involves the following steps: The grid-level displacements in the connected effective displacement field are mapped to the pixel coordinates corresponding to the center point of the grid, forming discrete pixel-level displacement seed points. For any target pixel in the image that is not at the center of the grid, locate the four nearest displacement seed points surrounding the target pixel; Based on the normalized distances between the target pixel and the four displacement seed points in the horizontal and vertical directions, the interpolation position of the target pixel in the horizontal X direction and the vertical Y direction is calculated by using the bilinear interpolation formula. By traversing all pixels, a pixel-level dense displacement field with the same resolution as the original image is generated. Based on the generated pixel-level dense displacement field, the current frame image is reversed spatially mapped to achieve image motion compensation. Since this invention decouples grayscale texture velocity calculation from subpixel stripe geometry extraction and physically repairs the distorted displacement field, it avoids the error compensation paradox caused by local distortion amplification, thereby obtaining a high-quality motion-compensated image.

[0018] Furthermore, the sub-pixel center points of the laser stripes in the compensated image are extracted. Combined with a spatial binary mask matrix, the material height is calculated for the effective region based on the visual stripe center, and the radial distance for the invalid region is extracted based on the radar power spectrum for height completion. The resulting fusion yields the material cross-sectional profile height function, including: Based on the spatial binary mask matrix, the effective contour of the identifier is generated, the binary mask image of the region is extracted, and the morphological closing operation is performed on the binary mask image to fill local holes and connect adjacent effective regions. In the motion-compensated image above, subsequent laser stripe center point extraction is performed only within the effective contour extraction area to reduce the impact of dust interference on stripe extraction accuracy. Within the effective contour extraction area, the motion-compensated image is subjected to Gaussian filtering and smoothing. The image grayscale gradient is calculated along the transverse direction corresponding to the laser stripe width. Pixels with gradient amplitudes higher than a gradient threshold are selected as initial candidate points for the laser stripes. This preset gradient threshold can be determined based on 3... The rules are established; For each initial candidate point, calculate the Hessian matrix of the gray-level distribution in the neighborhood of that point, and determine the normal direction of the laser stripe at that point through eigenvalue analysis. Local fitting of the grayscale distribution along the normal direction is performed to determine the sub-pixel offset corresponding to the grayscale extrema, thereby calculating the sub-pixel-level center coordinates of the laser stripe: ; in, For the extracted subpixel-level center coordinates of the laser stripes, The coordinates of the initial candidate pixels. The sub-pixel offset is obtained by fitting the extreme value of the gray-level distribution along the normal direction of the laser stripe, where n is the unit vector of the normal direction of the laser stripe determined by the Hessian matrix. Uniformly discrete sampling is performed along the transverse width of the material conveyor belt to obtain multiple transverse sampling positions. For each transverse sampling position, it is determined whether it is within the effective contour extraction area. If the sampling position is within the effective contour extraction area, the corresponding material height is calculated using visual measurement results. If the sampling position is within the invalid area, the height is completed by calling radar detection results. The reference plane is the reference plane corresponding to the upper surface of the unloaded conveyor belt, preferably obtained by collecting laser stripes and fitting them when the conveyor belt is unloaded; When a certain lateral sampling position is located within the effective contour extraction area, the three-dimensional coordinates of the visual space point corresponding to the sampling position in the camera coordinate system are obtained based on the sub-pixel center point of the laser stripe corresponding to the lateral sampling position, or the sub-pixel position obtained by interpolation based on the sub-pixel center points of the adjacent laser stripes on both sides of the lateral sampling position. The distance from the visual space point to the reference plane of the empty conveyor belt is taken as the visual material height value corresponding to the lateral sampling position, and the formula is: ; in, This represents the visual height of the material. These are the coordinate components of a point in the visual space in the camera coordinate system. , , as well as The parameters of the reference plane equation; When a certain horizontal sampling position is located in an invalid region, the initial height estimate is preferably obtained from the nearest valid sampling positions on its left and right sides, and the corresponding visual height value is obtained by linear interpolation. If there is only one valid sampling position on the side of the horizontal sampling position, the visual height value of the nearest valid sampling position on that side is used as the initial height estimate. If there are no valid sampling positions on either side, the empty reference height of the conveyor belt is used as the initial height estimate. Based on the initial height estimate, camera imaging model, and laser plane equation, construct the initial spatial point corresponding to the sampling location; Based on the initial spatial point, the radar radial unit direction vector pointing from the radar origin to the initial spatial point is calculated through the spatial transformation relationship between the radar and the camera. The radar radial velocity distribution range corresponding to each visual velocity sample within the effective area is statistically analyzed, and this distribution range is used as the reference search range for the radial velocity of the target within the invalid area. Preferably, the statistical mean and standard deviation of all visual radial velocity samples within the effective area are calculated. The statistical mean is used as the search center, and the standard deviation is used as a measure of the velocity dispersion. The search range is dynamically expanded to both sides of the search center according to a set confidence coefficient (specifically: search range = [mean - c times standard deviation, mean + c times standard deviation], where the preferred value of the confidence coefficient c is 2 to 3), thereby constructing a target radial velocity search range with a dynamic confidence interval. Furthermore, when the number of effective velocity samples in the effective area is lower than the preset threshold (i.e., the sample size is insufficient to characterize the true statistical distribution), the projection value of the conveyor belt drive motor's running speed in the radar radial direction is used as the center of the target radial velocity search interval, and a preset fixed velocity tolerance is given to extend to both sides to form the search interval. In a specific embodiment of the present invention, the preset quantity threshold is preferably limited by both relative proportion and absolute quantity; Specifically, the preset quantity threshold is preferably 10% to 15% of the total number of grid windows in the target detection area (i.e., the effective material distribution area of ​​the belt cross section) in the current frame image, and its absolute value is not less than 30. According to the law of large numbers and the central limit theorem, when the number of effective samples is greater than or equal to 30, the statistical characteristics (mean and standard deviation) of the samples can stably represent the true velocity distribution of the material at that moment. If, under extremely harsh conditions (such as encountering a sudden high concentration of fly ash), the number of effective velocity samples after mask screening drops sharply and falls below the preset threshold (such as less than 10% of the total number of windows or less than 30), it indicates that the very few effective samples at present are most likely interference noise at the edge of the dust or incorrect matching caused by local reflection, and can no longer represent the overall motion law of the material. At this point, the system will trigger an adaptive degradation protection strategy: abandon the use of unreliable visual samples for statistical expansion, and instead directly use the projection value of the rated operating speed of the conveyor belt drive motor in the radial direction of the radar as the search center, and give a preset fixed speed tolerance (preferably 10% of the motor speed) to expand to both sides to form a backup search range. This mechanism effectively prevents erroneous visual features from misleading the radar search under extreme obstruction.

[0019] In the two-dimensional power spectrum matrix of radar range and velocity, within the velocity cell range corresponding to the radial velocity interval of the target, the matrix cell with the largest two-dimensional power spectrum value is searched, and the radial distance value corresponding to the matrix cell is used as the radar complete radial distance. If there are multiple candidate cells with similar two-dimensional power spectrum values, the cell with the closest predicted distance to the initial spatial point is selected as the complete result. After determining the radar completion radial distance, the radar completion spatial point corresponding to the sampling position of the invalid region is constructed by combining the radar radial direction; Specifically, taking the radar origin as the starting point, extending along the determined radar radial direction, and taking a position point in this direction that is equal to the radar completion radial distance, the position point is determined as the three-dimensional coordinates of the completion point in the radar coordinate system, thus obtaining the radar completion spatial position corresponding to the sampling position of the invalid area; The three-dimensional coordinates of the completion point in the radar coordinate system are transformed into the camera coordinate system by the rotation matrix and translation vector between the camera and the radar obtained by calibration. The three-dimensional coordinates of the completion point in the camera coordinate system are obtained. Based on the positional relationship of the completion point in the camera coordinate system relative to the reference plane of the empty conveyor belt, the material completion height corresponding to the completion point is calculated, which is used to replace the height value that cannot be directly measured by vision in the invalid area. The radar completion altitude value is calculated based on the distance from the completion point to the reference plane, using the following formula: ; in, To complete the material height value obtained from the radar, , as well as To complete the coordinate components of the 3D coordinate vector of the point in the camera coordinate system; The visual material height values ​​and radar-completed material height values ​​at each lateral sampling position are merged according to the sampling order to form a discrete height data sequence. The sampling positions located within the effective contour extraction area use the visual material height value, while the sampling positions located within the invalid area use the radar-completed material height value. Spline interpolation is used to smooth the discrete height data sequence to generate a complete material cross-sectional profile height function for the current frame. , where k represents the current frame number and x represents the spatial coordinates along the transverse width direction of the conveyor belt.

[0020] Compared with the method of directly upsampling the sparse and distorted displacement field, the present invention uses a spatial binary mask matrix to remove unreliable visual displacement, and then forms a connected and effective displacement field through spatial neighborhood repair. It uses bilinear interpolation to generate a pixel-level dense displacement field, thus reducing the transmission and amplification effect of dust occlusion, stripe breaks and local distortion on the motion compensation results. This invention decouples the initial velocity estimation from the subpixel stripe geometry extraction, so that the stripe center extraction is based on the compensated stable image, rather than the reverse as a premise for the initial velocity calculation. This avoids circular dependencies in logic and improves the stability of motion compensation and the accuracy of contour reconstruction in implementation.

[0021] S4. Integrate the contour height function to obtain the instantaneous cross-sectional area. For the visual radial velocity of the effective area of ​​the mask, perform weighted averaging on the velocity components to obtain the cross-sectional weighted average velocity. Multiply the instantaneous cross-sectional area by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux and accumulate them to obtain the cumulative transport volume. Specifically, the instantaneous cross-sectional area is obtained by integrating the contour height function. The visual radial velocity of the effective mask region is then weighted and averaged to obtain the cross-sectional weighted average velocity. The instantaneous cross-sectional area is multiplied by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux, which is then accumulated to obtain the cumulative transport capacity, including: The material cross-sectional profile height function Hk(x) is preprocessed. For any lateral sampling point, if its corresponding height value is less than 0, the height value is set to 0 to avoid introducing non-physical negative heights due to noise, fitting errors, or coordinate transformation errors. The composite trapezoidal rule is used to numerically integrate the preprocessed discrete height sequence to calculate the instantaneous cross-sectional area of ​​the material section in the current frame. ; in, Let N be the instantaneous cross-sectional area of ​​the k-th frame, and N be the number of horizontal sampling points. For the i-th horizontal sampling position, This represents the height value at the i-th horizontal sampling position in the k-th frame; Based on the actual installation direction of the conveyor belt, the spatial direction of the belt running direction in the camera coordinate system is predetermined, and this direction is represented as a unit direction vector. The unit direction vector can be obtained through equipment installation calibration results, conveyor belt structural parameters or manual calibration. For each effective speed sample, its three-dimensional speed vector in the camera coordinate system is obtained, and the three-dimensional speed vector is projected along the belt running direction to obtain the speed component of the effective speed sample in the belt running direction. Traverse the spatial binary mask matrix. For each valid window with an element value of 1, obtain the velocity component corresponding to the window. Based on the center pixel coordinates of the window, combined with the camera intrinsic parameter matrix, distortion coefficients, and laser plane equation, calculate the spatial coordinates of the spatial point corresponding to the center of the window in the lateral width direction of the belt. Obtain the height value at this position by performing linear interpolation or spline interpolation on the material cross-section profile height function. When the interpolation result is less than 0, set the height value to 0. For all valid velocity samples, a weighted average is calculated based on the material height at their corresponding locations to obtain the cross-sectional weighted average velocity of the current frame. Specifically, the greater the height of the material, the greater the weight of its corresponding speed sample in the average speed calculation; the smaller the height of the material, the smaller the weight of its corresponding speed sample. This weighting method enables the calculated average speed to more realistically reflect the overall motion state of the entire material cross-section as the conveyor belt runs. If the sum of the heights corresponding to all valid speed samples in the current frame is zero, then the cross-sectional weighted average speed of the current frame is recorded as zero. Multiply the instantaneous cross-sectional area of ​​the current frame by the corresponding cross-sectional weighted average to obtain the instantaneous volumetric flux of the current frame; The cumulative transport volume is calculated by numerically integrating all instantaneous volumetric fluxes using the rectangular method.

[0022] Example 2 To further illustrate the technical solution of Example 1, its comprehensive technical effect and engineering feasibility in complex mine belt conveyor scenarios, this example adopts the same processing procedure as Example 1, conducts continuous field tests in the on-site testing scenario of underground mine belt conveyors, and sets up 4 control groups for comparison and verification with the preferred solution of Example 1.

[0023] 1. Experimental Scenario and Acquisition of Reference Truth Value This embodiment conducts continuous field tests on the main conveyor belt in an underground coal mine. The test parameters follow the value range disclosed in Embodiment 1: the camera frame period is 1.0ms, the grid window size is 32×32 pixels, the radar coherent processing interval is 128 pulse cycles, and the field conditions include intermittent high concentration of coal dust, local high reflectivity, and motion blur caused by the high speed of the conveyor belt. The proportion of random dust occlusion in the camera's field of view fluctuates between 10% and 30%, and the average speed of the conveyor belt is approximately 2.5m / s. To obtain a reference value for the cumulative transport volume, a calibrated weighing device was set up at the bottom of the coal unloading tower of the belt conveyor as a reference measuring unit. 180 seconds of continuous on-site operating data was selected as a test sample. The reference measuring unit measured that the actual cumulative transport volume of the material during this period was 345.6t. This value is used as the reference true value for the cumulative transport volume error analysis in this embodiment. 2. Experimental Group Setup Each group was processed based on the same field data source. Four control groups were formed by removing or replacing the key technical steps in Example 1, as follows: Control group 1 (traditional pure visual method): The initial two-dimensional velocity field is calculated based solely on continuous frame laser stripe images. Radar physical consistency verification is not performed, motion compensation is not performed, and the center of the laser stripe is directly extracted from the original blurred image and the subsequent cross-sectional area and momentum are calculated. Control Group 2 (Direct Zeroing of Invalid Displacement Method): A spatial binary mask matrix is ​​generated, but the displacement vector of invalid windows with a mask value of 0 is directly set to zero. Instead of using the displacement of the valid window in the spatial neighborhood for spatial smoothing interpolation repair, the displacement field of the invalid window is directly upsampled to generate a dense displacement field and then motion compensation is performed to verify the role of the invalid window spatial repair step. Control group 3 (no radar height completion method): The radar physical consistency verification, spatial smooth interpolation repair, bilinear interpolation to generate dense displacement field and motion compensation process were adopted. However, in the section reconstruction stage, the invalid area was only processed based on visual curve fitting, and the radar radial distance was not used for height completion to verify the role of the radar height completion step. Control group 4 (without height-weighted velocity fusion): The complete contour reconstruction process of this invention is adopted, including radar physical consistency verification, spatial repair, bilinear interpolation, motion compensation and radar height completion. However, in the volume throughput calculation stage, the visual velocity of the effective mask area and the velocity component in the belt running direction are not used to calculate the cross-sectional average velocity by the material height-weighted average method. Instead, the average linear velocity of the belt is directly multiplied by the cross-sectional area to verify the effect of the height-weighted velocity fusion step on the final cumulative transport volume calculation. Example 1 (Preferred Group of the Invention): The full-link processing scheme of multi-source fusion, physical consistency verification, spatial repair, motion compensation, radar altitude completion and altitude-weighted velocity fusion described in Example 1 is fully adopted.

[0024] 3. Experimental Results and Statistical Analysis To demonstrate the impact of each processing step on the final transport accuracy, this embodiment statistically analyzes indicators such as image quality after compensation, stripe center positioning accuracy, cross-sectional area calculation error, and cumulative transport error. The results for each group are shown in Table 1. Table 1 Comparison of core performance indicators of each test group under complex mining conditions.

[0025] Note: The “Image Distortion Rate After Compensation” in the table represents the proportion of local tearing, misalignment and discontinuous deformation areas in the striped region to the effective striped region after motion compensation. The actual cumulative transport volume reference value is 345.6t.

[0026] 4. Results Analysis As shown in Table 1, the preferred group of Example 1 achieved the best results in all indicators, especially the cumulative transport volume error of only -0.86% in 180s, which is significantly better than the control groups. This shows that the solution of the present invention has good measurement accuracy and engineering applicability under complex mine transport conditions. Control group 1 did not perform radar physical consistency verification or motion compensation. It directly extracted the center of the laser stripe from the original blurred image. Therefore, under the combined effects of dust obstruction, local high reflectivity and motion blur, the stripe center positioning error reached 0.85px, the relative error of the cross-sectional area reached 9.60%, and the cumulative transport error reached -11.35%. This shows that relying solely on pure visual processing is difficult to meet the transport accuracy requirements under complex working conditions. Although control group 2 generated a spatial binary mask matrix, it directly set the invalid window displacement to zero before upsampling, resulting in insufficient spatial continuity of the displacement field. The compensated image showed obvious local tearing and misalignment, with a distortion rate of 24.50%, a fringe center positioning error of 0.48px, and a cumulative motion error of -8.12%. Compared with example 1, it can be seen that using the effective window displacement in the spatial neighborhood for spatial smooth interpolation repair has a significant effect on improving the continuity of the displacement field and enhancing the quality of motion compensation. Control group 3 employed spatial restoration and motion compensation procedures, resulting in significant improvements in image quality and stripe center localization accuracy after compensation. The stripe center localization error was reduced to 0.22px. However, due to the failure to utilize radar radial distance for height completion in invalid areas, the cross-sectional contour restoration of locally obscured areas remained incomplete, with the relative error of cross-sectional area still at 5.33% and the cumulative transport error still at -5.93%. This indicates that radar height completion has a significant effect on improving the integrity of cross-sectional reconstruction and the accuracy of cumulative transport calculation. Control group 4 adopted the complete process in the contour reconstruction stage, so the image quality, stripe center positioning error and cross-sectional area error after compensation were close to those of Example 1. However, since it did not use the cross-sectional average velocity weighted by material height to calculate the volume throughput, but directly used a single average linear velocity multiplied by the cross-sectional area, the final cumulative throughput error was still -4.28%, which was significantly higher than -0.86% of Example 1. This shows that, under the condition of similar contour reconstruction accuracy, the cross-sectional average velocity weighted by material height still has a significant effect on improving the final cumulative throughput recognition accuracy. In summary, this invention, through a continuous processing chain—combining visual velocity estimation with physical consistency verification of radar range and velocity two-dimensional power spectrum, spatial binary mask screening with invalid window displacement space repair, connecting effective displacement fields with bilinear interpolation to generate pixel-level dense displacement fields, extracting effective region stripes after motion compensation with radar height completion of invalid regions, and integrating cross-sectional area with calculated cross-sectional average velocity weighted by material height—can effectively improve the stability of stripe extraction, the integrity of cross-sectional contour restoration, and the accuracy of cumulative transport volume recognition in mining transport environments characterized by dust obstruction, local high reflectivity, and motion blur. This demonstrates that the solution has good feasibility and engineering application value.

[0027] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for identifying the transport capacity of mine conveyor belts based on video volume measurement, characterized in that, include: Simultaneously acquire continuous frame laser stripe image data and radar intermediate frequency signal data of material cross section, divide the laser stripe image data into grid windows, calculate the window displacement of the corresponding window to generate an initial two-dimensional velocity field, and perform spectrum processing on radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar range and velocity. The coordinates of each window spatial point in the initial two-dimensional velocity field are transformed to the radar coordinate system to obtain radial distance and radial velocity. In the two-dimensional power spectrum matrix of distance and velocity, the corresponding power spectrum values ​​are extracted based on the visual radial distance and visual radial velocity index and compared with the preset threshold to determine the validity of the visual velocity and generate a spatial binary mask matrix. The initial two-dimensional velocity field is selected based on the spatial binary mask matrix to generate a dense displacement field. Motion compensation is performed on the laser stripe image, and the sub-pixel center points of the laser stripes in the compensated image are extracted. Combined with the spatial binary mask matrix, the material height is calculated based on the visual stripe center in the effective area, and the radial distance is extracted based on the radar power spectrum in the invalid area to complete the height. The fusion is then used to obtain the material cross-sectional contour height function. The instantaneous cross-sectional area is obtained by integrating the contour height function. The velocity component of the visual velocity of the effective area of ​​the mask in the direction of belt movement is weighted and averaged to obtain the cross-sectional weighted average velocity. The instantaneous cross-sectional area is multiplied by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux, and the cumulative transport volume is obtained by summing them.

2. The method for identifying the transport capacity of mine conveyor belts based on video volume measurement as described in claim 1, characterized in that, The process of dividing the laser stripe image data into grid windows, calculating the initial displacement field of the corresponding window to generate an initial two-dimensional velocity field, and performing spectral processing on the radar intermediate frequency signal data to generate a two-dimensional power spectrum matrix of radar range and velocity includes: Use a checkerboard calibration board to calibrate the camera intrinsic parameters and output the camera intrinsic parameter matrix and distortion coefficient vector. The rectangular calibration block is passed through the laser plane at a constant speed, and multiple laser stripe images are acquired. The center points of the laser stripes on the top and bottom surfaces of the calibration block are extracted, and the plane equation of the laser plane in the camera coordinate system is obtained by fitting. Two frames of laser stripe images are acquired at two consecutive acquisition times using a high-speed area array camera, referred to as the first frame and the second frame. The first and second frames are converted to grayscale and smoothed using a Gaussian filter. For each frame of the filtered image, the grayscale mean and grayscale standard deviation are calculated. The grayscale threshold is set by adding twice the standard deviation to the mean. Then, pixel regions with grayscale values ​​greater than the grayscale threshold are selected and marked as regions of interest. The region of interest is divided into grid windows. The percentage of effective pixels in the laser stripes within each window is counted, and the effective windows are filtered. For each valid window, a frequency domain cross-correlation algorithm based on fast Fourier transform is used to calculate the sub-pixel displacement between two frames of images, and the displacement is divided by the time interval to obtain the initial velocity, forming an initial two-dimensional velocity field matrix. The velocity of the invalid window is recorded as zero. The range-dimensional spectrum is obtained by performing a one-dimensional fast Fourier transform on the radar intermediate frequency signal along the fast time dimension. The intermediate frequency signals of multiple pulse periods are arranged into a two-dimensional data matrix, and a two-dimensional discrete Fourier transform is performed and the square of the modulus is calculated to obtain the distance and Doppler two-dimensional power spectrum matrix. Based on the radar carrier frequency, the Doppler dimension is converted to the velocity dimension, resulting in a two-dimensional power spectrum matrix of radar range and velocity.

3. The method for identifying the transport capacity of mine conveyor belts based on video volume measurement as described in claim 2, characterized in that, The process involves transforming the coordinates of each window point in the initial two-dimensional velocity field to the radar coordinate system to obtain radial range and radial velocity. Then, based on the visual radial range and visual radial velocity indices in the range-velocity two-dimensional power spectrum matrix, the corresponding power spectrum values ​​are extracted and compared with a preset threshold to determine the validity of the visual velocity, generating a spatial binary mask matrix. This includes: For each effective velocity sample in the initial two-dimensional velocity field matrix, obtain the image pixel coordinates of the center of the corresponding query window. Based on the camera intrinsic parameter matrix, distortion coefficient vector and laser plane equation, calculate the three-dimensional spatial coordinates of the image pixel coordinates in the camera coordinate system. Transform the three-dimensional coordinates to the radar coordinate system and calculate the radial distance and radial unit direction vector of the three-dimensional spatial coordinates relative to the radar. Based on the coordinates of three-dimensional spatial points and the time interval in two frames of images, the three-dimensional velocity vector in the camera coordinate system is calculated, transformed to the radar coordinate system, and projected onto the radial unit direction vector to obtain the radial velocity inferred from the visual data. Construct a coordinate pair between the radial distance and the inferred radial velocity, and read the corresponding power spectrum value from the two-dimensional power spectrum matrix of radar range and velocity; Set a noise decision threshold; Compare the power spectral density value with the noise decision threshold: If the power spectrum value is greater than or equal to the noise decision threshold, the visual velocity data is deemed valid. If the power spectrum value is less than the noise decision threshold, the visual velocity data is deemed invalid. Based on the determination results of all valid velocity samples, a spatial binary mask matrix with the same size as the initial two-dimensional velocity field matrix is ​​generated, where valid positions are assigned a value of 1 and invalid positions are assigned a value of 0.

4. The method for identifying the transport capacity of mine conveyor belts based on video volume measurement as described in claim 3, characterized in that, The process of filtering the initial two-dimensional velocity field based on a spatial binary mask matrix to generate a dense displacement field and performing motion compensation on the laser stripe image includes: Based on the spatial binary mask matrix, the window displacements are filtered, and the displacements of invalid windows are spatially interpolated and repaired to generate a connected effective displacement field, which is then fused to obtain the connected effective displacement field. The repaired connected effective displacement field is upsampled into a pixel-level dense displacement field. Based on the pixel-level dense displacement field, the current frame image is reverse-mapped and resampled to obtain the motion-compensated image.

5. The method for identifying the transport capacity of mine conveyor belts based on video volume measurement as described in claim 4, characterized in that, The process involves extracting the sub-pixel center points of laser stripes in the compensated image, combining them with a spatial binary mask matrix, calculating the material height in the effective region based on the visual stripe center, and performing height completion for the invalid region by extracting the radial distance based on the radar power spectrum. The resulting fusion yields a material cross-sectional contour height function, including: Based on the spatial binary mask matrix, the effective window with a mask value of 1 is mapped to the motion-compensated image and merged to form a pixel-level effective region mask. The sub-pixel center point of the laser stripe is extracted only within the pixel-level effective region. The plane containing the unloaded surface of the conveyor belt is designated as the reference plane, and discrete sampling points are constructed along the transverse direction of the belt. For each sampling point, if it is located within the pixel-level effective area mask, the center coordinates of the stripe are obtained by interpolation of the center points of adjacent sub-pixels. The three-dimensional coordinates are then calculated in reverse by combining the camera parameters and the laser plane equation, and the distance to the reference plane is calculated as the visual height value. If it is located outside the pixel-level effective area mask, radar height completion is performed. The radar height completion includes: determining the spatial search direction corresponding to the invalid sampling position based on the lateral spatial coordinates of the invalid sampling position, and constructing the corresponding radar radial direction; Based on the statistical values ​​of visual radial velocity within the effective area mask, estimate the target radial velocity search interval; Within the velocity range of the two-dimensional power spectrum matrix of radar range and velocity, search for the radial distance corresponding to the maximum power spectrum value; The spatial completion points are determined by combining the radial distance and radial direction, and then transformed to the camera coordinate system. The distance to the reference plane is calculated as the radar completion height value. The visual height values ​​of the effective area and the radar-completed height values ​​of the invalid area are merged to form a discrete height sequence, and the material cross-section profile height function is generated by spline interpolation.

6. The method for identifying the transport capacity of mine conveyor belts based on video volume measurement as described in claim 5, characterized in that, The instantaneous cross-sectional area is obtained by integrating the contour height function. The visual radial velocity of the effective mask area is weighted and averaged to obtain the cross-sectional weighted average velocity. The instantaneous cross-sectional area is multiplied by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux, which is then accumulated to obtain the cumulative throughput. The material cross-sectional profile height function is preprocessed, negative height values ​​are set to zero, and the preprocessed height is numerically integrated using the composite trapezoidal method to calculate the instantaneous cross-sectional area. Traverse the spatial binary mask matrix, and for each valid window with a mask value of 1, obtain the velocity component of the visual velocity in the unit vector of the belt running direction. Based on the window center pixel coordinates, camera intrinsic parameter matrix and laser plane equation, the belt lateral spatial coordinates are calculated inversely, and the height value is obtained by interpolation from the contour height function. The cross-sectional weighted average velocity is calculated by taking a weighted average of the velocity components of all valid velocity samples. Multiply the instantaneous cross-sectional area by the cross-sectional weighted average velocity to obtain the instantaneous volumetric flux; The instantaneous volumetric flux is multiplied by the image sampling time interval and then summed to obtain the cumulative flux.

Images