Mine vehicle safe driving detection system based on vision and radar fusion

Abstract

The invention relates to the technical field of vehicle safe driving detection, and provides a mining vehicle safe driving detection system based on vision and radar fusion, which comprises a camera module for extracting image edge gradient and road surface color features to generate a composite feature map, dynamic recognition of rigid or soft obstacles is achieved by combining connected domain marking and area mutation rate analysis, and meanwhile a pit candidate area is detected through a closed annular edge structure and color variance; and furthermore, through radar point cloud data, barrier reflection intensity abnormity and pit height difference characteristics are verified, and finally, multi-modal data are fused to generate a warning area and distance information thereof. Through a cooperative mechanism of visual feature pre-screening and radar data verification, the accuracy of obstacle detection in a complex mine environment is effectively improved, and thus the adaptability of safe driving of mining vehicles is improved.

Description

Technical Field

[0001] This application relates to the technical field of vehicle safe driving detection. Specifically, it relates to a safe driving detection system for mining vehicles based on vision and radar fusion. Background Technique

[0002] The content of this part only provides background information related to this application, and it may not constitute prior art.

[0003] In the field of safe driving of mining vehicles, obstacle recognition is of utmost importance. The mining area environment is complex and changeable, and factors such as rough terrain, low visibility, and dust flying increase the risk of vehicle driving. Accurately identifying obstacles ahead, such as obstacles, potholes, mud puddles, etc., can provide key warning information for drivers, enabling them to take timely braking, avoidance and other measures, thereby effectively avoiding the occurrence of collision accidents, ensuring the safety of personnel lives, reducing equipment damage, and reducing production interruptions and economic losses caused by accidents.

[0004] In the prior art, Chinese Patent No. CN118244291A discloses an obstacle detection method, terminal device, and storage medium. Although it can identify positive and negative obstacles to a certain extent, it mainly relies on the geometric features of point cloud data and lacks the comprehensive utilization of visual information, resulting in insufficient detection accuracy in dynamic scenarios and complex lighting conditions. In addition, this method has weak recognition ability for soft obstacles (such as loose soil piles or accumulated water), and only relies on the height difference of the point cloud in pothole detection, which is easily affected by the sparsity of the point cloud, resulting in misjudgment or missed judgment. Therefore, this patent has insufficient adaptability in complex scenarios. Therefore, there is an urgent need for a safe driving detection system for mining vehicles based on vision and radar fusion, which combines point cloud data and visual images to improve the adaptability of safe driving of mining vehicles. Summary of the Invention

[0005] To solve the above technical problems, the purpose of this application is to provide a safe driving detection system for mining vehicles based on vision and radar fusion. By fusing vision and radar to generate a composite feature map, combining the regional mutation rate and the reflection intensity anomaly coefficient to achieve accurate classification of rigid and soft obstacles, and confirming the pothole area through neighborhood connectivity scanning and height difference calculation. At the same time, using the prompt of real-time distance to enhance the practicality of the detection result, thus solving the problem of insufficient adaptability of the prior art in complex scenarios.

[0006] The purpose of this application is achieved through the following technical solutions: In the first aspect, the present invention provides a safe driving detection system for mining vehicles based on vision and radar fusion, including: A camera module for obtaining image data in front of a mining vehicle through a camera on the mining vehicle; calculating an edge gradient matrix of the image data using the Sobel operator, separating the road surface color features based on the HSV color space, extracting the saturation component to generate a color space conversion result; and superimposing the edge gradient matrix and the color space conversion result to generate a composite feature map. An obstacle marking module for performing connected component marking on pixel points with gradient values exceeding a first preset threshold based on the composite feature map, and generating a candidate box when a connected region with more than a preset number of pixel points continuously appears; calculating the regional mutation rate of the saturation component matrix corresponding to the candidate box, and if the regional mutation rate exceeds a preset multiple of the average mutation rate of the adjacent road surface region and is stably present in the detection of three consecutive frames of images, marking it as a rigid obstacle; if the regional mutation rate is lower than the average mutation rate of the adjacent road surface region and the intensity of the blue channel of the corresponding image data rises by more than a first preset ratio, marking it as a soft obstacle. A pothole area screening module for performing neighborhood connectivity scanning on the composite feature map, and when a closed circular edge structure is detected and the variance of the color space conversion result in the internal region is lower than a second preset ratio of the external region, marking it as a pothole area to be verified. An obstacle confirmation module for obtaining point cloud data in front of the mining vehicle through a radar installed on the mining vehicle, and extracting the reflection intensity anomaly coefficient in the regions corresponding to the rigid obstacle and the soft obstacle in the point cloud data according to the preset corresponding relationship between any position of the point cloud data and the image data; when the reflection intensity anomaly coefficient is greater than the corresponding second preset threshold, confirming the existence of the obstacle; otherwise, confirming that it does not exist. An actual pothole area confirmation module for calculating the height difference of all the point cloud data within the closed circular edge structure, and when the height difference exceeds the preset difference, confirming it as an actual pothole area. A prompt module for marking the regions corresponding to the confirmed rigid obstacle, soft obstacle, and actual pothole area as warning regions on the image data; calculating the distance between each warning region and the mining vehicle based on the point cloud data, and projecting the distance and the warning region on the display screen of the mining vehicle.

[0007] Furthermore, the camera module further includes: Before calculating the edge gradient matrix of the image data using the Sobel operator, dividing the image data into multiple local sub-blocks, calculating the gray histogram for each sub-block respectively and setting a preset contrast limit parameter, and eliminating the boundary effect between local sub-blocks through bilinear interpolation algorithm to generate image data with balanced illumination.

[0008] Furthermore, the system further includes: A steep slope recognition module, which is used to calculate the horizontal edge gradient of any image data. When the consistency of the horizontal edge gradient direction of more than 10 consecutive rows of pixels in any area of the image data exceeds the third preset ratio, it is determined that there is a slope change in the area; Calculate the edge density attenuation rate of the area in the vertical direction. When the attenuation rate drops by more than the fourth preset ratio per second and lasts for more than 3 frames, it is confirmed that a steep slope terrain is formed; Mark the confirmed steep slope terrain in the image data and display it on the display screen of the mining vehicle.

[0009] Furthermore, the system also includes: A muddy recognition module, which is used to extract the first ratio change curve of the saturation component and the brightness component in the HSV space. When the fluctuation amplitude of the first ratio in the set pixel window is less than the fifth preset ratio and the overall brightness drops by more than the sixth preset ratio, a corresponding candidate area is generated; Process the image data through the Canny edge detection algorithm to generate a texture feature map. Extract the edge pixel points of the candidate area in the texture feature map, and calculate the second ratio of the number of non - continuous edge segments to the total edge length of the edge pixel points. When the second ratio exceeds the third preset threshold, it is confirmed as a muddy area; Mark the confirmed muddy area in the image data and display it on the display screen of the mining vehicle.

[0010] Furthermore, the system also includes: An obstacle avoidance path generation module, which is used to generate an obstacle avoidance path that maintains a preset safety distance from obstacles based on the three - dimensional coordinates of rigid obstacles, soft obstacles, and actual pothole areas through a heuristic path search algorithm.

[0011] Furthermore, the generation of the obstacle avoidance path includes: Set the area where the slope change rate determined by the steep slope recognition module exceeds the preset threshold as an impassable area. According to the height difference parameters and the distance parameters from the mining vehicle corresponding to the rigid obstacles, soft obstacles, and actual pothole areas in the radar point cloud data, adjust the weight ratio of the obstacle distance cost term and the path smoothness cost term in the path search algorithm, so that the generated path preferentially maintains a preset safety distance from the rigid obstacles; In the real - time image data of the display screen, mark the obstacle avoidance path trajectory with a first identifier, mark the area corresponding to the steep slope terrain with a second identifier, and mark the muddy area with a third identifier.

[0012] Furthermore, the system also includes a synchronization module, which is used to send the first identifier, the second identifier, and the third identifier to the following vehicle through the vehicle - to - vehicle communication protocol; the specific processing process is: When generating the first identifier, the second identifier, or / and the third identifier on the display screen of any mining vehicle, synchronously extract the corresponding type attributes of the tag and the boundary parameters in the image coordinate system, and encapsulate them into a structured data packet; Broadcast the data packet at a fixed frequency through the vehicle-mounted communication unit on the mining vehicle, and append the current GPS coordinates of the mining vehicle and the real-time point cloud height parameters collected by the radar to the data packet; After the following vehicle receives the data packet, according to the distance information of the preceding vehicle obtained by its own radar, map the received tag coordinates to its own vehicle coordinate system to achieve dynamic position calibration among multiple vehicles. Further, the formula corresponding to the preset correspondence is:

[0013] Among them, are respectively the pixel horizontal and vertical coordinates of the target in the image coordinate system, is the depth value of the target in the camera coordinate system; 、 are respectively the x-axis and y-axis focal lengths of the camera, 、 are respectively the coordinates of the camera optical center on the image plane, is the rotation matrix, is the translation vector, 、 、 are respectively the three-dimensional coordinates of the target in the radar coordinate system.

[0014] In a second aspect, the present invention provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps corresponding to the method in the first aspect are implemented.

[0015] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by the processor, the steps corresponding to the method in the first aspect are implemented.

[0016] In summary, the technical solutions of the embodiments of the present application at least have the following advantages and beneficial effects: The present invention uses a camera module to extract the edge gradient matrix of an image by means of the Sobel operator, separates the road surface color features in the HSV color space to generate a composite feature map, and enhances the expression of environmental features; the obstacle marking module filters candidate regions based on the gradient threshold and connected component analysis, differentiates rigid / soft obstacles by the saturation component mutation rate and the change in blue channel intensity, and combines three-frame stability detection to reduce false positives; the pothole area screening module identifies potential potholes through closed-loop edge structure detection and color variance comparison; the obstacle confirmation module verifies the visual detection results through the radar point cloud reflection intensity anomaly coefficient and establishes the mapping relationship between the image coordinates and the point cloud space; the actual pothole area confirmation module quantifies the terrain mutation through the point cloud height difference; the prompt module fuses multi-sensor data to calculate the obstacle spacing and projects and displays it. The detection accuracy is improved through visual feature fusion and radar data verification, the feature extraction ability under complex lighting is enhanced by using color space conversion and gradient superposition, the false alarm rate is reduced by using multi-frame stability detection and cross-verification of the reflection intensity anomaly coefficient, and the physical features of potholes and planar obstacles are distinguished by combining point cloud height difference analysis, effectively improving the adaptability of obstacle detection in the unstructured road environment of mining areas. Brief Description of the Drawings

[0017] Figure 1 It is a schematic structural diagram of a safety driving detection system for mining vehicles based on vision and radar fusion provided by the present invention; Figure 2 It is another schematic structural diagram of a safety driving detection system for mining vehicles based on vision and radar fusion in the present invention; Figure 3 It is a schematic structural diagram of an electronic device provided by the present invention. Detailed Embodiment

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. The components of the embodiments of the present application described and illustrated herein are generally arranged and designed in various different configurations.

[0019] As Figure 1 shown, a safety driving detection system for mining vehicles based on vision and radar fusion proposed in the embodiments of the present application includes: A camera module 101, configured to obtain image data in front of the mining vehicle through a camera on the mining vehicle; calculate the edge gradient matrix of the image data by using the Sobel operator, separate the road surface color features based on the HSV color space, extract the saturation component to generate a color space conversion result; and superimpose the edge gradient matrix and the color space conversion result to generate a composite feature map.

[0020] Specifically, the camera captures the front road image data at a rate of 25 frames per second through a wide-angle lens installed at the front end of the vehicle, and uses a 1 / 2.8-inch CMOS sensor to ensure the imaging quality in the low-illumination mining area environment. After obtaining the original RGB image data, the system immediately applies the Sobel operator for edge detection, that is, performs biaxial gradient calculation by applying a convolution kernel in the horizontal direction and a transposed kernel in the vertical direction to generate an edge gradient matrix containing the gradient components in the x and y directions. This matrix can effectively capture the contour features of obstacles and is robust to light changes. That is, when the vehicle travels to the dust area, the traditional RGB image will become blurred, but the edge gradient matrix can still clearly show the edge of the metal bracket hidden in the dust.

[0021] Then, the HSV (Hue, Saturation, Value, that is, hue, saturation, and brightness) color space conversion technology is adopted, and the saturation component is extracted as the color feature index. After converting the original RGB image data into the HSV space, the system effectively separates the characteristic colors of the mining area road surface by setting a saturation threshold range of 0.4 - 0.6 (for example, the unique reddish-brown color of iron ore has a high saturation feature in the HSV space). For example, when the vehicle is driving on a gravel road with water reflection, the brightness component may fluctuate violently, but the saturation component can stably maintain the recognition ability of the road surface features. Through constructing a 128-level quantization histogram analysis, the system can dynamically adjust the saturation threshold to adapt to the differences in road surface materials in different mining areas.

[0022] Finally, when fusing the edge gradient matrix with the color space conversion result, a weighted superposition algorithm is adopted: a weight of 0.7 is assigned to the edge gradient matrix, and a weight of 0.3 is assigned to the saturation component to form a dual-information composite feature map containing spatial structure and color features.

[0023] In addition, before calculating the edge gradient matrix of the image data using the Sobel operator, the image data is divided into multiple local sub-blocks, the gray histogram is calculated for each sub-block respectively, and a preset contrast limit parameter is set. The boundary effect between local sub-blocks is eliminated through the bilinear interpolation algorithm to generate image data with balanced illumination. That is, the system performs the contrast-limited adaptive histogram equalization (CLAHE) algorithm on each sub-block. By setting the preset contrast limit parameter to 0.02, the stretching amplitude of the histogram distribution of a single sub-block is limited within 2% of the total frequency of adjacent gray levels, so as to enhance the local contrast while avoiding excessive amplification of noise. For example, in a section with both strong reflective water accumulation and dark iron ore, the traditional global histogram equalization will cause overexposure in the high-brightness area and loss of the metal bracket edge, while this solution can separately retain the texture details of the water accumulation area and the surface uneven features of the ore area through independent processing of each sub-block.

[0024] After completing the adaptive equalization of each sub-block, the system uses the bilinear interpolation algorithm to eliminate the boundary effect between sub-blocks. That is, calculate the weight coefficients of each pixel in its four adjacent sub-blocks, and generate a continuous and smooth illumination equalized image by weighted averaging the gray values after equalization of each sub-block. Specifically, in implementation, for any pixel point (x, y), the system first determines which overlapping region of the four adjacent sub-blocks it is located in, and then calculates the interpolation weight according to the distance ratio between this point and the centers of each sub-block. For example, when a pixel is located at the junction of two horizontally adjacent sub-blocks, the weight coefficient of its left sub-block is (1 - dx), and the right sub-block is dx, where dx is the normalized distance from this pixel to the boundary of the left sub-block. This interpolation mechanism effectively solves the blocky artifacts that may be generated due to block processing. Especially when processing the gradually changing illumination area caused by dust, it can maintain the spatial continuity of the image gray level. Verified by experiments, in a typical mining area scenario, this scheme can increase the local contrast of the image by 40% while reducing the occurrence rate of blocky artifacts to less than 0.5%. After generating the illumination equalized image data, the system continues to execute the Sobel operator edge detection process.

[0025] The obstacle marking module 102 performs connected component marking on the pixel points whose gradient values exceed the first preset threshold based on the composite feature map, and generates a candidate box when there is a connected region with more than the preset number of pixel points continuously; calculates the regional mutation rate of the saturation component matrix corresponding to the candidate box. If the regional mutation rate exceeds a preset multiple of the average mutation rate of the adjacent road surface area and exists stably in the detection of three consecutive frames of images, it is marked as a rigid obstacle; if the regional mutation rate is lower than the average mutation rate of the adjacent road surface area and the intensity of the blue channel of the corresponding image data increases by more than the first preset ratio, it is marked as a soft obstacle.

[0026] Specifically, the system performs threshold screening on the gradient amplitude of each pixel in the composite feature map. When the gradient value of a certain pixel exceeds the first preset threshold (for example, set to 60% of the gradient amplitude range), it is determined as a potential obstacle edge point. For example, when there is a broken conveyor belt metal bracket in front of the vehicle, its sharp edge will form a highlighted area with a gradient value exceeding 120 (assuming the gradient range is 0 - 200) in the composite feature map.

[0027] After obtaining the set of candidate pixel points, the module uses the eight-neighbor connected component labeling algorithm for region clustering. This algorithm recursively traverses adjacent pixels, aggregates spatially continuous candidate points into independent regions, and counts the number of pixels in each connected component. When the number of pixels in a connected component exceeds a preset number (for example, set to 50 pixel points, corresponding to the projection of an obstacle of about 0.3 meters by 0.3 meters in the actual environment), the system will generate a rectangular candidate box. Taking a metal bracket as an example, its X-shaped cross structure may form two adjacent connected components in the image. If each connected component contains more than 80 high-gradient pixels, they will be respectively framed as candidate obstacles.

[0028] After completing the preliminary positioning, the module further analyzes the material characteristics through the saturation component matrix. The system calculates the regional mutation rate of the saturation component within the candidate box. This index is defined as the ratio of the sum of the absolute values of the saturation differences between adjacent pixels within the candidate box to the regional area. Rigid obstacles (such as metal equipment debris) usually have a regional mutation rate more than 2.5 times that of the adjacent road surface area due to the uniform surface material and significant color difference from the road surface. For example, the average mutation rate within the candidate box of an iron obstacle is 85, while the average mutation rate of the surrounding road surface is 32. At this time, the system will trigger the rigid obstacle determination condition. For soft obstacles (such as scattered tarpaulins), due to the light absorption characteristics of the material, their saturation mutation rate is often less than 1.2 times that of the road surface, but the intensity of the blue channel will increase by more than 15% (the preset first ratio) due to the light reflection characteristics of the material.

[0029] To ensure the detection stability, the module introduces a timing verification mechanism. A candidate box will be finally confirmed as a valid obstacle only when it continuously exists in three consecutive frames of images (time span 120 ms) and its spatial position conforms to the trajectory deduced by the vehicle kinematic model. For example, a metal rod that appears and disappears in dust may form a candidate box in a single frame of image, but if the number of connected component pixels in this area suddenly decreases due to dust and fog occlusion in subsequent frames, it will be determined as interference noise. This multi-frame verification mechanism enables the system to maintain an obstacle continuous tracking accuracy of more than 92% in the typical 60% dust concentration environment in the mining area.

[0030] The pothole area screening module 103 is used to perform neighborhood connectivity scanning on the composite feature map. When a closed circular edge structure is detected and the variance of the color space conversion result of the internal area is lower than the second preset ratio of the external area, it is marked as a pothole area to be verified.

[0031] Specifically, first, an eight-direction neighborhood scan is performed on the composite feature map. By tracing the spatial continuity between pixel points whose gradient values exceed the first preset threshold, an edge structure forming a closed loop is searched for. For example, when a collapsed pothole with a diameter of 1.2 meters appears in front of a vehicle, its edge will present an elliptical contour composed of continuous high-gradient pixels in the composite feature map, and this geometric feature can be recognized as a closed-loop structure through the scanning algorithm.

[0032] After detecting the closed-loop structure, the module further analyzes the color space conversion result of its internal area (i.e., the distribution characteristics of the saturation component in the HSV color space). By calculating the saturation variance of the internal area of the loop structure and comparing it with the saturation variance of the external adjacent area (a circular area with a 20-pixel expansion outside the loop). When the variance of the internal area is lower than the second preset ratio of the external area (for example, set to 35% of the external variance), it indicates that the color distribution uniformity in this closed area is significantly higher than that of the surrounding road surface, which conforms to the color homogenization characteristics caused by water accumulation or sediment in the pothole area. In this solution, the closed-loop edge structure can effectively capture the unique geometric morphological characteristics of the pothole, avoiding misjudging a temporary water accumulation area (without a clear edge) or a debris accumulation area (with a non-closed edge) as a pothole; secondly, the internal color variance comparison can exclude false alarms caused by light changes. For example, when the vehicle headlights shine on an uneven road surface to form a ring-like shadow, although a temporary edge contour may be generated, its internal color variance is still of the same order of magnitude as that of the normal road surface.

[0033] The obstacle confirmation module 104 is used to obtain the point cloud data in front of the mining vehicle through the radar installed on the mining vehicle, and extract the reflection intensity anomaly coefficients in the corresponding areas of the rigid obstacles and soft obstacles in the point cloud data according to the preset correspondence relationship at any position between the point cloud data and the image data; when the reflection intensity anomaly coefficient is greater than the corresponding second preset threshold, it is confirmed that an obstacle exists; otherwise, it is confirmed that there is no obstacle.

[0034] Specifically, first, based on the joint calibration parameters of the camera module and the radar, a preset correspondence relationship matrix between the image pixel coordinate system and the radar polar coordinate system is established, so that the position of any candidate box in the image data can be mapped to the corresponding spatial area in the point cloud data. The formula corresponding to the preset correspondence relationship is:

[0035] Among them, are respectively the pixel horizontal and vertical coordinates of the target in the image coordinate system, is the depth value of the target in the camera coordinate system; 、 are respectively the x-axis and y-axis focal lengths of the camera, 、 are respectively the coordinates of the camera optical center on the image plane, is the rotation matrix, is the translation vector, , , are respectively the three-dimensional coordinates of the target in the radar coordinate system.

[0036] Then the module extracts the reflection intensity anomaly coefficients of the point cloud data in the corresponding areas of rigid obstacles and soft obstacles. This coefficient is calculated by statistically calculating the ratio of the standard deviation of the radar reflection intensity values in these areas to the reference reflection intensity of the mining area road surface. Its physical meaning is to quantify the difference in the reflection characteristics between the obstacle material and the conventional road surface. Specifically, for rigid obstacles such as metal brackets, due to the dense surface material and strong conductivity, their radar reflection intensity usually reaches more than 1.8 times the reference value, and the standard deviation of the reflection intensity distribution in the area is less than 15% (reflecting the material uniformity); while for soft obstacles such as rubber tarpaulins, due to their wave-absorbing characteristics, their reflection intensity is only 0.6 times the reference value, but the standard deviation may increase to 25% due to the influence of material wrinkles. For example, when the average reflection intensity of the point cloud area corresponding to a certain candidate box is 145 dB (the reference road surface is 80 dB) and the standard deviation is 12 dB, the calculated reflection intensity anomaly coefficient is (145 / 80)×(15% / 12%) = 2.81, which exceeds the rigid obstacle determination threshold of 2.5, and the system immediately confirms the real existence of this obstacle.

[0037] The actual pothole area confirmation module 105 is used to calculate the height difference of all the point cloud data within the closed loop edge structure. When the height difference exceeds the preset difference, it is confirmed as the actual pothole area.

[0038] Specifically, when the pothole area screening module identifies the closed loop edge structure through the composite feature map, this module immediately calls the radar to obtain the point cloud data of the corresponding spatial area. According to the preset sensor calibration parameters, the system maps the geometric position of the closed loop structure in the image coordinate system to the radar coordinate system, and accurately extracts the three-dimensional point cloud data set covered by this circular area. For example, when a 1.2-meter-diameter elliptical closed edge is visually detected, the system determines the spatial range of this area in the radar point cloud through the coordinate transformation matrix, usually corresponding to about 2000 - 3000 effective point cloud data points.

[0039] The module then performs elevation analysis on these point cloud data and calculates the statistical characteristics of their height differences. In specific implementation, first, the horizontal plane where the center point of the annular structure is located is used as the reference plane, and the least squares method is used to fit the ideal road surface plane equation. Then, the vertical distances from all the point cloud data within the annular region to this reference plane are calculated, and the maximum height difference index is statistically analyzed. When the maximum height difference exceeds a preset difference (for example, set to 0.5 meters, corresponding to the minimum dangerous depth of common collapse pits in the mining area), the system will confirm this as the actual pit area. Taking an iron ore transportation roadway as an example, when a collapse pit with a diameter of 1.5 meters formed due to geological activities appears in front of the vehicle, the radar point cloud data shows that the height difference between the lowest point of the pit bottom and the surrounding road surface reaches 0.82 meters, significantly exceeding the threshold. At this time, the system immediately triggers a pit warning; while the temporarily waterlogged area, although visually presenting a similar closed annular feature, its point cloud height difference is only 0.08 meters, far lower than the determination threshold, so it will not be misjudged as a dangerous pit.

[0040] The prompt module 106 is used to mark the areas corresponding to the confirmed rigid obstacles, soft obstacles, and actual pit areas as warning areas on the image data; calculate the distance between each warning area and the mining vehicle based on the point cloud data, and project the distance and the warning area onto the display screen of the mining vehicle.

[0041] Specifically, this module first receives the coordinate information of rigid obstacles (such as metal brackets) and soft obstacles (such as scattered tarpaulins) from the obstacle confirmation module, as well as the spatial parameters of the collapse area from the actual pit area confirmation module. By aligning the image data and the point cloud data in space and time, the module superimposes semi-transparent warning marks (i.e., warning areas) on the image data: rigid obstacles are marked with red rectangular frames, and the line width of the frame is positively correlated with the height of the obstacle; soft obstacles are marked with yellow dashed frames, and their transparency is dynamically adjusted according to the reflectivity of the obstacle material; the actual pit area is marked with blue concentric circles, the diameter of the inner circle accurately reflects the opening size of the pit, and the outer circle radius expands according to the depth of the pit (for example, for a 0.5-meter deep pit, the outer circle radius is 15 pixels larger than the inner circle).

[0042] In terms of spatial positioning, the module calculates the Euclidean distance between the center point of each warning area and the vehicle's centroid by analyzing the three-dimensional coordinates of the radar point cloud data and combining with the vehicle pose data output by the real-time positioning system for mining vehicles (RTK-GNSS, i.e., Real-Time Kinematic Global Navigation Satellite System). During specific implementation, the system takes the center of the vehicle's front bumper as the origin of the coordinate system, and uses the three-dimensional point set of the corresponding area of the obstacle in the point cloud data to determine its spatial position through the weighted centroid algorithm. For example, when a metal support is detected 12 meters ahead, the system extracts 200 radar point cloud data mapped by the candidate box of the obstacle. After removing the high-altitude noise points (points with Z-axis coordinates exceeding ±0.3 meters from the road surface reference), it calculates the average XYZ coordinates of the remaining points, and obtains that the actual distance of the obstacle is 12.3 meters, and the horizontal yaw angle is -2.1 degrees (indicating that it is in the front left of the vehicle's driving direction).

[0043] In the visualization projection stage, the system converts the calculated spatial parameters of the warning area into two-dimensional projection coordinates on the cab display screen. This process uses a perspective projection matrix and combines with the installation angle of the display screen (usually set at a 15° downward angle with respect to the driver's line of sight) for coordinate transformation to ensure that the position of the warning mark on the screen strictly corresponds to the real spatial orientation. For example, when an actual pothole is 8 meters ahead of the vehicle's right front wheel track, the system will dynamically display the blue concentric circle mark of the pothole in the right 1 / 3 area of the display screen, and calculate the flashing frequency (5Hz) of the mark according to the vehicle speed (assumed to be 20 km / h) to enhance the warning effect. This spatial mapping mechanism enables the driver to capture more than 80% of the warning information through peripheral vision without having to perform a visual focus switch (i.e., when observing the road surface and the display screen simultaneously).

[0044] Furthermore, the system also includes: A steep slope recognition module 107, which is used to calculate the horizontal edge gradient of any image data. When the consistency of the horizontal edge gradient direction of more than 10 consecutive rows of pixels in any area of the image data exceeds the third preset ratio, it is determined that there is a slope change in the area; calculate the edge density attenuation rate of the area in the vertical direction. When the attenuation rate drops by more than the fourth preset ratio per second and lasts for more than 3 frames, it is confirmed that a steep slope terrain is formed; mark the confirmed steep slope terrain in the image data and display it on the display screen of the mining vehicle.

[0045] Specifically, the module first extracts the horizontal gradient component from the edge gradient matrix generated by the camera module. When it is detected that the consistency of the horizontal edge gradient direction of more than 10 consecutive rows of pixels in a certain area of the image exceeds the third preset ratio (for example, set to 85%), it is preliminarily determined that there is a slope change in this area. The principle is as follows: During the driving of a mining vehicle, when there is a steep slope on the road, the horizontal edge formed on the road surface (such as the transition line between the top and bottom of the slope) will show the characteristic of high consistency in the gradient direction of multiple consecutive rows of pixels in the image. For example, in an uphill section with a slope angle exceeding 15°, a horizontal gradient direction-consistent edge band will be formed at the junction of the slope surface and the skyline in the image captured by the camera. More than 90% of the pixels in 10 consecutive rows of pixels have the horizontal gradient direction pointing to the same side (such as the right side), and at this time, the preliminary steep slope determination condition is triggered.

[0046] Then, the module verifies the terrain features by calculating the edge density attenuation rate in the vertical direction of this area. The edge density attenuation rate is defined as the decreasing rate of the number of pixels with the vertical gradient amplitude exceeding the first preset threshold in each row of pixels along the vertical direction (from top to bottom). When it is detected that the edge density attenuation rate of this area drops by more than the fourth preset ratio (for example, set to 30%) per second and lasts for more than 3 frames, the system confirms the formation of a steep slope terrain. The principle is as follows: When the vehicle approaches a steep slope, due to the sudden change in the geometric shape of the slope surface, the visible range of the vertical edge (such as the gravel texture on the slope surface) in the image will rapidly shrink as the vehicle approaches. For example, during the process of driving towards a 30° steep slope, the vertical edge density formed by the gravel on the slope surface in the image drops from 120 effective edge points per row to 84 (a 30% decrease) in the first three frames (time span of 120 ms), and continues to drop to 60 in the subsequent two frames. At this time, the system will meet the steep slope terrain confirmation condition.

[0047] After completing the terrain feature verification, the module marks the formed steep slope terrain in the image data with a green wavy line, and the marking range covers the circumscribed rectangle of the gradient direction consistency area. At the same time, the system predicts the slope angle of the current steep slope through a linear regression algorithm based on the change trend of the edge density attenuation rate within three consecutive frames, and superimposes and displays the angle value at the corresponding position on the display screen of the mining vehicle cab. For example, when a climbing section with a slope angle of 22° is detected, the system draws a green wavy line with a width of 5 pixels at the top of the slope in the image, and dynamically displays the warning text "Slope 22°" below it.

[0048] Furthermore, the system also includes a muddy recognition module 108.

[0049] This module is used to extract the first ratio change curve of the saturation component and the brightness component in the HSV space. When the fluctuation range of the first ratio within the set pixel window is less than the fifth preset ratio and the overall brightness drops by more than the sixth preset ratio, a corresponding candidate area is generated.

[0050] Specifically, first, the saturation component and the brightness component are extracted from the HSV color space conversion result obtained from the camera module, the first ratio (S / V ratio) of the saturation component to the brightness component at each pixel is calculated, and the fluctuation amplitude of this ratio in the spatial distribution is statistically calculated in units of a set pixel window (such as a 50×50 pixel sliding window). When the fluctuation amplitude of the first ratio in a certain area is less than the fifth preset ratio (such as set to 15%) and the overall brightness value drops by more than the sixth preset ratio (such as set to 30%) compared to the reference road surface, a corresponding candidate area is generated. The principle is as follows: Due to water infiltration, the surface color of the muddy area becomes homogenized, which is manifested as the ratio of the saturation component to the brightness component tending to be stable (the fluctuation amplitude decreases) in the HSV space. At the same time, the overall brightness significantly decreases due to the light diffuse reflection effect on the wet surface. For example, when the vehicle travels to a waterlogged section with a mud content of 40%, the reddish-brown color of the mud mixture has a saturation component stable in the range of 0.35 - 0.45 in the HSV space, and the brightness component drops from 0.7 of the normal road surface to 0.5. At this time, the fluctuation amplitude of the first ratio is only 12%, meeting the candidate area generation conditions.

[0051] Subsequently, the Canny edge detection algorithm is used to process the image data to generate a texture feature map. The edge pixels of the candidate area are extracted from the texture feature map, and the second ratio of the number of non - continuous edge segments to the total edge length of the edge pixels is calculated. When the second ratio exceeds the third preset threshold, it is confirmed as a muddy area; Specifically, this module performs multi - scale gradient calculations on the original image data through the Canny edge detection algorithm to generate a high - resolution texture feature map. In the texture feature map, edge pixels of the candidate area are extracted, and the second ratio of the number of non - continuous edge segments to the total edge length of the edge pixels is calculated. When the second ratio exceeds the third preset threshold (such as set to 0.25), it is confirmed as a muddy area. The principle is as follows: The surface texture damage caused by the wet and slippery muddy road surface will form a large number of fine and broken edge features, which are significantly different from the continuous texture of the normal road surface. In specific implementation, the Canny algorithm uses a double - threshold (such as setting the low threshold to 30 and the high threshold to 90) for gradient amplitude screening and retains the real edge information through non - maximum suppression technology. For example, in the muddy area, the junction between broken stones and the mud mixture will form non - continuous edge segments with a length less than 5 pixels, and the proportion of their number in the total edge length can reach 28%, exceeding the determination threshold to trigger the confirmation condition.

[0052] Finally, mark the confirmed muddy areas in the image data and display them on the display screen of the mining vehicle. Send the boundary coordinates of the muddy areas to be confirmed to the prompt module, mark them with green semi-transparent polygons in the image data, and overlay warning icons on the display screen of the mining vehicle. The warning icons adaptively adjust their display sizes according to the area and shape parameters of the muddy areas. When the area of the muddy area exceeds 2 square meters, a high-frequency flashing border is added to the periphery of the icon to enhance the warning effect.

[0053] Furthermore, the system also includes an obstacle avoidance path generation module 109. This module is used to generate an obstacle avoidance path that maintains a preset safe distance from obstacles based on the three-dimensional coordinates of rigid obstacles, soft obstacles, and actual pothole areas through a heuristic path search algorithm.

[0054]

[0055] Among them, represents the actual movement cost from the starting point to node n, is the heuristic estimated cost from node n to the target point, is the path smoothness penalty term. The dynamic weight adjustment mechanism is reflected in the real-time calculation of the three coefficients , , : When a rigid obstacle is detected, according to its horizontal distance d from the current path node, the path deviation cost is dynamically increased according to the formula to force the generated path to maintain a safe distance of at least 1.2 meters from the rigid obstacle; when the path passes through the soft obstacle area, by reducing the value to below 0.1, the path is allowed to have a greater curvature to bypass the soft area that may affect the vehicle's stability.

[0056] Among them, the generation of the obstacle avoidance path includes: Set the area where the slope change rate determined by the steep slope recognition module exceeds the preset threshold as an impassable area. According to the height difference parameters and the distance parameters from the mining vehicle corresponding to the rigid obstacles, soft obstacles, and actual pothole areas in the radar point cloud data, adjust the weight ratio of the obstacle distance cost term and the path smoothness cost term in the path search algorithm, so that the generated path preferentially maintains a preset safe distance from the rigid obstacles.

[0057] Specifically, when the elevation increase rate of three consecutive grid cells is detected to exceed 15% (corresponding to a slope angle > 8.5°), this area is marked as an impassable area. For example, in an iron ore transportation roadway, if there is a slope formed by a landslide 10 meters ahead, the radar point cloud data shows that the elevation in this area suddenly rises from +0.2 meters to +2.1 meters within a lateral range of 3 meters, and the elevation change rate reaches 63% / meter. At this time, the module will generate a red warning polygon covering this slope area and prohibit the path search algorithm from selecting nodes passing through this area.

[0058] In addition, the multi-constraint fusion mechanism during the path generation process is implemented through a hierarchical decision-making architecture: The first layer processes rigid obstacle constraints and uses the elliptical expansion method to expand the grid occupied by the obstacle. The expansion radius is: where, is the current vehicle speed in m / s. This ensures the dynamic safety distance; The second layer processes terrain constraints, sets the grid cost corresponding to the impassable area to infinity, and adjusts the grid movement cost coefficient according to the ground material classification result (jointly judged by the saturation component and the radar reflection intensity). For example, the movement cost coefficient for a muddy area is set to 2.5, and for a gravel road surface is set to 1.0; The third layer performs path smoothing optimization, uses B-spline curves to fit the preliminary path nodes to ensure that the path curvature is continuous and meets the minimum turning radius of the vehicle (set to 6 meters).

[0059] Finally, in the real-time image data of the display screen, the obstacle avoidance path trajectory is marked with a first identifier, the area corresponding to the steep terrain is marked with a second identifier, and the muddy area is marked with a third identifier.

[0060] Furthermore, the system also includes a synchronization module for sending the first identifier, the second identifier, and the third identifier to the following vehicles through the vehicle-to-vehicle communication protocol; The specific processing process is as follows: When generating the first identifier, the second identifier, or / and the third identifier on the display screen of any mining vehicle, the type attribute and the boundary parameters in the image coordinate system corresponding to the mark are synchronously extracted and encapsulated into a structured data packet.

[0061] Specifically, when the display screen of the host vehicle (i.e., the vehicle that detects the obstacle) generates the first identifier (rigid obstacle), the second identifier (soft obstacle), or the third identifier (actual pothole area), the synchronization module immediately extracts the type attributes of each identifier, the boundary parameters in the image coordinate system, and the associated sensor data to form a structured data packet containing multi-dimensional information. Among them, the type attributes include the identifier classification code (e.g., 01 represents a rigid obstacle, 02 represents a soft obstacle, 03 represents a pothole area), and the confidence level (generated based on the number of multi-frame validations and the sensor consistency score); the boundary parameters include the set of vertex coordinates of the warning area in the image coordinate system, represented in normalized proportion (e.g., the upper left corner coordinates (0.2, 0.5), the lower right corner coordinates (0.3, 0.7)); the additional data includes the current GPS coordinates of the host vehicle (in WGS84 format, with centimeter-level accuracy) and the real-time point cloud height parameters collected by the radar (including the extreme values and standard deviation of the point cloud elevation in the obstacle area). For example, when a certain host vehicle detects a rigid obstacle identifier from the upper left corner (80, 120) to the lower right corner (160, 180) in the image coordinate system (320×240), the module will extract the normalized coordinates of this rectangle (0.25, 0.5)-(0.5, 0.75), and append the current GPS coordinates (39°56'27.6" N, 116°20'15.8" E) and the radar point cloud height parameters of the corresponding area (maximum elevation difference 0.8 m, standard deviation 0.12 m).

[0062] Broadcast the data packet at a fixed frequency through the on-vehicle communication unit on the mining vehicle, and append the current GPS coordinates of the mining vehicle and the real-time point cloud height parameters collected by the radar; specifically, the data encapsulation and broadcast mechanism adopts a hierarchical coding strategy to encapsulate the above information into a data frame structure that conforms to the ISO 20078 standard. The on-vehicle communication unit broadcasts the data packet through the dedicated short-range communication (DSRC, i.e., Dedicated Short Range Communications) protocol at a fixed frequency (usually set to 10 Hz). Each data packet contains a 16-byte header (including a timestamp, vehicle ID, and check code) and a 128-byte payload. For example, in the data packet broadcast by a certain host vehicle at t = 1234567890 ms (this time is an absolute timestamp for example, representing the digital representation of a certain moment in the system), the type attribute field is "01-0.95" (indicating a rigid obstacle with a 95% confidence level), the boundary parameter field contains 4 sets of floating-point numbers, and the additional data field contains double-precision GPS coordinates and floating-point elevation parameters. This mechanism ensures that the following vehicle receives at least two updated data within 200 ms, meeting the information synchronization requirements under the maximum relative speed of 40 km / h for mining area vehicles.

[0063] After receiving the data packet, the rear vehicle maps the received marked coordinates to its own vehicle coordinate system according to the distance information of the vehicle ahead obtained by its own radar, realizing the dynamic position calibration among multiple vehicles. Specifically, the data fusion processing of the rear vehicle includes the following core steps: (1) Receive the data packet through the vehicle-mounted communication unit and parse each field; (2) Calculate the relative spatial position based on the GPS coordinate difference between the master and slave vehicles, and establish a coordinate transformation matrix in combination with the distance information of the vehicle ahead collected by the vehicle's own radar (accuracy ±0.3 m); (3) Map the boundary parameters of the received image coordinate system to the vehicle's own sensor coordinate system. In specific implementation, the rear vehicle first parses the GPS coordinates of the master vehicle, converts them into plane rectangular coordinates through UTM projection, and performs differential calculation with the vehicle's own real-time GPS coordinates to obtain the relative azimuth angle and distance between the two vehicles. For example, when the master vehicle is located 28.5 meters away in the direction of 15° east of north, the rear vehicle converts the received obstacle image coordinates (0.25, 0.5)-(0.5, 0.75) into (0.31, 0.53)-(0.55, 0.78) in its own image coordinate system through the coordinate transformation matrix. This conversion takes into account the vehicle heading angle deviation (obtained through the vehicle's own IMU sensor) and perspective projection distortion. At the same time, the module performs spatial registration on the radar point cloud height parameter of the master vehicle and the vehicle's own real-time point cloud data. When the detected elevation feature consistency exceeds 85%, it is determined as a credible mapping.

[0064] Based on the same inventive concept, the present invention provides an electronic device, including: a memory 202, a processor 201, and a computer program stored on the memory 202 and executable on the processor 201. When the processor 201 executes the computer program, a safety driving detection system for mining vehicles based on vision and radar fusion is realized.

[0065] Based on the same inventive concept, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, a safety driving detection system for mining vehicles based on vision and radar fusion is realized.

[0066] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A mining vehicle safe driving detection system based on vision and radar fusion, characterized in that: include: A camera module, used for acquiring image data in front of the mining vehicle through a camera on the mining vehicle; The Sobel operator is used to calculate the edge gradient matrix of the image data, the road color features are separated based on the HSV color space, and the saturation component is extracted to generate the color space conversion result; the edge gradient matrix and the color space conversion result are superimposed to generate a composite feature map; The obstacle marking module marks the connected domains of the pixels whose gradient values ​​exceed the first preset threshold based on the composite feature map, and generates a candidate box when a connected area with more than a preset number of pixels appears continuously; Calculate the regional mutation rate of the saturation component matrix corresponding to the candidate frame. If the regional mutation rate exceeds the preset multiple of the average mutation rate of the adjacent road area and exists stably in three consecutive image detections, it is marked as a rigid obstacle; if the regional mutation rate is lower than the average mutation rate of the adjacent road area and the blue channel intensity of the corresponding image data rises by more than the first preset ratio, it is marked as a soft obstacle; A pothole area screening module is used to scan the neighborhood connectivity of the composite feature map, and when a closed ring edge structure is detected and the variance of the color space conversion result of the inner area is lower than the second preset ratio of the outer area, it is marked as a pothole area to be verified; The obstacle confirmation module is used to obtain the point cloud data in front of the mining vehicle through the radar installed on the mining vehicle, and extract the reflection intensity abnormality coefficient in the corresponding area of ​​the rigid obstacle and the soft obstacle in the point cloud data according to the preset corresponding relationship between the point cloud data and any position of the image data; when the reflection intensity abnormality coefficient is greater than the corresponding second preset threshold value, it is confirmed that the obstacle exists; Otherwise, it is confirmed as non-existent; The actual pothole area confirmation module is used to calculate the height difference of all point cloud data within the closed ring edge structure. When the height difference exceeds the preset difference value, it is confirmed as an actual pothole area; A prompt module is used to mark the confirmed rigid obstacles, soft obstacles and areas corresponding to the actual pothole areas as warning areas on the image data; The distance between each warning area and the mining vehicle is calculated based on the point cloud data, and the distance and the warning area are projected on a display screen of the mining vehicle.

2. According to claim 1, a mining vehicle safe driving detection system based on vision and radar fusion is characterized in that: The camera module also includes: Before the Sobel operator is used to calculate the edge gradient matrix of the image data, the image data is divided into a plurality of local sub-blocks, a grayscale histogram is calculated for each sub-block and a preset contrast limit parameter is set, and a bilinear interpolation algorithm is used to eliminate the boundary effect between the local sub-blocks to generate image data with balanced illumination.

3. The mining vehicle safe driving detection system based on vision and radar fusion according to claim 1 is characterized in that: The system further comprises: A steep slope identification module, used to calculate the horizontal edge gradient of any image data, and when the consistency of the horizontal edge gradient direction of more than 10 consecutive rows of pixels in any area of ​​the image data exceeds a third preset ratio, it is determined that there is a slope change in the area; Calculating the edge density decay rate of the region in the vertical direction, and when the decay rate decreases by more than a fourth preset ratio per second and lasts for more than 3 frames, it is confirmed that a steep slope terrain is formed; The confirmed steep slope terrain is marked in the image data and displayed on the display screen of the mining vehicle.

4. The mining vehicle safe driving detection system based on vision and radar fusion according to claim 3 is characterized in that: The system further comprises: A muddy identification module is used to extract a first ratio change curve of the saturation component and the brightness component in the HSV space, and generate a corresponding candidate area when the fluctuation amplitude of the first ratio in the set pixel window is less than a fifth preset ratio and the overall brightness decreases by more than a sixth preset ratio; Processing the image data by using the Canny edge detection algorithm to generate a texture feature map, extracting edge pixel points of the candidate area from the texture feature map, calculating a second ratio of the number of discontinuous edge segments in the edge pixel points to the total edge length, and confirming it as a muddy area when the second ratio exceeds a third preset threshold; Confirmed muddy areas are marked in the image data and displayed on the display screen of the mining vehicle.

5. A mining vehicle safe driving detection system based on vision and radar fusion according to claim 4, characterized in that: The system further comprises: The obstacle avoidance path generation module is used to generate an obstacle avoidance path that maintains a preset safety distance from the obstacles through a heuristic path search algorithm based on the three-dimensional coordinates of the rigid obstacles, soft obstacles, and actual pothole areas.

6. The mining vehicle safe driving detection system based on vision and radar fusion according to claim 5 is characterized in that: The generation of the obstacle avoidance path includes: The area where the slope change rate determined by the steep slope recognition module exceeds the preset threshold is set as an impassable area. According to the height difference parameters corresponding to the rigid obstacles, soft obstacles and actual pothole areas in the radar point cloud data and the spacing parameters with the mining vehicle, the weight ratio of the obstacle distance cost item and the path smoothness cost item in the path search algorithm is adjusted so that the generated path preferentially maintains a preset safe distance from the rigid obstacles; In the real-time image data of the display screen, the obstacle avoidance path trajectory is marked with a first marker, the area corresponding to the steep slope terrain is marked with a second marker, and the muddy area is marked with a third marker.

7. The mining vehicle safe driving detection system based on vision and radar fusion according to claim 6 is characterized by: The system further includes a synchronization module for sending the first identifier, the second identifier and the third identifier to the rear vehicle through the inter-vehicle communication protocol; the specific processing process is: When the display screen of any mining vehicle generates the first mark, the second mark or / and the third mark, the type attribute corresponding to the mark and the boundary parameter in the image coordinate system are synchronously extracted and encapsulated into a structured data packet; The data packet is broadcasted at a fixed frequency by an on-board communication unit on the mining vehicle, and the current GPS coordinates of the mining vehicle and the real-time point cloud height parameters collected by the radar are attached to the data packet; After receiving the data packet, the rear vehicle maps the received marker coordinates to its own vehicle coordinate system based on the distance information of the preceding vehicle obtained by its own radar, thus realizing dynamic position calibration of multiple vehicles.

8. The mining vehicle safe driving detection system based on vision and radar fusion according to claim 1 is characterized in that: The formula corresponding to the preset corresponding relationship is: in, are the horizontal and vertical coordinates of the pixel of the target in the image coordinate system, is the depth value of the target in the camera coordinate system; , are the x-axis and y-axis focal lengths of the camera, respectively. , are the coordinates of the camera optical center in the image plane, is the rotation matrix, is the translation vector, , , are the three-dimensional coordinates of the target in the radar coordinate system.

9. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a mining vehicle safe driving detection system based on vision and radar fusion as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a mining vehicle safe driving detection system based on vision and radar fusion as described in any one of claims 1 to 8 is implemented.

Images