Pavement depression treatment method, device, equipment and medium

Abstract

The invention discloses a road surface depression processing method and device, equipment and a medium, and relates to the technical field of computers, and the method comprises the steps: obtaining a road surface scene image collected by a monocular camera, inputting the road surface scene image into a pre-trained depression detection model, and outputting depression visual perception data through the depression detection model; wherein the sag detection model is a multi-task deep neural network model constructed based on a lightweight coding and decoding architecture; determining concave geometrical characteristic parameters based on the camera calibration parameters and the concave visual perception data; evaluating the grade of the risk area where the corresponding recess is located through the preset vehicle driving risk area and the recess geometric feature parameters, and determining whether to trigger an early warning prompt or not based on the grade of the risk area. Therefore, the problems of high cost and limited stability caused by the use of a laser radar in the traditional technology are solved, the adaptability and robustness to complex scenes are remarkably improved, and the operation burden of a driver is effectively relieved at the same time.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, equipment and medium for treating road surface depressions. Background Technology

[0002] In heavy-duty vehicle operations, vehicles typically carry large loads and operate under high intensity. Road surfaces are subjected to the combined effects of heavy traffic and environmental factors (such as rain erosion and weathering), making them prone to potholes, subsidence, and cracks. If a vehicle tire enters these damaged areas, it can easily lead to tire blowouts, loss of vehicle control, and suspension system damage. This not only reduces transportation efficiency and increases tire replacement and equipment maintenance costs, but also threatens the safety of drivers and passengers and the integrity of the operating equipment.

[0003] In traditional operation modes, drivers must rely on their personal driving experience to subjectively judge road conditions and avoid damaged areas, which places extremely high demands on their operational skills and concentration. Furthermore, fatigue from prolonged work, poor visibility conditions such as rain, snow, or nighttime, or misjudgment can easily lead to accidents, while also significantly reducing driving comfort and overall work efficiency. In intelligent heavy-duty vehicle autonomous driving systems, the problem of road damage perception requires the collaboration of high-precision sensors and efficient algorithms to achieve accurate identification of road conditions. Therefore, real-time detection of road surface damage and triggering risk warnings are of great significance for reducing the driver's workload, improving the safety of the entire operation process, and reducing the accident rate.

[0004] Among the existing road damage detection technologies, (1) although lidar can achieve high-precision ground perception, the equipment cost is high and it is easily affected by debris such as soil piles, stones, and dust in the operation scene, and its stability is limited; (2) although traditional image processing methods (such as binarization, road cross section analysis, and edge detection) have low hardware costs, their environmental adaptability and robustness are poor, and they are difficult to cope with complex operation scenes such as sudden changes in illumination, uneven road surface materials, and debris occlusion, and cannot meet the requirements of accurate perception. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for treating road surface depressions, which solves the problems of high cost and limited stability associated with traditional technologies using lidar, and significantly improves adaptability and robustness to complex scenarios, while effectively reducing the driver's workload. The specific solution is as follows:

[0006] In a first aspect, this application discloses a method for treating road surface depressions, including:

[0007] A road scene image captured by a monocular camera is acquired, the road scene image is input into a pre-trained concavity detection model, and concavity visual perception data is output through the concavity detection model; the concavity detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture.

[0008] The geometric feature parameters of the depression are determined based on the camera calibration parameters and the visual perception data of the depression; wherein, the geometric feature parameters of the depression include the depression location, the depression diameter and the height difference of the depression itself;

[0009] The risk level of the corresponding depression is assessed by comparing the pre-defined vehicle driving risk area with the geometric feature parameters of the depression, and a warning prompt is determined based on the risk level.

[0010] Optionally, before inputting the road surface scene image into the pre-trained depression detection model, the method further includes:

[0011] To obtain a training dataset, sample images with different environmental conditions and different concavity morphologies are acquired.

[0012] The training dataset is annotated in multiple dimensions; wherein the multiple dimensions are annotated in three ways: first annotation of the depression area, second annotation of the road surface structure, and third annotation of the depth of the depression.

[0013] The labeled training dataset is input into the model to be trained to train the first task branch corresponding to the first label, the second task branch corresponding to the second label, and the third task branch corresponding to the third label in the model to be trained, and the first loss, the second loss and the third loss are obtained respectively.

[0014] When the sum of the loss values ​​of the first loss, the second loss, and the third loss satisfies the preset loss condition, the indentation detection model is obtained.

[0015] Optionally, the step of outputting visual perception data of the depression through the depression detection model includes:

[0016] The first task branch of the indentation detection model outputs the border information representing the indentation region.

[0017] The second task branch of the indentation detection model outputs pixel information characterizing the road surface structure.

[0018] The third task branch of the indentation detection model outputs depth information that characterizes the depth of the indentation.

[0019] Optionally, determining the geometric feature parameters of the depression based on the camera calibration parameters and the depression visual perception data includes:

[0020] The location of the depression is determined using a first calculation model, the camera calibration parameters, and the visual perception data of the depression; wherein, the first calculation model is:

[0021] ;

[0022] in, Indicates the location of the depression. This represents the image coordinates of the midpoint of the concave border. Indicates camera intrinsic parameters. This represents the depth value of the midpoint of the recessed border in the camera coordinate system. This indicates the position of the camera's optical center in the world coordinate system. This represents the camera pose rotation matrix;

[0023] The diameter of the depression is determined using the second calculation model and the location of the depression.

[0024] The second calculation model is as follows:

[0025] ;

[0026] in, Indicates the diameter of the depression. This represents the image coordinates of the points within the left frame of the depression. This represents the image coordinates of the points in the right frame of the depression. This represents the image coordinates of the midpoint of the upper border of the indentation. The image coordinates of the midpoint of the concave lower border;

[0027] The height difference of the depression itself is determined by the third calculation model and the location of the depression.

[0028] The third calculation model is as follows:

[0029] ;

[0030] The difference in the height of the depression It is the average of the depth values ​​of all pixels within the recessed area. It is an absolute value.

[0031] Optionally, the method for treating road surface depressions further includes:

[0032] The reference area is the area of ​​the road surface covered by the width of the vehicle itself when it is in motion;

[0033] The area formed by extending forward a first length from the reference area is identified as a high-risk area.

[0034] The area formed by extending forward from the reference area by a first length to a second length is defined as a medium-risk area; wherein the second length is greater than the first length.

[0035] The high-risk areas and other areas outside the medium-risk areas are designated as low-risk areas.

[0036] Optionally, the step of assessing the risk zone level of the corresponding depression by using a pre-defined vehicle driving risk zone and the depression's geometric feature parameters includes:

[0037] If the depression is located in the high-risk area, and the diameter of the depression is greater than a preset diameter threshold and the height difference of the depression itself is greater than a preset height difference threshold, then the depression is determined to be at a high-risk level.

[0038] If the depression is located in the medium-risk area, and the diameter of the depression is greater than a preset diameter threshold and the height difference of the depression itself is greater than a preset height difference threshold, then the depression is determined to be at the medium-risk level.

[0039] The remaining cases were classified as low-risk depressions.

[0040] Optionally, determining whether to trigger an early warning based on the risk area level includes:

[0041] If the dent is classified as high-risk, a Level 1 warning is triggered; the Level 1 warning is used to remind the driver to take evasive action.

[0042] If the dent is classified as medium risk, a level 2 warning is triggered; the level 2 warning is used to remind the driver to plan their route.

[0043] If the dent is at a low-risk level, there is no need to trigger an early warning.

[0044] Secondly, this application discloses a road surface depression treatment device, comprising:

[0045] The concave model detection module is used to acquire road scene images captured by a monocular camera, input the road scene images into a pre-trained concave detection model, and output concave visual perception data through the concave detection model; the concave detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture.

[0046] A geometric feature determination module is used to determine the geometric feature parameters of the depression based on camera calibration parameters and the depression visual perception data; wherein, the geometric feature parameters of the depression include the depression location, the depression diameter and the height difference of the depression itself;

[0047] The risk warning module is used to assess the risk level of the corresponding depression by comparing the pre-set vehicle driving risk area with the geometric feature parameters of the depression, and to determine whether to trigger a warning based on the risk level.

[0048] Thirdly, this application discloses an electronic device, including:

[0049] Memory, used to store computer programs;

[0050] A processor is used to execute the computer program to implement the aforementioned disclosed method for treating road surface depressions.

[0051] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for treating road surface depressions.

[0052] As can be seen, this application proposes a method for processing road surface depressions, including: acquiring a road scene image captured by a monocular camera, inputting the road scene image into a pre-trained depression detection model, and outputting depression visual perception data through the depression detection model; the depression detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture; determining depression geometric feature parameters based on camera calibration parameters and depression visual perception data; wherein, the depression geometric feature parameters include the depression location, the depression diameter, and the height difference between the depression and itself; evaluating the risk area level of the corresponding depression by using a pre-set vehicle driving risk area and the depression geometric feature parameters, and determining whether to trigger a warning prompt based on the risk area level. As can be seen, this application acquires road scene images using a monocular camera, which significantly reduces equipment costs compared to LiDAR and avoids interference from debris, thus solving the problems of high cost and limited stability of existing LiDAR. Simultaneously, it employs a multi-task deep neural network model based on a lightweight encoding / decoding architecture to output concave visual perception data, which significantly improves adaptability and robustness to complex scenes compared to traditional image processing methods. Furthermore, by using camera calibration parameters and concave visual perception data to determine geometric feature parameters such as concave location, diameter, and height difference, and then combining these with pre-defined vehicle driving risk areas to assess the risk level of the concave area and trigger corresponding warning prompts, this approach can replace the subjective judgment of drivers relying on personal experience in traditional operating modes, effectively reducing the driver's operational burden and lowering the accident rate caused by low visibility conditions or judgment errors. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a flowchart of a method for treating road surface depressions disclosed in this application;

[0055] Figure 2 This is a schematic diagram of the risk level of road surface depression disclosed in this application;

[0056] Figure 3 This is a schematic diagram of the structure of a road surface depression treatment device disclosed in this application;

[0057] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Among the existing road damage detection technologies, (1) although lidar can achieve high-precision ground perception, the equipment cost is high and it is easily affected by debris such as soil piles, stones, and dust in the operation scene, and its stability is limited; (2) traditional image processing methods have lower hardware costs, but poor environmental adaptability and robustness, and are difficult to cope with operation scenes such as sudden changes in lighting, uneven road surface materials, and debris obstruction, and cannot meet the needs of accurate perception.

[0060] To address this, this application proposes a road surface depression treatment solution that solves the problems of high cost and limited stability associated with traditional LiDAR technology, significantly improves adaptability and robustness to complex scenarios, and effectively reduces the driver's workload.

[0061] This application discloses a method for treating road surface depressions. See [link to relevant documentation]. Figure 1 As shown, the method includes:

[0062] Step S11: Acquire road scene images captured by a monocular camera, input the road scene images into a pre-trained depression detection model, and output depression visual perception data through the depression detection model; the depression detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture.

[0063] In this embodiment, a depression detection model is first trained. The specific steps include: acquiring sample images under different environmental conditions and with different depression morphologies to obtain a training dataset; performing multi-dimensional annotation on the training dataset; wherein, the multi-dimensional annotation includes a first annotation on the depression area, a second annotation on the road surface structure, and a third annotation on the depth of the depression; inputting the annotated training dataset into the model to be trained to train the first task branch corresponding to the first annotation, the second task branch corresponding to the second annotation, and the third task branch corresponding to the third annotation in the model to be trained, and obtaining the first loss, the second loss, and the third loss respectively; when the sum of the loss values ​​of the first loss, the second loss, and the third loss satisfies a preset loss condition, a depression detection model for multi-task detection is obtained.

[0064] The following is a detailed analysis: (1) Considering that the road surface for heavy-duty vehicles (such as mining roads) is easily affected by the natural environment and vehicle load, the potholes (i.e., depressions) formed on the road surface vary significantly in shape, and the visual presentation of potholes under different environmental conditions is also significantly different. Therefore, data collection needs to be completed by a monocular camera installed at the front of the vehicle. The camera is fixed in position and its field of view can completely cover the road surface in front, so as to truly capture the actual state of the road surface in front during the vehicle's driving process and ensure that the collected pothole images are consistent with the actual working scene. During the collection process, two aspects need to be considered to ensure the comprehensiveness of the dataset: First, different environmental conditions, specifically including different lighting and weather scenes such as strong light on sunny days, weak light on cloudy days, water accumulation and reflection on rainy days, and auxiliary lighting at night. Second, different pothole shapes, which need to cover pothole samples of varying depths (from shallow depressions of a few centimeters to deep depressions of tens of centimeters), different sizes (from small potholes with a diameter of less than 30 centimeters to large potholes of more than 1 meter), and diverse shapes (circular, irregular polygons). At the same time, it is also necessary to include samples with complex shapes such as potholes with gravel accumulation at the edge and potholes connected to road surface cracks. (2) Multi-dimensional annotation of the training dataset: First, the first annotation is performed by drawing a rectangular bounding box to determine the specific location of each depression in the image; then the second annotation is performed by using pixel-level semantic segmentation to divide all pixels in the image into two categories: depression area and non-depression area; finally, the third annotation is performed by first obtaining the true depth value of each depression area with the help of laser scanning equipment or high-precision rangefinder, and then associating and binding the obtained true depth value with the corresponding depression area in the image. (3) Model training: The training process needs to be implemented based on a lightweight encoding and decoding architecture. The choice of this architecture needs to take into account both the detection accuracy and the real-time requirements of the vehicle equipment. A lightweight architecture (such as an encoding and decoding structure based on MobileNet or ShuffleNet) can effectively reduce the number of model parameters and the amount of computation while ensuring the feature extraction capability, thereby adapting to the limited computing resources of the vehicle terminal. The model is designed with three parallel task branches, which correspond to the three types of information mentioned above in the multi-dimensional annotation. (4) During the training process, the loss value of the three task branches needs to be calculated separately.

[0065] The first loss is calculated as follows:

[0066] ;

[0067] ;

[0068] ;

[0069] in, To predict probabilities; For truth labels; The intersection-union ratio of the predicted bounding box and the ground truth bounding box; N is the number of feature maps in the detection branches. In a lightweight architecture, N can be set to 3 (corresponding to feature maps of different scales, such as 13×13, 26×26, and 52×52). For the first loss, Area represents the area.

[0070] The second loss is calculated as follows:

[0071] ;

[0072] The number of pixels in the segmentation image needs to be determined in conjunction with the image resolution. To determine the number of categories, C=2; For the index of the segmentation map; These are the probability values ​​output by the model. It is true; This is the second loss.

[0073] The third loss is calculated as follows:

[0074] ;

[0075] in, The number of pixels in the depth map needs to be determined in conjunction with the image resolution. This is the index of the depth map; The depth value output by the model; The truth depth; This is the third loss.

[0076] When the sum of the first loss, the second loss, and the third loss satisfies a preset loss condition, a concave detection model for multi-task detection is obtained. It can be understood that the concave detection model is a multi-task deep neural network model built on a lightweight encoding / decoding architecture, containing three parallel task branches (corresponding to three types of annotation information). During training, the loss values ​​of the three branches are jointly optimized to achieve a synergistic improvement in bounding box detection, semantic segmentation, and depth estimation.

[0077] After obtaining the depression detection model, the road scene image is input into the pre-trained depression detection model. The first task branch of the depression detection model outputs the bounding box information representing the depression region, the second task branch of the depression detection model outputs the pixel information representing the road structure, and the third task branch of the depression detection model outputs the depth information representing the depth of the depression.

[0078] Step S12: Determine the geometric feature parameters of the depression based on the camera calibration parameters and the depression visual perception data; the geometric feature parameters of the depression include the depression location, the depression diameter and the height difference of the depression itself.

[0079] In this embodiment, the location of the depression is determined by a first calculation model, the camera calibration parameters, and the depression visual perception data; wherein, the first calculation model is:

[0080] ;

[0081] in, Indicates the location of the depression. This represents the image coordinates of the midpoint of the concave border. Indicates camera intrinsic parameters. This represents the depth value of the midpoint of the recessed border in the camera coordinate system. This indicates the position of the camera's optical center in the world coordinate system. This represents the camera pose rotation matrix;

[0082] The diameter of the depression is determined using the second calculation model and the location of the depression.

[0083] The second calculation model is as follows:

[0084] ;

[0085] in, Indicates the diameter of the depression. This represents the image coordinates of the points within the left frame of the depression. This represents the image coordinates of the points in the right frame of the depression. This represents the image coordinates of the midpoint of the upper border of the indentation. The image coordinates of the midpoint of the concave lower border;

[0086] The height difference of the depression itself is determined by the third calculation model and the location of the depression.

[0087] The third calculation model is as follows:

[0088] ;

[0089] The difference in the height of the depression It is the average of the depth values ​​of all pixels within the recessed area. It is an absolute value.

[0090] Step S13: Evaluate the risk level of the corresponding depression by using the pre-set vehicle driving risk area and the geometric feature parameters of the depression, and determine whether to trigger a warning prompt based on the risk level.

[0091] In this embodiment, the road surface area covered by the vehicle's width during driving is used as the reference area. An area extending forward by a first length from the reference area is defined as a high-risk area. An area extending forward by the first length to a second length from the reference area is defined as a medium-risk area, where the second length is greater than the first length. Areas outside the high-risk and medium-risk areas are defined as low-risk areas. Further, if a depression is located in a high-risk area, and the depression diameter is greater than a preset diameter threshold and the depression's height difference is greater than a preset height difference threshold, then the depression is determined to be at a high-risk level. If a depression is located in a medium-risk area, and the depression diameter is greater than a preset diameter threshold and the depression's height difference is greater than a preset height difference threshold, then the depression is determined to be at a medium-risk level. All other cases are determined to be at a low-risk level. Furthermore, if the dent is at a high-risk level, a Level 1 warning is triggered; the Level 1 warning is used to remind the driver to take evasive action. If the dent is at a medium-risk level, a Level 2 warning is triggered; the Level 2 warning is used to remind the driver to plan their route. If the dent is at a low-risk level, no warning is triggered.

[0092] In some embodiments, the area extending 20 meters forward from the reference area is defined as a high-risk area; the area extending 20 to 40 meters forward from the reference area is defined as a medium-risk area (where 40 meters is greater than 20 meters); and other areas outside the high-risk and medium-risk areas are determined to be low-risk areas. Further, if a depression is located in a high-risk area, and the depression diameter is greater than a preset diameter threshold of 50 cm and the height difference of the depression itself is greater than a preset height difference threshold of 30 cm, then the depression is determined to be at a high-risk level; if a depression is located in a medium-risk area, and simultaneously meets the conditions of a depression diameter greater than 50 cm and a height difference of the depression greater than 30 cm, then it is determined to be at a medium-risk level; except for the above two cases, all other cases are determined to be at a low-risk level. Furthermore, if the dent is at a high-risk level, a Level 1 warning will be triggered, which reminds the driver to take immediate evasive action to avoid accidents caused by driving into the high-risk dent area. If the dent is at a medium-risk level, a Level 2 warning will be triggered, which reminds the driver to plan their route in advance and adjust their route before entering the medium-risk area. If the dent is at a low-risk level, no warning will be triggered to avoid unnecessary interference and ensure smooth driving.

[0093] Figure 2 The left side shows the risk zone division results in the mining road scenario: the red area is the high-risk area extending forward 20 meters, and the blue area is the medium-risk area extending forward 20 to 40 meters. Figure 2The right side displays a real-time risk monitoring visualization interface, using a three-color scale of "green-yellow-red" to distinguish risk levels: green represents low risk, yellow represents medium risk, and red represents high risk. A pointer pointing to the red area indicates that the currently detected indentation is at a high risk level, requiring a Level 1 warning to alert the driver to take immediate evasive action.

[0094] The following example, a mining road, illustrates the method for treating road surface depressions in this solution:

[0095] I. Mine Road Environment

[0096] Mine roads are mostly unpaved dirt roads, which are prone to forming numerous complex depressions (potholes) due to long-term exposure to heavy mining trucks, rainwater erosion, and ore accumulation. These road surfaces lack standardized markings such as lane lines and stop lines, and the mining areas have high dust levels and complex lighting conditions, with direct sunlight on sunny days and insufficient lighting at night, posing significant challenges to depression detection and risk assessment.

[0097] II. Depression Detection and Geometric Feature Calculation

[0098] A monocular camera mounted on the front of the mining truck continuously captures images of the road surface ahead, covering typical environmental conditions such as sunny days, rainy days, and nighttime, as well as samples of depressions of varying depths, sizes, and shapes. After the captured images are input into a pre-trained depression detection model, the model outputs three types of information: first, the bounding box information of the depression area, clearly defining the location and extent of the depression in the image; second, pixel segmentation information of the road surface structure, distinguishing the depression area from normal road surface; and third, depth information indicating the depth of the depression. Combined with pre-calibrated parameters of the camera (intrinsic and extrinsic parameters, rotation matrices, and translation vectors), the geometric feature parameters of the depression are further calculated, including the specific location of the depression on the actual road surface, the depression diameter, and the depression height difference.

[0099] III. Risk Level Assessment and Early Warning Trigger

[0100] The risk areas are divided as follows: the area extending 20 meters forward from the road surface covered by the width of the mining truck is the base area, the area between 20 and 40 meters is the medium-risk area, and the area beyond 40 meters is the low-risk area. The preset diameter threshold is set to 50 centimeters and the preset height difference threshold is set to 30 centimeters.

[0101] If a depression is located 15 meters in front of the mine car (a high-risk area), with a diameter of 60 centimeters and a height difference of 35 centimeters, it is classified as high-risk and triggers a Level 1 warning: the audible and visual alarm in the driver's cab will continuously emit a rapid alarm, reminding the driver to immediately avoid the depression. If a depression is located 30 meters in front of the mine car (a medium-risk area), with a diameter of 55 centimeters and a height difference of 32 centimeters, it is classified as medium-risk and triggers a Level 2 warning. The driver can adjust the driving route before entering the high-risk area and choose a path to avoid the depression. If the depression is located more than 40 meters in front of the mine car, or if the diameter and height difference do not reach the threshold, it is classified as low-risk and will not trigger a warning, ensuring that the mine car's movement is not disturbed.

[0102] As can be seen, this application proposes a method for processing road surface depressions, including: acquiring a road scene image captured by a monocular camera, inputting the road scene image into a pre-trained depression detection model, and outputting depression visual perception data through the depression detection model; the depression detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture; determining depression geometric feature parameters based on camera calibration parameters and depression visual perception data; wherein, the depression geometric feature parameters include the depression location, the depression diameter, and the height difference between the depression and itself; evaluating the risk area level of the corresponding depression by using a pre-set vehicle driving risk area and the depression geometric feature parameters, and determining whether to trigger a warning prompt based on the risk area level. As can be seen, this application acquires road scene images using a monocular camera, which significantly reduces equipment costs compared to LiDAR and avoids interference from debris, thus solving the problems of high cost and limited stability of existing LiDAR. Simultaneously, it employs a multi-task deep neural network model based on a lightweight encoding / decoding architecture to output concave visual perception data, which significantly improves adaptability and robustness to complex scenes compared to traditional image processing methods. Furthermore, by using camera calibration parameters and concave visual perception data to determine geometric feature parameters such as concave location, diameter, and height difference, and then combining these with pre-defined vehicle driving risk areas to assess the risk level of the concave area and trigger corresponding warning prompts, this approach can replace the subjective judgment of drivers relying on personal experience in traditional operating modes, effectively reducing the driver's operational burden and lowering the accident rate caused by low visibility conditions or judgment errors.

[0103] Accordingly, this application also discloses a road surface depression treatment device, see [link to relevant documentation]. Figure 3 As shown, the device includes:

[0104] The concave model detection module 11 is used to acquire road scene images captured by a monocular camera, input the road scene images into a pre-trained concave detection model, and output concave visual perception data through the concave detection model; the concave detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture.

[0105] The geometric feature determination module 12 is used to determine the geometric feature parameters of the depression based on the camera calibration parameters and the depression visual perception data; wherein, the geometric feature parameters of the depression include the depression location, the depression diameter and the height difference of the depression itself;

[0106] The risk warning module 13 is used to assess the risk level of the corresponding depression by using a pre-set vehicle driving risk area and the geometric feature parameters of the depression, and to determine whether to trigger a warning based on the risk level.

[0107] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0108] As can be seen, this application proposes a method for processing road surface depressions, including: acquiring a road scene image captured by a monocular camera, inputting the road scene image into a pre-trained depression detection model, and outputting depression visual perception data through the depression detection model; the depression detection model is a multi-task deep neural network model built on a lightweight encoding and decoding architecture; determining depression geometric feature parameters based on camera calibration parameters and depression visual perception data; wherein, the depression geometric feature parameters include the depression location, the depression diameter, and the height difference between the depression and itself; evaluating the risk area level of the corresponding depression by using a pre-set vehicle driving risk area and the depression geometric feature parameters, and determining whether to trigger a warning prompt based on the risk area level. As can be seen, this application acquires road scene images using a monocular camera, which significantly reduces equipment costs compared to LiDAR and avoids interference from debris, thus solving the problems of high cost and limited stability of existing LiDAR. Simultaneously, it employs a multi-task deep neural network model based on a lightweight encoding / decoding architecture to output concave visual perception data, which significantly improves adaptability and robustness to complex scenes compared to traditional image processing methods. Furthermore, by using camera calibration parameters and concave visual perception data to determine geometric feature parameters such as concave location, diameter, and height difference, and then combining these with pre-defined vehicle driving risk areas to assess the risk level of the concave area and trigger corresponding warning prompts, this approach can replace the subjective judgment of drivers relying on personal experience in traditional operating modes, effectively reducing the driver's operational burden and lowering the accident rate caused by low visibility conditions or judgment errors.

[0109] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0110] Figure 4This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the road surface depression treatment method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 26 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0112] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon may include computer programs 221, and the storage method may be temporary storage or permanent storage. In addition to including computer programs capable of performing the road surface depression treatment method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 221 may further include computer programs capable of performing other specific tasks.

[0113] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for treating road surface depressions.

[0114] For the specific steps of this method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0115] The various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to in the method section.

[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0119] The foregoing has provided a detailed description of a method, apparatus, device, and storage medium for treating road surface depressions. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of treating a road surface depression, characterized by, The method comprises the following steps: acquiring a road scene image collected by a monocular camera, inputting the road scene image into a pre-trained pit detection model, and outputting pit visual perception data through the pit detection model; the pit detection model is a multi-task deep neural network model constructed based on a lightweight coding architecture; determining pit geometric feature parameters based on camera calibration parameters and the pit visual perception data; wherein the pit geometric feature parameters include pit position, pit diameter, and pit height difference; evaluating the risk area level of the corresponding pit based on the pit geometric feature parameters and pre-set vehicle driving risk areas, and determining whether to trigger a warning prompt based on the risk area level.

2. The road surface depression treatment method according to claim 1, characterized by, Before inputting the road scene image into the pre-trained pit detection model, the method further comprises the following steps: acquiring sample images of different environmental conditions and different pit morphologies to obtain a training data set; performing multi-dimensional labeling on the training data set; wherein the multi-dimensional labeling includes first labeling of a pit area, second labeling of a road surface structure, and third labeling of pit depth; inputting the labeled training data set into a to-be-trained model to train a first task branch corresponding to the first labeling, a second task branch corresponding to the second labeling, and a third task branch corresponding to the third labeling in the to-be-trained model, and respectively obtaining a first loss, a second loss, and a third loss; when the sum of the loss values of the first loss, the second loss, and the third loss satisfies a pre-set loss condition, the pit detection model is obtained.

3. The method of claim 2, wherein The method further comprises the following steps: outputting bounding box information representing a pit area through the first task branch of the pit detection model; outputting pixel information representing a road surface structure through the second task branch of the pit detection model; outputting depth information representing pit depth through the third task branch of the pit detection model.

4. The method for treating a road surface depression according to claim 3, characterized by, The method further comprises the following steps: determining the pit position through a first calculation model, the camera calibration parameters, and the pit visual perception data; wherein the first calculation model is: ； wherein, denotes the position of the depression, denotes the image coordinate value of the midpoint of the depression frame, denotes the camera intrinsic parameters, denotes the depth value of the midpoint of the depression frame in the camera coordinate system, denotes the position of the camera optical center in the world coordinate system, denotes the camera pose rotation matrix; determining the pit diameter through a second calculation model and the pit position; wherein the second calculation model is: ； wherein, represents a recess diameter, represents an image coordinate value of a left edge midpoint of the recess, represents an image coordinate value of a right edge midpoint of the recess, represents an image coordinate value of an upper edge midpoint of the recess, represents an image coordinate value of a lower edge midpoint of the recess; determining the pit height difference through a third calculation model and the pit position; wherein the third calculation model is: ； is the height difference of the recess, is the average of all depth values of the pixels within the recess area, is the absolute value.

5. The method of claim 4, wherein The method further comprises the following steps: taking a road area covered by the width of the vehicle itself as a reference area when the vehicle is driving; determining a region formed by extending a first length forward from the reference area as a high-risk region; determining a region formed by extending a second length forward from the reference area as a medium-risk region; wherein the second length is greater than the first length; determining other regions outside the high-risk region and the medium-risk region as a low-risk region.

6. The method of claim 5, wherein The method further comprises the following steps: If the pit is located in the high-risk area, and the pit diameter is greater than the preset diameter threshold, and the height difference of the pit itself is greater than the preset height difference threshold, it is determined that the pit is in a high-risk level; If the pit is located in the medium-risk area, and the pit diameter is greater than the preset diameter threshold, and the height difference of the pit itself is greater than the preset height difference threshold, it is determined that the pit is in a medium-risk level; The remaining other cases are determined as pits in a low-risk level.

7. The method of claim 6, wherein the method further comprises, The determination of whether to trigger a warning prompt based on the risk area level includes: If the pit is in a high-risk level, a first-level warning prompt is triggered; wherein the first-level warning prompt is used to remind the driver to take evasive measures; If the pit is in a medium-risk level, a second-level warning prompt is triggered; wherein the second-level warning prompt is used to remind the driver to plan the driving route; If the pit is in a low-risk level, no warning prompt needs to be triggered.

8. A pothole treatment device, characterized by, It includes: A pit model detection module for obtaining a road surface scene image collected by a monocular camera, inputting the road surface scene image into a pre-trained pit detection model, and outputting pit visual perception data through the pit detection model; the pit detection model is a multi-task deep neural network model constructed based on a lightweight coding architecture; A geometric feature determination module for determining pit geometric feature parameters based on camera calibration parameters and the pit visual perception data; wherein the pit geometric feature parameters include pit position, pit diameter, and pit height difference; A risk warning prompt module for evaluating the risk area level of the corresponding pit by the pre-set vehicle driving risk area and the pit geometric feature parameters, and determining whether to trigger a warning prompt based on the risk area level.

9. An electronic device, comprising: It includes: A memory for saving a computer program; A processor for executing the computer program to implement the road pit processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A memory for saving a computer program; wherein the computer program is executed by a processor to implement the road pit processing method of any one of claims 1 to 7. A memory for saving a computer program; wherein the computer program is executed by a processor to implement the road pit processing method of any one of claims 1 to 7.

Images