Automatic driving risk assessment method and device, vehicle, equipment and medium

Abstract

The invention discloses an automatic driving risk assessment method and device, a vehicle, equipment and a medium, and belongs to the technical field of automatic driving. The method comprises the following steps: respectively acquiring a visual image and a radar reflectivity graph of a target road surface; after time synchronization and space calibration are carried out on the visual image and the radar reflectivity graph, visual features of an area where a lane line is located are extracted from the visual image by adopting a convolutional neural network, a target reflectivity area higher than a reflectivity threshold in the radar reflectivity graph is identified, and reflectivity distribution features of the target reflectivity area are extracted; fusing the visual features and the reflectivity distribution features according to time synchronization and space calibration results to obtain a lane line comprehensive feature set; generating a lane line sensing result according to the lane line comprehensive feature set; and matching is performed from the risk level relationship according to the aging degree of the lane line and the complexity of the road surface, and the target risk level corresponding to the lane line modeling information is obtained, so that the safety of automatic driving is improved.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a method, device, vehicle, equipment and medium for assessing autonomous driving risks. Background Technology

[0002] Lane lines serve as the geometric reference for autonomous vehicles to achieve lateral positioning, path planning, and driving control. They determine the positioning accuracy and driving safety of the autonomous driving system, especially in complex scenarios such as highway autonomous driving and urban road autonomous driving. Accurate lane line perception can reduce the risk of vehicles deviating from the driving corridor and avoid collisions. In related technologies, autonomous driving solutions generally assume that "lane lines are always clearly identifiable" when designing lane line perception functions. They use only a single visual sensor to acquire lane line information and use a "confidence threshold filtering" mechanism to judge the lane line information collected by the sensor. If the information is below the threshold, it is determined that "no usable lane line"; if it is above the threshold, subsequent path planning and driving control are completed based on the information.

[0003] However, in real-world road traffic scenarios where autonomous vehicles are driving, lane markings are subject to long-term effects from vehicle traffic, wind and rain erosion, and changes in lighting conditions and usage environment. These effects can lead to aging phenomena such as performance degradation, shape damage, and color fading. Furthermore, road surfaces may be subject to complex conditions such as water accumulation, snow accumulation, debris obstruction, and uneven road texture. The aforementioned autonomous driving technology solutions are prone to misjudging lane markings with geometric reference value as invalid, or judging incomplete or offset lane markings as valid, causing vehicles to deviate from the driving corridor and increasing the risk of collision. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, vehicle, device, and medium for assessing the risks of autonomous driving, in order to improve the safety of autonomous driving.

[0005] Firstly, this application provides a method for assessing the risks of autonomous driving, including: The vehicle uses visual sensors and lidar to collect visual images and radar reflectivity maps of the target road surface. After time synchronization and spatial calibration of the visual image and the radar reflectivity map, a convolutional neural network is used to extract visual features of the lane line area from the visual image and identify target reflectivity areas in the radar reflectivity map that are higher than the reflectivity threshold, extracting the reflectivity distribution features of the target reflectivity areas; the visual features and the reflectivity distribution features are fused according to the time synchronization and spatial calibration results to obtain a comprehensive lane line feature set; the reflectivity distribution features include at least average reflectivity and reflectivity variance; the visual features include at least edge features, color features, and geometric shape features. The lane line perception results are generated based on the comprehensive feature set of the lane lines; the lane line perception results include lane line modeling information, lane line aging degree, and road surface complexity of the area where the lane line is located; The target risk level corresponding to the lane line modeling information is obtained by matching the lane line aging degree and the road surface complexity from the risk level relationship; the risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree range and road surface complexity range.

[0006] The autonomous driving risk assessment method provided in this application, by acquiring visual images and radar reflectivity maps of the target road surface, can further analyze the aging degree of lane lines and the complexity of the road surface in the area based on multi-source heterogeneous data. It incorporates assessment dimensions such as lane line aging state and road surface state into the lane line perception results. Compared with the coarse perception using the "confidence threshold filtering" mechanism, it can improve the accuracy of lane line state recognition, enabling the autonomous driving system to stably output accurate lane line information and risk levels even in aging lane lines and complex road surface scenarios, reducing decision-making errors, improving the safety of autonomous driving, and can be adapted to different levels of autonomous driving vehicles. It is compatible with various road surface scenarios such as urban roads, highways, and rural roads, and is especially suitable for complex scenarios such as old roads and construction sections.

[0007] According to one embodiment of this application, the edge features include edge continuity and edge gradient, the color features include the color contrast between the lane lines and the road surface, and the geometric features include lane line width and curvature.

[0008] In this embodiment, by employing a convolutional neural network to extract visual features such as edge features, color features, and geometric features of the area where the lane line is located from the visual image, a visual feature representation of the lane line can be constructed from different dimensions, thereby improving the completeness of the visual features.

[0009] According to one embodiment of this application, generating lane line perception results based on the lane line integrated feature set includes: The lane line integrated feature set is input into a preset perception model to obtain the lane line perception result output by the perception model. The perception model is a multi-task neural network model based on multi-task joint learning, and the multi-task includes lane line modeling task, lane line aging degree modeling task, and road surface complexity modeling task.

[0010] In this embodiment, by inputting the comprehensive feature set of lane lines into a multi-task neural network model based on multi-task joint learning, lane line modeling, lane line aging degree modeling, and road surface complexity modeling tasks are completed. This enables multi-task feature sharing and joint reasoning, reduces the computational load of repeated feature extraction, improves perception efficiency, meets the real-time requirements of autonomous driving systems, and enables mutual constraints and enhancements among tasks, thereby improving the accuracy of lane line modeling information, aging degree, and road surface complexity.

[0011] According to one embodiment of this application, the perception model is trained in the following manner: Obtain the modeling logic and output targets corresponding to each task; The network structure of the perception model is constructed, which includes a shared feature extraction layer and output layers corresponding to each task. The shared feature extraction layer is used to extract features of the comprehensive feature set of sample lane lines, and the output layers corresponding to each task correspond to the modeling logic of each task. The comprehensive feature set of sample lane lines is obtained based on sample visual images and sample radar reflectivity maps. The perception model is trained using the sample lane line comprehensive feature set as training input and the output target corresponding to each task as label, based on the joint loss function; wherein, the joint loss function includes lane line modeling loss term, lane line aging degree modeling loss term and road surface complexity modeling loss term.

[0012] In this embodiment, by constructing a multi-task neural network structure that includes a shared feature extraction layer and output layers for each task, lane line modeling, lane line aging degree modeling, and road surface complexity modeling tasks can share the underlying feature extraction process, reducing the work of repeatedly extracting the same features, thereby reducing the number of model parameters and computational load. Furthermore, by using a joint loss function that integrates the loss terms of each task for training, mutual constraints and collaborative optimization among multiple tasks can be achieved during the training process. This allows the model to learn the geometric and semantic features of lane lines while referring to the feature representations of aging degree and road surface complexity, thereby enhancing the model's joint prediction ability for multiple tasks and improving the accuracy of lane line perception.

[0013] According to one embodiment of this application, the output target of the lane line modeling task includes the geometric coordinates and semantic category of the lane line; the semantic category includes at least one of solid line, dashed line, and guide line. The modeling logic for the lane line modeling task includes: The first evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the first evaluation index includes at least one of lane line edge continuity, color contrast, width, curvature, average reflectance, and reflectance variance. The geometric coordinates and semantic category of the lane lines are determined based on the first evaluation metric.

[0014] In this embodiment, by fusing visual features and radar reflectivity distribution features, the first evaluation indicators such as edge continuity, color contrast, width, curvature, average reflectivity, and reflectivity variance are calculated. This enables the analysis of the physical properties and reflectivity characteristics of lane lines from multiple dimensions of vision and radar. Based on the multi-dimensional evaluation indicators, the geometric coordinates of lane lines and semantic categories such as solid lines, dashed lines, and guide lines are determined, which can improve the accuracy of lane line geometric positioning and the accuracy of semantic category recognition.

[0015] According to one embodiment of this application, the output objective of the lane line aging degree modeling task includes: a quantitative value characterizing the lane line aging degree; The modeling logic for the lane line aging modeling task includes: A second evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the second evaluation index includes at least one of reflectance attenuation coefficient, edge integrity coefficient, and material uniformity coefficient. The quantitative value representing the degree of lane line aging is calculated based on the second evaluation index.

[0016] In this embodiment, by fusing visual features and radar reflectivity distribution features, second evaluation indicators such as reflectivity attenuation coefficient, edge integrity coefficient, and material uniformity coefficient are calculated. This can quantitatively characterize the aging degree of lane lines from multiple dimensions such as reflectivity, edge state, and material consistency, thereby improving the accuracy of lane line aging degree quantification.

[0017] According to one embodiment of this application, the output objective of the road surface complexity modeling task includes: a quantified value characterizing the road surface complexity of the area where the lane line is located; The modeling logic for the road surface complexity modeling task includes: A third evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the third evaluation index includes at least one of the following: density of interfering objects within the target range of the lane line, unevenness of road surface texture, and semantic interference degree of the lane line. The quantitative value representing the road surface complexity is calculated based on the third evaluation index.

[0018] In this embodiment, by fusing visual features and radar reflectivity distribution features, a third evaluation index is calculated, which includes interference density, road surface texture non-uniformity, and lane line semantic interference. This index can quantitatively characterize road surface complexity from multiple dimensions such as target interference, texture changes, and semantic confusion, thereby improving the accuracy of road surface complexity quantification.

[0019] Secondly, this application provides an autonomous driving risk assessment device, the device comprising: The acquisition module is used to collect visual images and radar reflectivity maps of the target road surface through the visual sensors and lidar mounted on the autonomous vehicle. An extraction module is used to perform time synchronization and spatial calibration on the visual image and the radar reflectivity map, and then use a convolutional neural network to extract visual features of the lane line area from the visual image, identify target reflectivity areas in the radar reflectivity map that are higher than the reflectivity threshold, and extract the reflectivity distribution features of the target reflectivity areas; based on the time synchronization and spatial calibration results, the visual features and the reflectivity distribution features are fused to obtain a comprehensive feature set for the lane lines; the reflectivity distribution features include at least average reflectivity and reflectivity variance; the visual features include at least edge features, color features, and geometric morphology features. The generation module is used to generate lane line perception results based on the lane line comprehensive feature set; the lane line perception results include lane line modeling information, lane line aging degree, and road surface complexity of the area where the lane line is located; The evaluation module is used to match the lane line aging degree and the road surface complexity from the risk level relationship to obtain the target risk level corresponding to the lane line modeling information; the risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree range and road surface complexity range.

[0020] The autonomous driving risk assessment device provided in this application, by acquiring visual images and radar reflectivity maps of the target road surface, can further analyze the aging degree of lane lines and the complexity of the road surface in the area based on multi-source heterogeneous data. It incorporates assessment dimensions such as lane line aging state and road surface state into the lane line perception results. Compared with the coarse perception using the "confidence threshold filtering" mechanism, it can improve the accuracy of lane line state recognition, enabling the autonomous driving system to stably output accurate lane line information and risk levels even in aging lane lines and complex road surface scenarios, reducing decision-making errors, improving the safety of autonomous driving, and can be adapted to different levels of autonomous driving vehicles. It is compatible with various road surface scenarios such as urban roads, highways, and rural roads, and is especially suitable for complex scenarios such as old roads and construction sections.

[0021] Thirdly, this application provides a vehicle including a vision sensor, a lidar, and a controller; The visual sensor is used to acquire visual images of the target road surface; The lidar is used to collect radar reflectivity maps of the target road surface; The controller is used to perform the aforementioned method for assessing autonomous driving risks.

[0022] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method for assessing autonomous driving risks.

[0023] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for assessing autonomous driving risks.

[0024] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for assessing autonomous driving risks.

[0025] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects: The autonomous driving risk assessment method provided in this application, by acquiring visual images and radar reflectivity maps of the target road surface, can further analyze the aging degree of lane lines and the complexity of the road surface in the area based on multi-source heterogeneous data. It incorporates assessment dimensions such as lane line aging state and road surface state into the lane line perception results. Compared with the coarse perception using the "confidence threshold filtering" mechanism, it can improve the accuracy of lane line state recognition, enabling the autonomous driving system to stably output accurate lane line information and risk levels even in aging lane lines and complex road surface scenarios, reducing decision-making errors, improving the safety of autonomous driving, and can be adapted to different levels of autonomous driving vehicles. It is compatible with various road surface scenarios such as urban roads, highways, and rural roads, and is especially suitable for complex scenarios such as old roads and construction sections.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the method for assessing autonomous driving risks provided in an embodiment of this application; Figure 2 This is a schematic diagram of the modeling logic for the lane line aging degree modeling task provided in the embodiments of this application; Figure 3 This is a schematic diagram of the modeling logic for the road surface complexity modeling task provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the autonomous driving risk assessment device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The following description, in conjunction with the accompanying drawings, details the autonomous driving risk assessment method, apparatus, vehicle, equipment, and medium provided in this application through specific embodiments and application scenarios.

[0032] Among them, the risk assessment method for autonomous driving can be applied to the terminal, specifically by the hardware or software in the terminal.

[0033] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0034] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0035] The autonomous driving risk assessment method provided in this application can be executed by an electronic device or a functional module or entity within an electronic device capable of implementing the method. The electronic device mentioned in this application includes servers, in-vehicle computers, etc. The following description uses an electronic device as an example to illustrate the autonomous driving risk assessment method provided in this application.

[0036] like Figure 1 As shown, the method for assessing the risks of autonomous driving includes steps 110, 120, 130, and 140.

[0037] Step 110: Collect visual images and radar reflectivity maps of the target road surface using the vehicle's onboard visual sensors and lidar.

[0038] In this embodiment of the application, the target road surface is the road surface area on the vehicle's driving path that is to be subject to risk assessment, such as the main road, secondary road, and branch road of urban roads, the driving lane and emergency lane of highways, the single-vehicle road section or temporary construction access road of rural roads, etc. The scope of the target road surface can be determined according to the current driving speed of the vehicle, the detection range of the sensing sensor, etc. For example, the scope of the target road surface can be the driving road surface within 50 meters, 100 meters, 200 meters in front of the vehicle, etc., or it can include the road surface area of ​​the vehicle's current lane and adjacent lanes.

[0039] Visual images can be acquired using vehicle-mounted vision sensors, such as monocular cameras, binocular cameras, and surround-view cameras. These sensors can capture visual information about the target road surface, including color, texture, and contour, to obtain the shape and position of lane lines and the basic appearance features of the road surface. Radar reflectivity maps can be generated using vehicle-mounted radar sensors, such as millimeter-wave radar and lidar. These sensors emit radar signals towards the target road surface and receive reflected signals. Based on the intensity distribution of the reflected signals, a radar reflectivity map is generated. This map reflects information such as the material properties, surface roughness, and obstacle distribution in different areas of the target road surface.

[0040] In some embodiments, the acquisition actions of the visual sensor and radar can be triggered synchronously, enabling the visual image and radar reflectivity map to be acquired simultaneously within the same spatiotemporal window, and performing time synchronization and spatial calibration. The time synchronization accuracy can be less than or equal to 50 ms, so that the visual image and radar reflectivity map correspond to the same vehicle pose; spatial calibration can be performed by obtaining the extrinsic parameter matrices of the visual sensor and radar through joint calibration, and mapping the visual image and radar reflectivity map to the same road plane coordinate system.

[0041] In some embodiments, if the vehicle is equipped with multiple sets of visual sensors or radar, the information collected by the multiple sensors can be integrated by a data fusion unit to obtain visual images and radar reflectivity maps with wider coverage and more comprehensive information.

[0042] Step 120: After time synchronization and spatial calibration of the visual image and radar reflectivity map, a convolutional neural network is used to extract the visual features of the lane line area from the visual image and identify the target reflectivity area above the reflectivity threshold in the radar reflectivity map, and extract the reflectivity distribution features of the target reflectivity area; based on the time synchronization and spatial calibration results, the visual features and reflectivity distribution features are fused to obtain the comprehensive feature set of the lane line; the reflectivity distribution features include at least the average reflectivity and reflectivity variance; the visual features include at least edge features, color features and geometric features.

[0043] In this embodiment, because the visual sensor and radar sensor have different acquisition frame rates and triggering mechanisms, directly extracting features from visual images and radar reflectivity maps acquired at different timestamps can easily lead to a mismatch between the road surface state at the corresponding moment in the visual image and the road surface state at the corresponding moment in the radar reflectivity map. Therefore, time synchronization and spatial calibration algorithms can be used to synchronize the visual image and radar reflectivity map in time and space. For example, a high-precision synchronization clock can be allocated to the visual sensor and radar sensor through the unified clock module of the vehicle domain controller, so that the timetamp deviation between the visual sensor and radar sensor is controlled within milliseconds. If the visual sensor and radar sensor do not trigger data acquisition synchronously, time alignment can be achieved by using methods such as timestamp interpolation and nearest frame matching to select the visual image and radar reflectivity map with the closest timestamps.

[0044] Visual sensors (such as cameras) collect two-dimensional data based on the image pixel coordinate system, while radar sensors (such as lidar and millimeter-wave radar) collect reflectivity data based on the radar's own coordinate system. The installation positions, pitch angles, and azimuth angles of visual and radar sensors may differ, making it difficult to directly correlate extracted features spatially. Therefore, sensor calibration can be used to obtain the intrinsic parameters (such as focal length and principal point) and extrinsic parameters (such as rotation matrix and translation vector) of the visual sensor, as well as the coordinate transformation relationship between the radar sensor and the visual sensor. Then, the radar reflectivity map can be transformed from the radar coordinate system to the pixel coordinate system of the visual image, or both the visual image and the radar reflectivity map can be transformed to the vehicle's own coordinate system. This achieves spatial calibration between the visual image and the radar reflectivity map, ensuring that the pixel positions of lane lines in the visual image correspond spatially to the reflectivity regions in the radar reflectivity map, reducing feature extraction errors caused by spatial misalignment.

[0045] After completing time synchronization and spatial calibration, feature extraction is performed on the visual image and radar reflectivity map respectively to obtain visual features and reflectivity distribution features.

[0046] By synchronizing and spatially calibrating visual images and radar reflectivity maps, and then performing joint feature extraction on them, the deviations between visual images and radar reflectivity maps in the time and spatial dimensions can be reduced, improving the spatiotemporal consistency of multi-source heterogeneous data, and thus improving the accuracy of lane line integrated feature set acquisition.

[0047] In this embodiment, joint feature extraction is achieved by associating, fusing, and extracting feature information from visual images and radar reflectivity maps to construct a comprehensive lane line feature set containing multi-dimensional information.

[0048] The comprehensive feature set of lane markings includes at least the visual features and reflectance distribution features of the area where the lane markings are located. Visual features are appearance features related to lane markings extracted from visual images, and can include color features (such as color saturation, brightness, and color contrast between white and yellow lane markings and the road surface), geometric features (such as straightness, curvature, width, and discontinuity distribution), texture features (such as wear texture and dirt coverage texture of the lane marking surface), edge features (such as edge continuity and edge gradient), and positional features (such as lateral offset and longitudinal distance of the lane markings relative to vehicles). Reflectance distribution features are radar signal reflection characteristics of the lane marking area extracted from radar reflectance maps, and can include the average reflectance of the lane marking area, reflectance gradient changes, reflectance variance, reflectance peak distribution, and reflectance difference with surrounding road surface areas. Reflectance distribution features can reflect the degree of material wear of the lane markings, the presence of surface debris, and differences in road surface materials. For example, severely worn lane markings have lower reflectance than intact lane markings, and lane markings covered with oil or snow will show abnormal fluctuations in reflectance.

[0049] In some embodiments, deep learning models, such as convolutional neural networks and multimodal fusion networks, can be used for joint feature extraction. For example, a visual feature extraction branch can perform convolution and pooling operations on the visual image to extract the visual feature vector of the lane lines, while a radar feature extraction branch can encode the radar reflectivity map to extract the reflectivity distribution feature vector of the lane lines. Then, a feature fusion module, such as a stitching layer or an attention mechanism fusion layer, can combine the temporal synchronization and spatial calibration results to weightedly fuse the visual feature vector and the reflectivity distribution feature vector to generate a comprehensive lane line feature set. Alternatively, traditional feature extraction methods can be combined with data fusion algorithms. For example, edge detection and Hough transform can be used to extract the visual features of lane lines from the visual image, threshold segmentation and region growing can be used to extract the reflectivity distribution features from the radar reflectivity map, and then temporal synchronization and spatial calibration results can be combined with a feature matching algorithm to associate and integrate the visual features and the reflectivity distribution features to form a comprehensive lane line feature set.

[0050] In some embodiments, the convolutional neural network can employ a lightweight backbone network, such as MobileNetV2 or EfficientNet-Lite, which extracts low-dimensional basic features and high-dimensional semantic features of visual images step by step through multi-layer convolution, pooling, and non-linear activation operations.

[0051] The extracted visual features can include edge features, which include edge continuity and edge gradient. Edge features characterize the clarity and integrity of lane line boundaries. A larger and more concentrated edge gradient amplitude indicates a clearer lane line edge; a smaller and more dispersed edge gradient amplitude indicates that the lane line edge is blurred due to wear or dirt. Edge continuity features represent the degree of edge continuity. If the edge gradient features are continuously distributed without obvious breaks, it indicates high edge continuity; if the edge gradient features are discontinuous or missing (e.g., partial peeling of lane lines or obstruction by debris), it indicates low edge continuity.

[0052] The extracted visual features can also include color features, such as the color contrast between lane lines and the road surface. Color features are the visual characteristics that distinguish lane lines from the background road surface. The higher the color contrast between lane lines and the road surface, the more vibrant the lane line color and the more obvious the difference from the road surface, making it easier to identify. Conversely, the lower the color contrast between lane lines and the road surface, the more difficult it is to identify, as the lane lines may have faded or aged, resulting in a color similar to the road surface.

[0053] The extracted visual features can also include geometric features, such as lane width and curvature. Geometric features characterize the spatial structure and physical dimensions of lane lines. Lane width reflects whether the physical dimensions of the lane line conform to specifications and whether there are problems such as narrowing or missing widths due to local wear. For example, the standard lane line width is 15-20cm. If the extracted lane line width is less than 15-20cm, it indicates that there is local peeling or wear on the lane line. Lane curvature indicates the degree of curvature of the lane line. The larger the radius of curvature, the closer the lane line is to a straight line. The smaller the radius of curvature, the higher the degree of curvature of the lane line (such as lane lines at curves or ramps).

[0054] By employing convolutional neural networks to extract visual features such as edge features, color features, and geometric features of the area where lane lines are located from visual images, it is possible to construct visual feature representations of lane lines from different dimensions and improve the completeness of visual features.

[0055] In some embodiments, radar reflectivity maps can reflect physical information such as material properties, surface roughness, and attached debris in different areas of the target road surface. Lane lines are usually made of highly reflective materials, and the reflectivity of lane lines is significantly higher than that of the surrounding ordinary road surface. Therefore, the area where the lane line is located can be located by identifying the target reflectivity area with a reflectivity higher than the reflectivity threshold.

[0056] For example, high reflectivity regions in radar reflectivity maps can be identified using threshold segmentation, region growing, or machine learning-based semantic segmentation models. Threshold segmentation uses a preset reflectivity threshold to segment regions with significantly higher reflectivity than the background, i.e., target reflectivity regions. Region growing starts from a seed point (such as a local maximum reflectivity point) and gradually merges adjacent pixels according to reflectivity similarity criteria to form connected high reflectivity regions, i.e., target reflectivity regions. Machine learning-based semantic segmentation models can segment high reflectivity regions corresponding to lane lines from radar reflectivity maps through end-to-end learning. Of course, other methods can also be used to identify target reflectivity regions in radar reflectivity maps, and this application does not limit this approach.

[0057] The reflectivity distribution characteristics of the target reflectivity region can be extracted. These reflectivity distribution characteristics include at least the average reflectivity and the reflectivity variance. The average reflectivity characterizes the overall reflectivity intensity of the lane lines, while the reflectivity variance characterizes the uniformity of the lane line reflectivity.

[0058] By identifying high-reflectivity areas corresponding to lane lines under radar perception, and then extracting the average reflectivity and reflectivity variance of these areas, low-reflectivity interference information such as road background can be filtered out, highlighting the radar reflection characteristics of lane lines. Characterizing lane lines from dimensions such as average reflectivity and reflectivity variance can improve the accuracy of subsequent lane line perception results.

[0059] Step 130: Generate lane line perception results based on the comprehensive feature set of lane lines; the lane line perception results include lane line modeling information, lane line aging degree and road surface complexity of the area where the lane line is located.

[0060] In this embodiment, the lane line perception result is multi-dimensional perception information obtained through feature parsing, model calculation, and state evaluation based on a comprehensive lane line feature set. This includes not only lane line modeling information but also lane line aging degree and road surface complexity. The lane line modeling information is the fundamental output of lane line perception, including geometric modeling parameters (such as the starting and ending coordinates, radius of curvature, and tilt angle) and semantic categories (such as solid lines, dashed lines, double yellow lines, and guide lines). Lane line modeling information can be obtained by fitting, clustering, and classifying visual features and reflectance distribution features from the comprehensive lane line feature set. For example, curve fitting can be performed based on the shape and position features of the lane lines to generate a geometric model, and lane line categories can be classified and labeled based on reflectance distribution and color features.

[0061] Lane line aging degree is a quantitative assessment of the physical wear, fading, peeling, and other aging conditions of lane lines. It can be evaluated based on visual features (such as the degree of brightness decay and texture wear) and reflectivity distribution features (such as the magnitude of reflectivity reduction and reflectivity uniformity in the lane line area) from a comprehensive feature set of lane lines. For example, the comprehensive feature set of lane lines can be input into a trained lane line aging degree assessment model, and the model's inference and calculation can output the lane line aging degree. Lane line aging degree can be represented by a quantitative value, such as 0-1, where 0 represents no aging and 1 represents complete aging. Lane line aging degree can also be divided into multiple levels, such as intact, slightly aged, moderately aged, and severely aged.

[0062] The road surface complexity of the area where lane markings are located is a quantitative assessment of the complexity of the road environment surrounding the lane markings. It can be comprehensively determined by combining visual features (such as cracks, potholes, construction markings, and debris distribution) and reflectivity distribution features (such as reflectivity differences in different areas of the road and peak reflectivity of obstacles) from the comprehensive feature set of the lane markings. For example, the comprehensive feature set of lane markings can be input into a trained road surface complexity assessment model. Through the model's inference and calculation, the road surface complexity of the area where the lane markings are located can be output. Road surface complexity can be represented by a quantitative value, such as a range of 0-1, where 0 indicates a simple road surface and 1 indicates an extremely complex road surface. Road surface complexity can also be divided into multiple levels, such as low, medium, and high. For example, when the road surface is smooth, without obstacles, and without construction areas, it is judged as low complexity; when the road surface has a few cracks, scattered debris, or slight construction traces, it is judged as medium complexity; when the road surface has large areas of potholes, dense construction areas, multiple types of obstacles, or complex traffic markings, it is judged as high complexity.

[0063] Step 140: Match the lane line aging degree and road surface complexity from the risk level relationship to obtain the target risk level corresponding to the lane line modeling information; the risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree range and road surface complexity range.

[0064] Lane line modeling information is the input for autonomous driving systems to perform lane keeping, lane changing decisions, and path planning. The accuracy of lane line modeling information affects the decision-making accuracy of the autonomous driving system. This application embodiment considers that the aging degree of lane lines and road surface complexity are important factors affecting the accuracy of lane line modeling information. When the aging degree of lane lines is high, the visual features and reflectivity distribution characteristics of the lane lines may be weakened or distorted, potentially leading to problems such as positional deviation and misclassification in the lane line modeling information. When the road surface complexity is high, obstacles, construction marks, and other interference factors on the road surface can affect the extraction and recognition of lane line features, also reducing the accuracy of lane line modeling information, thereby causing decision-making errors in the autonomous driving system and increasing driving risks. Therefore, this application embodiment, after outputting lane line modeling information, further assesses the autonomous driving risk by combining the aging degree of lane lines and road surface complexity.

[0065] In some embodiments, lane line aging and road surface complexity can be comprehensively analyzed using preset risk assessment rules or machine learning algorithms to output corresponding autonomous driving risks. Autonomous driving risks can be represented by risk levels, such as low, medium, and high risks. When lane line aging is intact or slightly aged, and road surface complexity is low, the accuracy of lane line modeling information is high, resulting in a low-risk assessment. The autonomous driving system can then normally execute lane keeping and lane changing decisions based on the lane line modeling information. When lane line aging is moderate and road surface complexity is medium, the lane line modeling information has some deviation, resulting in a medium-risk assessment. The autonomous driving system can reduce decision confidence, increase verification of lane line modeling information, or combine information from other perception sensors (such as high-precision maps and inertial measurement units) for assisted decision-making. When lane line aging is severe and road surface complexity is high, the accuracy of lane line modeling information is poor, resulting in a high-risk assessment. The autonomous driving system can trigger a risk warning, prompting the driver to take over the vehicle or adjust driving strategies (such as reducing speed, maintaining the current lane, and avoiding lane changes) to reduce risk. Of course, the risks of autonomous driving can also be represented by quantitative values, ranging from 0 to 1, where 0 represents no risk and 1 represents high risk.

[0066] In some embodiments, the assessment results of autonomous driving risks can be dynamically corrected by combining factors such as vehicle speed, autonomous driving level, and surrounding traffic conditions. For example, in high-speed scenarios, even if the assessment result is medium risk, the vehicle's reaction time is short due to the high speed, so the medium risk can be upgraded to high risk.

[0067] In this embodiment, the aging degree of lane lines and the complexity of the road surface can be quantified values ​​within the range of [0,1]. The risks of autonomous driving can be pre-classified into multiple risk levels, such as low risk, medium risk, high risk, and extremely high risk. The relationship between risk levels is shown in the table below:

[0068] After obtaining the lane line aging level and road surface complexity, matching can be performed according to the rules in the table above to output the target risk level. The lane line modeling information and target risk level can be transmitted to the downstream autonomous driving decision-making and planning module to provide a basis for operations such as path adjustment, speed control, and warning prompts.

[0069] By classifying autonomous driving risks into multiple different risk levels and determining the corresponding risk level based on the degree of lane line aging and road surface complexity, early warnings can be provided to downstream decision-making modules. This enables the autonomous driving system to adjust vehicle speed and optimize routes in advance, effectively avoiding accidents such as lane departure and collisions, and improving the safety of autonomous driving.

[0070] The autonomous driving risk assessment method provided in this application, by acquiring visual images and radar reflectivity maps of the target road surface, can further analyze the aging degree of lane lines and the complexity of the road surface in the area based on multi-source heterogeneous data. It incorporates assessment dimensions such as lane line aging state and road surface state into the lane line perception results. Compared with the coarse perception using the "confidence threshold filtering" mechanism, it can improve the accuracy of lane line state recognition, enabling the autonomous driving system to stably output accurate lane line information and risk levels even in aging lane lines and complex road surface scenarios, reducing decision-making errors, improving the safety of autonomous driving, and can be adapted to different levels of autonomous driving vehicles. It is compatible with various road surface scenarios such as urban roads, highways, and rural roads, and is especially suitable for complex scenarios such as old roads and construction sections.

[0071] In some embodiments, generating lane line perception results based on a comprehensive lane line feature set includes: The lane line integrated feature set is input into the preset perception model to obtain the lane line perception result output by the perception model. The perception model is a multi-task neural network model based on multi-task joint learning. The multi-tasks include lane line modeling, lane line aging modeling, and road surface complexity modeling.

[0072] In this embodiment, a multi-task neural network model based on multi-task joint learning is used as the perception model. This model enables end-to-end joint reasoning for lane line modeling, lane line aging degree modeling, and road surface complexity modeling tasks. It fully utilizes the feature sharing and complementary relationships between tasks, which not only improves the reasoning accuracy of each task but also reduces the number of model parameters and computational overhead, thereby improving the real-time performance and accuracy of the autonomous driving system's perception.

[0073] The perception model employs a multi-task joint learning design, using a shared feature backbone + multi-task head architecture, including a shared feature extraction layer and output layers corresponding to each task. The shared feature extraction layer is responsible for feature encoding and fusion of the input lane line comprehensive feature set. It can use a lightweight convolutional neural network (such as MobileNetV3, EfficientNet-B0) or a Transformer encoder as the backbone network. After receiving the lane line comprehensive feature set, it gradually extracts low-dimensional basic features and high-dimensional semantic features through multi-layer convolution, pooling, non-linear activation, or self-attention mechanisms to generate a shared feature map.

[0074] The output layers for each task implement inference based on shared feature maps. Specifically, the output layer for the lane line modeling task can employ regression or classification structures to output lane line modeling information; the output layer for the lane line aging degree modeling task can employ classification or regression structures to output the quantitative results of lane line aging; and the output layer for the road surface complexity modeling task can also employ classification or regression structures to output the quantitative results of road surface complexity in the area where the lane line is located. The output layers for each task share the output results of the feature extraction layer and perform task inference and computation through independent fully connected layers, convolutional layers, or decoders, respectively outputting lane line modeling information, lane line aging degree, and road surface complexity of the area where the lane line is located, forming a lane line perception result containing multi-dimensional information.

[0075] In this embodiment, by inputting the comprehensive feature set of lane lines into a multi-task neural network model based on multi-task joint learning, lane line modeling, lane line aging degree modeling, and road surface complexity modeling tasks are completed. This enables multi-task feature sharing and joint reasoning, reduces the computational load of repeated feature extraction, improves perception efficiency, meets the real-time requirements of autonomous driving systems, and enables mutual constraints and enhancements among tasks, thereby improving the accuracy of lane line modeling information, aging degree, and road surface complexity.

[0076] In some embodiments, the perception model is trained in the following manner: Obtain the modeling logic and output targets corresponding to each task; The network structure of the perception model is constructed, which includes a shared feature extraction layer and output layers corresponding to each task. The shared feature extraction layer is used to extract features of the comprehensive feature set of sample lane lines, and the output layers corresponding to each task correspond to the modeling logic of each task. The comprehensive feature set of sample lane lines is obtained based on sample visual images and sample radar reflectivity maps. The perception model is trained using a comprehensive feature set of sample lane lines as the training input and the output target corresponding to each task as the label. The joint loss function includes lane line modeling loss term, lane line aging degree modeling loss term, and road surface complexity modeling loss term.

[0077] In some embodiments, the output target of the lane line modeling task includes the geometric coordinates and semantic category of the lane line; the semantic category includes at least one of solid line, dashed line, and guide line. The modeling logic for the lane line modeling task includes: The first evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the first evaluation index includes at least one of lane line edge continuity, color contrast, width, curvature, average reflectance, and reflectance variance. The geometric coordinates and semantic category of lane lines are determined based on the first evaluation metric.

[0078] In this embodiment, the first evaluation index can characterize the appearance, physical, and geometric attributes of the lane lines. It can be obtained by calculating visual features and reflectance distribution features. The weighted fusion and logical reasoning of each first evaluation index outputs the spatial geometric coordinates of the lane lines. Furthermore, the semantic category of the lane lines is matched based on features. For example, edge continuity is used to determine whether the lane lines are continuously distributed solid lines or discontinuously distributed dashed lines, and geometric shape and color features are used to determine whether they are guide lines, etc.

[0079] In this embodiment, by fusing visual features and radar reflectivity distribution features, the first evaluation indicators such as edge continuity, color contrast, width, curvature, average reflectivity, and reflectivity variance are calculated. This enables the analysis of the physical properties and reflectivity characteristics of lane lines from multiple dimensions of vision and radar. Based on the multi-dimensional evaluation indicators, the geometric coordinates of lane lines and semantic categories such as solid lines, dashed lines, and guide lines are determined, which can improve the accuracy of lane line geometric positioning and the accuracy of semantic category recognition.

[0080] In some embodiments, the output objective of the lane line aging modeling task includes: a quantitative value characterizing the aging degree of the lane lines; The modeling logic for the lane line aging modeling task includes: The second evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the second evaluation index includes at least one of the reflectance attenuation coefficient, edge integrity coefficient, and material uniformity coefficient. The quantitative value representing the degree of lane line aging is calculated based on the second evaluation index.

[0081] In this embodiment, the quantitative value characterizing the aging degree of lane lines can be represented by a continuous value between 0 and 1. The closer the value is to 0, the better the condition of the lane lines; the closer the value is to 1, the more severe the aging degree of the lane lines.

[0082] like Figure 2 As shown, the second evaluation index may include reflectivity attenuation coefficient, edge integrity coefficient, and material uniformity coefficient. The reflectivity attenuation coefficient is calculated based on radar reflectivity distribution characteristics. For example, the reflectivity benchmark value of a newly painted standard lane line can be obtained in advance. The average reflectivity of the target lane line area can be extracted from the comprehensive feature set of lane lines. The difference between the benchmark value and the average reflectivity can be calculated and then divided by the benchmark value to obtain the attenuation ratio. This attenuation ratio is then normalized to the [0,1] interval to obtain the reflectivity attenuation coefficient. If the average reflectivity is greater than or equal to the benchmark value, the reflectivity attenuation coefficient is equal to 0, indicating no attenuation. The closer the reflectivity attenuation coefficient is to 1, the more severe the loss of the highly reflective material of the lane line, the lower the radar perception reliability, and the higher the degree of aging.

[0083] The edge integrity coefficient can be calculated based on the continuity features of lane line edges in a visual image. It is obtained by statistically analyzing the proportion of edge damage length to the total length, and then normalizing this proportion to the [0,1] interval to obtain the edge integrity coefficient. The closer the edge integrity coefficient is to 1, the more severe the damage and discontinuity of the lane line edges, the more likely visual perception will miss or misdetect them, and the higher the degree of aging.

[0084] The material uniformity coefficient is calculated based on the reflectance variance and the color uniformity of the visual image, and is used to evaluate the wear uniformity of lane line materials. For example, the reflectance variance of the lane line area can be extracted from the radar reflectance distribution characteristics, and the color variance of the lane line area can be extracted from the visual characteristics. The reflectance variance and color variance are normalized respectively, and then a weighted sum is performed so that the value of the weighted sum is in the range of [0,1] to obtain the material uniformity coefficient. The closer the material uniformity coefficient is to 1, the more uneven the wear, fading, and contamination of the lane line surface, and the greater the local aging differences.

[0085] A weighted summation algorithm can be used, setting the weights of the reflectivity attenuation coefficient, edge integrity coefficient, and material uniformity coefficient. For example, the weight of the reflectivity attenuation coefficient is 0.4, the weight of the edge integrity coefficient is 0.3, and the weight of the material uniformity coefficient is 0.3. The weighted summation is used to calculate the quantitative value of the lane line aging degree, with a value range of 0-1, where 0 represents no aging and 1 represents complete aging.

[0086] In this embodiment, by fusing visual features and radar reflectivity distribution features, second evaluation indicators such as reflectivity attenuation coefficient, edge integrity coefficient, and material uniformity coefficient are calculated. This can quantitatively characterize the aging degree of lane lines from multiple dimensions such as reflectivity, edge state, and material consistency, thereby improving the accuracy of lane line aging degree quantification.

[0087] In some embodiments, the output objective of the road surface complexity modeling task includes: a quantified value representing the road surface complexity of the area where the lane line is located; The modeling logic for the road surface complexity modeling task includes: The third evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the third evaluation index includes at least one of the following: density of interference within the target range of the lane line, unevenness of road texture, and semantic interference degree of the lane line. The quantitative value representing the complexity of the road surface is calculated based on the third evaluation index.

[0088] In this embodiment, the quantitative value characterizing the road surface complexity of the area where the lane line is located can be represented by a continuous value between 0 and 1. The closer the value is to 0, the simpler the road surface environment is; the closer the value is to 1, the more complex the road surface environment is.

[0089] like Figure 3 As shown, the third evaluation index can include the density of interfering objects within the target range of the lane line, the unevenness of road surface texture, and the semantic interference degree of the lane line. Among these, the density of interfering objects within the target range of the lane line reflects the quantity and spatial distribution density of interfering objects such as stones, oil stains, potholes, marking residue, and construction waste within the target range surrounding the lane line. The target range can be pre-defined, such as a rectangular area extending 0.5m to the left and right and 5m forward and backward from the lane line as the center. Then, features within the rectangular area are extracted from visual features to calculate the number of visual interfering objects, and features within the rectangular area are extracted from radar reflectivity distribution features to calculate the number of radar interfering objects. Through spatiotemporal calibration and feature fusion, the visual and radar detection results are deduplicated and fused to obtain the total number of interfering objects within the target range, and then the interfering object density is calculated. The interfering object density can be normalized to the [0,1] interval. The closer the interfering object density is to 1, the denser the interfering objects within the target range, the more severe the occlusion, segmentation, and confusion of the lane line, and the higher the road surface complexity.

[0090] Road surface texture inhomogeneity can be calculated from the texture variance of the radar reflectance map and the road surface texture features of the visual image. For example, the reflectance variance of the road surface area around the lane line can be extracted from the radar reflectance distribution features, and the texture variance of the same area can be extracted from the visual features. The ratio of the reflectance variance to the preset maximum reflectance variance is determined as the radar texture inhomogeneity, and the ratio of the texture variance to the preset maximum texture variance is determined as the visual texture inhomogeneity. Then, the radar texture inhomogeneity and the visual texture inhomogeneity are weighted and summed so that the weighted sum value is in the range of [0,1] to obtain the road surface texture inhomogeneity. The closer the road surface texture inhomogeneity is to 1, the more chaotic the road surface texture, the greater the undulation, the more uneven the material, the more blurred the boundary between the lane line and the background, and the higher the road surface complexity.

[0091] Lane line semantic interference degree can be calculated based on semantic and geometric information from visual features. It can reflect the degree of semantic confusion caused by overlapping adjacent lane lines, temporary markings covering, old marking residues, and confusion caused by construction markings. For example, geometric and semantic information of the target lane line and surrounding interfering markings can be extracted from visual features. Then, according to preset semantic interference judgment rules, such as spacing less than a threshold, directional angle less than a threshold, similar colors, and overlapping line types, various semantic interference degrees are calculated. Then, the various semantic interference degrees are normalized and weighted summed so that the weighted sum value is in the range of [0,1] to obtain the lane line semantic interference degree. The closer the lane line semantic interference degree is to 1, the more serious the confusion, covering, and overlap of the target lane line by other markings, and the higher the road surface complexity.

[0092] In this embodiment, a multi-feature fusion algorithm can be used to calculate each third evaluation index to obtain a quantitative value representing the road surface complexity. For example, the weighted sum of each third evaluation index can be calculated so that the value of the sum is in the range of [0,1] to obtain a quantitative value representing the road surface complexity. The quantitative value ranges from 0 to 1, where 0 indicates a simple road surface and 1 indicates an extremely complex road surface.

[0093] In this embodiment, by fusing visual features and radar reflectivity distribution features, a third evaluation index is calculated, which includes interference density, road surface texture non-uniformity, and lane line semantic interference. This index can quantitatively characterize road surface complexity from multiple dimensions such as target interference, texture changes, and semantic confusion, thereby improving the accuracy of road surface complexity quantification.

[0094] The network structure of the perception model includes a shared feature extraction layer and output layers corresponding to each task. The shared feature extraction layer can use a lightweight convolutional neural network (such as MobileNetV3 or EfficientNet-B0) or a Transformer encoder as the backbone network. If a convolutional neural network is used, low-dimensional basic features (such as edges, textures, and reflectivity distributions) and high-dimensional semantic features (such as the overall structure of lane lines and road surface type) in the sample feature set can be extracted step by step through multiple convolutions, pooling, and non-linear activation operations. If a Transformer encoder is used, long-distance dependencies in the comprehensive feature set of sample lane lines can be captured through self-attention mechanisms and feedforward neural networks, improving the ability to model lane lines and road surface features in complex scenes.

[0095] The output layer for each task needs to be constructed in conjunction with the modeling logic corresponding to that task. Specifically, the output layer for the lane line modeling task can employ a network with regression and classification structures. This network adapts to the calculation of the first evaluation metric and the rules for deriving the output target from the first evaluation metric in the modeling logic. For example, the regression structure can use convolutional layers and fully connected layers to map the output of the shared feature extraction layer to predicted values ​​of lane line geometric coordinates through the inference of the modeling logic; the classification structure can use a fully connected layer + softmax layer to map the output of the shared feature extraction layer to the probability distribution of lane line semantic categories through the inference of the modeling logic.

[0096] The output layer for the lane line aging modeling task can employ a regression-structured network. This network adapts to the calculation of the second evaluation metric and the rules governing the transition from the second evaluation metric to the output target within the modeling logic. For example, the regression structure uses fully connected layers and non-linear activation functions to map shared feature maps to predicted values ​​of aging quantification through the modeling logic.

[0097] The output layer for road surface complexity modeling tasks can also employ a regression-structured network. This network adapts to the calculation of the third evaluation metric and the rules for mapping the third evaluation metric to the output target in the modeling logic. For example, the regression structure uses fully connected layers and a sigmoid activation function to map shared feature maps to predicted values ​​of road surface complexity quantifications through the modeling logic.

[0098] It should be noted that the comprehensive feature set of sample lane lines is obtained based on sample visual images and sample radar reflectance maps. Specifically, sample visual images and sample radar reflectance maps covering different scenarios (such as urban roads, highways, rural roads, construction sections, etc.), different working conditions (such as sunny days, rainy days, foggy days, morning and evening rush hours, etc.), and different lane line conditions (such as intact, slightly aged, moderately aged, and severely aged) can be collected. Then, visual features and reflectance distribution features are extracted separately, and the visual features and reflectance distribution features are spliced ​​or weighted and fused to generate the comprehensive feature set of sample lane lines. Labels are then established for the comprehensive feature set of sample lane lines according to the output target of the above task.

[0099] During training, the comprehensive feature set of sample lane lines can be input into the perception model. The perception model encodes the comprehensive feature set of sample lane lines through a shared feature extraction layer to generate a shared feature map. The shared feature map is then input into three task output layers. Each output layer performs inference operations based on its own modeling logic. Specifically, the output layer corresponding to the lane line modeling task outputs the predicted geometric coordinates and semantic category probability distribution; the output layer corresponding to the lane line aging degree modeling task outputs the predicted value of the aging degree quantification; and the output layer corresponding to the road surface complexity modeling task outputs the predicted value of the road surface complexity quantification.

[0100] Next, joint loss calculation is performed. Based on the prediction results and labels of each task, lane line modeling loss term, lane line aging degree modeling loss term, and road surface complexity modeling loss term are calculated separately, thereby calculating the total loss value corresponding to the joint loss function. Then, the backpropagation algorithm is used to calculate the gradient of the total loss with respect to the parameters of each layer of the model. The network parameters are updated through the optimizer to reduce the total loss until the perceptual model converges or the training epochs reach their maximum value.

[0101] In some embodiments, the lane line modeling loss term can be calculated using cross-entropy loss, and the lane line aging degree modeling loss term and the road surface complexity modeling loss term can be calculated using mean squared error loss. Of course, the lane line modeling loss term, the lane line aging degree modeling loss term, and the road surface complexity modeling loss term can also be calculated in other ways, and this application embodiment does not limit them.

[0102] The joint loss function is a weighted sum of the lane line modeling loss term, the lane line aging degree modeling loss term, and the road surface complexity modeling loss term. Weight coefficients can be set in advance for each type of loss term, or the weight coefficients of each type of loss term can be dynamically adjusted according to factors such as task importance, data type, and training effect.

[0103] In this embodiment, by constructing a multi-task neural network structure that includes a shared feature extraction layer and output layers for each task, lane line modeling, lane line aging degree modeling, and road surface complexity modeling tasks can share the underlying feature extraction process, reducing the work of repeatedly extracting the same features, thereby reducing the number of model parameters and computational overhead. Furthermore, by using a joint loss function that integrates the loss terms of each task for training, mutual constraints and collaborative optimization among multiple tasks can be achieved during the training process. This allows the model to learn the geometric and semantic features of lane lines while referring to the feature representations of aging degree and road surface complexity, thereby enhancing the model's joint prediction ability for multiple tasks and improving the accuracy of lane line perception.

[0104] The autonomous driving risk assessment method provided in this application can be executed by an autonomous driving risk assessment device. This application uses an autonomous driving risk assessment device executing the autonomous driving risk assessment method as an example to illustrate the autonomous driving risk assessment device provided in this application.

[0105] This application also provides an apparatus for assessing the risks of autonomous driving.

[0106] like Figure 4 As shown, the autonomous driving risk assessment device includes: The acquisition module 410 is used to acquire visual images and radar reflectivity maps of the target road surface through the vehicle-mounted vision sensors and lidar, respectively. The extraction module 420 is used to extract visual features of the lane line area from the visual image after performing time synchronization and spatial calibration on the visual image and radar reflectivity map, and to identify target reflectivity areas in the radar reflectivity map that are higher than the reflectivity threshold, and extract the reflectivity distribution features of the target reflectivity areas; and to fuse the visual features and reflectivity distribution features according to the time synchronization and spatial calibration results to obtain a comprehensive feature set of the lane line; the reflectivity distribution features include at least the average reflectivity and reflectivity variance; the visual features include at least edge features, color features and geometric features. The generation module 430 is used to generate lane line perception results based on the comprehensive feature set of lane lines; the lane line perception results include lane line modeling information, lane line aging degree and road surface complexity of the area where the lane line is located; The evaluation module 440 is used to match the lane line aging degree and road surface complexity from the risk level relationship to obtain the target risk level corresponding to the lane line modeling information; the risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree range and road surface complexity range.

[0107] The autonomous driving risk assessment device provided in this application, by acquiring visual images and radar reflectivity maps of the target road surface, can further analyze the aging degree of lane lines and the complexity of the road surface in the area based on multi-source heterogeneous data. It incorporates assessment dimensions such as lane line aging state and road surface state into the lane line perception results. Compared with the coarse perception using the "confidence threshold filtering" mechanism, it can improve the accuracy of lane line state recognition, enabling the autonomous driving system to stably output accurate lane line information and risk levels even in aging lane lines and complex road surface scenarios, reducing decision-making errors, improving the safety of autonomous driving, and can be adapted to different levels of autonomous driving vehicles. It is compatible with various road surface scenarios such as urban roads, highways, and rural roads, and is especially suitable for complex scenarios such as old roads and construction sections.

[0108] The autonomous driving risk assessment device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0109] The autonomous driving risk assessment device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0110] This application also provides a vehicle, including a vision sensor, a lidar, and a controller; A visual sensor is used to acquire visual images of the target road surface; LiDAR is used to collect radar reflectivity maps of target road surfaces. The controller is used to perform the aforementioned risk assessment methods for autonomous driving.

[0111] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described autonomous driving risk assessment method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0112] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0113] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described autonomous driving risk assessment method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0114] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0115] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-mentioned method for assessing autonomous driving risks.

[0116] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0117] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described autonomous driving risk assessment method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0118] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0119] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0121] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0122] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for assessing the risks of autonomous driving, characterized in that, include: The vehicle uses visual sensors and lidar to collect visual images and radar reflectivity maps of the target road surface. After time synchronization and spatial calibration of the visual image and the radar reflectivity map, a convolutional neural network is used to extract visual features of the lane line area from the visual image and identify target reflectivity areas in the radar reflectivity map that are higher than the reflectivity threshold, extracting the reflectivity distribution features of the target reflectivity areas; the visual features and the reflectivity distribution features are fused according to the time synchronization and spatial calibration results to obtain a comprehensive lane line feature set; the reflectivity distribution features include at least average reflectivity and reflectivity variance; the visual features include at least edge features, color features, and geometric shape features. The lane line perception results are generated based on the comprehensive feature set of the lane lines; the lane line perception results include lane line modeling information, lane line aging degree, and road surface complexity of the area where the lane line is located; The target risk level corresponding to the lane line modeling information is obtained by matching the lane line aging degree and the road surface complexity from the risk level relationship. The risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree ranges and road surface complexity ranges.

2. The method according to claim 1, characterized in that, The edge features include edge continuity and edge gradient, the color features include the color contrast between the lane lines and the road surface, and the geometric features include lane line width and curvature.

3. The method according to claim 1, characterized in that, The process of generating lane line perception results based on the lane line integrated feature set includes: The lane line integrated feature set is input into a preset perception model to obtain the lane line perception result output by the perception model. The perception model is a multi-task neural network model based on multi-task joint learning, and the multi-task includes lane line modeling task, lane line aging degree modeling task, and road surface complexity modeling task.

4. The method according to claim 3, characterized in that, The perception model is trained in the following manner: Obtain the modeling logic and output targets corresponding to each task; The network structure of the perception model is constructed, which includes a shared feature extraction layer and output layers corresponding to each task. The shared feature extraction layer is used to extract features of the comprehensive feature set of sample lane lines, and the output layers corresponding to each task correspond to the modeling logic of each task. The comprehensive feature set of sample lane lines is obtained based on sample visual images and sample radar reflectivity maps. The perception model is trained using the sample lane line comprehensive feature set as training input and the output target corresponding to each task as label, based on the joint loss function; wherein, the joint loss function includes lane line modeling loss term, lane line aging degree modeling loss term and road surface complexity modeling loss term.

5. The method according to claim 4, characterized in that, The output target of the lane line modeling task includes the geometric coordinates and semantic category of the lane line; the semantic category includes at least one of solid line, dashed line and guide line. The modeling logic for the lane line modeling task includes: The first evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the first evaluation index includes at least one of lane line edge continuity, color contrast, width, curvature, average reflectance, and reflectance variance. The geometric coordinates and semantic category of the lane lines are determined based on the first evaluation metric.

6. The method according to claim 4, characterized in that, The output objective of the lane line aging modeling task includes: a quantitative value characterizing the aging degree of the lane lines; The modeling logic for the lane line aging modeling task includes: A second evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the second evaluation index includes at least one of reflectance attenuation coefficient, edge integrity coefficient, and material uniformity coefficient. The quantitative value representing the degree of lane line aging is calculated based on the second evaluation index.

7. The method according to claim 4, characterized in that, The output objective of the road surface complexity modeling task includes: a quantitative value representing the road surface complexity of the area where the lane line is located; The modeling logic for the road surface complexity modeling task includes: A third evaluation index is calculated based on the visual characteristics and reflectance distribution characteristics of the area where the lane line is located; the third evaluation index includes at least one of the following: density of interfering objects within the target range of the lane line, unevenness of road surface texture, and semantic interference degree of the lane line. The quantitative value representing the road surface complexity is calculated based on the third evaluation index.

8. A device for assessing the risks of autonomous driving, characterized in that, include: The acquisition module is used to collect visual images and radar reflectivity maps of the target road surface through the visual sensors and lidar carried by the autonomous vehicle. An extraction module is used to perform time synchronization and spatial calibration on the visual image and the radar reflectivity map, and then use a convolutional neural network to extract visual features of the lane line area from the visual image, identify target reflectivity areas in the radar reflectivity map that are higher than the reflectivity threshold, and extract the reflectivity distribution features of the target reflectivity areas; based on the time synchronization and spatial calibration results, the visual features and the reflectivity distribution features are fused to obtain a comprehensive feature set for the lane lines; the reflectivity distribution features include at least average reflectivity and reflectivity variance; the visual features include at least edge features, color features, and geometric morphology features. The generation module is used to generate lane line perception results based on the lane line comprehensive feature set; the lane line perception results include lane line modeling information, lane line aging degree, and road surface complexity of the area where the lane line is located; The assessment module is used to match the lane line aging degree and the road surface complexity from the risk level relationship to obtain the target risk level corresponding to the lane line modeling information; The risk level relationship includes multiple risk levels, and each risk level corresponds to a set of lane line aging degree ranges and road surface complexity ranges.

9. A vehicle, characterized in that, Includes visual sensors, LiDAR, and controllers; The visual sensor is used to acquire visual images of the target road surface; The lidar is used to collect radar reflectivity maps of the target road surface; The controller is configured to perform the method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images