Vehicle driving state monitoring method and related device

Abstract

The embodiment of the invention provides a vehicle driving state monitoring method and a related device, which are used for improving the safety in an intelligent vehicle driving process. The method provided by the embodiment of the invention comprises the following steps: monitoring whether sensing information output by a sensing network model is abnormal or not; and / or monitoring whether a prediction track output by the prediction and decision network model is abnormal; and if at least one of the sensing information and the predicted trajectory is abnormal, performing a degradation decision of the vehicle, and controlling a driving mode of the vehicle according to a degradation decision result.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method and related device for monitoring the driving status of a vehicle. Background Technology

[0002] Intelligent driving vehicles, including vehicles, mobile robots, and aircraft equipped with intelligent driving systems, are intelligent entities that achieve intelligent driving through computer systems. With the increasing prevalence of intelligent driving vehicles, they can be used as taxis or public transportation. When using an intelligent driving vehicle, passengers need to input their destination, and the vehicle generates a route based on the current location and destination, then follows that route.

[0003] With the development of automated driving technology, end-to-end models are being deployed in intelligent driving systems. An end-to-end model integrates the independent modules of perception, decision-making, and planning found in traditional intelligent driving into a unified neural network. It directly generates vehicle control commands (such as steering, acceleration, and braking) from raw sensor data (e.g., cameras, LiDAR, millimeter-wave radar). This design simulates the "input-processing-output" mechanism of the human brain, completely changing the pipeline-like operation of traditional modular systems.

[0004] Currently, the end-to-end model of intelligent driving system is still in its infancy. Its black-box nature will bring a large number of unknown safety issues to the intelligent driving system. Moreover, since the end-to-end model is not yet mature, safety problems in the end-to-end model will cause safety hazards to the entire vehicle. Summary of the Invention

[0005] This invention provides a method and related device for monitoring the driving status of a vehicle, which can improve the safety of intelligent driving vehicles during the driving process.

[0006] The first aspect of this application provides a method for monitoring the driving state of a vehicle, wherein an end-to-end model is deployed on the vehicle, and the end-to-end model includes at least a perception network model and a prediction and decision network model. The method includes:

[0007] Monitor whether the perception information output by the perception network model is abnormal; and / or, monitor whether the predicted trajectory output by the prediction and decision network model is abnormal;

[0008] If at least one of the perceived information or predicted trajectory is abnormal, a vehicle degradation decision is executed.

[0009] Based on the downgrade decision, control the vehicle's driving mode.

[0010] As an optional embodiment, the perception network model includes at least one of an object detection network model, an obstacle detection network model, and a road element detection network model, and the monitoring of whether the perception information output by the perception network model is abnormal includes:

[0011] Monitor whether the object information output by the object detection network model is abnormal;

[0012] And / or,

[0013] Monitor whether the obstacle information output by the obstacle detection network model is abnormal;

[0014] And / or,

[0015] Monitor whether the road element information output by the road element detection network model is abnormal;

[0016] If at least one of the perceived information and the predicted trajectory is abnormal, then a vehicle degradation decision is executed, including:

[0017] If at least one of the object information, obstacle information, road element information, and predicted trajectory is abnormal, a degrade decision is made for the vehicle.

[0018] As an optional embodiment, the prediction and decision network model is used to output at least one of the predicted trajectories of vehicles, objects, and obstacles based on at least one of object information, obstacle information, and road element information.

[0019] Monitoring whether the predicted trajectory output by the prediction and decision network model is abnormal includes:

[0020] Monitor whether at least one of the predicted trajectories of vehicles, objects, and obstacles is abnormal;

[0021] If at least one of the perceived information and the predicted trajectory is abnormal, then a vehicle degradation decision is executed, including:

[0022] If at least one of the following is abnormal: object information, obstacle information, road element information, vehicle predicted trajectory, object predicted trajectory, and obstacle predicted trajectory, then a vehicle degradation decision is executed.

[0023] As an optional embodiment, the object information includes at least one of the following: the object's position range, the object's size range, the object's velocity range, the object's type range, the object's characteristic values, and whether the object type matches the object parameters.

[0024] Monitor whether the object information output by the object detection network model is abnormal, including:

[0025] The monitoring function checks whether the position range of an object exceeds a first preset range in a first direction, a second preset range in a second direction, or a third preset range in a third direction. The first, second, and third directions are perpendicular to each other. If any one of them exceeds the preset range, the object information is determined to be abnormal.

[0026] And / or,

[0027] Monitor whether the size range of the same target object exceeds a fourth preset range; if it does, determine that the object information is abnormal.

[0028] And / or,

[0029] The system monitors whether the horizontal and vertical velocity ranges of the object exceed a fifth preset range. If they do, the system determines that the object information is abnormal.

[0030] And / or,

[0031] The system monitors whether the range of object types exceeds a preset range. If it does, the system determines that the object information is abnormal.

[0032] And / or,

[0033] The system monitors whether the characteristic values ​​of an object are empty. If they are empty, the system determines that the object information is abnormal.

[0034] And / or,

[0035] The system monitors whether the object type matches the object parameters. If they do not match, the system determines that the object information is abnormal.

[0036] As an optional embodiment, obstacle information includes at least one of the following: speed at different locations within the obstacle, obstacle category and current driving mode, category of different locations within the same obstacle, obstacle location range, obstacle speed range, and whether the obstacle category and speed match.

[0037] Monitor whether the obstacle information output by the obstacle detection network model is abnormal, including:

[0038] Monitor whether the velocity direction is different at different locations within the obstacle; if it is different, determine that the obstacle information is abnormal.

[0039] And / or,

[0040] The speed difference between different points in the obstacle is monitored to see if it exceeds a first preset threshold. If it does, the obstacle information is determined to be abnormal.

[0041] And / or,

[0042] The system monitors whether the type of obstacle matches the current driving mode; if they do not match, the obstacle information is determined to be abnormal.

[0043] And / or,

[0044] Monitor whether the categories of different locations of the same obstacle are different; if they are different, the obstacle information is determined to be abnormal.

[0045] And / or,

[0046] The system monitors whether the location range of obstacles exceeds the fifth preset range. If it does, the obstacle information is determined to be abnormal.

[0047] And / or,

[0048] The system monitors whether the speed range of obstacles exceeds the sixth preset range. If it does, the obstacle information is determined to be abnormal.

[0049] And / or,

[0050] Monitor whether the category of the obstacle exceeds the preset category range; if it does, determine that the obstacle information is abnormal.

[0051] And / or,

[0052] The system monitors whether the obstacle type and speed match; if they do not match, the obstacle information is determined to be abnormal.

[0053] And / or,

[0054] The system checks whether the obstacle at the same location output by the obstacle detection network model is the same as the obstacle at the same location output by the object detection network model. If they are different, the obstacle information is determined to be abnormal.

[0055] As an optional embodiment, the road element information includes lane line point chains and feature values ​​of the lane line point chains. The feature values ​​include road markings, lane markings, the location of the lane line point chains and / or the type of the lane line point chains, and the shape type of the lane lines.

[0056] Monitor whether the road element information output by the road element detection network model is abnormal, including:

[0057] Based on the position and type of lane line dot chain, determine whether the lane line and the lane center line intersect. If they intersect, then the road element information is determined to be abnormal.

[0058] And / or,

[0059] Based on the position and type of lane line point chain, determine whether the road edge line and lane center line intersect. If they intersect, then the road element information is determined to be abnormal.

[0060] And / or,

[0061] Based on the position and type of lane line point chain, determine whether the dividing line intersects with the center line of the lane and the lane line. If they intersect, the road element information is determined to be abnormal.

[0062] And / or,

[0063] Based on the feature values ​​of lane line point chains and lane line point chains, determine whether the same lane line point chain corresponds to multiple lane categories. If so, determine that the road element information is abnormal.

[0064] And / or,

[0065] Based on the feature values ​​of lane line point chains and lane line point chains, determine whether the same lane line point chain corresponds to multiple lane line categories. If so, determine that the road element information is abnormal.

[0066] And / or,

[0067] Based on the feature values ​​of the lane line point chain, determine whether the feature values ​​are empty. If they are empty, then the road element information is determined to be abnormal.

[0068] As an optional embodiment, monitoring whether at least one of the predicted trajectory of the vehicle, the predicted trajectory of the detected object, and the predicted trajectory of the detected obstacle is abnormal includes:

[0069] Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is empty. If it is empty, determine that the predicted trajectory is abnormal.

[0070] And / or,

[0071] Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is a preset abnormal trajectory. If so, determine that the predicted trajectory is abnormal.

[0072] And / or,

[0073] The total probability of the predicted trajectory of the monitored target is greater than 1. The target trajectory includes the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle. If so, the predicted trajectory is determined to be abnormal.

[0074] And / or,

[0075] The system monitors whether the number of predicted trajectories for the target exceeds a preset value. The predicted trajectories for the target include the predicted trajectories of vehicles, objects, and / or obstacles. If so, the predicted trajectory is determined to be abnormal.

[0076] As an optional embodiment, if at least one of the perceived information and the predicted trajectory is abnormal, a vehicle degradation decision is executed, and the driving mode of the vehicle is controlled according to the degradation decision result, including:

[0077] If at least one of the perceived information and the predicted trajectory is abnormal, the vehicle's fault information is determined based on the number of times the abnormality occurs and the duration of the abnormality.

[0078] Based on the vehicle's fault information, the vehicle's driving mode is downgraded.

[0079] As an optional implementation, the degradation process includes:

[0080] Limit the speed of vehicles;

[0081] And / or,

[0082] Limit the distance between vehicles;

[0083] And / or,

[0084] Activation of some intelligent driving functions is restricted;

[0085] And / or,

[0086] Switch the vehicle's driving mode;

[0087] And / or,

[0088] Remind the driver of the vehicle;

[0089] And / or,

[0090] Exit the intelligent driving function.

[0091] A second aspect of this application provides a vehicle driving state monitoring device, wherein an end-to-end model is deployed on the vehicle, the end-to-end model including at least a perception network model and a prediction and decision network model, and the monitoring device includes:

[0092] The perception network model monitoring module is used to monitor whether the perception information output by the perception network model is abnormal.

[0093] And / or,

[0094] The prediction and decision network model monitoring module is used to monitor whether the prediction trajectory output by the prediction and decision network model is abnormal.

[0095] A degradation decision module is used to execute a degradation decision for the vehicle if at least one of the perceived information and the predicted trajectory is abnormal; and to control the driving mode of the vehicle based on the degradation decision result.

[0096] A third aspect of this application provides a controller, including a processor, a memory, and a computer program stored in the memory. When executed by the processor, the computer program is used to implement the vehicle driving state monitoring method provided in the first aspect of this application.

[0097] The fourth aspect of this application provides an intelligent driving vehicle, including at least the controller provided in the third aspect of this application.

[0098] The fifth aspect of this application provides a computer-readable storage medium storing a computer program thereon, characterized in that, when executed by a processor, the computer program is used to implement the vehicle driving state monitoring method provided in the first aspect of this application.

[0099] The sixth aspect of this application provides a computer program product having a computer program stored thereon, characterized in that, when executed by a controller, the computer program is used to implement the vehicle driving state monitoring method provided in the first aspect of this application.

[0100] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0101] In this embodiment, the perception information and / or predicted trajectory information output by the perception network model and the prediction and decision network model are monitored, and when any output information is abnormal, the driving mode of the vehicle is downgraded to achieve safety monitoring and downgrade decision-making of the end-to-end network, which can improve the safety of intelligent driving vehicles with end-to-end models during driving. Attached Figure Description

[0102] Figure 1 This is a schematic diagram of the architecture of an end-to-end model deployed in a vehicle in an embodiment of this application;

[0103] Figure 2 This is a schematic diagram of one embodiment of a vehicle driving status monitoring method according to the present application.

[0104] Figure 3 This is a schematic diagram of an embodiment of monitoring whether the perceptual information output by the perceptual network model is abnormal, as described in this application.

[0105] Figure 4 This is a schematic diagram of another architecture of the end-to-end model deployed in the vehicle in the embodiments of this application;

[0106] Figure 5 This is a schematic diagram of an embodiment of monitoring whether the object information output by the object detection network model is abnormal, as described in this application.

[0107] Figure 6 This is a schematic diagram of an embodiment of the present application, illustrating whether the obstacle information output by the obstacle detection network model is abnormal.

[0108] Figure 7This is a schematic diagram of an embodiment of monitoring whether the road element information output by the road element detection network model is abnormal, as described in this application.

[0109] Figure 8 This is a schematic diagram of an embodiment of the present application that monitors whether at least one of the predicted trajectories of vehicles, objects, and obstacles is abnormal.

[0110] Figure 9 This is a schematic diagram of one embodiment of a vehicle driving status monitoring device according to the present application.

[0111] Figure 10 This is a schematic diagram of one embodiment of an intelligent driving vehicle in this application. Detailed Implementation

[0112] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0113] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0114] For ease of understanding, the method for monitoring the vehicle driving status in the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 :

[0115] Specifically, in this embodiment, the vehicle driving state monitoring method is executed by a controller deployed in the vehicle. Furthermore, an end-to-end network model is also deployed on the vehicle in this embodiment. This end-to-end network model specifically includes a perception network model and a prediction and decision network model. The perception network model is used to perceive and identify surrounding objects, obstacles during driving, and / or road elements during driving, providing decision-making basis for the intelligent driving vehicle to improve driving safety. The prediction and decision network model receives the output of the perception network model and, based on the specific information output by the perception network model, outputs a predicted trajectory for the vehicle to assist in safe driving. For example, for ease of understanding, Figure 1 A schematic diagram of the end-to-end network model deployed on a vehicle is presented. The perception network model receives images of surrounding objects, road obstacles, or road elements from sensors on the vehicle (such as LiDAR, cameras, and millimeter-wave radar), and then processes these images to output object, obstacle, or road element information. The prediction and decision network model, based on the output of the perception network model and combined with sensor calibration data and map positioning information, performs trajectory prediction processing and outputs the predicted trajectories of the vehicle, surrounding vehicles, and obstacles. The planning processing module plans the vehicle's trajectory based on the trajectory output by the prediction and decision network model and outputs the vehicle's trajectory planning result. The vehicle control module controls the vehicle based on the vehicle trajectory planning result output by the planning processing module and the degradation decision result output by the vehicle degradation strategy module. To enable safety monitoring of the end-to-end model, the end-to-end network model also includes a perception network model monitoring module and a prediction and decision network monitoring module. These modules are used to monitor whether the outputs of the perception network model and the prediction and decision network model are abnormal, respectively. The vehicle degradation strategy module is used to make degradation decisions and output the degradation decision results, which are then provided to the vehicle control module for vehicle control decisions.

[0116] It is understood that the embodiments of this application Figure 1 For ease of understanding, the end-to-end model is presented as two models: a perception network model and a prediction and decision network model. However, the end-to-end model can also be a holistic model that includes both perception and prediction and decision-making functions.

[0117] For ease of understanding, the method for monitoring the vehicle driving state in the embodiments of this application is described below. Please refer to [link / reference]. Figure 2 :

[0118] 201. Monitor whether the perceptual information output by the perceptual network model is abnormal; and / or,

[0119] Specifically, the perception network model in this application embodiment is used to perceive and identify environmental information of the intelligent vehicle during driving, so as to provide a basis for vehicle driving decisions and assist the safe driving of the vehicle. This includes surrounding objects encountered by the vehicle during driving (such as vehicles and pedestrians), obstacles encountered during driving (such as construction zones or animals that suddenly enter), or road information (such as lane lines) encountered during driving. The model outputs the specific location, speed, acceleration, size and other perception information of the environmental information to assist the safe driving of the vehicle.

[0120] Optionally, the perception network model in this application embodiment can be an integrated network that outputs the above-mentioned various information (such as surrounding objects, obstacles and road information), or it can include multiple different network models to output the above-mentioned various information respectively. Here, there is no specific limitation on the number of models in the perception network model. Furthermore, the perception network model in this application embodiment can be an AI model or various types of neural network models. Here, there is no specific limitation on the model architecture of the perception network model.

[0121] For example, such as Figure 1 As shown, in this embodiment of the application, a perceptual network model monitoring module can be added to receive the output information of the perceptual network model and make anomaly judgment on the output information according to a preset anomaly monitoring method (such as monitoring the anomaly threshold of the output information), so as to detect the output anomaly of the perceptual network model in a timely manner.

[0122] 202. Monitor whether the predicted trajectory output by the prediction and decision-making network model is abnormal;

[0123] In the end-to-end model, a prediction and decision network model is also deployed that communicates with the perception network model. Specifically, the prediction and decision network model is used to receive the output information of the perception network model and predict the vehicle's driving trajectory based on the output information of the perception network model, and / or predict the trajectories of various dynamic objects or dynamic obstacles output by the perception network model, so as to assist the safe driving of the vehicle.

[0124] For example, such as Figure 1 As shown, in this embodiment of the application, a prediction and decision network model monitoring module can be added to receive the output information of the prediction and decision network model and make anomaly judgments on the output information according to a preset anomaly monitoring method (such as monitoring the anomaly threshold of the output trajectory), so as to detect the output anomaly of the prediction and decision network model in a timely manner.

[0125] 203. If at least one of the perceived information and the predicted trajectory is abnormal, a vehicle degradation decision is executed, and the vehicle's driving mode is controlled based on the degradation decision result.

[0126] It is easy to understand that, in order to ensure the safety of the vehicle during driving, if the output information of either the perception network model or the prediction and decision network model is abnormal, it indicates that either the perception network model or the prediction and decision network model may be malfunctioning, or that the objects, obstacles, or road information encountered by the vehicle during driving exceeds the information range that the perception network model and the prediction and decision network model can recognize. In this case, the information output by the perception network model and the prediction and decision network model will interfere with the safe driving of the vehicle.

[0127] To ensure safe driving, it is necessary to implement a downgrade decision for the vehicle and control the vehicle's driving mode based on the result of the downgrade decision.

[0128] It should be noted that the downgrade decision in this application embodiment is to determine the dangerous situation of the current vehicle driving process based on abnormal information and to determine the corresponding safe driving plan, including downgrading the vehicle's speed, lane changing behavior, distance from surrounding objects, or intelligent driving mode, so as to better assist the vehicle in safe driving.

[0129] In this embodiment, the perception information and / or predicted trajectory information output by the perception network model and the prediction and decision network model are monitored, and when any output information is abnormal, a downgrade decision is made, and the driving mode of the vehicle is downgraded according to the decision result, so as to realize the safety monitoring and downgrade decision of the end-to-end network, which can improve the safety of intelligent driving vehicles with end-to-end models during the driving process.

[0130] based on Figure 1 The embodiments described below, using a perception network model comprising multiple models as an example, will provide a detailed description of the specific network models deployed in the vehicle. To assist in safe vehicle operation, this embodiment sets up an object detection network model, an obstacle detection network model, and a road element detection network model within the perception network model. These models separately monitor objects, obstacles, and road elements encountered by the vehicle during operation, ensuring the accuracy of the detected objects, obstacles, and road elements. It should be noted that the information output by each model in this embodiment is only used to assist in safe vehicle operation, not directly to control the vehicle's movement. Furthermore, the personal information involved in the specific information collected by each model during output or storage is confidential or anonymized and used only to assist in safe vehicle operation. The processed personal information complies with relevant legal regulations.

[0131] Specifically, the object detection network model is used to detect dynamic traffic participants (such as vehicles, pedestrians, and cyclists) and static landmarks (such as traffic signs and traffic lights) on the road in real time, outputting the location, category, and motion state of the targets; the obstacle detection network model is used to identify all objects that may affect vehicle driving safety, including dynamic obstacles (such as animals that suddenly run into the road) and static obstacles (such as construction zones and potholes), outputting their spatial location and passability assessment; the road element detection network model is used to analyze the static geometric structure of the road (such as lane lines, shoulders, zebra crossings, and lane centerlines) and topological relationships (such as intersection types and road boundaries), providing basic map information for path planning. The process of monitoring whether the perception information output by the perception network model is abnormal in the embodiments of this application is described in detail below. Please refer to [link to relevant documentation]. Figure 3 :

[0132] 301. Monitor whether the object information output by the object detection network model is abnormal; and / or,

[0133] Specifically, in order to monitor the output information of the object detection network model, obstacle detection network model, and road element detection network model, this application embodiment sets up a corresponding safety monitoring module for each of the three models, such as setting up an object detection network model monitoring module for the object detection network model, an obstacle detection network model monitoring module for the obstacle detection network model, and a road element detection network model monitoring module for the road element detection network model, so as to monitor whether the output of each model is abnormal.

[0134] Specifically, in this embodiment, the object detection network model monitoring module can be controlled to monitor whether the object information output by the object detection network model is abnormal. The object information includes at least one of the following: the object's position, the object's size, the object's speed, the object's type, the object's feature values, and the object's parameters. The process of monitoring the object information will be described in the following embodiments and will not be repeated here.

[0135] For ease of understanding, Figure 4The system architecture is presented when the end-to-end model simultaneously includes an object detection network model, an obstacle detection network model, a road element detection network model, and a prediction and decision network model, with a safety monitoring module set up for each model. Of course, if the end-to-end model only includes one or two of the above three network models, then a safety monitoring module is set up for one or two network models. Here, there is no specific limitation on the number of network models set up in the end-to-end model or the corresponding number of safety monitoring modules. In other embodiments, the perception network model may also include a single network model whose output may include the output information of the above three network models. A safety monitoring module can be set up for this single network model. The monitoring function and process of this module can be referred to the above embodiments and will not be repeated here.

[0136] Furthermore, in Figure 4 In addition to the prediction and decision network model, a planning and processing model and a vehicle control module are also set up. The planning and processing model selects the vehicle's driving trajectory based on the multiple predicted trajectories output by the prediction and decision network model and the probability value of each predicted trajectory. The final selected vehicle driving trajectory is then input to the vehicle control module so that the vehicle control module can control the vehicle according to the final selected vehicle driving trajectory.

[0137] 302. Monitor whether the obstacle information output by the obstacle detection network model is abnormal; and / or,

[0138] Specifically, when an obstacle detection network model and an obstacle detection network model monitoring module are deployed in the end-to-end model, the obstacle detection network model monitoring module is controlled to monitor the obstacle information output by the obstacle detection network model. Specifically, the obstacle information in this embodiment includes at least one of the following: the speed of different points in the obstacle, the type of the obstacle and the current driving mode, the type of different points of the same obstacle, the position of the obstacle, and the speed of the obstacle. The process of monitoring the obstacle information will be described in the following embodiments and will not be repeated here.

[0139] 303. Monitor whether the road element information output by the road element detection network model is abnormal;

[0140] Specifically, when a road element detection network model and a road element detection network model monitoring module are deployed in the end-to-end model, the monitoring module is controlled to monitor the road element information output by the road element detection network model to assist the intelligent driving of the vehicle. In this embodiment, the road element information includes lane line point chains and feature values ​​of lane line point chains. The feature values ​​include road markings, lane markings, the position of lane line point chains and / or the category of lane line point chains.

[0141] Specifically, in intelligent driving technology, lane line point chains are a technical solution that dynamically describes lane lines through a continuous sequence of coordinate points. It abstracts lane lines into a chain structure of ordered key points, accurately capturing the geometric features of lane lines (such as curvature and width) and adapting to dynamic changes (such as road diversions during construction or tidal flow lane adjustments). It is one of the core technologies for realizing intelligent driving perception and planning. For example, during the point chain acquisition process, the detected lane lines are uniformly sampled or sampled at key locations to generate point chains containing coordinate and direction information, which are then used to characterize the geometric features of the lane lines. Furthermore, the road element detection network model in this embodiment, combined with information such as the coordinates and directions of the lane lines, can also output feature values ​​of the lane line point chains, such as road markings (e.g., bus lanes, truck lanes), lane markings (e.g., left-turn lanes, right-turn lanes), the category of the lane line point chain (e.g., lane center line, lane edge line, or dividing line), and the shape category of the lane lines (e.g., white dashed lines, white solid lines, yellow dashed lines).

[0142] c

[0143] 304. Monitor whether the predicted trajectory output by the prediction and decision-making network model is abnormal;

[0144] In the end-to-end model, a prediction and decision network model is also deployed, which is communicatively connected to the perception network model. Specifically, the prediction and decision network model receives the output information from the perception network model and predicts the vehicle's trajectory based on this output information, and / or predicts the trajectories of various dynamic objects or obstacles output by the perception network model, to assist in safe vehicle operation. Specifically, the end-to-end model includes a prediction and decision network model monitoring module, which monitors the trajectory information output by the prediction and decision network model. Specifically, the trajectory information in this embodiment includes at least one of the following: vehicle trajectory, object trajectory, and obstacle trajectory. The process of monitoring the trajectory output by the prediction and decision network model will be described in the following embodiments and will not be repeated here.

[0145] 305. If at least one of the object information, obstacle information, road element information, and predicted trajectory is abnormal, then the vehicle's degrading strategy is executed, and the vehicle's driving mode is controlled according to the degrading decision result.

[0146] It is easy to understand that when any of the object detection network model, obstacle detection network model, road element detection network model, and prediction and decision network model outputs abnormal object information, obstacle information, road element information, or predicted trajectory, it indicates that the vehicle may face danger during intelligent driving. In this case, a degraded strategy is implemented for the vehicle, and the driving mode of the vehicle is controlled according to the result of the degraded strategy to improve the safety of the vehicle during driving.

[0147] Optionally, when anomalies occur in the object information, obstacle information, road element information, or predicted trajectory output by any of the object detection network model, obstacle detection network model, road element detection network model, and prediction and decision network model, the process of implementing a downgrade strategy for the vehicle can involve determining the vehicle's fault information based on the number of times the anomaly occurs and / or its duration (e.g., occurring twice consecutively, or lasting for 10 seconds), and downgrading the vehicle's driving mode based on the fault information. The fault information can be a fault code or other information that can transmit anomalies. The following embodiment uses a fault code as an example for illustration.

[0148] Specifically, the downgrade processing in this application embodiment includes, but is not limited to:

[0149] The system can limit the vehicle's speed, such as limiting the vehicle's speed to no more than x kilometers per hour; and / or limit the vehicle's distance, such as limiting the distance between the vehicle and surrounding objects or obstacles to no less than x meters; and / or restrict the activation of some intelligent driving functions, such as restricting the activation of the vehicle's active lane change function; and / or downgrade the vehicle's current driving mode, such as downgrading from NOCD (No-map End-to-End Urban Navigation) mode to single-lane driving mode, or from single-lane driving mode to ACC (Adaptive Cruise Control); and / or remind the driver, such as automatically providing text, voice, or animation reminders to the driver to alert them to intelligent driving abnormalities and request the driver to take over immediately; and / or deactivate intelligent driving functions and request the driver to take over immediately.

[0150] For example, each monitoring module can be equipped with an anomaly counter and an anomaly timer, which can record the number of times the same anomaly occurs and its duration after an anomaly is determined, and then upload the record to the vehicle degradation decision module. In other embodiments, the vehicle degradation decision module can also be equipped with an anomaly counter and an anomaly timer, which can record the number of times the same anomaly occurs and its duration after receiving the anomaly information reported by each monitoring module.

[0151] Optionally, the vehicle degradation decision module may pre-store a mapping relationship between the frequency and duration of different abnormal information and fault codes. This mapping relationship may include the correspondence between single or multiple abnormal information combinations and multiple fault codes. Each fault code may correspond to a degradation strategy (degradation result). For example, if one or more abnormal information occurs infrequently or for short durations, it may correspond to a low-risk level fault and a low-risk level fault code, thus requiring small-scale degradation or no degradation. Alternatively, if some important information is abnormal, or if one or more abnormal information occurs frequently or for long durations, it may correspond to a high-risk level fault and a high-risk level fault code, thus requiring large-scale degradation.

[0152] Furthermore, during the process of downgrading vehicle driving based on vehicle fault codes, the specific downgrading type can be selected based on the type of fault code, fault level, and / or the duration and frequency of the abnormality, such as:

[0153] 1. If the frequency and duration of the abnormality are short and / or the level of the abnormality is low, it indicates that the vehicle may be facing a low-level fault. Based on the low-level fault, the vehicle speed can be limited and / or the distance between the vehicle and surrounding objects or obstacles can be restricted.

[0154] 2. When the number of occurrences of anomalies continues to increase and / or the duration gradually increases, the fault level is upgraded, and the vehicle is further downgraded according to the upgraded fault level, such as restricting the vehicle from actively changing lanes, or even directly performing a downgrade switch on the vehicle's current driving mode.

[0155] 3. If the level of the anomaly is high, the vehicle will be downgraded to the highest level, such as by giving the driver a voice or animation reminder to take over immediately, or by disabling the intelligent driving function and asking the driver to take over immediately.

[0156] In this embodiment, the vehicle is downgraded according to the type, level, duration and frequency of the fault, thereby improving the intelligence and comfort of intelligent driving while ensuring safe driving.

[0157] In this embodiment, the system monitors whether the object information, obstacle information, road element information, and predicted trajectory output by the object detection network model, obstacle detection network model, road element detection network model, or prediction and decision network model are abnormal. When any output information is abnormal, the driving mode of the vehicle is downgraded to improve the safety of the vehicle during intelligent driving.

[0158] For ease of understanding, the following sections describe in detail the process of checking whether the object information output by the monitoring object detection network model is abnormal, whether the obstacle information output by the monitoring obstacle detection network model is abnormal, whether the road element information output by the monitoring road element detection network model is abnormal, and whether at least one of the predicted trajectories of vehicles, objects, and obstacles is abnormal:

[0159] 1. Monitor whether the object information output by the object detection network model is abnormal.

[0160] Please see Figure 5 :

[0161] 501. Monitor whether the position range of an object exceeds a first preset range in a first direction, exceeds a second preset range in a second direction, or exceeds a third preset range in a third direction, wherein the first direction, the second direction, and the third direction are perpendicular to each other. If any one of them exceeds the preset range, the object information is determined to be abnormal; and / or;

[0162] Optionally, in this embodiment, the position range of an object in three mutually perpendicular directions can be monitored. If the object detection network model monitoring module detects that the position range of the object output by the object detection network model exceeds the first preset range, the second preset range, and the third preset range in one or more of the three mutually perpendicular directions, such as exceeding [-100, 120] meters in the longitudinal position range, or exceeding [-60, 60] meters in the lateral position range, or exceeding [-10, 10] meters in the vertical position range, it indicates that the object detected by the object detection network model exceeds the normal object size encountered in daily driving scenarios. This indicates that the object information output by the object detection network model is incorrect, possibly due to an error in the object detection network model, and thus prompts that the output object information is abnormal. Among them, the first direction, the second direction, and the third direction are preset coordinate system directions (such as the direction of the vehicle coordinate system), and the first preset range, the second preset range, and the third preset range are preset object position ranges, such as the detection range of the object detection network model. This detection range is usually related to the perception range of the sensors mounted on the vehicle and can be set according to the specific perception range. Here, there is no specific limitation on the size of the first preset range, the second preset range, and the third preset range.

[0163] 502. Monitor whether the size range of the same target object exceeds the fourth preset range. If it does, determine that the object information is abnormal; and / or;

[0164] Optionally, during the process of monitoring object information by the object detection network model monitoring module, it can also monitor whether the size range of the same target object exceeds the fourth preset range. For example, if the length range of the same target object exceeds [0,20] meters, or the width range exceeds [0,10] meters, or the height range exceeds [0,10] meters in multiple consecutive frames, it indicates that the object information output by the object detection network model exceeds the normal object size range, which may be due to an error in the object detection network model, prompting that the output object information is abnormal.

[0165] Specifically, the fourth preset range in this application embodiment is related to the sensing range of the sensors mounted on the vehicle and the experienced size of objects in the actual driving scenario. It can be set comprehensively based on the sensing range of the sensors mounted on the vehicle and the experienced size of objects in the actual driving scenario. Here, the size of the fourth preset range is not specifically limited.

[0166] 503. Monitor whether the horizontal and vertical velocity range of the object exceeds the fifth preset range. If it does, determine that the object information is abnormal; and / or;

[0167] Optionally, embodiments of this application can also monitor whether the object's speed range exceeds a fifth preset range, such as whether the longitudinal speed parameter range of the object exceeds [-200, 200] m / s, or whether the lateral speed range exceeds [-200, 200] m / s. If it exceeds these ranges, it indicates that the detected object is rapidly crossing the road or is moving abnormally fast, which may pose a danger. Alternatively, it may indicate that the object information output by the object detection network model is incorrect, thus prompting an error in the output object information.

[0168] Specifically, the fifth preset range in this embodiment is related to the speed sensed by the sensors (such as lidar) on the vehicle and the empirical speed of objects measured in the actual driving scenario. It can be set comprehensively based on the sensing range of the sensors on the vehicle and the empirical speed of objects in the actual driving scenario. Here, the size of the fifth preset range is not specifically limited.

[0169] 504. Monitor whether the range of object types exceeds the preset range. If it does, determine that the object information is abnormal; and / or;

[0170] Furthermore, the embodiments of this application can further determine whether the types of objects output by the object detection network model exceed the preset range of object types. For example, if the object detection network model normally outputs 100 types of objects, but the object detection network model outputs more than the preset 100 types of objects, it indicates that the object detection network model may be malfunctioning or the types of objects output may be incorrect, and the output object information will be indicated as abnormal.

[0171] Specifically, the preset range of categories in this application embodiment is related to the range of categories that the object detection network model can output. For example, if the object detection network model can identify 1,000 object types during the training process, then the preset range of categories in this application embodiment includes 1,000 object types. Of course, the object detection network model should be able to identify various types of objects in the vehicle driving scene during the training process.

[0172] 505. Monitor whether the characteristic representation value of the object is empty. If it is empty, determine that the object information is abnormal; and / or;

[0173] Optionally, the object detection network model can also output the object's feature values ​​during the process of outputting object information. These feature values ​​can represent the object's size, speed, type, name, etc. Therefore, this embodiment can also monitor whether the object's feature values ​​are empty. If they are empty, it indicates that the information output by the object detection network model is incorrect, or that the object detection network model itself is incorrect, and thus prompts that the output object information is abnormal.

[0174] 506. Monitor whether the object type and object parameters match. If they do not match, determine that the object information is abnormal.

[0175] Optionally, embodiments of this application can also monitor whether the object type and object parameters output by the object detection network model match. For example, if the object detection network model outputs a bicycle with a width of 2 meters, or a car with a length of 30 meters, or a truck with a width of 1 meter, it indicates that the logic of the object information output by the object detection network model is incorrect, and the object detection network model may be faulty. Therefore, the output object information is prompted to be abnormal.

[0176] Specifically, in the process of monitoring whether the object type and object parameters match, the object detection network model monitoring module should have a pre-set object type table and a corresponding object parameter table. Through the relationship between the object type table and the object parameter table, it should record the possible parameter range for each object type. This is used to check the parameter range corresponding to the object type when the object detection network model outputs the object type and object parameters, so as to monitor whether the object type and object parameters match.

[0177] In this embodiment of the application, the monitoring process of object information output by the object detection network model is described in detail. During the monitoring process, not only the speed, scale, feature value and type of the object are monitored, but also the matching logic between the object type and the object parameters is monitored, thereby improving the comprehensiveness of the monitoring process, that is, increasing the probability of detecting abnormal objects, which is equivalent to improving the safety of the vehicle driving process.

[0178] II. Monitor whether the obstacle information output by the obstacle detection network model is abnormal.

[0179] Please see Figure 6 :

[0180] 601. Monitor whether the velocity directions of different points within an obstacle are different; if they are different, determine that the obstacle information is abnormal; and / or,

[0181] Optionally, when an obstacle detection network model is deployed in the end-to-end model, this embodiment can also monitor whether the velocity directions of different points on the obstacle output by the obstacle detection network model are the same. Whether the velocity directions are the same refers to whether the angle between the directions indicated by the two velocities is less than a preset velocity angle range. The preset velocity angle range can be set according to the model's detection accuracy and is not limited here. For example, if a vehicle detects a dog ahead, but during the monitoring process, it is found that the dog's first leg runs to the left and the second to the right, meaning the velocity directions of the dog's two legs are inconsistent, or the angle between the velocity directions of the two legs exceeds the preset velocity angle range, then it indicates that the obstacle information output by the obstacle detection network model is incorrect, or that the obstacle detection network model is malfunctioning, and thus an obstacle information anomaly is indicated.

[0182] 602. Monitor whether the speed difference between different points in the obstacle exceeds the first preset threshold. If it does, determine that the obstacle information is abnormal.

[0183] Optionally, in addition to monitoring the velocity direction of different points in the obstacle, this embodiment of the application can also monitor the velocity difference of different points in the obstacle and determine whether the velocity difference of different points exceeds a first preset threshold. For example, if a dog is detected in front of the vehicle while driving, but the velocity difference between the dog's head and tail exceeds 10% of the dog's head velocity, it can be determined that the detected obstacle information is incorrect or that the obstacle detection network model is faulty, and an abnormal obstacle information is indicated.

[0184] 603. Monitor whether the category of the obstacle matches the current driving mode; if they do not match, determine that the obstacle information is abnormal, and / or,

[0185] Optionally, embodiments of this application may also monitor whether the type of obstacle matches the current driving model. For example, if the obstacle detected during the monitoring process is a limit switch, but the current driving mode of the vehicle is driving, it indicates that the type of obstacle does not match the current driving mode, because a limit switch cannot appear during normal driving of the vehicle, and thus an obstacle information abnormality is indicated.

[0186] Specifically, in the process of monitoring the category of obstacles and the current driving mode, the obstacle detection network model monitoring module can store the category of obstacles stored in each driving mode. Then, when an obstacle is detected, the category of each obstacle is matched with the current driving mode to determine whether the category of the obstacle matches the current driving mode.

[0187] 604. Monitor whether the categories of different locations of the same obstacle are different; if they are different, determine that the obstacle information is abnormal, and / or,

[0188] Optionally, embodiments of this application can also monitor the category of different locations of the same obstacle and determine whether the categories of different locations of the obstacle are the same. For example, when a vehicle is driving, it is determined that an obstacle appears in front. However, after detecting different locations of the same obstacle, it is found that the first location point represents the obstacle as a dog, while the second location point represents the obstacle as a cat. In this case, it is determined that the detected obstacle information is incorrect, or the obstacle detection network model is faulty, and an abnormal obstacle information is indicated.

[0189] 605. Monitor whether the location range of the obstacle exceeds the sixth preset range; if it does, determine that the obstacle information is abnormal, and / or,

[0190] Optionally, embodiments of this application can also determine the location range of obstacles output by the obstacle detection network model to determine the size of the obstacles. When the location range of obstacles output by the obstacle detection network model exceeds the sixth preset range, it can be determined that the size of the obstacle is abnormal. For example, if the location range of the obstacle is: the longitudinal location range exceeds [-120, 120] meters, or the lateral location range exceeds [-60, 60] meters, or the vertical location range exceeds [-5, 5] meters, it is determined that the location range of the obstacle exceeds the detection range, and it is determined that the detected obstacle information is incorrect, or the obstacle detection network model is faulty, and an abnormal obstacle information is indicated.

[0191] Specifically, the sixth preset range in this application embodiment is related to the position range that can be sensed by various sensors mounted on the vehicle. Therefore, the sixth preset range in this application embodiment can be specifically limited according to the range sensed by the sensors.

[0192] 606. Monitor whether the speed range of the obstacle exceeds the seventh preset range. If it does, determine that the obstacle information is abnormal; and / or,

[0193] Optionally, embodiments of this application may also judge the speed range of obstacles output by the obstacle detection network model to determine whether the speed of the obstacle is abnormal, or to determine the impact of the obstacle speed on the vehicle. For example, if the longitudinal speed parameter range of the obstacle exceeds [-80, 80] m / s or the lateral speed range exceeds [-10, 10] m / s, it is determined that the speed of the obstacle is abnormal, or if the obstacle detection network model is faulty, it will prompt that the obstacle information is abnormal.

[0194] Specifically, the seventh preset range in this application embodiment is related to the speed sensed by the sensors (such as lidar) mounted on the vehicle and the empirical speed of objects measured in the actual driving scenario. It can be set comprehensively based on the sensing range of the sensors mounted on the vehicle and the empirical speed of objects in the actual driving scenario. Here, the size of the seventh preset range is not specifically limited.

[0195] 607. Monitor whether the category of the obstacle exceeds the preset category range; if it does, determine that the obstacle information is abnormal; and / or,

[0196] Optionally, embodiments of this application can also determine the category of obstacles. For example, if the obstacle detection network model normally outputs 100 types of objects, but the number of types of objects output by the obstacle detection network model exceeds the preset 100 types, it indicates that the obstacle detection network model may be malfunctioning, or the type of obstacle output may be incorrect, and the output obstacle information will be indicated as abnormal.

[0197] Specifically, in this embodiment, the preset category range is related to the types of objects that the obstacle detection network model can identify. For example, if the obstacle detection network model can identify 100 types during training, then the preset category range records 100 object types. Of course, the 100 object types here are just an explanation of the preset category range and not a limitation.

[0198] 608. Monitor whether the obstacle category and speed match; if they do not match, determine that the obstacle information is abnormal; and / or,

[0199] Optionally, embodiments of this application may also determine whether the type of obstacle matches the speed. For example, if the obstacle detection network model outputs that the type of obstacle is a static obstacle, but the speed of the obstacle is greater than zero, then it is determined that the obstacle information output by the obstacle detection network model is incorrect. Alternatively, if the obstacle detection network model is incorrect, then the output obstacle information is indicated as abnormal.

[0200] Specifically, the obstacle detection network model monitoring module can pre-store the categories of obstacles and the corresponding speed ranges for each category. Therefore, in the process of determining whether the obstacle category and speed match, the obstacle detection network model monitoring module can compare the obstacle category and speed range output by the obstacle detection network model with the pre-stored obstacle category and speed range for each category, and prompt an error message if there is a mismatch.

[0201] 609. Monitor whether the objects at the same location output by the obstacle detection network model and the object detection network model are the same. If not, determine that the obstacle information is abnormal.

[0202] Optionally, embodiments of this application can also determine whether the objects at the same location output by the obstacle detection network model and the object detection network model are the same, in order to detect whether the judgment of the same object or the same obstacle is accurate. For example, for objects at the same coordinates, if the obstacle detection network model outputs an obstacle as a bicycle, while the object detection network model outputs an object as a large dog, it means that the outputs of the two models are inconsistent for objects at the same coordinates. This indicates that the obstacle detection network model is wrong, or the output obstacle information is wrong, and then an abnormal output obstacle information is indicated.

[0203] In this embodiment, not only are the output speed, output type, output logic, and output position of obstacles judged, but also the speed and category of different positions of the same obstacle are judged, as well as whether the objects at the same position output by the obstacle detection network model and the object detection network model are the same. This further improves the effective judgment of obstacle information output by the obstacle detection network model and enhances the accuracy of obstacle information judgment.

[0204] III. Monitor whether the road element information output by the road element detection network model is abnormal.

[0205] Please see Figure 7 :

[0206] 701. Based on the position and type of lane line point chain, determine whether the lane line and lane center line intersect. If they intersect, determine that the road element information is abnormal; and / or;

[0207] Optionally, when a road element detection network model is deployed in the end-to-end model, the output objects of the road element detection network model generally include lane line point chains and lane line point chain feature values. The feature values ​​include road markings, lane line point chain markings, lane line point chain positions, and / or lane line point chain categories.

[0208] Specifically, in intelligent driving technology, lane line point chains are a technical solution that dynamically describes lane lines using a continuous sequence of coordinate points. It abstracts lane lines into a chain structure of ordered key points, accurately capturing the geometric features of lane lines (such as curvature and width) and adapting to dynamic changes (such as road diversions during construction or tidal lane adjustments). It is one of the core technologies for realizing intelligent driving perception and planning. For example, during the point chain acquisition process, the detected lane lines are uniformly sampled or sampled at key locations to generate point chains containing coordinate and direction information, which are then used to characterize the geometric features of the lane lines. Furthermore, the road element detection network model in this embodiment, combined with the coordinates and direction of the lane lines, can also output feature values ​​of the lane line point chains, such as road markings (slow down), lane markings (left-turn lanes), the category of the lane line point chains (such as lane center lines, lane edge lines, or dividing lines), and the category of lane lines (such as dashed lines or solid lines). Among these, dividing lines are generally placed between lane edge lines and lane lines, lane center lines are placed at the center of the lane, and road edge lines (lane edge lines) are placed at the road edge, such as curb markings.

[0209] When the feature values ​​of the lane line point chain include the position and category of the lane line point chain, it can be determined whether the lane line and the lane center line intersect. This is because the lane line and the lane center line are parallel to each other during vehicle movement. Therefore, when the lane line and the lane center line intersect, it indicates that the road element information output by the road element detection network model is incorrect.

[0210] It should be noted that the lane lines in this embodiment are used to divide roads into lanes, such as dividing a 50m wide road into a first lane and a second lane. The lane center lines are used to mark the center position of each lane, such as the first lane center line and the second lane center line, to assist in determining the relative position of vehicles within the lanes during travel. The type of the lane line point chain can determine whether it belongs to a lane line or a lane center line. The position of the lane line point chain can determine whether a lane line and a lane center line intersect (e.g., the position of the lane line point chain can be used to fit the corresponding lane line equation or lane center line equation, and mathematical calculations can be used to determine whether two straight lines intersect).

[0211] 702. Based on the position and type of lane line point chain, determine whether the road edge line and lane center line intersect. If they intersect, determine that the road element information is abnormal; and / or,

[0212] Optionally, embodiments of this application can also determine whether the road edge line and the lane center line intersect based on the position and type of the lane line point chain. Specifically, the road edge line is the marker of the road boundary, and the lane center line is the marker of the middle position of the lane. Therefore, when it is determined that the road edge line and the lane center line intersect based on the position and type of the lane line point chain, it is determined that the road element information output by the road element detection network model is incorrect. The process of determining whether the road edge line and the lane center line intersect can refer to the process of determining whether the lane line and the lane center line intersect in step 701 above, and will not be repeated here.

[0213] 703. Based on the position and type of lane line point chain, determine whether the dividing line intersects with the lane centerline and whether the dividing line intersects with the lane lines. If they intersect, determine that the road element information is abnormal; and / or,

[0214] Optionally, embodiments of this application can also determine whether the dividing line intersects the lane center line and the lane line based on the position and type of the lane line point chain. Specifically, in road markings, dividing lines are generally placed between lane lines and road edge lines as markers for the road edge lines. Therefore, if, during vehicle operation, the dividing line output by the road element detection network model intersects with the lane center line, or if the dividing line intersects with a lane line, then the road element information output by the road element detection network model is determined to be incorrect. The process for determining whether the dividing line intersects with the lane center line and the lane line can refer to the process for determining whether the lane line and the lane center line intersect in step 701 above, and will not be repeated here.

[0215] Specifically, when determining whether a lane line intersects with the lane center line, it can also determine whether a lane line intersects with the lane center line within a preset range (such as within 10 meters to the left and right, 90 meters forward, or 50 meters backward). If an intersection is detected, it indicates that there may be a road construction lane change in the forward, backward, or left and right directions, or that the road element detection network model is malfunctioning. In such cases, it will prompt that the road element information is incorrect to assist the vehicle in performing safe driving.

[0216] Specifically, when determining whether the road edge line intersects with the lane center line, it can also determine whether the road edge line intersects with the lane center line within a preset range (such as within 10 meters to the left and right, 90 meters forward, or 50 meters backward). If an intersection is detected, it indicates that there may be a construction lane change in the forward, backward, or left and right directions, or that the road element detection network model is malfunctioning. In such cases, it will prompt that the road element information is incorrect to assist the vehicle in performing safe driving.

[0217] Similarly, when determining whether the dividing line intersects with the lane center line or the lane line, it is also possible to determine whether the dividing line intersects with the lane center line or the lane line within a preset range.

[0218] The above-mentioned scheme for determining whether two of the lane lines, lane center lines, dividing lines, and road edge lines within a preset range intersect can determine the preset range based on the perception range of the intelligent driving system. This allows the system to determine whether the road information in the area of ​​greatest concern to the system (the area with the greatest impact on driving safety) is abnormal, reducing the computational burden and making it suitable for devices with limited computing power.

[0219] 704. Based on the lane line point chain and its feature values, determine whether the same lane line point chain corresponds to multiple lane categories; if so, determine that the road element information is abnormal; and / or,

[0220] Optionally, embodiments of this application can also determine whether the same lane line point chain corresponds to multiple lane categories based on the lane line point chain and its feature values. Lane categories include lane direction categories such as left-turn lane, right-turn lane, and both left and right-turn lanes, and lane usage categories such as bus lanes and truck lanes. Specifically, when the road element detection network model outputs lane line point chains and their feature values, the feature values ​​(such as lane markings and road markings) can be used to further determine whether the same lane line point chain output by the road element detection network model corresponds to multiple lane categories (multiple lane direction categories or multiple lane usage categories). For example, if the road element detection network model outputs that the same lane line point chain is both a left-turn lane and a right-turn lane, or that the same lane line point chain is both a bus lane and a truck lane, then the road element information output by the road element detection network model is incorrect.

[0221] 705. Based on the lane line point chain and its feature values, determine whether the same lane line point chain corresponds to multiple lane line categories. If so, determine that the road element information is incorrect; and / or,

[0222] Optionally, embodiments of this application can also determine whether the same lane line point chain corresponds to multiple lane line categories based on the lane line point chain and its feature values. Lane lines include: lane line point chain categories such as lane center lines, lane edge lines, or dividing lines, and lane line shape categories such as white dashed lines and white solid lines. Specifically, when the road element detection network model outputs lane line point chains and road markings, it can further determine whether the same lane line point chain output by the road element detection network model corresponds to multiple lane line categories (multiple lane line point chain categories or multiple lane line shape categories) based on the feature values ​​of the lane line circuit (such as the lane line point chain category and lane line shape category). For example, if the same lane line point chain is both a "white solid line" and a "yellow dashed line," or both a lane center line and a dividing line, it indicates that the road element information output by the road element detection network model is incorrect.

[0223] 706. Based on the feature values ​​of the lane line point chain, determine whether the feature values ​​are empty. If they are empty, then the road element information is incorrect.

[0224] Optionally, in this embodiment of the application, the feature value of the lane line point chain can also be used to determine whether the feature value is empty. Specifically, when the road element detection network model outputs the lane line point chain, it will also output the feature value of the lane line point chain at the same time. When the feature value of the lane line point chain output by the road element detection network model is empty, it is determined that the road element information output by the road element detection network model is incorrect.

[0225] In this embodiment, various abnormal situations of lane line point chains encountered by the vehicle during driving are judged, thereby improving the effective judgment of road element information output by the road element detection network model and improving the accuracy of road element information judgment.

[0226] IV. Monitor whether at least one of the predicted trajectories of vehicles, objects, and obstacles is abnormal;

[0227] Please see Figure 8 :

[0228] 801. Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is empty; if empty, determine that the predicted trajectory is abnormal; and / or,

[0229] Optionally, when a prediction and decision network model is deployed in the end-to-end model, and the prediction and decision network model is used to output the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle based on the object information, obstacle information, and / or road element information output by the detection network model, the obstacle detection network model, and / or the road element detection network model, then in the process of monitoring the vehicle driving state, this embodiment of the application can also monitor at least one of the predicted trajectory of the vehicle, the predicted trajectory of the object, and the predicted trajectory of the obstacle.

[0230] Because the prediction and decision network model is specifically used to predict the driving trajectory of vehicles, objects, and / or obstacles based on the object detection network model, obstacle detection network model, and road element detection network model's output information on objects, obstacles, and road elements, when the detected driving trajectory of a vehicle, object, or obstacle is empty, it is determined that the output information of the prediction and decision network model is incorrect and the predicted trajectory is abnormal.

[0231] 802. Monitor whether the predicted trajectory of vehicles, objects, and / or obstacles is a preset abnormal trajectory; if so, determine that the predicted trajectory is abnormal; and / or,

[0232] Optionally, after the prediction and decision network model outputs the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle, it can further determine whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is a preset abnormal trajectory. For example, it can determine whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is a fishhook shape or an S-shape. If the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is a fishhook shape or an S-shape or other abnormal trajectory, it is determined that the output information of the prediction and decision network model is incorrect.

[0233] It should be noted that the fishhook-shaped and S-shaped patterns in this application embodiment are merely illustrative examples of abnormal trajectories and not limitations. Any other trajectories that do not conform to the normal driving trajectory of a vehicle are considered abnormal trajectories in this application embodiment.

[0234] 803. Whether the sum of probabilities of the predicted trajectories of the monitored targets is greater than 1, where the target trajectory includes the predicted trajectory of vehicles, objects, and / or obstacles; if so, then determine that the predicted trajectory is abnormal, and / or...

[0235] Optionally, when outputting the predicted trajectories of vehicles, objects, and / or obstacles, the prediction and decision network model outputs X predicted trajectories for each target (vehicle, object, or obstacle) and evaluates the probability of each trajectory. For example, for the predicted trajectory of a vehicle, the prediction and decision network model may output 5 predicted trajectories for each vehicle and output the corresponding probability for each predicted trajectory. The model then selects the trajectory with the highest probability from all the predicted trajectories as the driving trajectory. Therefore, for the target predicted trajectory output by the prediction and decision network model, it is possible to calculate whether the sum of the probabilities of the target predicted trajectories is greater than 1. If it is greater than 1, it is determined that the output information of the prediction and decision network model is incorrect.

[0236] It should be noted that the target predicted trajectory in the embodiments of this application is any one of the predicted trajectories of vehicles, objects, and obstacles.

[0237] 804. Monitor whether the number of predicted trajectories of the target exceeds the preset value. The predicted trajectories of the target include the predicted trajectories of vehicles, objects, and / or obstacles. If so, the predicted trajectory is determined to be abnormal.

[0238] Optionally, embodiments of this application may also determine the number of target prediction trajectories output by the prediction and decision network model. For example, if the model outputs 5 prediction trajectories for each target by default, and the number of target prediction trajectories for each target output by the prediction and decision network model exceeds 5, then it is determined that the output information of the prediction and decision network model is incorrect.

[0239] Furthermore, in the embodiments of this application, the preset value of the prediction estimate generally depends on the specific value set by the prediction and decision network model during the training process. If the prediction and decision network model can only output 5 prediction trajectories during the training process, then the preset value here is 5. If it can output 10 prediction trajectories during the training process, then the preset value here is 10. The size of the preset value here mainly depends on the capability of the model, and there is no specific limitation on the size of the preset value here.

[0240] It should be noted that the target predicted trajectory in the embodiments of this application is any one of the predicted trajectories of vehicles, objects, and obstacles.

[0241] In this embodiment of the application, by monitoring and judging the predicted trajectory output by the prediction and decision network model, the correctness of the predicted trajectory of the vehicle, the predicted trajectory of the object, and the predicted trajectory of the obstacle output by the prediction and decision network model is ensured, thereby improving the safety of the vehicle during the autonomous driving process.

[0242] It should be noted that, in the embodiments of this application, when judging the above multiple models, the vehicle's driving mode can be downgraded based on the output information of one model, or the vehicle's driving mode can be downgraded jointly based on the output information of multiple models. Here, there are no specific restrictions on the judgment conditions for performing downgrade processing on the vehicle's driving model.

[0243] Specifically, in the process of controlling the vehicle's driving mode, the abnormal information output by the object detection network model monitoring module, obstacle detection network model monitoring module, road element detection network model monitoring module, and prediction and decision-making network model monitoring module can be judged separately and comprehensively. For example, a corresponding fault code can be set for the abnormal information output by each monitoring module, the vehicle's fault code can be determined based on the abnormal information output by each monitoring module, and the vehicle's driving mode can be controlled based on the vehicle's fault code. As for how to control the vehicle's driving mode based on the fault code, it is similar to that described in the previous embodiments and will not be repeated here.

[0244] Furthermore, embodiments of this application can also combine abnormal information output by multiple monitoring modules to comprehensively output vehicle fault codes, and control the vehicle's driving mode based on the vehicle's fault codes. Specifically, each fault code of an abnormal information can be assigned a corresponding weight based on the frequency, duration, or importance of the abnormal information (the more frequent, longer the duration, or the higher the importance, the greater the weight). By comprehensively considering the weights of each fault code, the final vehicle fault code can be obtained (e.g., the fault code with the highest weight or a downgraded fault code when multiple weights reach 100%).

[0245] If the object detection network model monitoring module detects anomalies in object information in 30% of consecutive X frames (e.g., 5 consecutive frames) and anomalies in predicted trajectory in 70% of consecutive Y frames (e.g., 3 consecutive frames), then the anomalies in both network models reach 100%, and the current driving mode of the vehicle is downgraded, that is, downgraded to another mode that is lower than the current mode.

[0246] Alternatively, when the output information of the first network model in an interconnected network model is abnormal, considering the impact of the first network model on other interconnected network models (abnormal information in interconnected network models is of higher importance), the current driving mode of the entire vehicle can be downgraded to improve the safety of vehicle driving. For example, in the embodiments of this application, the obstacle detection network model and the object detection network model are two interconnected network models. When one of the network models fails, the current driving mode of the entire vehicle can be downgraded to avoid affecting other network models in the interconnected network models.

[0247] It is understood that, in various embodiments of the present invention, the order of the steps does not imply the order of execution. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0248] The vehicle driving state monitoring method in the embodiments of this application has been described in detail above. The vehicle driving state monitoring device in the embodiments of this application will be described below. Please refer to [link / reference]. Figure 9 One embodiment of the vehicle driving state monitoring device in this application includes an end-to-end model deployed on the vehicle. The end-to-end model includes at least one of an object detection network model, an obstacle detection network model, and a road element detection network model. The vehicle driving state monitoring device includes:

[0249] The perception network model monitoring module 901 is used to monitor whether the perception information output by the perception network model is abnormal.

[0250] And / or,

[0251] The prediction and decision network model monitoring module 902 is used to monitor whether the prediction trajectory output by the prediction and decision network model is abnormal.

[0252] The degradation decision module 903 is used to execute a degradation decision for the vehicle if at least one of the perceived information and the predicted trajectory is abnormal; and to control the driving mode of the vehicle based on the degradation decision result.

[0253] As an optional embodiment, the perception network model includes at least one of an object detection network model, an obstacle detection network model, and a road element detection network model, and the perception network model monitoring module 901 specifically includes:

[0254] The object detection network model monitoring module 9011 is used to monitor whether the object information output by the object detection network model is abnormal.

[0255] And / or,

[0256] The obstacle detection network model monitoring module 9012 is used to monitor whether the obstacle information output by the obstacle detection network model is abnormal.

[0257] And / or,

[0258] The Road Element Detection Network Model Monitoring Module 9013 is used to monitor whether the road element information output by the road element detection network model is abnormal.

[0259] Degradation decision module 903 is specifically used for:

[0260] If at least one of the object information, obstacle information, road element information, and predicted trajectory is abnormal, a downgrade decision is made for the vehicle.

[0261] As an optional embodiment, the prediction and decision network model is used to output at least one of the predicted trajectories of vehicles, objects, and obstacles based on at least one of object information, obstacle information, and road element information.

[0262] The Prediction and Decision Network Model Monitoring Module 902 is specifically used for:

[0263] Monitor whether at least one of the predicted trajectories of the vehicle, the object, and the obstacle is abnormal;

[0264] Degradation decision module 903 is specifically used for:

[0265] If at least one of the object information, obstacle information, road element information, predicted trajectory of the vehicle, predicted trajectory of the object, and predicted trajectory of the obstacle is abnormal, a downgrade decision is made for the vehicle.

[0266] As an optional embodiment, the object information includes at least one of the following: the object's position range, the object's size range, the object's velocity range, the object's type range, the object's characteristic values, and whether the object type matches the object parameters.

[0267] The 9011 object detection network model monitoring module is specifically used for:

[0268] The monitoring function checks whether the position range of an object exceeds a first preset range in a first direction, a second preset range in a second direction, or a third preset range in a third direction, wherein the first, second, and third directions are perpendicular to each other. If any of the above exceeds the range, the object information is abnormal.

[0269] And / or,

[0270] Monitor whether the size range of the same target object exceeds a fourth preset range; if it does, determine that the object information is abnormal.

[0271] And / or,

[0272] The system monitors whether the horizontal and vertical velocity ranges of the object exceed a fifth preset range. If they do, the system determines that the object information is abnormal.

[0273] And / or,

[0274] The system monitors whether the range of object types exceeds a preset range. If it does, the system determines that the object information is abnormal.

[0275] And / or,

[0276] If the feature representation value of the monitored object is empty, it is determined that the object information is abnormal.

[0277] And / or,

[0278] The system monitors whether the object type and object parameters match. If they do not match, the object information is determined to be abnormal.

[0279] As an optional embodiment, obstacle information includes at least one of the following: speed at different locations within the obstacle, obstacle category and current driving mode, category of different locations within the same obstacle, obstacle location range, obstacle speed range, and whether the obstacle category and speed match; the obstacle detection network model monitoring module 9012 is specifically used for:

[0280] Monitor whether the velocity direction is different at different locations within the obstacle; if it is different, determine that the obstacle information is abnormal.

[0281] And / or,

[0282] The speed difference between different points in the obstacle is monitored to see if it exceeds a first preset threshold. If it does, the obstacle information is determined to be abnormal.

[0283] And / or,

[0284] The system monitors whether the type of obstacle matches the current driving mode; if they do not match, the obstacle information is determined to be abnormal.

[0285] And / or,

[0286] Monitor whether the categories of different locations of the same obstacle are different; if they are different, the obstacle information is determined to be abnormal.

[0287] And / or,

[0288] The system monitors whether the location range of obstacles exceeds the sixth preset range. If it does, the obstacle information is determined to be abnormal.

[0289] And / or,

[0290] The system monitors whether the speed range of obstacles exceeds the seventh preset range. If it does, the obstacle information is determined to be abnormal.

[0291] And / or,

[0292] Monitor whether the category of the obstacle exceeds the preset category range; if it does, determine that the obstacle information is abnormal.

[0293] And / or,

[0294] The system monitors whether the obstacle type and speed match; if they do not match, the obstacle information is determined to be abnormal.

[0295] And / or,

[0296] The system checks whether the obstacle at the same location output by the obstacle detection network model is the same as the obstacle at the same location output by the object detection network model. If they are different, the obstacle information is determined to be abnormal.

[0297] As an optional embodiment, the road element information includes lane line point chains and feature values ​​of the lane line point chains. The feature values ​​include road markings, lane markings, the location of the lane line point chains and / or the type of the lane line point chains, and the shape type of the lane lines.

[0298] The 9013 road element detection network model monitoring module is specifically used for:

[0299] Based on the position and type of lane line dot chain, determine whether the lane line and the lane center line intersect. If they intersect, then the road element information is determined to be abnormal.

[0300] And / or,

[0301] Based on the position and type of lane line point chain, determine whether the road edge line and lane center line intersect. If they intersect, then the road element information is determined to be abnormal.

[0302] And / or,

[0303] Based on the position and type of lane line point chain, determine whether the dividing line intersects with the center line of the lane and the lane line. If they intersect, the road element information is determined to be abnormal.

[0304] And / or,

[0305] Based on the feature values ​​of lane line point chains and lane line point chains, determine whether the same lane line point chain corresponds to multiple lane categories. If so, determine that the road element information is abnormal.

[0306] And / or,

[0307] Based on the feature values ​​of lane line point chains and lane line point chains, determine whether the same lane line point chain corresponds to multiple lane line categories. If so, determine that the road element information is abnormal.

[0308] And / or,

[0309] Based on the feature values ​​of the lane line point chain, determine whether the feature values ​​are empty. If they are empty, then the road element information is determined to be abnormal.

[0310] As an optional embodiment, the prediction and decision network model monitoring module 805 is specifically used for:

[0311] Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is empty. If it is empty, the predicted trajectory is determined to be abnormal.

[0312] And / or,

[0313] Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle is a preset abnormal trajectory. If so, determine that the predicted trajectory is abnormal.

[0314] And / or,

[0315] The total probability of the predicted trajectory of the monitored target is greater than 1. The target trajectory includes the predicted trajectory of the vehicle, the predicted trajectory of the object, or the predicted trajectory of the obstacle. If so, the predicted trajectory is determined to be abnormal.

[0316] And / or,

[0317] The system monitors whether the number of predicted trajectories for the target exceeds a preset value. The predicted trajectories include those for vehicles, objects, or obstacles. If so, the predicted trajectory is determined to be abnormal.

[0318] As an optional embodiment, the degradation decision module 903 is specifically used for:

[0319] If at least one of the perceived information and the predicted trajectory is abnormal, the vehicle's fault information is determined based on the number of times the abnormality occurs and the duration of the abnormality.

[0320] Based on the vehicle's fault information, the vehicle's driving mode is downgraded.

[0321] As an optional implementation, downgraded driving includes:

[0322] Limit the speed of vehicles;

[0323] And / or,

[0324] Limit the distance between vehicles;

[0325] And / or,

[0326] Activation of some intelligent driving functions is restricted;

[0327] And / or,

[0328] Switch the vehicle's driving mode;

[0329] And / or,

[0330] Remind the driver of the vehicle;

[0331] And / or,

[0332] Exit the intelligent driving function.

[0333] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0334] In this embodiment, the perception information and / or predicted trajectory information output by the perception network model and the prediction and decision network model are monitored, and when any output information is abnormal, the driving mode of the vehicle is downgraded to achieve safety monitoring and downgrade decision-making of the end-to-end network, which can improve the safety of intelligent driving vehicles with end-to-end models during driving.

[0335] This application also provides a computer program product on which a computer program is stored, wherein when the computer program is executed by a processor, it is used to implement the various steps in the above-described method embodiments of this application.

[0336] This application also provides an intelligent driving vehicle; please refer to [link / reference]. Figure 10 The system includes at least a controller 1001, which has a built-in computer program. When executed by the controller 1001, the computer program is used to implement the vehicle driving state monitoring method in the above-described method embodiments. Furthermore, the intelligent driving vehicle may also include a memory 1002 and a communication interface 1003. The communication interface 1003 can acquire sensor data, and the memory 1002 can store sensor calibration data or positioning information, etc., for use by the controller 1001 when executing the computer program.

[0337] Intelligent driving vehicles can be mobile platforms such as vehicles equipped with intelligent driving systems, mobile robots, and aircraft. Vehicles can be driverless or manned, and aircraft can be drones, etc.

[0338] The vehicle in the embodiments of the present invention has been described above. The controller in the embodiments of the present invention will be described below from the perspective of hardware processing:

[0339] One embodiment of the controller in this invention includes:

[0340] Processor, memory, and computer programs stored in memory;

[0341] The memory is used to store computer programs, and when the processor executes the computer programs stored in the memory, it can implement the various steps in the above method embodiments.

[0342] It is understood that when the processor in the computer device described above executes the computer program, it can also realize the functions of each unit in the corresponding device embodiments described above, which will not be repeated here. For example, the computer program can be divided into one or more modules / units, one or more of which are stored in memory and executed by the processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a vehicle. For example, the computer program can be divided into units in the vehicle described above, and each unit can realize the specific functions as described in the corresponding vehicle description above.

[0343] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the processor and memory are merely examples of a computer device and do not constitute a limitation on the computer device. It may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0344] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0345] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0346] The present invention also provides a computer-readable storage medium for implementing vehicle functions, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the processor can be used to perform the various steps in the above method embodiments.

[0347] It is understood that if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a corresponding computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0348] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0349] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0350] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0351] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0352] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring the driving status of a vehicle, characterized in that, The vehicle is equipped with an end-to-end model, which includes a perception network model and a prediction and decision network model. The method includes: Monitor whether the perception information output by the perception network model is abnormal; and / or monitor whether the predicted trajectory output by the prediction and decision network model is abnormal; If at least one of the perceived information and the predicted trajectory is abnormal, a downgrade decision is made for the vehicle, and the driving mode of the vehicle is controlled according to the downgrade decision result.

2. The method according to claim 1, characterized in that, The perception network model includes at least one of an object detection network model, an obstacle detection network model, and a road element detection network model, and the monitoring of whether the perception information output by the perception network model is abnormal includes: Monitor whether the object information output by the object detection network model is abnormal; And / or, Monitor whether the obstacle information output by the obstacle detection network model is abnormal; And / or, Monitor whether the road element information output by the road element detection network model is abnormal; If at least one of the perceived information and the predicted trajectory is abnormal, a degrading decision is made for the vehicle, including: If at least one of the object information, obstacle information, road element information, and predicted trajectory is abnormal, a downgrade decision is made for the vehicle.

3. The method according to claim 1, characterized in that, The prediction and decision network model is used to output at least one of the predicted trajectories of the vehicle, the object, and the obstacle based on at least one of the object information, the obstacle information, and the road element information. Monitoring whether the predicted trajectory output by the prediction and decision network model is abnormal includes: Monitor whether at least one of the predicted trajectories of the vehicle, the object, and the obstacle is abnormal; If at least one of the perceived information and the predicted trajectory is abnormal, a degrading decision is made for the vehicle, including: If at least one of the object information, obstacle information, road element information, predicted trajectory of the vehicle, predicted trajectory of the object, and predicted trajectory of the obstacle is abnormal, a downgrade decision is made for the vehicle.

4. The method according to claim 2, characterized in that, The object information includes at least one of the following: the object's position range, the object's size range, the object's velocity range, the object's type range, the object's characteristic values, and whether the object type matches the object parameters. The monitoring of whether the object information output by the object detection network model is abnormal includes: The position range of the object is monitored to see if it exceeds a first preset range in a first direction, a second preset range in a second direction, or a third preset range in a third direction, wherein the first direction, the second direction, and the third direction are perpendicular to each other. If any of the above exceeds the range, the object information is determined to be abnormal. And / or, Monitor whether the size range of the same target object exceeds a fourth preset range; if it does, determine that the object information is abnormal. And / or, Monitor whether the horizontal and vertical velocity range of the object exceeds the fifth preset range. If it does, determine that the object information is abnormal. And / or, The system monitors whether the range of object types exceeds a preset range. If it does, the system determines that the object information is abnormal. And / or, Monitor whether the feature representation value of the object is empty; if it is empty, determine that the object information is abnormal. And / or, The system monitors whether the object type and object parameters match. If they do not match, the object information is determined to be abnormal.

5. The method according to claim 2, characterized in that, The obstacle information includes at least one of the following: speed at different locations within the obstacle, obstacle category and current driving mode, category of different locations within the same obstacle, obstacle location range, obstacle speed range, and whether the obstacle category and speed match. The monitoring of whether the obstacle information output by the obstacle detection network model is abnormal includes: Monitor whether the velocity directions of different points in the obstacle are different; if they are different, determine that the obstacle information is abnormal. And / or, Monitor whether the speed difference between different points in the obstacle exceeds a first preset threshold. If it does, determine that the obstacle information is abnormal. And / or, The system monitors whether the category of the obstacle matches the current driving mode; if they do not match, the obstacle information is determined to be abnormal. And / or, Monitor whether the categories of different locations of the same obstacle are different; if they are different, determine that the obstacle information is abnormal. And / or, Monitor whether the location range of the obstacle exceeds the sixth preset range; if it does, determine that the obstacle information is abnormal. And / or, Monitor whether the speed range of the obstacle exceeds the seventh preset range; if it does, determine that the obstacle information is abnormal. And / or, Monitor whether the category of the obstacle exceeds the preset category range; if it does, determine that the obstacle information is abnormal. And / or, Monitor whether the type and speed of the obstacle match; if they do not match, determine that the obstacle information is abnormal. And / or, The system detects whether the obstacle at the same location output by the obstacle detection network model is the same as the obstacle at the same location output by the object detection network model. If they are different, the obstacle information is determined to be abnormal.

6. The method according to claim 2, characterized in that, The road element information includes lane line point chains and lane line point chain feature values. The feature values ​​include road markings, lane markings, the position of lane line point chains and / or the type of lane line point chains, and the shape type of lane lines. The monitoring of whether the road element information output by the road element detection network model is abnormal includes: Based on the position and type of the lane line dot chain, determine whether the lane line and the lane center line intersect. If they intersect, determine that the road element information is abnormal. And / or, Based on the position and type of the lane line dot chain, determine whether the road edge line and the lane center line intersect. If they intersect, determine that the road element information is abnormal. And / or, Based on the position and type of the lane line dot chain, determine whether the dividing line intersects with the center line of the lane and the lane line. If they intersect, determine that the road element information is abnormal. And / or, Based on the lane line point chain and the feature value of the lane line point chain, determine whether the same lane line point chain corresponds to multiple lane categories. If so, determine that the road element information is abnormal. And / or, Based on the lane line point chain and the feature value of the lane line point chain, determine whether the same lane line point chain corresponds to multiple lane line categories. If so, determine that the road element information is abnormal. And / or, Based on the feature value of the lane line point chain, determine whether the feature value is empty. If it is empty, determine that the road element information is abnormal.

7. The method according to claim 3, characterized in that, The monitoring of whether at least one of the predicted trajectory of the vehicle, the predicted trajectory of the detected object, and the predicted trajectory of the detected obstacle is abnormal includes: Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle are empty; if they are empty, determine that the predicted trajectory is abnormal. And / or, Monitor whether the predicted trajectory of the vehicle, the predicted trajectory of the object, and / or the predicted trajectory of the obstacle are preset abnormal trajectories; if so, determine that the predicted trajectory is abnormal. And / or, The total probability of the predicted trajectory of the monitored target is greater than 1. The target trajectory includes the predicted trajectory of the vehicle, the predicted trajectory of the object, or the predicted trajectory of the obstacle. If so, the predicted trajectory is determined to be abnormal. And / or, The system monitors whether the number of predicted trajectories of the monitored targets exceeds a preset value. The predicted trajectories of the targets include the predicted trajectories of the vehicles, the objects, or the obstacles. If so, the predicted trajectories are determined to be abnormal.

8. The method according to any one of claims 1 to 7, characterized in that, If at least one of the perceived information and the predicted trajectory is abnormal, a degrade decision is made for the vehicle, and based on the degrade decision result, the driving mode of the vehicle is controlled, including: If at least one of the perceived information and the predicted trajectory is abnormal, the vehicle's fault information is determined based on the number of times the abnormality occurs and the duration of the abnormality. Based on the vehicle's fault information, the vehicle's driving mode is downgraded.

9. The method according to claim 7, characterized in that, The downgrade process includes: The speed of the vehicle shall be limited; And / or, The distance between the vehicles is limited; And / or, Activation of some intelligent driving functions is restricted; And / or, Perform a driving mode switch on the vehicle; And / or, Remind the driver of the vehicle; And / or, Exit the intelligent driving function.

10. A vehicle driving status monitoring device, characterized in that, The vehicle is equipped with an end-to-end model, which includes at least a perception network model and a prediction and decision network model. The monitoring device includes: The perception network model monitoring module is used to monitor whether the perception information output by the perception network model is abnormal. and / or; The prediction and decision network model monitoring module is used to monitor whether the prediction trajectory output by the prediction and decision network model is abnormal. A degradation decision module is used to make a degradation decision for the vehicle if at least one of the perceived information and the predicted trajectory is abnormal, and to control the driving mode of the vehicle based on the degradation decision result.

11. A controller, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory, wherein when the processor executes the computer program, it is used to implement the method for monitoring the driving state of a vehicle as described in any one of claims 1 to 9.

12. An intelligent driving vehicle, characterized in that, It includes at least the controller as described in claim 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it is used to implement the method for monitoring the vehicle driving state as described in any one of claims 1 to 9.

14. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the controller, it is used to implement the vehicle driving state monitoring method as described in any one of claims 1 to 9.

Images