Fault identification method and apparatus for autonomous driving system

Abstract

The present application provides a fault identification method and apparatus for an autonomous driving system. The method comprises: acquiring travelling-related data of an ego vehicle (at least including ego vehicle travelling data and target obstacle motion data); respectively performing feature extraction on the travelling-related data of the ego vehicle on the basis of a plurality of feature extraction networks to obtain feature vectors of the ego vehicle from a plurality of viewing angles; fusing the feature vectors from the plurality of viewing angles to obtain a global driving scenario feature vector of the ego vehicle; and on the basis of the global driving scenario feature vector, determining a target driving scenario where the ego vehicle is located in a current driving mode and whether a fault occurs to an autonomous driving system in the target driving scenario. In the present application, dynamic change information of an ego vehicle and a surrounding environment of the ego vehicle is captured from a plurality of viewing angles, the information from the plurality of viewing angles is adaptively fused to obtain comprehensive representations of driving scenarios, and a target driving scenario where the ego vehicle is currently located and whether a fault occurs to an autonomous driving system in the target driving scenario are identified on the basis of the comprehensive representations, thereby improving the accuracy of fault identification results of the autonomous driving system.

Description

A method and apparatus for fault identification in an autonomous driving system

[0001] This application claims priority to Chinese Patent Application No. 202410591542.4, filed on May 13, 2024, entitled "A Fault Identification Method and Apparatus for an Autonomous Driving System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of autonomous driving technology, and in particular to a fault identification method and apparatus for an autonomous driving system. Background Technology

[0003] In recent years, with the continuous development of autonomous driving (also known as intelligent driving) technology and the widespread application of 5G communication technology, intelligent connected vehicles have become an important development trend in the automotive industry. Software is gradually becoming the core of automobiles, and the demand for remote vehicle maintenance capabilities is becoming increasingly prominent. Various remote maintenance platforms have sprung up like mushrooms after rain, but most platforms in the industry differ significantly in terms of technology and capabilities. This has led to uneven remote maintenance capabilities among major automakers, and their safety and reliability are difficult to guarantee. Among these, timely identification of problems in the autonomous driving system under various driving scenarios is crucial for the remote maintenance of autonomous vehicles, timely detection of vehicle software vulnerabilities, and over-the-air (OTA) system upgrades.

[0004] However, existing solutions have many problems in fault identification for autonomous driving systems, such as poor accuracy in fault identification and inaccurate identification of the driving scenario. Summary of the Invention

[0005] The embodiments of this application provide a fault identification method and apparatus for an autonomous driving system. It captures dynamic change information of the vehicle and its surrounding environment from multiple perspectives, adaptively fuses multi-perspective information to obtain a comprehensive representation of the driving scenario, and identifies the target driving scenario in which the vehicle is currently located and whether the intelligent driving system is malfunctioning in the target driving scenario, thereby increasing the accuracy of fault identification results of the autonomous driving system.

[0006] Firstly, this application provides a fault identification method for an autonomous driving system, comprising: acquiring driving-related data of the vehicle, which includes at least driving data of the vehicle and motion data of a target obstacle, wherein the target obstacle is an obstacle around the vehicle that affects the driving of the vehicle; extracting features from the driving-related data of the vehicle based on multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle, wherein the feature vectors from multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, wherein the predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle; fusing the feature vectors from multiple perspectives to obtain a global driving scene feature vector of the vehicle; and determining, based on the global driving scene feature vector, whether the target driving scene in which the vehicle is located under the current driving mode and whether the autonomous driving system is faulty in the target driving scene.

[0007] This application captures the interactive features of the predicted trajectory, simulates the perspective of the control module to "predict" whether there is a risk of trajectory intersection and collision between the vehicle and obstacles, captures the interactive features of the real trajectory, obtains the real trajectory relationship between the vehicle and obstacles (including whether they intersect), and combines the predicted trajectory and the real trajectory to improve the accuracy of fault identification results of the autonomous driving system.

[0008] In one possible implementation, the vehicle's driving data includes the vehicle's predicted trajectory data and the vehicle's actual trajectory data. The predicted trajectory data indicates the vehicle's predicted driving trajectory within a target time period, and the actual trajectory data indicates the vehicle's actual driving trajectory within the target time period. The target obstacle's motion data includes the target obstacle's motion trajectory data, which indicates the target obstacle's motion trajectory within the target time period. A specific implementation of extracting features from the vehicle's driving-related data using multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle is as follows: the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as inputs to a first feature extraction network, which outputs a predicted trajectory interaction feature vector; the actual trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as inputs to a second feature extraction network, which outputs an actual trajectory interaction feature vector.

[0009] In this possible implementation, the complex changes and mutual influences of the trajectories of the vehicle and the target obstacle are fully modeled, and a trajectory interaction feature extraction network is designed. This network can effectively capture the dynamic influence relationship between the target obstacle trajectory and the vehicle's driving trajectory, thereby helping subsequent steps to accurately identify the fault results of the autonomous driving system.

[0010] In another possible implementation, a specific implementation of using the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as input to the first feature extraction network and outputting a predicted trajectory interaction feature vector is as follows: the predicted trajectory data of the vehicle is used as input to the first temporal representation network, which outputs a predicted driving trajectory feature vector of the vehicle; the motion trajectory data of the target obstacle is used as input to the second temporal representation network, which outputs a motion trajectory feature vector of the target obstacle; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a first trajectory fusion feature vector; the first trajectory fusion feature vector is used as input to the first trajectory interaction network, which outputs a predicted trajectory interaction feature vector.

[0011] In another possible implementation, a specific implementation of using the vehicle's real trajectory data and the target obstacle's motion trajectory data as input to the second feature extraction network to output a real trajectory interaction feature vector is as follows: The vehicle's real trajectory data is used as input to the third temporal representation network to output the vehicle's real driving trajectory feature vector; the target obstacle's motion trajectory data is used as input to the fourth temporal representation network to output the target obstacle's motion trajectory feature vector; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; the second trajectory fusion feature vector is used as input to the second trajectory interaction network to output a real trajectory interaction feature vector.

[0012] The vehicle's trajectory features and the target obstacle's motion trajectory features are extracted separately through a temporal representation network. Then, the two trajectory features are concatenated and input into a trajectory interaction network to accurately extract the interaction features between the vehicle's trajectory and the target obstacle's motion trajectory, thus obtaining the correlation between the vehicle's trajectory and the target obstacle's motion trajectory, that is, the influence of the target obstacle's dynamic motion on the vehicle's trajectory.

[0013] Optionally, the first temporal representation network, the second temporal representation network, the third temporal representation network, and the fourth temporal representation network are all constructed based on a temporal convolutional network (TCN); and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on a multi-layer perceptron (MLP) network.

[0014] Of course, in some other examples, the temporal representation network (including the first temporal representation network, the second temporal representation network, the third temporal representation network, and the fourth temporal representation network) and the trajectory interaction network (including the first trajectory interaction network and the second trajectory interaction network) can also be implemented by other neural network models. For example, the temporal representation network can be a recurrent neural network (RNN) or a Transformer neural network model, and the trajectory interaction network can be a fully connected neural network (FCN), etc. This application does not limit the specific implementation structure of the temporal representation network and the trajectory interaction network.

[0015] In another possible implementation, a specific method for fusing feature vectors from multiple perspectives to obtain the global driving scene feature vector of the vehicle is as follows: determine the first attention weight corresponding to the predicted trajectory interaction feature vector and the second attention weight corresponding to the real trajectory interaction feature vector; based on the first attention weight and the second attention weight, perform weighted fusion of the predicted trajectory interaction feature vector and the real trajectory interaction feature vector to obtain the global driving scene feature vector of the vehicle.

[0016] By adaptively fusing feature vectors from multiple perspectives and considering the influence of different perspectives, a perspective dimension attention mechanism is designed to weightedly fuse information from multiple perspectives, thereby obtaining a global representation of the driving scene and improving the accuracy of intelligent driving fault identification.

[0017] In another possible implementation, the target obstacle is retrieved from the vehicle's operation log. For example, in a specific scenario, the vehicle automatically records and uploads the target obstacle ID, which can then be retrieved from the vehicle's operation log when determining the target obstacle. For instance, in a scenario where advanced emergency braking (AEB) is falsely triggered, the vehicle experiences a sudden braking or stop, and automatically records obstacles before and after the sudden braking in the vehicle log (e.g., obstacles in front of the vehicle). Alternatively, the target obstacle can be selected from obstacles around the vehicle based on preset rules. For example, the target obstacle can be determined based on the distance of the obstacles around the vehicle, such as identifying the closest obstacles within a 50-meter radius of the vehicle as the target obstacle.

[0018] In another possible implementation, the vehicle's driving-related data also includes road environment data, which indicates the road environment information in which the vehicle is driving; the feature vectors from multiple perspectives also include road environment feature vectors, and the vehicle's global driving scene feature vectors are also related to the road environment feature vectors, which are obtained by feature extraction from the road environment data.

[0019] By taking into account the influence of road environment information, and fully modeling the complex state changes and mutual influence relationships of the vehicle, target obstacles and road environment, a more comprehensive representation of the driving scenario can be obtained, thereby further improving the accuracy of fault identification in the autonomous driving system.

[0020] Optionally, road environment data may include one or more of the following: lane number, lane type, distance from intersection, and distance from traffic light.

[0021] In another possible implementation, the vehicle driving data also includes the vehicle's driving state data, which includes one or more of the vehicle's intelligent driving state, speed, acceleration, position coordinates, orientation angle, and braking value; and / or, the target obstacle motion data also includes the target obstacle's motion state data, which includes one or more of the target obstacle's speed, acceleration, and position coordinates.

[0022] In another possible implementation, the feature vectors from multiple perspectives also include driving scene feature vectors; the driving scene feature vectors are obtained by feature extraction from first driving scene data and / or second driving scene data; wherein, the first driving scene data includes the result data calculated based on driving data and target obstacle motion data based on dynamic equations and / or prior knowledge, and the result data is related to the driving mode of the vehicle; the second driving scene data includes the vehicle driving data and target obstacle motion data corresponding to the time of failure, and the time of failure indicates the time when the vehicle's driving may be abnormal; the global driving scene feature vector of the vehicle is also related to the driving scene feature vector.

[0023] By leveraging prior knowledge (such as certain rules) and / or dynamic equations to enhance driving scene features, for example by extracting fault point information (i.e., second driving scene data), focusing on changes in the vehicle and its surrounding environment before and after a fault, amplifying fault modes, and thus improving the accuracy of fault identification in autonomous driving systems.

[0024] In another possible implementation, the resulting data includes one or more of the following: the collision time between the vehicle and the target obstacle, the predicted trajectory of the vehicle, the duration of the target obstacle, whether the target obstacle is a phantom obstacle, whether autonomous braking is performed, braking value, and steering angle.

[0025] Based on perception and control experience, the system calculates the collision time between the vehicle and the target obstacle, whether the target obstacle is a phantom obstacle, and whether the vehicle brakes autonomously, thus constructing scene information and improving the richness of scene information.

[0026] Secondly, this application also provides a driving risk assessment method, including acquiring driving-related data of a vehicle, which includes at least driving data of the vehicle and motion data of a target obstacle, wherein the target obstacle is an obstacle around the vehicle that affects the driving of the vehicle; extracting features from the driving-related data of the vehicle based on multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle, wherein the feature vectors from multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, wherein the predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle; fusing the feature vectors from multiple perspectives to obtain a global driving scene feature vector of the vehicle; and determining the driving risk assessment result based on the global driving scene feature vector.

[0027] In other words, the global driving scene features obtained by multi-view feature fusion in this application can also be used for driving risk assessment to obtain more accurate driving risk assessment results.

[0028] Optionally, the driving risk assessment results may include one or more of the following: traffic accident risk assessment results, vehicle risk assessment results, and driver driving risk assessment results.

[0029] Thirdly, this application also provides a fault identification device for an autonomous driving system, including a first acquisition module, a first multi-view feature extraction module, a first feature fusion module, and a fault identification module. The first acquisition module acquires driving-related data of the vehicle, which includes at least driving data of the vehicle and motion data of a target obstacle. The target obstacle is an obstacle surrounding the vehicle that affects its driving. The first multi-view feature extraction module extracts features from the driving-related data of the vehicle using multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle. These feature vectors include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, while the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The first feature fusion module fuses the feature vectors from multiple perspectives to obtain a global driving scene feature vector for the vehicle. The fault identification module determines, based on the global driving scene feature vector, whether the vehicle is in the target driving scene under the current driving mode and whether the autonomous driving system is faulty in the target driving scene.

[0030] In one possible implementation, the vehicle's driving data includes the vehicle's predicted trajectory data and the vehicle's actual trajectory data. The predicted trajectory data indicates the vehicle's predicted driving trajectory data within a target time period, and the actual trajectory data indicates the vehicle's actual driving trajectory data within the target time period. The target obstacle's motion data includes the target obstacle's motion trajectory data, which indicates the target obstacle's motion trajectory data within the target time period. The first multi-view feature extraction module is specifically used to: take the vehicle's predicted trajectory data and the target obstacle's motion trajectory data as input to a first feature extraction network and output a predicted trajectory interaction feature vector; take the vehicle's actual trajectory data and the target obstacle's motion trajectory data as input to a second feature extraction network and output an actual trajectory interaction feature vector.

[0031] In another possible implementation, a specific implementation of using the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as input to the first feature extraction network and outputting a predicted trajectory interaction feature vector is as follows: the predicted trajectory data of the vehicle is used as input to the first temporal representation network, which outputs a predicted driving trajectory feature vector of the vehicle; the motion trajectory data of the target obstacle is used as input to the second temporal representation network, which outputs a motion trajectory feature vector of the target obstacle; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a first trajectory fusion feature vector; the first trajectory fusion feature vector is used as input to the first trajectory interaction network, which outputs a predicted trajectory interaction feature vector.

[0032] In another possible implementation, a specific implementation of using the vehicle's real trajectory data and the target obstacle's motion trajectory data as input to the second feature extraction network to output a real trajectory interaction feature vector is as follows: The vehicle's real trajectory data is used as input to the third temporal representation network to output the vehicle's real driving trajectory feature vector; the target obstacle's motion trajectory data is used as input to the fourth temporal representation network to output the target obstacle's motion trajectory feature vector; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; the second trajectory fusion feature vector is used as input to the second trajectory interaction network to output a real trajectory interaction feature vector.

[0033] Optionally, the first temporal representation network, the second temporal representation network, the third temporal representation network, and the fourth temporal representation network are all constructed based on TCN; and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on MLP network.

[0034] In another possible implementation, the first feature fusion module is specifically used to: determine the first attention weight corresponding to the predicted trajectory interaction feature vector and the second attention weight corresponding to the real trajectory interaction feature vector; and, based on the first attention weight and the second attention weight, perform weighted fusion of the predicted trajectory interaction feature vector and the real trajectory interaction feature vector to obtain the vehicle's global driving scene feature vector.

[0035] In another possible implementation, the target obstacle is retrieved from the vehicle's operation log; or, the target obstacle is selected from obstacles around the vehicle based on preset rules.

[0036] In another possible implementation, the vehicle's driving-related data also includes road environment data, which indicates the road environment information in which the vehicle is driving; the feature vectors from multiple perspectives also include road environment feature vectors, and the vehicle's global driving scene feature vectors are also related to the road environment feature vectors, which are obtained by feature extraction from the road environment data.

[0037] Optionally, road environment data may include one or more of the following: lane number, lane type, distance from intersection, and distance from traffic light.

[0038] In another possible implementation, the vehicle driving data may also include one or more of the vehicle's intelligent driving status, speed, acceleration, position coordinates, orientation angle, and braking value; and / or, the target obstacle motion data may also include one or more of the target obstacle's speed, acceleration, and position coordinates.

[0039] In another possible implementation, the feature vectors from multiple perspectives also include driving scene feature vectors; the driving scene feature vectors are obtained by feature extraction from first driving scene data and / or second driving scene data; wherein, the first driving scene data includes the result data calculated based on driving data and target obstacle motion data based on dynamic equations and / or prior knowledge, and the result data is related to the driving mode of the vehicle; the second driving scene data includes the vehicle driving data and target obstacle motion data corresponding to the time of failure, and the time of failure indicates the time when the vehicle's driving may be abnormal; the global driving scene feature vector of the vehicle is also related to the driving scene feature vector.

[0040] In another possible implementation, the resulting data includes one or more of the following: the collision time between the vehicle and the target obstacle, the predicted trajectory of the vehicle, whether the target obstacle is a phantom obstacle, whether autonomous braking is performed, the duration of the target obstacle, the braking value, and the steering angle.

[0041] Fourthly, this application also provides a driving risk assessment device, including a second acquisition module, a second multi-view feature extraction module, a second feature fusion module, and a risk assessment module. The second acquisition module acquires driving-related data of the vehicle, which includes at least the vehicle's driving data and the motion data of a target obstacle. The target obstacle is any obstacle around the vehicle that affects the vehicle's driving. The second multi-view feature extraction module extracts features from the driving-related data of the vehicle from multiple perspectives to obtain feature vectors for multiple perspectives. These feature vectors include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, while the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The second feature fusion module fuses the feature vectors from multiple perspectives to obtain a global driving scene feature vector for the vehicle. The risk assessment module determines the driving risk assessment result based on the global driving scene feature vector.

[0042] Optionally, the driving risk assessment results may include one or more of the following: traffic accident risk assessment results, vehicle risk assessment results, and driver driving risk assessment results.

[0043] Fifthly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the methods described in the first aspect and / or the second aspect to be implemented.

[0044] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first or second aspect to be implemented.

[0045] In a seventh aspect, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the method described in the first aspect or the second aspect.

[0046] Eighthly, embodiments of this application also provide a chip including at least one processor and a communication interface, the processor being configured to execute the method described in the first or second aspect. Attached Figure Description

[0047] Figure 1 is a functional block diagram of the vehicle provided in an embodiment of this application;

[0048] Figure 2 is a schematic diagram of the computer system provided in an embodiment of this application;

[0049] Figure 3 is a framework diagram of the autonomous driving system provided in this application;

[0050] Figure 4 shows a system architecture diagram of a fault identification method system for an autonomous driving system provided in an embodiment of this application.

[0051] Figure 5 is a flowchart illustrating a fault identification method for an autonomous driving system provided in an embodiment of this application;

[0052] Figure 6 shows a schematic diagram of the system implementation architecture of a fault identification method for an autonomous driving system provided in an embodiment of this application;

[0053] Figure 7 shows a schematic diagram of the input data processing process of a fault identification model;

[0054] Figure 8 is a structural schematic diagram of a fault identification device for an autonomous driving system provided in an embodiment of this application;

[0055] Figure 9 is a schematic diagram of the structure of a driving risk assessment device provided in an embodiment of this application. Detailed Implementation

[0056] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0057] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature extraction network" and "second feature extraction network," etc., are used to distinguish different feature extraction networks, not to describe a specific order of feature extraction networks.

[0058] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0060] To facilitate understanding of the solutions in the embodiments of this application, the technical terms involved in this document will be explained first.

[0061] Autonomous vehicles (also known as driverless cars, computer-driven cars, intelligent cars, or wheeled mobile robots) are intelligent vehicles that achieve driverless operation through computer systems. Autonomous vehicles rely on the collaborative efforts of artificial intelligence, computer vision, radar, monitoring devices, and global positioning systems to enable computers to automatically and safely operate motor vehicles without any active human intervention.

[0062] Autonomous driving system fault identification: Based on various data reported by autonomous vehicles, identify the driving scenario in which the autonomous vehicle is located, and further identify whether the autonomous driving system is handling the scenario normally. Fault events can correspond to the autonomous driving system's operation such as incorrect emergency braking, failure to decelerate in time leading to vehicle collision, and incorrect identification of obstacles.

[0063] Time to Collision (TTC): This refers to the time it takes for a vehicle to reach the vehicle in front at its current speed, usually measured in seconds.

[0064] AEB: A system that monitors the driving environment ahead of the vehicle and automatically activates the vehicle's braking system to slow the vehicle down in order to avoid or mitigate a collision when a collision is possible.

[0065] Adaptive cruise control (ACC) is a comfort-oriented driver assistance feature. If the road ahead is clear, adaptive cruise control will maintain the set maximum cruising speed. If a vehicle is detected ahead, adaptive cruise control will reduce speed as needed, maintaining a distance from the vehicle ahead based on a selected time interval until a suitable cruising speed is reached.

[0066] There are many problems with the technical solutions for fault identification in autonomous driving systems. For example, in one related technology, an AI module is used to process trajectory data, vibration, sound and other signals; a rule base is built, and the processed signals are passed through the rule module, the results of each rule module are integrated, and then the fault is identified through higher-order rules.

[0067] However, the relevant technologies face the challenge of complex and ever-changing autonomous driving scenarios, which include various operational characteristics such as navigation cruise assist, ACC, and AEB, with more than a hundred fault scenarios. Rules / policies obtained manually through analysis of a small amount of data often can only cover a small part of the scenarios or some situations within a scenario, resulting in low fault identification accuracy. Furthermore, the descriptions of some fault scenarios are extremely complex or even impossible to describe, such as fault scenarios involving subjective human feelings like "unreasonable" or "unexpected," which cannot be quantified using rules.

[0068] Related technology two involves adjusting the model's receptive field by combining convolutions with various kernel sizes, simultaneously capturing multi-scale information from temporal data. The structure can be flexibly extended and integrated with other deep learning frameworks. However, when this approach is extended to fault identification in autonomous driving operation and maintenance scenarios, it cannot simultaneously process multiple sets of time-series (e.g., trajectories) and non-time-series data of variable length (e.g., scenario rule-based judgments such as whether it's a curve or a ghost obstacle), and it cannot achieve interaction between vehicle status information and dynamic environmental information.

[0069] In addition, fault identification in autonomous driving systems faces challenges such as each business area focusing on its own internal positioning, strong coupling between different modules in real-world scenarios, and difficulty in determining which fault level or module is at fault based on the results of a single module. Manually defining boundary rules is time-consuming and labor-intensive, requiring significant development manpower to maintain and update the rule base.

[0070] Therefore, embodiments of this application provide a fault identification method and apparatus for an autonomous driving system. It captures dynamic change information of the vehicle and surrounding obstacles from multiple perspectives, adaptively fuses multi-perspective information to obtain a comprehensive representation of the driving scene, and identifies the target driving scene in which the vehicle is currently located and whether the intelligent driving system is faulty in the target driving scene. Without the need for a manually maintained rule base, it can accurately identify the target driving scene in which the vehicle is currently located and whether the autonomous driving system is faulty in the target driving scene.

[0071] The following detailed description, with reference to the accompanying drawings, illustrates the specific implementation of a fault identification method for an autonomous driving system provided in this application.

[0072] Figure 1 is a functional block diagram of a vehicle provided in an embodiment of this application. As shown in Figure 1, the vehicle 100 includes various subsystems, such as a driving system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116.

[0073] Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0074] The propulsion system 102 includes components that provide powered motion to the vehicle 100. In one example, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels 121. The engine 118 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 118 converts the energy source 119 into mechanical energy.

[0075] Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide energy to other systems of vehicle 100.

[0076] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission may include a gearbox, a differential, and a drive shaft.

[0077] In one example, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

[0078] The sensor system 104 may include several sensors that sense information about the environment surrounding the vehicle 100.

[0079] For example, sensor system 104 may include positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), inertial measurement unit (IMU) 124, radar 126, laser rangefinder 128, and camera 130. Sensor system 104 may also include sensors from the vehicle 100's internal systems (e.g., in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a critical function for the safe operation of vehicle 100.

[0080] The positioning system can be used to estimate the geographic location of vehicle 100. IMU 124 is used to sense changes in the position and orientation of vehicle 100 based on inertial acceleration. In one example, IMU 124 can be a combination of an accelerometer and a gyroscope.

[0081] Radar 126 can use radio signals to sense objects in the surrounding environment of vehicle 100. In one example, in addition to sensing objects, radar 126 can also be used to sense the speed and / or direction of travel of objects.

[0082] The laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In one example, the laser rangefinder 128 may include one or more laser sources, a laser scanner, one or more detectors, and other system components.

[0083] Camera 130 can be used to capture multiple images of the surrounding environment of vehicle 100. Camera 130 can be a still camera or a video camera.

[0084] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various elements, including a steering system 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0085] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one example, the steering system 132 may include a steering wheel system.

[0086] Throttle 134 is used to control the operating speed of engine 118 and thus the speed of vehicle 100.

[0087] Braking unit 136 is used to control the deceleration of vehicle 100. Braking unit 136 can use friction to slow down wheel 121. In other examples, braking unit 136 can convert the kinetic energy of wheel 121 into electrical energy. Braking unit 136 may also take other forms to slow down the rotational speed of wheel 121 to control the speed of vehicle 100.

[0088] The computer vision system 140 can process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles, etc. The computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some examples, the computer vision system 140 may be used to map the environment, track objects, estimate object velocities, etc.

[0089] The route control system 142 is used to determine the driving route of the vehicle 100. In some examples, the route control system 142 may combine data from sensor 138, positioning system 122 and one or more predetermined maps to determine the driving route for the vehicle 100.

[0090] The obstacle avoidance system 144 is used to identify, assess and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100.

[0091] Of course, in one example, the control system 106 may add or replace components other than those shown and described. Alternatively, some of the components shown above may be removed.

[0092] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an on-board computer 148, a microphone 150, and / or a speaker 152.

[0093] In some examples, peripheral device 108 provides a means for users of vehicle 100 to interact with user interface 116. For example, on-board computer 148 can provide information to users of vehicle 100. User interface 116 can also operate on-board computer 148 to receive user input. On-board computer 148 can be operated via touchscreen. In other cases, peripheral device 108 provides a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 can receive audio (e.g., voice commands or other audio input) from users of vehicle 100. Similarly, speaker 152 can output audio to users of vehicle 100.

[0094] The wireless communication system 146 can communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular networks, such as LTE, or 5G cellular communication. The wireless communication system 146 can communicate using WiFi and a wireless local area network (WLAN). In one example, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are also possible. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices that can enable public and / or private data communication between vehicles and / or roadside stations.

[0095] Power source 110 can provide power to various components of vehicle 100. In one example, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 100. In some examples, power source 110 and energy source 119 can be implemented together, as is the case in some pure electric vehicles.

[0096] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable storage medium such as memory 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0097] Processor 113 can be any conventional processor, such as a commercially available CPU. Alternatively, the processor can be a special-purpose device such as an ASIC or other hardware-based processor. Although Figure 1 functionally illustrates the processor, memory, and other elements of computer 110 in the same block, those skilled in the art will understand that the processor, computer, or memory can actually include multiple processors, computers, or memories stored in the same / different physical enclosures. For example, memory can be a hard disk drive or other storage media located in enclosures of different computers 110. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, may each have their own processor that performs calculations only relevant to the function of a particular component.

[0098] In each aspect described herein, the processor may be located remotely from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor located within the vehicle, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0099] In some examples, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108.

[0100] In addition to instruction 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. This information can be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0101] User interface 116 is used to provide information to or receive information from users of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker.

[0102] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, the computer system can utilize input from control system 106 to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some examples, computer system 112 can be operated to provide control over multiple aspects of vehicle 100 and its subsystems.

[0103] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0104] Optionally, the above components are just an example. In actual applications, the components in each of the above modules may be added or removed according to actual needs. Figure 1 should not be construed as a limitation on the embodiments of this application.

[0105] Autonomous vehicles traveling on roads, such as vehicle 100 above, can identify objects (also known as obstacles) in their surrounding environment to determine driving strategies, such as braking, deceleration, acceleration, and steering. These objects can be other vehicles, pedestrians, animals, traffic equipment (such as traffic signs, road signs, speed bumps, and streetlights), or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed that the autonomous vehicle needs to adjust can be determined.

[0106] Optionally, the autonomous vehicle 100 or its associated computing devices (such as the computer system 112, computer vision system 140, and memory 114 as shown in Figure 1) can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, snow, ice on the road, etc.). Optionally, each identified object depends on the behavior of the others, so all identified objects can be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what state the vehicle needs to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 on the road, the curvature of the road, and the proximity of static and dynamic objects.

[0107] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a specified trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0108] The aforementioned vehicle 100 can be a sedan, van, truck, or any other type of vehicle, and this application embodiment does not impose any special limitations.

[0109] Figure 2 is a schematic diagram of the computer system provided in an embodiment of this application.

[0110] The computer system 112 shown in Figure 2 includes a processor 113 coupled to a system bus. The processor 113 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to a system bus 105. The system bus 105 is coupled to the input / output (I / O) bus 113 via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with various devices, such as input devices 117 (e.g., keyboard, mouse, touchscreen, etc.), a media tray 121 (e.g., CD-ROM), a multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and moving digital video images), and an external USB interface 125. Optionally, the interface connected to the I / O interface 115 can be a USB interface.

[0111] The processor 113 can be any conventional processor, including a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an application-specific integrated circuit (ASIC). Optionally, the processor 113 can be a neural network processor or a combination of a neural network processor and the aforementioned conventional processors.

[0112] Optionally, in the embodiments described herein, the computer system 112 may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle, or it may be deployed within the autonomous vehicle to facilitate communication with other systems within the vehicle and reduce communication latency. In other aspects, some of the processes described herein are executed on a processor located within the autonomous vehicle, while others are executed by a remote processor, including taking actions necessary to perform a single operation.

[0113] Computer system 112 can communicate with software deployment server 149 via network interface 129. Network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0114] The hard disk drive interface is coupled to the system bus 105. The hardware drive interface is connected to the hard disk drive. The memory 114 is coupled to the system bus 105. The data running in the memory 114 may include the operating system 137 and application programs 143 of the computer system 112.

[0115] An operating system consists of a shell (139) and a kernel (141). The shell (139) is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system, waits for user input, interprets the user input for the operating system, and processes various operating system outputs.

[0116] The kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and more.

[0117] Application 143 includes programs related to controlling the autonomous driving of the vehicle, such as programs managing the interaction between the autonomous vehicle and obstacles on the road, programs controlling the route or speed of the autonomous vehicle, and programs controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road. Application 143 also exists on the system of software deployment server 149. In one example, when application 147 needs to be executed, computer system 112 can download application 143 from software deployment server 149.

[0118] For example, application 141 could also be a program that controls autonomous vehicles to avoid collisions with other vehicles and safely pass through intersections.

[0119] Sensor 153 is associated with computer system 112. Sensor 153 is used to detect the environment surrounding computer system 112. For example, sensor 153 can detect animals, vehicles, pedestrians, and crosswalks, etc., and further sensors can also detect the environment around the aforementioned animals, vehicles, pedestrians, and crosswalks. Optionally, if computer system 112 is located on an autonomous vehicle, the sensor can be a camera, infrared sensor, chemical detector, microphone, etc.

[0120] For example, the processor 113 can predict the driving trajectory of other vehicles based on the surrounding road conditions and other vehicle conditions detected by the sensor 153.

[0121] Figure 3 is a framework diagram of the autonomous driving system provided in this application. As shown in Figure 3, the autonomous driving system includes a perception unit 31, a target fusion unit 32, a prediction unit 33, a planning unit 34, a control unit 35, a map unit 36, and a navigation unit 37. The prediction unit 33 connects with the perception unit 31, the target fusion unit 32, and the map unit 36. Through prediction algorithms, it can provide the future behavioral intent and trajectory of obstacles, and then output it to the downstream planning module. This can help the vehicle predict the future trajectory of other vehicles and determine the importance of targets. In dangerous scenarios, this is beneficial for planning and control to take emergency safety measures to ensure vehicle safety and avoid collisions.

[0122] The function of the perception unit 31 is based on the sensor system 104 in Figure 1 or the sensor 153 in Figure 2. It senses environmental information around the vehicle 100, such as obstacles (e.g., other vehicles, pedestrians, animals, etc.) and road information (e.g., pedestrian crossings, lane lines, traffic lights, etc.) that affect driving. The target fusion unit 32 processes the environmental information around the vehicle sensed by the perception unit 31 and outputs obstacle target information. The map unit is stored in the memory 114 in Figure 1 or Figure 2. The prediction unit 33 predicts the behavioral intention and future trajectory of the target vehicle based on the current map information and the target information sensed by the perception unit. The planning unit 34 plans the vehicle's driving route based on the prediction results of the prediction unit and / or the output information of the navigation unit 37. The control unit 35 controls the vehicle to drive along the planned driving route based on the driving route planned by the planning unit. The target fusion unit 32, prediction unit 33, planning unit 33, and control unit 35 are all implemented in the processor in Figure 1 or Figure 2. Real-time, accurate, and reliable prediction of other vehicles' intentions during vehicle operation can help the vehicle anticipate traffic conditions ahead, establish traffic patterns around the vehicle, assess the importance of targets in other vehicles, filter key targets for interaction, facilitate advance route planning, and safely navigate complex road conditions.

[0123] Due to the complexity and variability of driving scenarios, autonomous driving systems may encounter problems in real-world applications, leading to situations where they cannot properly handle driving risks. Therefore, it is crucial to promptly identify problems with autonomous driving systems in various driving scenarios—that is, to perform fault identification, promptly discover vulnerabilities (bugs) in the vehicle's autonomous driving software, and then fix these vulnerabilities to improve the safety of the autonomous driving system and the user experience. Thus, timely and accurate identification of autonomous driving system faults (i.e., identifying faults before they lead to accidents) and timely repair are essential for safe vehicle operation and the user's driving experience.

[0124] This application provides a fault identification method and apparatus for an autonomous driving system, which can identify faults in the autonomous driving system in a timely and accurate manner, and repair the fault in time before the vehicle accident is caused by the fault, thereby improving driving safety and user driving experience.

[0125] The fault identification method for autonomous driving systems provided in this application can be applied to autonomous driving systems of any level, such as L1, L2, L3, L4 and L5 autonomous driving systems. That is, this application does not limit the autonomous driving level of the autonomous driving system.

[0126] The fault identification device for the autonomous driving system provided in this application embodiment can be deployed in the cloud to help the remote operation and maintenance system to detect faults in the autonomous driving system in a timely manner and perform OTA on the vehicle's autonomous driving system in a timely manner, thereby improving driving safety and autonomous driving experience.

[0127] Figure 4 illustrates a system architecture diagram of a fault identification method system for an autonomous driving system provided in this application embodiment. As shown in Figure 4, the vehicle generates various types of data during driving, such as driving trajectory, dashcam video, and road environment information. The vehicle collects and reports this data to the cloud in a timely manner. The cloud, based on corresponding filtering rules or triggered by the vehicle owner, reports vehicle driving data for a certain time period to the maintenance interface and packages the relevant maintenance event data into a database. After the massive maintenance event data stored in the database is checked and classified by maintenance personnel, a fault identification model can be built offline using this application embodiment and packaged into the cloud system. During the real-time operation of the cloud system, once a maintenance event is reported, the cloud system can read the vehicle data related to the maintenance event and call the packaged fault identification model to identify the driving scenario of the vehicle in the maintenance event and whether the vehicle's autonomous driving system is operating normally during the reporting time period. For maintenance events involving abnormal handling of the autonomous driving system, the cloud-based system dispatches a work order containing fault identification results and related data to the root cause localization module. This module further confirms the cause and module leading to the abnormality, adds root cause information, and then dispatches the work order to the maintenance engineer. The engineer can then repair the relevant module code based on the work order information and trigger a remote software upgrade for the autonomous driving system. This remote fault identification and localization ensures the reliability of the autonomous driving system while improving fault handling efficiency and user experience.

[0128] Of course, in some other examples, the fault identification device of the autonomous driving system can also be deployed on the vehicle side, such as in the computer system 12 in Figure 1, to realize real-time fault identification of the autonomous driving system.

[0129] Figure 5 is a flowchart illustrating a fault identification method for an autonomous driving system provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application embodiment does not specifically limit the specific computing device executing the method; a suitable computing device can be selected for execution as needed. For example, it can be executed on a vehicle-side device, such as a vehicle-side computer system, or it can be executed on both the vehicle-side device and a cloud device (such as a cloud-side server, like a remote maintenance server for the autonomous driving system), i.e., using an end-to-cloud collaborative architecture. Alternatively, it can be completed on a cloud device, providing fault identification services for the autonomous driving system to users in the form of cloud services. For ease of description, the form of the executing entity will not be distinguished in the following text; all instances will be described as a fault identification system for an autonomous driving system. As shown in Figure 5, the fault identification method for an autonomous driving system provided in this application embodiment includes steps S501 to S504.

[0130] In step S501, driving-related data of the vehicle are acquired.

[0131] During the course of driving, the vehicle generates various data related to its movement, such as the vehicle's own driving data and the movement data of obstacles around the vehicle.

[0132] Vehicle driving data can include the vehicle's driving status data, the vehicle's actual driving trajectory (also known as the posterior driving trajectory) data, and the vehicle's predicted driving trajectory.

[0133] The vehicle's driving status data includes its intelligent driving status, speed, acceleration, position coordinates, heading angle, and braking value. This data can be obtained directly from the vehicle's driving log. For example, the vehicle's computer system records driving logs in real time, which include data such as the vehicle's intelligent driving status, speed, acceleration, position coordinates, heading angle, and braking value during driving. The vehicle's intelligent driving status indicates its current driving mode, such as ACC mode, NCA mode, and automatic parking mode.

[0134] The vehicle's actual driving trajectory can be calculated from its motion state data. For example, by using the vehicle's position coordinates from time t1 to time tk, the vehicle's position coordinate sequence can be obtained, and the vehicle's actual driving trajectory can be obtained from the position coordinate sequence.

[0135] The predicted trajectory of a vehicle can be estimated from its motion state at time t1 to its predicted trajectory from time t1 to time tk. For example, the vehicle's velocity, heading angle, and position coordinates at time t1 can be calculated using dynamic equations to obtain the predicted trajectory from time t1 to time tk. Alternatively, the vehicle's motion state data at time t1 can be used as input to the trajectory prediction model to obtain the predicted trajectory from time t1 to time tk. The trajectory prediction model is a trained neural network model that can map the vehicle's motion state data at a certain moment to its predicted trajectory over a certain time period, such as mapping the motion state data at time t1 to the predicted trajectory from time t1 to time tk.

[0136] Understandably, time t1 is the current time of the vehicle, while time tk is related to the current driving mode of the vehicle. For example, in ACC mode, time tk is the time to time collision (TTC), which is the time it takes for the vehicle to reach the vehicle in front at its current speed.

[0137] The vehicle's actual and predicted driving trajectories can be calculated on the vehicle itself. Specifically, the vehicle calculates its actual and predicted trajectories based on its driving status data, and then uploads these data to the fault identification system. Alternatively, the vehicle's actual and predicted driving trajectories can also be calculated by the fault identification system. For instance, the vehicle uploads its driving status data to the fault identification system, which then calculates the actual and predicted driving trajectories based on this data.

[0138] Motion data of obstacles around the vehicle includes obstacle trajectory data and obstacle motion state data. Obstacle motion state data can be obtained from various sensor data in a sensor system. Sensor data may include data acquired by cameras, LiDAR, millimeter-wave radar, etc. The sensor data in this application can be raw video streams, radar point cloud data, or analyzed structured motion state data of obstacles around the vehicle, such as position, velocity, acceleration, steering speed, size, and position coordinates. For raw video stream data and radar point cloud data, the perception unit can process this sensor data into identifiable structured motion state data of obstacles around the vehicle, including position, velocity, acceleration, steering speed, size, and position coordinates.

[0139] The motion trajectory data of an obstacle can be calculated from its motion state data. For example, based on the position coordinates of obstacle a from time t1 to time tk, the motion trajectory data of obstacle a from time t1 to time tk can be obtained.

[0140] Obstacles can be understood as objects that may affect the movement of a vehicle, such as other vehicles, pedestrians, animals, and traffic facilities (e.g., streetlights, cones, road signs, etc.) around the vehicle.

[0141] Typically, a vehicle is surrounded by multiple obstacles, such as other vehicles, pedestrians, and traffic facilities within a 200-meter radius. However, the driving decisions made by the vehicle's autonomous driving system are often influenced by a small number of specific key obstacles, such as vehicles or pedestrians directly in front of the vehicle. Therefore, this embodiment of the application needs to select and identify the key obstacles that will affect driving decisions from the multiple obstacles sensed around the vehicle. These key obstacles are then designated as target obstacles for the vehicle during a target time period (e.g., from time t1 to time tk). This allows for a focus on the impact of key obstacles on the vehicle, improving the accuracy of subsequent obstacle recognition and prediction.

[0142] There are several methods to identify target obstacles. One option is to retrieve them from the vehicle's logbook. The logbook records the IDs of obstacles that influenced the vehicle's actions when a specific event occurs. For example, if the vehicle experiences a sudden braking or stop, it automatically records the IDs of obstacles in front of it before and after the event in the logbook. For instance, if vehicle A in front of the vehicle suddenly decelerates, the vehicle's autonomous driving system, in response to this deceleration, implements a braking strategy to avoid a collision and controls the vehicle to brake suddenly. In this case, the logbook records the ID of vehicle A. By retrieving the ID of vehicle A from the logbook, vehicle A becomes the target obstacle for the vehicle.

[0143] Alternatively, target obstacles can be selected from multiple obstacles around the vehicle according to preset rules. For example, key obstacles can be determined based on the distance between the obstacles around the vehicle and the vehicle. For instance, the several obstacles closest to the vehicle within 50 meters of the vehicle can be identified as target obstacles.

[0144] The motion data of the target obstacle is obtained from the motion data of multiple obstacles around the vehicle. In subsequent steps, the vehicle's driving data and the motion data of the target obstacle are processed to obtain the fault identification result of the autonomous driving system.

[0145] In step S502, features are extracted from the driving-related data of the vehicle based on multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle.

[0146] This application embodiment extracts features from the vehicle's driving-related data from multiple perspectives to obtain feature vectors from multiple perspectives of the vehicle, expressing the driving scene features of the vehicle from multiple perspectives, making the obtained driving scene features of the vehicle more comprehensive, and thus using them to identify faults in the autonomous driving system more accurately.

[0147] For example, the multiple perspectives include the predicted trajectory interaction perspective between the vehicle and the target obstacle, and the actual trajectory interaction perspective between the vehicle and the target obstacle. By extracting features from the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle, a predicted trajectory interaction feature vector is obtained. This feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, effectively capturing the dynamic influence of the motion trajectory of the target obstacle on the predicted trajectory of the vehicle. Similarly, by extracting features from the actual trajectory data of the vehicle and the motion trajectory data of the target obstacle, a actual trajectory interaction feature vector is obtained. This feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle, effectively capturing the dynamic influence of the motion trajectory of the target obstacle on the actual driving trajectory of the vehicle.

[0148] In one example, a first feature extraction network can be used to extract features from the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle to obtain a predicted trajectory interaction feature vector; a second feature extraction network can be used to extract features from the actual trajectory data of the vehicle and the motion trajectory data of the target obstacle to obtain an actual trajectory interaction feature vector. That is, the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as input to the first feature extraction network, which outputs a predicted trajectory interaction feature vector; the actual trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as input to the second feature extraction network, which outputs an actual trajectory interaction feature vector.

[0149] Optionally, the first feature extraction network includes a first temporal representation network, a second temporal representation network, and a first trajectory interaction network. The process of the first feature extraction network extracting features from the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle can be as follows: the predicted trajectory data of the vehicle (which can be the position coordinate sequence of the vehicle at consecutive times from t1 to tk in the target time period) is used as the input of the first temporal representation network, and the predicted driving trajectory feature vector of the vehicle is output; the motion trajectory data of the target obstacle is used as the input of the second temporal representation network, and the motion trajectory feature vector of the target obstacle is output; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused (for example, the predicted driving trajectory feature vector and the motion trajectory feature vector of the target obstacle are concatenated) to obtain the first trajectory fusion feature vector; the first trajectory fusion feature vector is used as the input of the first trajectory interaction network, and the predicted trajectory interaction feature vector is output.

[0150] The second feature extraction network includes a third temporal representation network, a fourth temporal representation network, and a second trajectory interaction network. The process of the second feature extraction network extracting features from the vehicle's real trajectory data and the target obstacle's motion trajectory data can be as follows: the vehicle's real trajectory data is used as input to the third temporal representation network, and the output is the vehicle's real driving trajectory feature vector; the target obstacle's motion trajectory data is used as input to the fourth temporal representation network, and the output is the target obstacle's motion trajectory feature vector; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain the second trajectory fusion feature vector; the second trajectory fusion feature vector is used as input to the second trajectory interaction network, and the output is the real trajectory interaction feature vector.

[0151] The vehicle's trajectory features and the target obstacle's motion trajectory features are extracted separately through a temporal representation network. Then, the two trajectory features are concatenated and input into a trajectory interaction network to accurately extract the interaction features between the vehicle's trajectory and the target obstacle's motion trajectory, thus obtaining the correlation between the vehicle's trajectory and the target obstacle's motion trajectory, that is, the influence of the target obstacle's dynamic motion on the vehicle's trajectory.

[0152] The temporal representation network (including the first, second, third, and fourth temporal representation networks) and the trajectory interaction network (including the first and second trajectory interaction networks) can be constructed using various neural network structures. For example, the temporal representation network can be a TCN network, and the trajectory interaction network can be constructed based on an MLP network. Alternatively, the temporal representation network can be an RNN network or a Transformer neural network model, and the trajectory interaction network can be an FCN network. This application does not limit the specific implementation of the temporal representation network and the trajectory interaction network; appropriate neural network structures can be selected for training and implementation according to actual needs.

[0153] In some other examples, to further enhance the comprehensiveness of the driving scenario representation, the vehicle's driving-related data also includes road environment data, which indicates the road environment information in which the vehicle is driving; the feature vectors from multiple perspectives also include road environment feature vectors, which are obtained by feature extraction from the road environment data.

[0154] Road environment information can be obtained by the vehicle's sensor system sensing the road environment around the vehicle. For example, the vehicle's sensor system includes sensors such as LiDAR, millimeter-wave radar, and cameras, which can sense road environment information such as the lane number, lane line type, distance to intersection, and distance to traffic lights.

[0155] Of course, road environment information can also be obtained through other means, such as obtaining the road environment information of the road where the vehicle is located through high-precision maps. This application does not limit this, and you can choose the appropriate way to obtain road environment information according to the actual situation.

[0156] Optionally, features can be extracted from road environment data using a TCN network to obtain road environment feature vectors.

[0157] In another example, to further enrich the driving scenario feature information of the vehicle, the feature vectors of the multiple perspectives implemented in this application also include driving scenario feature vectors, which are obtained based on feature extraction from first driving scenario data and / or second driving scenario data. The first driving scenario data includes result data calculated from driving data and target obstacle motion data based on dynamic equations and / or prior knowledge, and the result data is related to the vehicle's driving mode. The second driving scenario data includes the vehicle's driving data and target obstacle motion data corresponding to the fault time, indicating the moment when the vehicle's driving is likely to be abnormal. The fault time can be predicted by the autonomous driving system; for example, if the autonomous driving system predicts that there is a high probability of abnormal driving at a certain moment, such as a rear-end collision or lane departure, the autonomous driving system will automatically record these moments along with the driving data and target obstacle motion data at those moments.

[0158] For example, in AEB mode, braking value, whether autonomous braking is used (e.g., if the braking value is non-zero and the duration is greater than 5 seconds, it is considered autonomous braking; otherwise, it is considered non-autonomous braking), whether the target obstacle is a phantom obstacle (e.g., if the duration of the target obstacle exceeds 10 seconds, it is considered a non-phantom obstacle; otherwise, it is considered a phantom obstacle), and the duration of the target obstacle are injected into the first driving scenario data as driving scenario information. In ACC mode, steering angle, steering speed, and the time to collision (TTC) between the vehicle and the target obstacle (e.g., the vehicle in front) are injected into the first driving scenario data as driving scenario information. The first driving scenario data also includes some data calculated based on kinematic equations, such as the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle.

[0159] Feature extraction networks can be used to extract features from the first and second driving scenario data, and then the two extracted features can be fused (e.g., concatenated) to obtain a driving scenario feature vector. For example, an MLP can be used to extract features from the first and second driving scenario data to obtain a driving scenario feature vector.

[0160] By injecting driving scene information through prior rules and / or dynamic equations, the characteristics of the driving scene are enhanced, and the richness of scene information is improved by combining human experience, thereby improving the accuracy of fault identification in autonomous driving systems.

[0161] In step S503, the feature vectors from multiple perspectives are fused to obtain the global driving scene feature vector of the vehicle.

[0162] In this embodiment, the feature vectors from multiple perspectives can be weighted and fused through an attention mechanism to achieve adaptive fusion of multi-perspective information, thereby obtaining the global driving scene feature vector of the vehicle and improving the accuracy of fault identification in the autonomous driving system.

[0163] In one example, when the feature vectors from multiple perspectives only include the predicted trajectory interaction feature vector and the real trajectory interaction feature vector, the feature vector fusion process from multiple perspectives is as follows: determine the first attention weight corresponding to the predicted trajectory interaction feature vector and the second attention weight corresponding to the real trajectory interaction feature vector; then, based on the first attention weight and the second attention weight, perform weighted fusion of the predicted trajectory interaction feature vector and the real trajectory interaction feature vector to obtain the global driving scene feature vector of the vehicle.

[0164] When the feature vectors from multiple perspectives include the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, and the driving scene feature vector, the feature vector fusion process from multiple perspectives is as follows: determine the first attention weight corresponding to the predicted trajectory interaction feature vector, the second attention weight corresponding to the real trajectory interaction feature vector, and the third attention weight corresponding to the driving scene feature vector; then, based on the first attention weight, the second attention weight, and the third attention weight, the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, and the driving scene feature vector are weighted and fused to obtain the global driving scene feature vector of the vehicle.

[0165] When the feature vectors from multiple perspectives include the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, the driving scenario feature vector, and the road environment feature vector, the feature vector fusion process from multiple perspectives is as follows: determine the first attention weight corresponding to the predicted trajectory interaction feature vector, the second attention weight corresponding to the real trajectory interaction feature vector, the third attention weight corresponding to the driving scenario feature vector, and the fourth attention weight corresponding to the road environment feature vector; then, based on the first attention weight, the second attention weight, the third attention weight, and the fourth attention weight, the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, the driving scenario feature vector, and the road environment feature vector are weighted and fused to obtain the vehicle's global driving scenario feature vector.

[0166] Thus, by fully representing information from multiple perspectives, the complex state changes and mutual influences of the vehicle, obstacles, and road environment are fully modeled. A trajectory interaction network is designed to effectively capture the dynamic influence between the trajectories of surrounding obstacles and the vehicle's trajectory. An obstacle-dimensional attention mechanism is designed to focus on key obstacles that cause the vehicle to accelerate, brake suddenly, or change lanes, extracting the impact of surrounding key obstacles and environmental information on the vehicle's behavior, thereby assisting in determining whether the autonomous driving system is operating correctly. Adaptive fusion of multi-perspective information considers the influence of different perspectives, designing a perspective-dimensional attention mechanism to weightedly fuse information from multiple perspectives, obtaining a global representation of the driving scene, and improving recognition accuracy.

[0167] In one possible implementation, feature vectors from multiple perspectives (e.g., predicted trajectory interaction feature vectors, real trajectory interaction feature vectors, driving scene feature vectors, and road environment feature vectors) are first concatenated and fused. Then, the concatenated feature vectors are passed through a fusion network of feature encoder and perspective dimension attention mechanism to obtain the final global driving scene feature vector of the vehicle.

[0168] In step S504, based on the global driving scene feature vector, the target driving scene in which the vehicle is in the current driving mode and whether the autonomous driving system is malfunctioning in the target driving scene are determined.

[0169] By obtaining the global driving scene feature vector of the vehicle through the aforementioned steps, it is possible to accurately identify the target driving scene in which the vehicle is currently driving in the current driving mode and whether the autonomous driving system is malfunctioning in the target driving scene.

[0170] For example, the global driving scene feature vector of the vehicle is used as input to the classifier, and the output is the fault identification result of the vehicle's autonomous driving system. This fault identification result includes the target driving scene in which the vehicle is currently driving in the current driving mode and whether the autonomous driving system malfunctions in the target driving scene. Whether the autonomous driving system malfunctions in the target driving scene means whether the autonomous driving system can make the correct operation and processing for the vehicle in the target driving scene. For example, if the vehicle is in a rear-end collision scenario, can the autonomous driving system control the vehicle to brake correctly in the rear-end collision scenario to avoid a rear-end collision with the vehicle in front?

[0171] In other words, the fault results of the autonomous driving system identified in the embodiments of this application include the target driving scenario and whether there is a fault. The target driving scenario indicates the driving scenario in which the vehicle is in the current driving mode, and whether there is a fault indicates whether the autonomous driving system of the vehicle is faulty in the target driving scenario.

[0172] The driving scenarios here are more granular. For example, if the vehicle's current driving mode is AEB mode, the driving scenarios include granular scenarios such as AEB false triggering, AEB late triggering, and AEB missed triggering. If the vehicle's current driving mode is ACC mode, the driving scenarios include granular scenarios such as driving out of center, following collision, cut-in collision, unexpected acceleration, unexpected deceleration, inability of function intervention, unreasonable obstacle avoidance, and abnormal function termination. If the vehicle's current driving mode is Auto Parking Assistant (APA) mode, the driving scenarios include granular scenarios such as parking failure, parking failure, abnormal parking exit posture, abnormal parking entry posture, and poor human-vehicle interaction experience. If the vehicle's current driving mode is NCA mode, the driving scenarios include granular scenarios such as unreasonable lane changing, running a red light at an intersection, unreasonable signaling, and unreasonable obstacle avoidance.

[0173] It should be clarified that the driving scenarios mentioned in the embodiments of this application do not refer to actual, already occurring scenarios, but rather to scenarios involving potential risk events. For example, a rear-end collision scenario refers to a driving scenario where the vehicle may rear-end another vehicle, not that a rear-end collision has already occurred; therefore, it can also be called a rear-end collision risk driving scenario. For instance, if there is vehicle A 100 meters ahead of the vehicle, considering both the vehicle's speed and vehicle A's speed, the driving scenario is determined to be a rear-end collision scenario. In other words, the embodiments of this application aim to detect malfunctions in the autonomous driving system before an accident occurs due to a failure in the system, and then repair those malfunctions to ensure the reliability of the autonomous driving system and guarantee driving safety.

[0174] The method in this application embodiment further includes issuing a fault alarm message or a manual takeover request when the autonomous driving system is detected to be malfunctioning in the target driving scenario and the fault has not been resolved (e.g., the code or module causing the fault is resolved after an OTA upgrade). The fault alarm message indicates that the vehicle's current autonomous driving system is in a faulty state in the target driving scenario and is not suitable for activating autonomous driving in the target driving scenario. The manual takeover request can be used to request the user to manually control the vehicle to drive.

[0175] For example, if the embodiments of this application identify a fault in the autonomous driving system during AEB false triggering scenarios in AEB mode, i.e., the autonomous driving system frequently triggers false emergency braking in AEB mode, a fault warning message is issued to the user, informing the user that the vehicle's current autonomous driving system is malfunctioning in AEB false triggering scenarios in AEB mode, and that this function is not suitable for use in driving scenarios where AEB false triggering may occur. The user is advised to disable this function and wait for an upgrade before using it again.

[0176] The following describes a practical application of the fault identification method for an autonomous driving system provided in this application embodiment.

[0177] Figure 6 illustrates a schematic diagram of the system implementation architecture of a fault identification method for an autonomous driving system provided in an embodiment of this application. As shown in Figure 6, the input data required in this embodiment includes vehicle driving data, surrounding obstacle movement data, and road environment data. Then, target obstacles are selected from the surrounding obstacles of the vehicle through a target obstacle screening sequence, thereby obtaining the movement data of the target obstacles. Driving scene information is then injected, and human experience is integrated to improve the richness of scene information. Finally, the vehicle driving data, target obstacle movement data, road environment data, and the driving scene information injected are used as input to the fault identification model, and the fault identification result is obtained through reasoning by the fault identification model.

[0178] The specific methods for acquiring vehicle driving data, surrounding obstacle movement data, and road environment data, as well as the methods for selecting target obstacles and injecting driving scenario information, can be found in the description above. For the sake of brevity, they will not be repeated here.

[0179] In one example, the fault identification model includes a preprocessing module that preprocesses vehicle driving data, target obstacle motion data, road environment data, and driving scenario information. For instance, the vehicle driving data includes the vehicle's driving state data, the vehicle's actual driving trajectory data, and the vehicle's predicted driving trajectory. Preprocessing the vehicle driving data includes selecting the vehicle's actual driving trajectory data and the vehicle's predicted driving trajectory from the vehicle driving data. As another example, the vehicle driving data may only include the vehicle's driving state data; preprocessing the vehicle driving data includes calculating the vehicle's actual driving trajectory data and the vehicle's predicted driving trajectory using the vehicle's driving state data. Similarly, the target obstacle motion data includes the target obstacle's motion trajectory data and the target obstacle's motion state data; preprocessing the target obstacle's motion data includes selecting the target obstacle's motion trajectory data from the target obstacle's motion data. As yet another example, the target obstacle motion data may only include the target obstacle's motion state data; preprocessing the target obstacle's motion trajectory data includes calculating the target obstacle's motion state data using the target obstacle's motion state data. For example, prior knowledge can be used to process vehicle driving data and target obstacle motion data to obtain driving scenario information. For instance, for the current driving mode, specific rule-based judgment results (e.g., time-of-collision (TCC)) and / or dynamic equations can be used to perform dynamic calculations on the vehicle's driving state data to obtain predicted driving trajectory data. Another example is using rules to extract the vehicle's driving data and target obstacle motion data at the time of the malfunction as driving scenario information. Yet another example is road environment information, including the lane number, lane line type, distance to intersections, and distance to traffic lights. Based on the current driving mode, the corresponding road environment information can be selected from multiple sources; for example, in ACC mode, lane number and lane line type are selected as the road environment information.

[0180] The preprocessed data is then used as input data for the fault identification model. The fault identification model processes the input data to predict the fault identification results of the autonomous driving system.

[0181] Figure 7 shows a schematic diagram of the input data processing process of a fault identification model.

[0182] Taking the current driving model of the autonomous vehicle as AEB mode as an example, this paper introduces the fault identification process of the autonomous driving system. In AEB mode, the time to collision (TTC), the predicted trajectory of the autonomous vehicle, and the duration of obstacle collisions can be calculated based on the speed and coordinate changes of the autonomous vehicle and obstacles, thus enriching the scene information. Road environment information includes lane line type, distance to intersection, and distance to traffic lights.

[0183] For the predicted trajectory of the autonomous vehicle, a temporal representation and trajectory interaction network is used to interact with the movement trajectory of the target obstacle within the corresponding time period, extracting the correlation between the two and obtaining the predicted trajectory interaction feature vector. For the posterior trajectory of the autonomous vehicle and the obstacle, the correlation between the two is extracted using the temporal representation and trajectory interaction network to obtain the real trajectory interaction feature vector. The fault point time information and scene features are processed using MLP to increase the dimensionality and obtain the driving scene feature vector. The road information is processed using TCN network to obtain the road environment feature vector. The four feature vectors are concatenated and fused through a feature encoder and a view dimension attention mechanism to obtain the final global scene feature. Finally, the global scene feature is processed by a classifier to output the classification result as the fault identification result, that is, whether the autonomous vehicle is in the target driving scene in the current driving mode and whether the autonomous driving system is faulty in the target driving scene.

[0184] The fault identification method for autonomous driving systems provided in this application integrates human experience to enhance the richness of scene information, captures the dynamic changes of the vehicle and surrounding obstacles and road environment from multiple perspectives, and adaptively integrates multi-perspective information to obtain a comprehensive representation of the driving scene for fault identification. This greatly improves the accuracy of fault identification in autonomous driving systems, facilitates iterative updates of the autonomous driving system by maintenance personnel based on the identified faults, enhances the reliability of the autonomous driving system, and ensures the driving safety of the vehicle.

[0185] This application embodiment obtains a global driving scene feature vector through multi-view fusion, which can also be used for other purposes, such as user driving behavior analysis, constructing driving behavior profiles and vehicle owner profiles, and helping to conduct comprehensive driving risk assessments, such as traffic operation level accident risk assessment (whether it is dangerous driving), vehicle and vehicle owner driving level risk assessment (whether the vehicle owner is fatigued / driving under the influence of alcohol / whether it is a novice driver), etc.

[0186] Therefore, this application also provides a driving risk assessment method, including obtaining a global driving scene feature vector of the vehicle and determining a driving risk assessment result based on the global driving scene feature vector. Optionally, the driving risk assessment result includes one or more of the following: traffic accident risk assessment result, vehicle risk assessment result, and driver driving risk assessment result.

[0187] Optionally, one way to determine the driving risk assessment result based on the global driving scene feature vector is to use the global driving scene feature vector as input to a classifier and output the driving risk assessment result. For example, if the probability distribution of the classifier's output risk assessment result is such that the probability of traffic accident risk is 0.8, the probability of vehicle risk is 0.6, and the probability of driver driving risk is 0.7, then the traffic accident risk assessment is high risk, the vehicle risk assessment is medium risk, and the driver driving risk assessment is medium-high risk.

[0188] In another example, the classifier can also output more detailed evaluation results based on the global driving scene feature vector. For instance, in driver risk assessment, the classifier outputs risk probability distributions for multiple categories. For example, if the classifier outputs a risk of driver fatigue of 0.8, a probability of drunk driving of 0.5, and a probability of the driver being a novice driver of 0.3, then the risk assessment is driver fatigue of 0.8. Of course, the classifier's output depends on the training labels of the training data; the more detailed the training labels, the more detailed the classifier's output.

[0189] The method for obtaining the global driving scene feature vector of the vehicle is described above, and will not be repeated here for the sake of brevity.

[0190] In one example, the driving risk assessment method provided in this application embodiment further includes issuing an alarm to the driver when a high driving risk assessment is detected. For example, if the driver has been driving for more than 5 hours, and the driving risk assessment method of this application embodiment assesses the driver's driving risk as high, an alarm is issued to the driver. The alarm includes, but is not limited to, one or more alarm operations such as voice alarm, light alarm, and seat belt tightening.

[0191] Based on the same concept as the aforementioned embodiment of a fault identification method for an autonomous driving system, this application also provides a fault identification device 800 for an autonomous driving system. This fault identification device 800 can be deployed in a terminal device or a cloud server to achieve accurate identification of faults in the autonomous driving system. The fault identification device 800 includes units or modules for implementing the various steps in the fault identification method for the autonomous driving system shown in Figures 5-7.

[0192] Figure 8 is a schematic diagram of the structure of a fault identification device for an autonomous driving system provided in an embodiment of this application. As shown in Figure 8, the fault identification device 800 for an autonomous driving system includes at least a first acquisition module 801, a first multi-view feature extraction module 802, a first feature fusion module 803, and a fault identification module 804. The first acquisition module 801 is used to acquire driving-related data of the vehicle, which includes at least the vehicle's driving data and the motion data of a target obstacle. The target obstacle is any obstacle around the vehicle that affects the vehicle's driving. The first multi-view feature extraction module 802 is used to extract features from the vehicle's driving-related data based on multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle. The feature vectors from multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, while the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The first feature fusion module 803 is used to fuse the feature vectors from multiple perspectives to obtain the global driving scene feature vector of the vehicle. The fault identification module 804 is used to determine, based on the global driving scene feature vector, the target driving scene in which the vehicle is located in the current driving mode and whether the autonomous driving system is faulty in the target driving scene.

[0193] In one possible implementation, the vehicle's driving data includes the vehicle's predicted trajectory data and the vehicle's actual trajectory data. The predicted trajectory data indicates the vehicle's predicted driving trajectory data within a target time period, and the actual trajectory data indicates the vehicle's actual driving trajectory data within the target time period. The target obstacle's motion data includes the target obstacle's motion trajectory data, which indicates the target obstacle's motion trajectory data within a target time period. The first multi-view feature extraction module 802 is specifically used to: take the vehicle's predicted trajectory data and the target obstacle's motion trajectory data as input to a first feature extraction network and output a predicted trajectory interaction feature vector; take the vehicle's actual trajectory data and the target obstacle's motion trajectory data as input to a second feature extraction network and output an actual trajectory interaction feature vector.

[0194] In another possible implementation, a specific implementation of using the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as input to the first feature extraction network and outputting a predicted trajectory interaction feature vector is as follows: the predicted trajectory data of the vehicle is used as input to the first temporal representation network, which outputs a predicted driving trajectory feature vector of the vehicle; the motion trajectory data of the target obstacle is used as input to the second temporal representation network, which outputs a motion trajectory feature vector of the target obstacle; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a first trajectory fusion feature vector; the first trajectory fusion feature vector is used as input to the first trajectory interaction network, which outputs a predicted trajectory interaction feature vector.

[0195] In another possible implementation, a specific implementation of using the vehicle's real trajectory data and the target obstacle's motion trajectory data as input to the second feature extraction network to output a real trajectory interaction feature vector is as follows: The vehicle's real trajectory data is used as input to the third temporal representation network to output the vehicle's real driving trajectory feature vector; the target obstacle's motion trajectory data is used as input to the fourth temporal representation network to output the target obstacle's motion trajectory feature vector; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; the second trajectory fusion feature vector is used as input to the second trajectory interaction network to output a real trajectory interaction feature vector.

[0196] Optionally, the first temporal representation network, the second temporal representation network, the third temporal representation network, and the fourth temporal representation network are all constructed based on TCN; and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on MLP network.

[0197] In another possible implementation, the first feature fusion module 803 is specifically used to: determine the first attention weight corresponding to the predicted trajectory interaction feature vector and the second attention weight corresponding to the real trajectory interaction feature vector; and, based on the first attention weight and the second attention weight, perform weighted fusion of the predicted trajectory interaction feature vector and the real trajectory interaction feature vector to obtain the global driving scene feature vector of the vehicle.

[0198] In another possible implementation, the target obstacle is retrieved from the vehicle's operation log; or, the target obstacle is selected from obstacles around the vehicle based on preset rules.

[0199] In another possible implementation, the vehicle's driving-related data also includes road environment data, which indicates the road environment information in which the vehicle is driving; the feature vectors from multiple perspectives also include road environment feature vectors, and the vehicle's global driving scene feature vectors are also related to the road environment feature vectors, which are obtained by feature extraction from the road environment data.

[0200] Optionally, road environment data may include one or more of the following: lane number, lane type, distance from intersection, and distance from traffic light.

[0201] In another possible implementation, the vehicle driving data may also include one or more of the vehicle's intelligent driving status, speed, acceleration, position coordinates, orientation angle, and braking value; and / or, the target obstacle motion data may also include one or more of the target obstacle's speed, acceleration, and position coordinates.

[0202] In another possible implementation, the feature vectors from multiple perspectives also include driving scene feature vectors; the driving scene feature vectors are obtained by feature extraction from first driving scene data and / or second driving scene data; wherein, the first driving scene data includes the result data calculated based on driving data and target obstacle motion data based on dynamic equations and / or prior knowledge, and the result data is related to the driving mode of the vehicle; the second driving scene data includes the vehicle driving data and target obstacle motion data corresponding to the time of failure, and the time of failure indicates the time when the vehicle's driving may be abnormal; the global driving scene feature vector of the vehicle is also related to the driving scene feature vector.

[0203] In another possible implementation, the resulting data includes one or more of the following: the collision time between the vehicle and the target obstacle, the predicted trajectory of the vehicle, the duration of the collision with the target obstacle, the braking value, and the steering angle.

[0204] The fault identification device 800 of the autonomous driving system according to the embodiments of this application can be used to execute the methods described in the embodiments of this application. The above and other operations and / or functions of each module in the fault identification device 800 for the autonomous driving system are respectively to implement the corresponding processes of each method in FIG5-7. For the sake of brevity, they will not be described again here.

[0205] Based on the same concept as the aforementioned embodiment of a driving risk assessment method, this application also provides a driving risk assessment device 900, which can be deployed in a terminal device or a cloud server to achieve accurate assessment of driving risks.

[0206] Figure 9 is a schematic diagram of the structure of a driving risk assessment device provided in an embodiment of this application. As shown in Figure 9, the driving risk assessment device 900 includes at least a second acquisition module 901, a second multi-view feature extraction module 902, a second feature fusion module 903, and a risk assessment module 904. The second acquisition module 901 acquires driving-related data of the vehicle, including at least the vehicle's driving data and the motion data of a target obstacle. The target obstacle is any obstacle around the vehicle that affects its driving. The second multi-view feature extraction module 902 extracts features from the driving-related data of the vehicle from multiple perspectives to obtain feature vectors for multiple perspectives. These feature vectors include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, while the real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The second feature fusion module 903 fuses the feature vectors from multiple perspectives to obtain a global driving scene feature vector for the vehicle. The risk assessment module 904 determines the driving risk assessment result based on the global driving scene feature vector.

[0207] Optionally, the driving risk assessment results may include one or more of the following: traffic accident risk assessment results, vehicle risk assessment results, and driver driving risk assessment results.

[0208] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0209] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0210] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0211] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0213] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A fault identification method for an autonomous driving system, characterized in that, include: Acquire driving-related data of the vehicle, which includes at least the vehicle's driving data and the movement data of the target obstacle, wherein the target obstacle is an obstacle around the vehicle that affects the vehicle's driving. Based on multiple feature extraction networks, the driving-related data of the vehicle are extracted to obtain feature vectors from multiple perspectives of the vehicle. The feature vectors from multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle. The real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The feature vectors from the multiple perspectives are fused to obtain the global driving scene feature vector of the vehicle. Based on the global driving scene feature vector, the target driving scene in which the vehicle is located in the current driving mode and whether the autonomous driving system is malfunctioning in the target driving scene are determined.

2. The method according to claim 1, characterized in that, The vehicle driving data includes the vehicle's predicted trajectory data and the vehicle's actual trajectory data. The vehicle's predicted trajectory data indicates the vehicle's predicted driving trajectory data within a target time period, and the vehicle's actual trajectory data indicates the vehicle's actual driving trajectory data within the target time period. The target obstacle motion data includes the target obstacle's motion trajectory data, which indicates the target obstacle's motion trajectory data during the target time period; The method involves extracting features from the vehicle's driving-related data using multiple feature extraction networks to obtain feature vectors from multiple perspectives of the vehicle, including: The predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as inputs to the first feature extraction network, and the predicted trajectory interaction feature vector is output. The actual trajectory data of the vehicle and the motion trajectory data of the target obstacle are used as inputs to the second feature extraction network, which outputs the actual trajectory interaction feature vector.

3. The method according to claim 2, characterized in that, The step of using the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as input to a first feature extraction network, and outputting the predicted trajectory interaction feature vector, includes: The predicted trajectory data of the vehicle is used as the input of the first temporal representation network, and the predicted driving trajectory feature vector of the vehicle is output. The motion trajectory data of the target obstacle is used as the input of the second temporal representation network, and the output is the motion trajectory feature vector of the target obstacle; The predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain the first trajectory fusion feature vector; The first trajectory fusion feature vector is used as the input to the first trajectory interaction network, and the predicted trajectory interaction feature vector is output.

4. The method according to claim 2 or 3, characterized in that, The step of using the vehicle's real trajectory data and the target obstacle's motion trajectory data as input to the second feature extraction network, and outputting the real trajectory interaction feature vector, includes: The actual trajectory data of the vehicle is used as the input of the third temporal representation network to output the actual driving trajectory feature vector of the vehicle. The motion trajectory data of the target obstacle is used as the input of the fourth temporal representation network, and the motion trajectory feature vector of the target obstacle is output. The real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; The second trajectory fusion feature vector is used as the input to the second trajectory interaction network, and the real trajectory interaction feature vector is output.

5. The method according to claim 4, characterized in that, The first temporal representation network, the second temporal representation network, the third temporal representation network, and the fourth temporal representation network are all constructed based on temporal convolutional networks; And / or, both the first trajectory interaction network and the second trajectory interaction network are constructed based on a multilayer perceptron network.

6. The method according to any one of claims 1-5, characterized in that, The process of fusing the feature vectors from the multiple perspectives to obtain the global driving scene feature vector of the vehicle includes: Determine the first attention weight corresponding to the predicted trajectory interaction feature vector and the second attention weight corresponding to the real trajectory interaction feature vector; Based on the first attention weight and the second attention weight, the predicted trajectory interaction feature vector and the real trajectory interaction feature vector are weighted and fused to obtain the global driving scene feature vector of the vehicle.

7. The method according to any one of claims 1-6, characterized in that, The target obstacle is retrieved from the vehicle's operation log; Alternatively, the target obstacle may be selected from obstacles around the vehicle based on preset rules.

8. The method according to any one of claims 1-7, characterized in that, The vehicle's driving-related data also includes road environment data, which indicates the road environment information where the vehicle is driving; The feature vectors from multiple perspectives also include road environment feature vectors. The global driving scene feature vector of the vehicle is also related to the road environment feature vectors. The road environment feature vectors are obtained by feature extraction from the road environment data.

9. The method according to claim 8, characterized in that, The road environment data includes one or more of the following: lane number, lane line type, distance from intersection, and distance from traffic lights.

10. The method according to any one of claims 1-9, characterized in that, The vehicle driving data also includes the vehicle's driving status data, which includes one or more of the following: intelligent driving status, speed, acceleration, position coordinates, orientation angle, and braking value. And / or, the target obstacle motion data further includes the target obstacle motion state data, which includes one or more of velocity, acceleration, and position coordinates.

11. The method according to claim 10, characterized in that, The feature vectors from multiple perspectives also include driving scene feature vectors; The driving scenario feature vector is obtained based on feature extraction from the first driving scenario data and / or the second driving scenario data; The first driving scenario data includes result data calculated from the driving state data and the motion state data based on dynamic equations and / or prior knowledge, and the result data is related to the driving mode of the vehicle. The second driving scenario data includes the vehicle's driving data and the target obstacle's movement data corresponding to the time of the fault, wherein the time of the fault indicates the moment when the vehicle's driving may be abnormal; The global driving scene feature vector of the vehicle is also related to the driving scene feature vector.

12. The method according to claim 11, characterized in that, The result data includes one or more of the following: the collision time between the vehicle and the target obstacle, the predicted driving trajectory of the vehicle, the duration of the target obstacle, whether the target obstacle is a ghost obstacle, the braking value, whether autonomous braking is performed, and the steering angle.

13. A driving risk assessment method, characterized in that, include: Acquire driving-related data of the vehicle, which includes at least the vehicle's driving data and the movement data of the target obstacle, wherein the target obstacle is an obstacle around the vehicle that affects the vehicle's driving. Based on multiple feature extraction networks, the driving-related data of the vehicle are extracted to obtain feature vectors from multiple perspectives of the vehicle. The feature vectors from multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle. The real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The feature vectors from the multiple perspectives are fused to obtain the global driving scene feature vector of the vehicle. Based on the global driving scenario feature vector, the driving risk assessment result is determined.

14. The method according to claim 13, characterized in that, The driving risk assessment results include one or more of the following: traffic accident risk assessment results, vehicle risk assessment results, and driver driving risk assessment results.

15. A fault identification device for an autonomous driving system, characterized in that, include: The first acquisition module is used to acquire driving-related data of the vehicle. The driving-related data of the vehicle includes at least the driving data of the vehicle and the movement data of the target obstacle. The target obstacle is an obstacle among the obstacles around the vehicle that affects the driving of the vehicle. The first multi-view feature extraction module is used to extract features from the driving-related data of the vehicle based on multiple feature extraction networks to obtain feature vectors of the vehicle from multiple perspectives. The feature vectors of the multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle. The real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The first feature fusion module is used to fuse the feature vectors from the multiple perspectives to obtain the global driving scene feature vector of the vehicle. The fault identification module is used to determine, based on the global driving scene feature vector, the target driving scene in which the vehicle is located in the current driving mode and whether the autonomous driving system is faulty in the target driving scene.

16. A driving risk assessment device, characterized in that, include: The second acquisition module is used to acquire driving-related data of the vehicle. The driving-related data of the vehicle includes at least the driving data of the vehicle and the movement data of the target obstacle. The target obstacle is an obstacle among the obstacles around the vehicle that affects the driving of the vehicle. The second multi-view feature extraction module is used to extract features from the driving-related data of the vehicle based on multiple feature extraction networks to obtain feature vectors of the vehicle from multiple perspectives. The feature vectors of the multiple perspectives include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector. The predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle. The real trajectory interaction feature vector represents the correlation between the actual driving trajectory of the vehicle and the motion trajectory of the target obstacle. The second feature fusion module is used to fuse the feature vectors from the multiple perspectives to obtain the global driving scene feature vector of the vehicle. The risk assessment module is used to determine the driving risk assessment result based on the global driving scenario feature vector.

17. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-14 to be implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-14 to be implemented.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the method as described in any one of claims 1-14 to be implemented.

Images