Fault identification method and apparatus for autonomous driving system

Abstract

The present application provides a fault identification method and apparatus for an autonomous driving system. The method comprises: acquiring travelling-related data of an ego vehicle (at least including ego vehicle travelling data and target obstacle motion data); respectively performing feature extraction on the travelling-related data of the ego vehicle on the basis of a plurality of feature extraction networks to obtain feature vectors of the ego vehicle from a plurality of viewing angles; fusing the feature vectors from the plurality of viewing angles to obtain a global driving scenario feature vector of the ego vehicle; and on the basis of the global driving scenario feature vector, determining a target driving scenario where the ego vehicle is located in a current driving mode and whether a fault occurs to an autonomous driving system in the target driving scenario. In the present application, dynamic change information of an ego vehicle and a surrounding environment of the ego vehicle is captured from a plurality of viewing angles, the information from the plurality of viewing angles is adaptively fused to obtain comprehensive representations of driving scenarios, and a target driving scenario where the ego vehicle is currently located and whether a fault occurs to an autonomous driving system in the target driving scenario are identified on the basis of the comprehensive representations, thereby improving the accuracy of fault identification results of the autonomous driving system.

Description

Fault identification method and device of automatic driving system

[0001] The present application claims priority from the Chinese patent application No. 202410591542.4, filed on May 13, 2024, and entitled "Fault identification method and device of automatic driving system", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of automatic driving, in particular to a fault identification method and device of an automatic driving system. BACKGROUND

[0003] In recent years, with the continuous development of automatic driving (also known as intelligent driving) technology and the wide application of 5G communication technology, intelligent networked vehicles have become an important development trend in the automotive industry. Software has gradually become the core of the vehicle, and the demand for vehicle remote operation capability has become increasingly prominent. Various remote operation platforms have emerged like mushrooms after rain, but most of the platforms in the industry have great differences in technology and capability. This has led to uneven remote operation capabilities among major vehicle manufacturers, and it is difficult to guarantee the safety and reliability. Among them, as an important part of vehicle remote diagnosis, timely identification of whether the automatic driving system in various driving scenarios has problems is of great significance to the remote operation of automatic driving vehicles, timely detection of vehicle software vulnerabilities and over-the-air (OTA) system.

[0004] However, there are many problems in the existing scheme for fault identification of the automatic driving system, such as poor accuracy of fault identification of the automatic driving system and inaccurate identification of the driving scenario. SUMMARY

[0005] Embodiments of the present application provide a fault identification method and device of an automatic driving system, which captures dynamic change information of a vehicle and its surrounding environment from multiple perspectives, adaptively fuses multi-perspective information to obtain a comprehensive representation of the driving scenario, and identifies the target driving scenario where the vehicle is currently located and whether the intelligent driving system is faulty in the target driving scenario, thereby increasing the accuracy of the fault identification result of the automatic driving system.

[0006] In a first aspect, the present application provides a fault identification method of an automatic driving system, comprising: obtaining driving-related data of a vehicle, the driving-related data of the vehicle at least comprising vehicle driving data and target obstacle motion data, the target obstacle being an obstacle around the vehicle that has an impact on the driving of the vehicle; performing feature extraction on the driving-related data of the vehicle based on a plurality of feature extraction networks respectively, to obtain a plurality of perspective feature vectors of the vehicle, the plurality of perspective feature vectors at least comprising a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing the correlation between the predicted driving trajectory of the vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing the correlation between the real driving trajectory of the vehicle and the motion trajectory of the target obstacle; fusing the plurality of perspective feature vectors to obtain a global driving scene feature vector of the vehicle; and determining, based on the global driving scene feature vector, a target driving scene in which the vehicle is currently located and whether the automatic driving system is faulty in the target driving scene.

[0007] The present application captures the predicted trajectory interaction feature, simulates the perspective of the rule control module to "predict" whether the vehicle and the obstacle have a trajectory intersection and a collision risk, captures the real trajectory interaction feature, obtains the real trajectory relationship (including whether to intersect, etc.) of the vehicle and the obstacle, and combines the predicted trajectory and the real trajectory to improve the accuracy of the fault identification result of the automatic driving system.

[0008] In one possible implementation, the vehicle driving data comprises predicted trajectory data of the vehicle and real trajectory data of the vehicle, the predicted trajectory data of the vehicle indicating predicted driving trajectory data of the vehicle in a target time period, and the real trajectory data of the vehicle indicating real driving trajectory data of the vehicle in the target time period; the target obstacle motion data comprises motion trajectory data of the target obstacle, the motion trajectory data of the target obstacle indicating motion trajectory data of the target obstacle in the target time period; and one specific implementation of performing feature extraction on the driving-related data of the vehicle based on the plurality of feature extraction networks respectively to obtain the plurality of perspective feature vectors of the vehicle is: taking the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a first feature extraction network, and outputting the predicted trajectory interaction feature vector; and taking the real trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a second feature extraction network, and outputting the real trajectory interaction feature vector.

[0009] In this possible implementation, the complex changes and mutual influences of the trajectories of the vehicle and the target obstacle are fully modeled, and the trajectory interaction feature extraction network is designed, which can effectively capture the dynamic influence relationship between the trajectory of the target obstacle and the driving trajectory of the vehicle, and then help to accurately identify the fault result of the automatic driving system in the subsequent steps.

[0010] In another possible implementation, the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are taken as inputs of the first feature extraction network, and a specific implementation of outputting the predicted trajectory interaction feature vector is as follows: the predicted trajectory data of the ego vehicle is taken as input of the first time sequence representation network, and a predicted driving trajectory feature vector of the ego vehicle is output; the motion trajectory data of the target obstacle is taken as input of the second time sequence representation network, and a motion trajectory feature vector of the target obstacle is output; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a first trajectory fusion feature vector; and the first trajectory fusion feature vector is taken as input of the first trajectory interaction network, and the predicted trajectory interaction feature vector is output.

[0011] In another possible implementation, the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are taken as inputs of the second feature extraction network, and a specific implementation of outputting the real trajectory interaction feature vector is as follows: the real trajectory data of the ego vehicle is taken as input of the third time sequence representation network, and a real driving trajectory feature vector of the ego vehicle is output; the motion trajectory data of the target obstacle is taken as input of the fourth time sequence representation network, and a motion trajectory feature vector of the target obstacle is output; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; and the second trajectory fusion feature vector is taken as input of the second trajectory interaction network, and the real trajectory interaction feature vector is output.

[0012] The driving trajectory feature of the ego vehicle and the motion trajectory feature of the target obstacle are respectively extracted through the time sequence representation network, then the two trajectory features are spliced to input the trajectory interaction network, and the interaction feature of the driving trajectory of the ego vehicle and the motion trajectory of the target obstacle is accurately extracted, the correlation of the driving trajectory of the ego vehicle and the motion trajectory of the target obstacle is obtained, that is, the influence of the dynamic motion of the target obstacle on the driving trajectory of the ego vehicle is obtained.

[0013] Optionally, the first time sequence representation network, the second time sequence representation network, the third time sequence representation network and the fourth time sequence representation network are all constructed based on a temporal convolutional network (TCN); and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on a multi-layer perceptron (MLP) network.

[0014] Of course, in some other examples, the time sequence representation network (including the first time sequence representation network, the second time sequence representation network, the third time sequence representation network, and the fourth time sequence representation network) and the trajectory interaction network (including the first trajectory interaction network and the second trajectory interaction network) can also be implemented by other neural network models. For example, the time sequence representation network can also be a recurrent neural network (RNN) or a Transformer neural network model, and the trajectory interaction network can also be a fully connected neural network (FCN), and the specific implementation structure of the time sequence representation network and the trajectory interaction network is not limited in the present application.

[0015] In another possible implementation, a specific implementation of fusing the feature vectors of multiple perspectives to obtain the global driving scene feature vector of the ego vehicle is as follows: determining a first attention weight corresponding to the predicted trajectory interaction feature vector and a second attention weight corresponding to the real trajectory interaction feature vector; and based on the first attention weight and the second attention weight, performing weighted fusion on the predicted trajectory interaction feature vector and the real trajectory interaction feature vector to obtain the global driving scene feature vector of the ego vehicle.

[0016] Through adaptive fusion of the feature vectors of multiple perspectives, the influence of different perspectives is considered, a perspective dimension attention mechanism is designed to weightedly fuse multiple perspective information, the global representation of the driving scene is obtained, and the intelligent driving fault recognition accuracy is improved.

[0017] In another possible implementation, the target obstacle is retrieved from the operation log of the ego vehicle; for example, in a specific scenario, the ego vehicle automatically records and uploads the target obstacle id, and the target obstacle can be obtained by retrieving from the vehicle operation log when the target obstacle of the ego vehicle is determined. For example, in an advanced emergency braking (AEB) false triggering scenario, the vehicle has a sudden braking or sudden stop event, and the vehicle automatically records the obstacles before and after the sudden braking in the driving log (such as recording the obstacles in front of the vehicle). Alternatively, the target obstacle is selected from the obstacles around the ego vehicle based on a preset rule, for example, the target obstacle is determined according to the distance between the obstacles around the ego vehicle and the ego vehicle, such as determining a plurality of obstacles closest to the ego vehicle within a range of 50 meters from the ego vehicle as the target obstacle.

[0018] In another possible implementation, the driving-related data of the ego vehicle further includes road environment data, the road environment data being indicative of road environment information in which the ego vehicle drives; the feature vectors of the multiple perspectives further include a road environment feature vector, and the global driving scene feature vector of the ego vehicle is further related to the road environment feature vector, the road environment feature vector being obtained based on feature extraction on the road environment data.

[0019] Considering the influence of the road environment information, the complex state changes of the ego vehicle, the target obstacle and the road environment, i.e., the mutual influence relationship, are fully modeled, a more comprehensive representation of the driving scene is obtained, and the accuracy of fault identification of the autonomous driving system is further improved.

[0020] Optionally, the road environment data includes one or more of a lane number, a lane line type, a distance to an intersection, and a distance to a traffic light of a road on which the ego vehicle drives.

[0021] In another possible implementation, the driving-related data of the ego vehicle further includes driving state data of the ego vehicle, the driving state data including one or more of an intelligent driving state, a speed, an acceleration, a position coordinate, an orientation angle and a brake value of the ego vehicle; and / or, the target obstacle motion data further includes motion state data of the target obstacle, the motion state data including one or more of a speed, an acceleration, a position coordinate of the target obstacle.

[0022] In another possible implementation, the feature vectors of the multiple perspectives further include a driving scene feature vector; the driving scene feature vector is obtained based on feature extraction on first driving scene data and / or second driving scene data; the first driving scene data includes result data calculated based on a dynamic equation and / or priori knowledge on the driving data and the target obstacle motion data, the result data being related to a driving mode of the ego vehicle; the second driving scene data includes the driving data and the target obstacle motion data corresponding to a fault time point, the fault time point indicating a time point at which an abnormality is likely to occur in driving of the ego vehicle; and the global driving scene feature vector of the ego vehicle is further related to the driving scene feature vector.

[0023] The driving scene features are enhanced by using priori knowledge (for example, certain rules) and / or a dynamic equation, for example, by fault point information extraction (i.e., the second driving scene data), focusing on changes in the ego vehicle and the surrounding environment before and after the fault, amplifying the fault mode, and thus improving the accuracy of fault identification of the autonomous driving system.

[0024] In another possible implementation, the result data includes one or more of a collision time of the ego vehicle and the target obstacle, a predicted driving trajectory of the ego vehicle, a duration of the target obstacle, whether the target obstacle is a ghost obstacle, whether to autonomously brake, a brake value, and a direction angle.

[0025] According to the perception and control experience, the collision time of the ego vehicle and the target obstacle, whether the target obstacle is a ghost obstacle, whether to autonomously brake, and the like are constructed to build scene information, so as to improve the richness of the scene information.

[0026] In a second aspect, the present application also provides a driving risk assessment method, including obtaining driving related data of an ego vehicle, the driving related data of the ego vehicle at least including ego vehicle driving data and target obstacle motion data, the target obstacle being an obstacle in the surroundings of the ego vehicle that has an influence on the driving of the ego vehicle; performing feature extraction on the driving related data of the ego vehicle based on a plurality of feature extraction networks respectively to obtain a plurality of perspective feature vectors of the ego vehicle, the plurality of perspective feature vectors at least including a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing the correlation between the predicted driving trajectory of the ego vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing the correlation between the real driving trajectory of the ego vehicle and the motion trajectory of the target obstacle; fusing the plurality of perspective feature vectors to obtain a global driving scene feature vector of the ego vehicle; and determining a driving risk assessment result based on the global driving scene feature vector.

[0027] That is, the global driving scene feature obtained by fusing the multi-perspective features is also used for driving risk assessment, so as to obtain a more accurate driving risk assessment result.

[0028] Optionally, the driving risk assessment result includes one or more of a traffic accident risk assessment result, a vehicle risk assessment result, and a driver driving risk assessment result.

[0029] In a third aspect, the present application also provides a fault identification device of an automatic driving system, including a first obtaining module, a first multi-perspective feature extraction module, a first feature fusion module, and a fault identification module, wherein the first obtaining module is configured to obtain driving related data of an ego vehicle, the driving related data of the ego vehicle at least including ego vehicle driving data and target obstacle motion data, the target obstacle being an obstacle in the surroundings of the ego vehicle that has an influence on the driving of the ego vehicle; the first multi-perspective feature extraction module is configured to perform feature extraction on the driving related data of the ego vehicle based on a plurality of feature extraction networks respectively to obtain a plurality of perspective feature vectors of the ego vehicle, the plurality of perspective feature vectors at least including a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing the correlation between the predicted driving trajectory of the ego vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing the correlation between the real driving trajectory of the ego vehicle and the motion trajectory of the target obstacle; the first feature fusion module is configured to fuse the plurality of perspective feature vectors to obtain a global driving scene feature vector of the ego vehicle; and the fault identification module is configured to determine, based on the global driving scene feature vector, a target driving scene in which the ego vehicle is currently located and whether the automatic driving system is faulty in the target driving scene.

[0030] In a possible implementation, the ego vehicle driving data includes predicted trajectory data of the ego vehicle and real trajectory data of the ego vehicle, the predicted trajectory data of the ego vehicle indicating predicted driving trajectory data of the ego vehicle in a target time period, and the real trajectory data of the ego vehicle indicating real driving trajectory data of the ego vehicle in the target time period; the target obstacle motion data includes motion trajectory data of the target obstacle, the motion trajectory data of the target obstacle indicating motion trajectory data of the target obstacle in the target time period; the first multi-view feature extraction module is specifically configured to: take the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle as inputs of a first feature extraction network, and output a predicted trajectory interaction feature vector; and take the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle as inputs of a second feature extraction network, and output a real trajectory interaction feature vector.

[0031] In another possible implementation, one specific implementation of taking the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle as inputs of the first feature extraction network and outputting the predicted trajectory interaction feature vector is as follows: taking the predicted trajectory data of the ego vehicle as an input of a first time sequence representation network, and outputting a predicted driving trajectory feature vector of the ego vehicle; taking the motion trajectory data of the target obstacle as an input of a second time sequence representation network, and outputting a motion trajectory feature vector of the target obstacle; fusing the predicted driving trajectory feature vector and the motion trajectory feature vector to obtain a first trajectory fusion feature vector; and taking the first trajectory fusion feature vector as an input of a first trajectory interaction network, and outputting the predicted trajectory interaction feature vector.

[0032] In another possible implementation, one specific implementation of taking the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle as inputs of the second feature extraction network and outputting the real trajectory interaction feature vector is as follows: taking the real trajectory data of the ego vehicle as an input of a third time sequence representation network, and outputting a real driving trajectory feature vector of the ego vehicle; taking the motion trajectory data of the target obstacle as an input of a fourth time sequence representation network, and outputting a motion trajectory feature vector of the target obstacle; fusing the real driving trajectory feature vector and the motion trajectory feature vector to obtain a second trajectory fusion feature vector; and taking the second trajectory fusion feature vector as an input of a second trajectory interaction network, and outputting the real trajectory interaction feature vector.

[0033] Optionally, the first time sequence representation network, the second time sequence representation network, the third time sequence representation network, and the fourth time sequence representation network are all constructed based on a TCN; and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on an MLP network.

[0034] In another possible implementation, the first feature fusion module is specifically configured to: determine a first attention weight corresponding to the predicted trajectory interaction feature vector and a second attention weight corresponding to the real trajectory interaction feature vector; and perform weighted fusion on the predicted trajectory interaction feature vector and the real trajectory interaction feature vector based on the first attention weight and the second attention weight to obtain the global driving scene feature vector of the ego vehicle.

[0035] In another possible implementation, the target obstacle is retrieved from a running log of the ego vehicle, or is filtered from obstacles around the ego vehicle based on a preset rule.

[0036] In another possible implementation, the driving related data of the ego vehicle further includes road environment data, the road environment data being indicative of road environment information in which the ego vehicle drives; the feature vectors of the multiple perspectives further include a road environment feature vector, the global driving scene feature vector of the ego vehicle being further related to the road environment feature vector, and the road environment feature vector being obtained based on feature extraction on the road environment data.

[0037] Optionally, the road environment data includes one or more of a lane number, a lane line type, a distance to an intersection, and a distance to a traffic light of a road on which the ego vehicle is located.

[0038] In another possible implementation, the driving data of the ego vehicle further includes one or more of an intelligent driving state, a speed, an acceleration, a position coordinate, an orientation angle, and a brake value of the ego vehicle; and / or, the target obstacle motion data further includes one or more of a speed, an acceleration, and a position coordinate of the target obstacle.

[0039] In another possible implementation, the feature vectors of the multiple perspectives further include a driving scene feature vector, the driving scene feature vector being obtained based on feature extraction on first driving scene data and / or second driving scene data, the first driving scene data including result data calculated based on a dynamics equation and / or priori knowledge on the driving data and the target obstacle motion data, the result data being related to a driving mode of the ego vehicle, the second driving scene data including the driving data and the target obstacle motion data corresponding to a failure time point, the failure time point being indicative of a time point at which the driving of the ego vehicle is likely to be abnormal, and the global driving scene feature vector of the ego vehicle being further related to the driving scene feature vector.

[0040] In another possible implementation, the result data includes one or more of a collision time of the ego vehicle and the target obstacle, a predicted driving trajectory of the ego vehicle, whether the target obstacle is a ghost obstacle, whether to autonomously brake, a duration of the target obstacle, a brake value, and a direction angle.

[0041] In a fourth aspect, the present application provides a driving risk assessment device, comprising a second acquisition module, a second multi-view feature extraction module, a second feature fusion module and a risk assessment module. The second acquisition module is configured to acquire driving related data of a host vehicle, wherein the driving related data of the host vehicle at least includes host vehicle driving data and target obstacle motion data, and the target obstacle is an obstacle in the surroundings of the host vehicle which has an influence on driving of the host vehicle. The second multi-view feature extraction module is configured to perform feature extraction on the driving related data of the host vehicle from multiple views to obtain feature vectors of multiple views of the host vehicle, wherein the feature vectors of multiple views at least include a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector represents a correlation between a predicted driving trajectory of the host vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector represents a correlation between a real driving trajectory of the host vehicle and the motion trajectory of the target obstacle. The second feature fusion module is configured to fuse the feature vectors of multiple views to obtain a global driving scene feature vector of the host vehicle. The risk assessment module is configured to determine a driving risk assessment result based on the global driving scene feature vector.

[0042] Optionally, the driving risk assessment result includes one or more of a traffic accident risk assessment result, a vehicle risk assessment result and a driver driving risk assessment result.

[0043] In a fifth aspect, an embodiment of the present application provides a computing device, comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method of the first aspect and / or the second aspect is implemented.

[0044] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method of the first aspect or the second aspect is implemented.

[0045] In a seventh aspect, an embodiment of the present application further provides a computer program or a computer program product, which comprises instructions, and when the instructions are executed, the computer executes the method of the first aspect or the second aspect.

[0046] In an eighth aspect, an embodiment of the present application further provides a chip, comprising at least one processor and a communication interface, wherein the processor is configured to execute the method of the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0047] FIG. 1 is a functional block diagram of a vehicle according to an embodiment of the present application;

[0048] FIG. 2 is a schematic diagram of a computer system according to an embodiment of the present application;

[0049] FIG. 3 is a framework diagram of an automatic driving system provided by the present application;

[0050] FIG. 4 shows a system architecture diagram of an automatic driving system fault identification method system provided by an embodiment of the present application;

[0051] FIG. 5 is a flow diagram of an automatic driving system fault identification method provided by an embodiment of the present application;

[0052] FIG. 6 shows a system implementation architecture diagram of an automatic driving system fault identification method provided by an embodiment of the present application;

[0053] FIG. 7 shows a processing process diagram of an input data of a fault identification model;

[0054] FIG. 8 is a structure diagram of an automatic driving system fault identification apparatus provided by an embodiment of the present application;

[0055] FIG. 9 is a structure diagram of a driving risk assessment apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0056] The term “and / or” mentioned in the present document is a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol “ / ” in the present document represents an or relationship of the associated objects, for example, A / B represents A or B.

[0057] The terms “first” and “second” and the like in the description and claims of the present document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first feature extraction network and the second feature extraction network are used to distinguish different feature extraction networks, and are not used to describe a specific order of the feature extraction networks.

[0058] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of “exemplary” or “for example” is intended to present concepts in a concrete manner. The words “exemplary” or “for example” are used to present concepts in a concrete manner.

[0059] In the description of the embodiments of the present application, unless otherwise specified, “a plurality of” means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0060] In order to facilitate understanding of the schemes of the embodiments of the present application, the technical terms involved in the present document will be explained first as follows.

[0061] Autonomous vehicles: Autonomous vehicles (Self-driving automobile) is also called unmanned vehicles, computer driving vehicles, intelligent driving vehicles or wheeled mobile robots, which is a kind of intelligent vehicle that realizes unmanned driving through computer system. Autonomous vehicles rely on artificial intelligence, visual computing, radar, monitoring devices and global positioning system to cooperate, so that the computer can automatically and safely operate motor vehicles without any human initiative.

[0062] Fault identification of autonomous driving system: Based on various data reported by autonomous driving vehicles, the driving scene where the autonomous driving vehicle is located is identified, and whether the autonomous driving system is normal in this scene is further identified. Fault events can correspond to error brake, vehicle collision caused by not timely deceleration, wrong obstacle identification and other operations of the autonomous driving system.

[0063] Time to collision (TTC): refers to the time for the vehicle to reach the front vehicle at the current speed, usually in seconds.

[0064] AEB: A system that monitors the driving environment in front of the vehicle and automatically activates the vehicle braking system to slow down the vehicle when there is a risk of collision to avoid or mitigate the collision.

[0065] Adaptive cruise control (ACC): is a comfortable auxiliary driving function. If the front of the vehicle is smooth, adaptive cruise control will maintain the maximum cruise speed set forward. If a vehicle is detected in front, adaptive cruise control will reduce the vehicle speed as needed to maintain a distance based on the selected time from the front vehicle until the appropriate cruise speed is reached.

[0066] There are many problems in the related technical solutions for fault identification of autonomous driving system. For example, in related technology one, AI module is used to process trajectory data, vibration, sound and other signals; a rule base is constructed, the processed signals are fused through the rule module, the results of each rule module are fused, and high-order rules are used for fault identification.

[0067] However, the related art has problems that the automatic driving scene is complex and changeable, contains various running characteristics such as intelligent driving, navigation cruise assist driving, ACC, AEB and the like, there are more than one hundred fault scenes, the rules / policies obtained by manual analysis of a small amount of data can only cover a small part of the scene or part of the scene, resulting in low fault recognition accuracy; and the description of some fault scenes is extremely complex and even cannot be described, such as "unreasonable", "unexpected" and the like, which involve subjective feelings of people, and cannot be quantified by rules.

[0068] The related art two adjusts the receptive field of the model by combining various kernel sizes of convolution, simultaneously captures multi-scale information in the time series data, and the structure can be flexibly expanded and combined with other deep learning frameworks. When this scheme is expanded to the fault recognition of the automatic driving operation and maintenance scene, it cannot simultaneously process multiple groups of time series (such as trajectories) and non-time series data (such as whether a curve or a ghost obstacle and the like), and cannot realize the interaction of the self-vehicle state information and the dynamic environment information.

[0069] In addition, the automatic driving system fault recognition also faces the problem that each business field focuses on the internal positioning of its own field, different modules are strongly coupled in the actual scene, and it is difficult to determine which fault level scene or module according to the result of a single module. It takes a lot of time and effort to manually develop and maintain the rule library.

[0070] Therefore, the embodiments of the present application provide a fault recognition method and device of an automatic driving system, which captures the dynamic change information of the self-vehicle and the surrounding obstacles from multiple perspectives, adaptively fuses the multi-perspective information to obtain a comprehensive representation of the driving scene, and identifies the target driving scene where the self-vehicle is currently located and whether the intelligent driving system is faulty in the target driving scene, without the need for a manually maintained rule library, so as to accurately identify the target driving scene where the self-vehicle is currently located and whether the automatic driving system is faulty in the target driving scene.

[0071] The specific implementation of the fault recognition method of the automatic driving system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0072] FIG. 1 is a functional block diagram of a vehicle according to an embodiment of the present application. As shown in FIG. 1, the vehicle 100 includes various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112, and a user interface 116.

[0073] Optionally, the vehicle 100 can include more or fewer subsystems, and each subsystem can include multiple elements. Additionally, each subsystem and element of the vehicle 100 can be interconnected by wires or wirelessly.

[0074] The travel system 102 includes components that provide powered movement for the vehicle 100. In one example, the travel system 102 can include an engine 118, an energy source 119, a transmission 120, and wheels 121. The engine 118 can be a combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine composed of a gasoline engine and an electric motor, a hybrid engine composed of a combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy.

[0075] Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity, etc. The energy source 119 can also provide energy for other systems of the vehicle 100.

[0076] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission can include a gearbox, a differential, and a drive shaft.

[0077] In one example, the transmission 120 can also include other devices, such as a clutch. The drive shaft can include one or more shafts that can be coupled to one or more wheels 121.

[0078] The sensor system 104 can include several sensors that sense environmental information about the surroundings of the vehicle 100.

[0079] For example, the sensor system 104 can include a positioning system 122 (which can be a GPS system, a Beidou system, or other positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 can also include sensors of internal systems of the vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the vehicle 100.

[0080] The positioning system can be used to estimate the geographic position of the vehicle 100. The IMU 124 is used to sense changes in position and orientation of the vehicle 100 based on inertial acceleration. In one example, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0081] The radar 126 can utilize radio signals to sense objects within the vehicle’s 100 surrounding environment. In one example, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or heading of the objects.

[0082] The laser rangefinder 128 can utilize laser light to sense objects in the environment in which the vehicle 100 is located. In one example, the laser rangefinder 128 can include one or more laser sources, a laser scanner, and one or more detectors, among other system components.

[0083] The camera 130 can be used to capture multiple images of the vehicle’s 100 surrounding environment. The camera 130 can be a still camera or a video camera.

[0084] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 can include various elements, including a steering system 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0085] The steering system 132 is operable to adjust the heading of the vehicle 100. For example, in one example the steering system 132 can include a steering wheel system.

[0086] The throttle 134 is used to control the operational speed of the engine 118 and, in turn, the speed of the vehicle 100.

[0087] The braking unit 136 is used to control the deceleration of the vehicle 100. The braking unit 136 can use friction to slow the wheels 121. In other examples, the braking unit 136 can convert the kinetic energy of the wheels 121 into electrical energy. The braking unit 136 can also take other forms to slow the rotational speed of the wheels 121 to control the speed of the vehicle 100.

[0088] The computer vision system 140 can operate to process and analyze images captured by the camera 130 to identify objects and / or features in the vehicle’s 100 surrounding environment. The objects and / or features can include traffic signals, road boundaries, and obstacles, among others. The computer vision system 140 can use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some examples, the computer vision system 140 can be used to map the environment, track objects, estimate the speed of objects, and the like.

[0089] The route control system 142 is used to determine the travel route of the vehicle 100. In some examples, the route control system 142 can determine the travel route for the vehicle 100 in conjunction with data from the sensors 138, the positioning system 122, and one or more predetermined maps.

[0090] The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise navigate around potential obstacles in the environment of the vehicle 100.

[0091] Of course, in one example, the control system 106 can include or replace components other than those shown and described. Or some of the components shown above can be reduced.

[0092] The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripherals 108. The peripherals 108 can include a wireless communication system 146, an on-board computer 148, a microphone 150, and / or a speaker 152.

[0093] In some examples, the peripherals 108 provide a means for a user of the vehicle 100 to interact with the user interface 116. For example, the on-board computer 148 can provide information to a user of the vehicle 100. The user interface 116 can also operate the on-board computer 148 to receive input from the user. The on-board computer 148 can be operated through a touch screen. In other cases, the peripherals 108 can provide a means for the vehicle 100 to communicate with other devices located within the vehicle. For example, the microphone 150 can receive audio (e.g., voice commands or other audio input) from a user of the vehicle 100. Similarly, the speaker 152 can output audio to a user of the vehicle 100.

[0094] The wireless communication system 146 can wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular networks, such as LTE. Or 5G cellular communication. The wireless communication system 146 can utilize WiFi for wireless local area network (WLAN) communication. In one example, the wireless communication system 146 can utilize an infrared link, Bluetooth, or ZigBee for direct communication with devices. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 can include one or more dedicated short range communications (DSRC) devices, which can include public and / or private data communication between vehicles and / or roadside stations.

[0095] Power source 110 can provide power to various components of vehicle 100. In one example, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such a battery can be configured as the power source to provide power to various components of vehicle 100. In some examples, power source 110 and energy source 119 can be implemented together, such as in some all-electric vehicles.

[0096] Some or all of the functionality of vehicle 100 is controlled by computer system 112. Computer system 112 can include at least one processor 113 that executes instructions 115 stored in a non-transitory computer readable storage medium, such as memory 114. Computer system 112 can also be a plurality of computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0097] Processor 113 can be any conventional processor, such as a commercially available CPU. Alternatively, the processor can be a dedicated device such as an ASIC or other hardware-based processor. Although FIG. 1 functionally illustrates the processor, the memory, and other elements of computer 110 in the same block, such a processor, computer, or memory can actually include multiple processors, computers, or memories that operate in coordination. For example, the memory can be a hard drive or other storage medium located in a housing different from that of the computer 110. References to the processor or computer shall thus be taken to include references to a collection of processors or computers or memories that can or can not operate in parallel. Unlike using a single processor to perform the steps described herein, some components such as the steering assembly and the deceleration assembly can each have their own processor that only performs calculations related to the function of the particular component.

[0098] In each aspect described herein, the processor can be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking the necessary steps to perform a single operation.

[0099] In some examples, memory 114 can contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 can also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of travel system 102, sensor system 104, control system 106, and peripherals 108.

[0100] In addition to instructions 115, memory 114 can store data, such as road maps, route information, the location, direction, speed, and other similar vehicle data of the vehicle, and other information. Such information can be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0101] User interface 116 is used to provide information to or receive information from a user of vehicle 100. Optionally, user interface 116 can include one or more input / output devices within the set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker.

[0102] Computer system 112 can control the functions of vehicle 100 based on inputs received from various subsystems (e.g., travel system 102, sensor system 104, and control system 106), as well as from user interface 116. For example, computer system can utilize inputs from control system 106 in order to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some examples, computer system 112 can be operable to provide control over multiple aspects of vehicle 100 and its subsystems.

[0103] Optionally, one or more of the above-described components can be installed separately from or associated with vehicle 100. For example, memory 114 can exist partially or entirely separately from vehicle 100. The above-described components can be communicatively coupled together in a wired and / or wireless manner.

[0104] Optionally, the above-described components are just one example, and in actual applications, components in each of the above-described modules can be added or deleted according to actual needs, and FIG. 1 should not be understood as a limitation on the embodiments of the present application.

[0105] An autonomous vehicle that travels on a road, such as vehicle 100 above, can identify objects (also referred to as obstacles) within its surrounding environment in order to determine a driving strategy for vehicle 100, such as braking, decelerating, accelerating, steering, etc. The objects can be other vehicles, pedestrians, animals, traffic devices (such as traffic signs, road signs, speed bumps, and street lights, etc.), or other types of objects. In some examples, each identified object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine a speed at which the autonomous vehicle is to adjust.

[0106] Optionally, the autonomous vehicle 100 or a computing device associated with the autonomous vehicle 100 (e.g., the computer system 112, the computer vision system 140, the memory 114 of FIG. 1) can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, snow, ice on the road, etc.). Optionally, each identified object depends on the behavior of the other identified objects, and thus all the identified objects can be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified object. In other words, the autonomous vehicle can determine what state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop, etc.) based on the predicted behavior of the object. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc.

[0107] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the vehicle 100 to cause the autonomous vehicle to follow a specified trajectory and / or maintain a safe lateral and longitudinal distance from objects in the vicinity of the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).

[0108] The vehicle 100 described above can be a car, a van, a truck, or any other type of vehicle, etc., and the embodiments of the present application are not particularly limited.

[0109] FIG. 2 is a schematic diagram of a computer system according to an embodiment of the present application.

[0110] The computer system 112 as shown in FIG. 2 includes a processor 113 coupled to a system bus. The processor 113 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 can drive a display 109 coupled to the system bus 105. The system bus 105 is coupled via a bus bridge 111 and an input / output (I / O) bus 113 to an I / O interface 115. The I / O interface 115 is coupled to a variety of devices, such as input devices 117 (e.g., keyboard, mouse, touchscreen, etc.), a media tray 121 (e.g., CD-ROM, DVD, media interface, etc.), a transceiver 123 (which can send and / or receive radio communication signals), a camera 155 (which can capture still and moving digital video images), and an external USB interface 125. Optionally, the interface connected to the I / O interface 115 can be a USB interface.

[0111] The processor 113 can be any conventional processor, including a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, or a combination of the above. Alternatively, the processor can be a special-purpose device, such as an application specific integrated circuit (ASIC). Alternatively, the processor 113 can be a neural network processor or a combination of a neural network processor and a conventional processor as described above.

[0112] Optionally, in various embodiments described herein, the computer system 112 can be located remotely from the autonomous vehicle and can communicate wirelessly with the autonomous vehicle or can be deployed in the autonomous vehicle to facilitate communication with other systems in the vehicle and to reduce communication latency. In other aspects, some of the processes described herein are performed on a processor disposed within the autonomous vehicle and others are performed by a remote processor, including taking actions required to perform a single operation.

[0113] The computer system 112 can communicate with a software deployment server 149 through a network interface 129 and a network 127. The network interface 129 is a hardware network interface, such as a network card. The network 127 can be an external network, such as the Internet, or an internal network, such as an Ethernet or virtual private network (VPN). Optionally, the network 127 can also be a wireless network, such as a WiFi network, a cellular network, or the like.

[0114] The hard drive interface and the system bus 105 are coupled. The hard drive interface and the hard drive are connected. The memory 114 and the system bus 105 are coupled. Data running on the memory 114 can include an operating system 137 and an application 143 of the computer system 112.

[0115] The operating system includes a shell 139 and a kernel 141. The shell 139 is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system, waits for the user's input, interprets the user's input to the operating system, and processes various outputs of the operating system.

[0116] The kernel 141 is composed of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Directly interacting with hardware, the operating system kernel usually runs processes and provides inter-process communication, provides CPU time slice management, interrupts, memory management, IO management, etc.

[0117] The application programs 143 include programs related to controlling the vehicle to drive autonomously, such as programs for managing the interaction between the autonomous vehicle and the obstacles on the road, programs for controlling the route or speed of the autonomous vehicle, and programs for controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road. The application programs 143 also exist on the system of the software deploying server 149. In one example, when the application programs 147 need to be executed, the computer system 112 can download the application programs 143 from the software deploying server 149.

[0118] For example, the application programs 141 can also be programs for controlling the autonomous vehicle to avoid collision with other vehicles and safely pass through intersections.

[0119] The sensors 153 are associated with the computer system 112. The sensors 153 are used to detect the environment around the computer system 112. For example, the sensors 153 can detect animals, vehicles, pedestrians, crosswalks, and the like, and further can detect the environment around the above-mentioned animals, vehicles, pedestrians, crosswalks, and the like. Optionally, if the computer system 112 is located on an autonomous vehicle, the sensors can be cameras, infrared sensors, chemical detectors, microphones, and the like.

[0120] For example, the processor 113 can predict the driving trajectory of other vehicles according to the surrounding road conditions and other vehicle conditions detected by the sensors 153.

[0121] FIG. 3 is a block diagram of an autonomous driving system provided by the present application. As shown in FIG. 3, the autonomous driving system includes a perception unit 31, a target fusion unit 32, a prediction unit 33, a planning unit 34, a control unit 35, a map unit 36, and a navigation unit 37. The prediction unit 33 receives the perception unit 31, the target fusion unit 32, and the map unit 36, and through a prediction algorithm, can give the future behavior intention and trajectory of the obstacles, and then outputs them to the downstream planning module. This can help the ego vehicle to predict the future trajectory of the target vehicle and the importance of the target, and in dangerous scenarios, is conducive to planning and control to take emergency safety measures to ensure the safety of the vehicle and avoid collision.

[0122] The function of the perception unit 31 is implemented based on the sensor system 104 in FIG. 1 or the sensor 153 in FIG. 2, and senses environmental information around the vehicle 100, such as obstacles (for example, other vehicles, pedestrians, animals, etc.), road information (for example, crosswalks, lane lines, traffic signal lights, etc.), and other information that affects driving. The target fusion unit 32 processes the environmental information around the vehicle sensed by the perception unit 31 and outputs obstacle target information. The map unit is stored in the memory 114 in FIG. 1 or FIG. 2, and the prediction unit 33 predicts the behavior intention and future trajectory of the target vehicle based on the current map information and the target information sensed by the perception unit. The planning unit 34 plans the driving route of the vehicle based on the prediction result of the prediction unit and / or the output information of the navigation unit 37, and the control unit 35 controls the vehicle to drive according to the driving route planned by the planning unit. The target fusion unit 32, the prediction unit 33, the planning unit 33, and the control unit 35 are all implemented in the processor in FIG. 1 or FIG. 2. During the driving of the vehicle, real-time, accurate, and reliable prediction of the intention of other vehicles can help the ego vehicle to predict the traffic conditions in front of the vehicle, establish the traffic situation around the ego vehicle, help to judge the importance of the surrounding target vehicles, filter the key targets for interaction, facilitate the ego vehicle to plan the path in advance, and safely pass through complex road scenarios.

[0123] Due to the complexity and variability of vehicle driving scenarios, problems may occur in the actual application scenarios of the automatic driving system, causing the automatic driving system to fail to correctly handle the driving risk scenarios. Therefore, it is necessary to identify whether the automatic driving system has problems in various driving scenarios in a timely manner, that is, to identify the faults of the automatic driving system, discover the bugs of the automatic driving software of the vehicle in a timely manner, and then repair the bugs to improve the safety of the automatic driving system and the driving experience of the user. Therefore, timely and accurate identification of the faults of the automatic driving system (that is, identification of the faults of the automatic driving system before the faults of the automatic driving system cause the vehicle to have an accident) and timely repair are crucial for the safe driving of the vehicle and the driving experience of the user.

[0124] The embodiments of the present application provide a fault identification method and device for an automatic driving system, which can timely and accurately identify the faults of the automatic driving system, repair the faults in a timely manner before the faults of the automatic driving system cause the vehicle to have an accident, and improve the driving safety and the driving experience of the user.

[0125] The fault identification method for the automatic driving system provided by the embodiments of the present application can be applied to any level of automatic driving system, such as an L1 level automatic driving system, an L2 level automatic driving system, an L3 level automatic driving system, an L4 level automatic driving system, and an L5 level automatic driving system. That is, the embodiments of the present application do not limit the automatic driving level of the automatic driving system.

[0126] The fault identification device of the automatic driving system provided by the embodiments of the present application can be deployed in the cloud to help the remote operation and maintenance system to discover the faults of the automatic driving system in time and to perform OTA on the automatic driving system of the vehicle in time, thereby improving the driving safety and the automatic driving experience.

[0127] FIG. 4 shows a system architecture diagram of an application of the fault identification method of the automatic driving system provided by the embodiments of the present application. As shown in FIG. 4, the vehicle end generates various types of data in the driving process, for example, including the running trajectory, the video of the driving recorder, the road environment information, etc. The vehicle end collects and reports the data to the cloud end in time. The cloud end reports the vehicle driving data of a certain time period to the operation and maintenance interface and stores the related operation and maintenance event data to the database according to the corresponding screening rules or the self-triggering of the vehicle owner. After the massive operation and maintenance event data stored in the database are checked and classified by the operation and maintenance personnel, the fault identification model can be constructed offline and packaged and encapsulated to the cloud system. In the real-time running process of the cloud system, once the operation and maintenance event is reported, the cloud system can read the vehicle data related to the operation and maintenance event and call the packaged fault identification model to identify the driving scene of the vehicle in the operation and maintenance event and whether the automatic driving system of the vehicle is normally running in the reporting time period. For the operation and maintenance event of the abnormal processing of the automatic driving system, the cloud end dispatches the work order containing the fault identification result and the related data to the root cause positioning module to further confirm the cause and the module causing the abnormality of the automatic driving system, adds the fault root cause information, and dispatches it to the operation and maintenance engineer. The operation and maintenance engineer can repair the related module code of the automatic driving system according to the work order information and trigger the remote software upgrade of the automatic driving system. Through the remote fault identification and positioning, on the one hand, the reliability of the automatic driving system is ensured, and on the other hand, the fault processing efficiency and the user experience are improved.

[0128] Of course, in some other examples, the fault identification device of the automatic driving system can also be deployed in the vehicle end, for example, in the computer system 12 in FIG. 1 to realize the real-time fault identification of the automatic driving system.

[0129] FIG. 5 is a flowchart of a fault identification method of an autonomous driving system according to an embodiment of the present application. The method can be executed by any device, apparatus, platform or cluster of devices with computing capability. The specific computing device for executing the method is not limited in the embodiments of the present application, and a suitable computing device can be selected as needed for execution. For example, the method can be executed on a vehicle-side device, such as a computer system on the vehicle side, or on a vehicle-side device and a cloud-side device (e.g., a server on the cloud side, such as a remote operation and maintenance server of the autonomous driving system), that is, the method can be executed in an end-cloud collaborative architecture, or the method can be executed on a cloud-side device to provide a fault identification service of the autonomous driving system to users in the form of a cloud service. Hereinafter, for the convenience of description, the form of the execution subject is not distinguished, and the fault identification system of the autonomous driving system is described. As shown in FIG. 5, the fault identification method of the autonomous driving system according to the embodiments of the present application includes steps S501 to S504.

[0130] In step S501, driving-related data of a host vehicle is obtained.

[0131] The host vehicle generates various data related to driving of the vehicle during driving, such as host vehicle driving data and motion data of obstacles around the host vehicle.

[0132] The host vehicle driving data can include driving state data of the host vehicle, real driving trajectory (also referred to as posterior driving trajectory) data of the host vehicle, and predicted driving trajectory data of the host vehicle.

[0133] The driving state data of the host vehicle includes intelligent driving state, speed, acceleration, position coordinates, orientation angle, and brake value of the host vehicle. The driving state data of the host vehicle can be directly obtained from the driving log of the host vehicle. For example, the computer system of the vehicle records the driving log of the vehicle in real time, and the driving log includes the driving state data of the vehicle, such as intelligent driving state, speed, acceleration, position coordinates, orientation angle, and brake value of the vehicle during driving. The intelligent driving state of the host vehicle can indicate the driving mode of the host vehicle, such as ACC mode, NCA mode, automatic parking mode, etc.

[0134] The real driving trajectory of the host vehicle can be calculated from the motion state data of the host vehicle, such as the position coordinate sequence of the host vehicle obtained from the position coordinates of the host vehicle at continuous time t1 to tk, and the real driving trajectory of the host vehicle can be obtained from the position coordinate sequence.

[0135] The predicted driving trajectory of the ego vehicle can be obtained by estimating the motion state of the ego vehicle at time t1 to obtain the predicted driving trajectory of the ego vehicle from time t1 to time tk. For example, by calculating the speed, heading angle and position coordinates of the ego vehicle at time t1 through the dynamics equation, the predicted driving trajectory of the ego vehicle from time t1 to time tk is obtained. Alternatively, the driving state data of the ego vehicle at time t1 can be input into the trajectory prediction model to obtain the predicted driving trajectory of the ego vehicle from time t1 to time tk. The trajectory prediction model is a neural network model obtained through training, which can map the driving state data of the ego vehicle at a certain time to the predicted driving trajectory of the ego vehicle in a certain time period, for example, mapping the driving state data at time t1 to the predicted driving trajectory from time t1 to time tk.

[0136] It can be understood that time t1 is the current time at which the ego vehicle is located, and time tk is related to the current driving mode of the ego vehicle, for example, in the ACC mode, time tk is the time to collision (ttc), that is, the time for the vehicle to reach the front vehicle at the current speed.

[0137] The real driving trajectory and the predicted driving trajectory of the ego vehicle can be calculated at the vehicle end, that is, the vehicle end calculates the real driving trajectory and the predicted driving trajectory of the ego vehicle according to the driving state data of the ego vehicle, and then uploads the driving state data of the ego vehicle, the real driving trajectory data of the ego vehicle and the predicted driving trajectory of the ego vehicle to the fault identification system. Alternatively, the real driving trajectory and the predicted driving trajectory of the ego vehicle can also be calculated by the fault identification system, for example, the vehicle end uploads the driving state data of the ego vehicle to the fault identification system, and the fault identification system calculates the real driving trajectory and the predicted driving trajectory of the ego vehicle according to the driving state data of the ego vehicle.

[0138] The motion data of the obstacles around the ego vehicle includes the motion trajectory data of the obstacles and the motion state data of the obstacles. The motion state data of the obstacles can be obtained through various sensor data in the sensor system. The sensor data may, for example, include data obtained by a camera, a laser radar, a millimeter wave radar, etc. The sensor data in the present application can be raw video stream, point cloud data of the radar or structured motion state data of the obstacles around the vehicle such as position, speed, acceleration, steering speed, size, position coordinates, etc. For raw video stream data and point cloud data of the radar, the perception unit can process these sensor data into structured motion state data of the obstacles around the vehicle such as position, speed, acceleration, steering speed, size, position coordinates, etc.

[0139] The motion trajectory data of the obstacles can be calculated from the motion state data of the obstacles, for example, the motion trajectory data of obstacle a from time t1 to time tk is obtained according to the position coordinates of obstacle a from time t1 to time tk.

[0140] The obstacle can be understood as an object that can affect the driving of the ego vehicle, such as other vehicles, pedestrians, animals and traffic facilities (such as traffic facilities such as street lamp poles, cone barrels, road signs, etc.) around the vehicle.

[0141] Generally, there are multiple obstacles around the vehicle, such as multiple other vehicles, multiple pedestrians, multiple traffic facilities, etc. within 200 meters of the vehicle, but the automatic driving system of the vehicle is usually affected by a small number of specific key obstacles, such as vehicles in front of the vehicle, pedestrians, etc. Therefore, the embodiments of the present application need to select and determine the key obstacles that will affect the driving decision from the multiple obstacles sensed around the vehicle, and take the key obstacles as the target obstacles of the ego vehicle in the target time period (such as from t1 moment to tk moment), so as to focus on the influence of the key obstacles on the ego vehicle subsequently, and improve the accuracy of subsequent obstacle identification and prediction.

[0142] There are many ways to determine the target obstacle, which can be retrieved from the obstacle recorded in the driving log of the vehicle. The driving log of the vehicle records the obstacle id that affects the vehicle to make a specific event when the vehicle occurs. For example, when the vehicle appears an emergency braking or emergency stopping event, the vehicle will automatically record the obstacle id in front of the vehicle before and after the emergency braking event in the driving log. For example, vehicle a driving in front of the vehicle suddenly slows down, and the automatic driving system of the ego vehicle makes an emergency braking strategy to avoid collision with the front vehicle, and controls the control system to perform emergency braking operation on the ego vehicle. The ego vehicle occurs an emergency braking event, and the driving log records the id of vehicle a. The id of vehicle a is obtained by retrieving the driving log, and vehicle a is taken as the target obstacle of the ego vehicle.

[0143] The target obstacle can also be selected from the multiple obstacles around the ego vehicle according to a preset rule. For example, according to the distance between the obstacles around the ego vehicle and the ego vehicle, the key obstacles are determined, such as determining the target obstacle as the obstacles closest to the ego vehicle within 50 meters of the ego vehicle.

[0144] The motion data of the target obstacle is obtained from the motion data of the multiple obstacles around the ego vehicle, and the fault identification result of the automatic driving system is obtained by processing the driving data of the ego vehicle and the motion data of the target obstacle in the subsequent steps.

[0145] In step S502, a plurality of feature extraction networks are used to extract features from the driving related data of the ego vehicle respectively, and a plurality of feature vectors of the ego vehicle are obtained.

[0146] The embodiment of the present application extracts features of driving related data of the ego vehicle from multiple perspectives, obtains feature vectors of multiple perspectives of the ego vehicle, and expresses driving scene features of the ego vehicle from multiple perspectives, so that the obtained driving scene features of the ego vehicle are more comprehensive, and the identified faults of the automatic driving system are more accurate.

[0147] For example, the multiple perspectives include a predicted trajectory interaction perspective of the ego vehicle and the target obstacle and a real trajectory interaction perspective of the ego vehicle and the target obstacle. By extracting features of the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle, a predicted trajectory interaction feature vector is obtained, which represents the correlation between the predicted driving trajectory of the ego vehicle and the motion trajectory of the target obstacle, and can effectively capture the dynamic influence relationship of the motion trajectory of the target obstacle on the predicted trajectory of the ego vehicle. By extracting features of the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle, a real trajectory interaction feature vector is obtained, which represents the correlation between the real driving trajectory of the ego vehicle and the motion trajectory of the target obstacle, and can effectively capture the dynamic influence relationship of the motion trajectory of the target obstacle on the real driving trajectory of the ego vehicle.

[0148] In one example, the predicted trajectory interaction feature vector can be obtained by extracting features of the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle through a first feature extraction network; and the real trajectory interaction feature vector can be obtained by extracting features of the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle through a second feature extraction network. That is, the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are input into the first feature extraction network, and the predicted trajectory interaction feature vector is output; and the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are input into the second feature extraction network, and the real trajectory interaction feature vector is output.

[0149] Optionally, the first feature extraction network includes a first time sequence feature network, a second time sequence feature network and a first trajectory interaction network. The process of extracting features of the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle by the first feature extraction network can be: inputting the predicted trajectory data of the ego vehicle (which can be a position coordinate sequence of the ego vehicle at continuous time points in a target time period, for example, t1 to tk) into the first time sequence feature network as input, and outputting a predicted driving trajectory feature vector of the ego vehicle; inputting the motion trajectory data of the target obstacle into the second time sequence feature network as input, and outputting a motion trajectory feature vector of the target obstacle; fusing the predicted driving trajectory feature vector and the motion trajectory feature vector (for example, splicing the predicted driving trajectory feature vector and the motion trajectory feature vector of the target obstacle), to obtain a first trajectory fusion feature vector; and inputting the first trajectory fusion feature vector into the first trajectory interaction network as input, and outputting a predicted trajectory interaction feature vector.

[0150] The second feature extraction network comprises a third time sequence feature network, a fourth time sequence feature network and a second trajectory interaction network. The process of feature extraction of the second feature extraction network on the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle can be: taking the real trajectory data of the ego vehicle as the input of the third time sequence feature network, outputting a real driving trajectory feature vector of the ego vehicle; taking the motion trajectory data of the target obstacle as the input of the fourth time sequence feature network, outputting a motion trajectory feature vector of the target obstacle; fusing the real driving trajectory feature vector and the motion trajectory feature vector to obtain a second trajectory fusion feature vector; taking the second trajectory fusion feature vector as the input of the second trajectory interaction network, outputting a real trajectory interaction feature vector.

[0151] The driving trajectory feature of the ego vehicle and the motion trajectory feature of the target obstacle are respectively extracted by the time sequence feature network, then the two trajectory features are spliced to input the trajectory interaction network, and the interaction feature of the driving trajectory of the ego vehicle and the motion trajectory of the target obstacle is accurately extracted, the correlation of the driving trajectory of the ego vehicle and the motion trajectory of the target obstacle is obtained, that is, the influence of the dynamic motion of the target obstacle on the driving trajectory of the ego vehicle is obtained.

[0152] The time sequence feature network (including the first time sequence feature network, the second time sequence feature network, the third time sequence feature network and the fourth time sequence feature network) and the trajectory interaction network (including the first trajectory interaction network and the second trajectory interaction network) can be constructed by various structure neural networks; for example, the time sequence feature network can be a TCN network, and the trajectory interaction network can be constructed based on an MLP network. For another example, the time sequence feature network can be an RNN network or a Transformer neural network model, and the trajectory interaction network can be an FCN network, and the specific implementation mode of the time sequence feature network and the trajectory interaction network is not limited in the application, and a suitable neural network structure can be selected for training according to actual needs.

[0153] In some other examples, in order to further increase the comprehensiveness of the driving scene representation, the driving related data of the ego vehicle further comprises road environment data, and the road environment data indicates road environment information of a road where the ego vehicle drives; the feature vectors of multiple perspectives further comprise a road environment feature vector, and the road environment feature vector is obtained based on feature extraction on the road environment data.

[0154] The road environment information can be sensed by a sensor system of the ego vehicle, for example, the sensor system of the ego vehicle comprises a laser radar, a millimeter wave radar, a camera and other sensors, and road environment information such as a lane number, a lane line type, a distance from an intersection and a distance from a traffic light of a road where the ego vehicle is located is sensed.

[0155] Of course, the road environment information can also be obtained in other ways, such as obtaining the road environment information of the road where the vehicle is located through a high-precision map, and the application does not limit this. The appropriate way of obtaining road environment information can be selected according to the actual situation.

[0156] Optionally, the road environment data can be feature extracted through the TCN network to obtain a road environment feature vector.

[0157] In another example, in order to further enrich the driving scene feature information of the ego vehicle, the plurality of view feature vectors implemented by the application also include a driving scene feature vector, which is obtained based on feature extraction of first driving scene data and / or second driving scene data; wherein the first driving scene data includes result data calculated based on a dynamic equation and / or prior knowledge on driving data and target obstacle motion data, and the result data is related to the driving mode of the ego vehicle; the second driving scene data includes ego vehicle driving data and target obstacle motion data corresponding to a fault time, and the fault time indicates a time when the driving of the ego vehicle is likely to be abnormal. The fault time can be predicted by the automatic driving system, such as the automatic driving system predicting that the driving is likely to be abnormal at a certain time, for example, rear-end collision, lane deviation, etc., and the automatic driving system will automatically record these times and the driving data and target obstacle motion data at these times.

[0158] For example, in the AEB mode, the brake value, whether to autonomously brake (for example, if the brake value is non-zero for more than 5s, it is judged as autonomous braking, otherwise it is non-autonomous braking), whether the target obstacle is a ghost obstacle (for example, whether the target obstacle duration exceeds 10s, if yes, it is judged that the target obstacle is a non-ghost obstacle, if not, it is judged that the target obstacle is a ghost obstacle), and the duration of the target obstacle are taken as driving scene information and injected into the first driving scene data. In the ACC mode, the steering angle, steering speed, and collision time ttc between the ego vehicle and the target obstacle (for example, the preceding vehicle) are taken as driving scene information and injected into the first driving scene data. The first driving scene data also includes some data calculated based on the kinematic equation, such as the predicted trajectory of the ego vehicle and the motion trajectory of the target obstacle.

[0159] The first driving scene data and the second driving scene data can be feature extracted through a feature extraction network, and then the two extracted features are fused (for example, spliced) to obtain a driving scene feature vector. For example, the first driving scene data and the second driving scene data are feature extracted through an MLP to obtain a driving scene feature vector.

[0160] The driving scene information injection is performed by prior rules and / or dynamic equations to enhance the driving scene features, the scene information richness is improved in combination with artificial experience, and the accuracy of fault identification of the automatic driving system is improved.

[0161] In step S503, the feature vectors of multiple perspectives are fused to obtain a global driving scene feature vector of the ego vehicle.

[0162] In the embodiments of the present application, the feature vectors of multiple perspectives can be weighted and fused through the attention mechanism to realize adaptive fusion of multi-perspective information, and then the global driving scene feature vector of the ego vehicle is obtained, and the accuracy of fault identification of the automatic driving system is improved.

[0163] In one example, when the feature vectors of multiple perspectives only include the predicted trajectory interaction feature vector and the real trajectory interaction feature vector, the feature vector fusion process of multiple perspectives is: determining a first attention weight corresponding to the predicted trajectory interaction feature vector and a second attention weight corresponding to the real trajectory interaction feature vector; and then based on the first attention weight and the second attention weight, the predicted trajectory interaction feature vector and the real trajectory interaction feature vector are weighted and fused to obtain the global driving scene feature vector of the ego vehicle.

[0164] When the feature vectors of multiple perspectives include the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, and the driving scene feature vector, the feature vector fusion process of multiple perspectives is: determining a first attention weight corresponding to the predicted trajectory interaction feature vector, a second attention weight corresponding to the real trajectory interaction feature vector, and a third attention weight corresponding to the driving scene feature vector; and then based on the first attention weight, the second attention weight, and the third attention weight, the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, and the driving scene feature vector are weighted and fused to obtain the global driving scene feature vector of the ego vehicle.

[0165] When the feature vectors of multiple perspectives include the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, the driving scene feature vector, and the road environment feature vector, the feature vector fusion process of multiple perspectives is: determining a first attention weight corresponding to the predicted trajectory interaction feature vector, a second attention weight corresponding to the real trajectory interaction feature vector, a third attention weight corresponding to the driving scene feature vector, and a fourth attention weight corresponding to the road environment feature vector; and then based on the first attention weight, the second attention weight, the third attention weight, and the fourth attention weight, the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, the driving scene feature vector, and the road environment feature vector are weighted and fused to obtain the global driving scene feature vector of the ego vehicle.

[0166] Thus, by fully representing multi-view information, fully modeling the complex state changes and mutual influence relationships of the ego vehicle, obstacles and road environment, and designing a trajectory interaction network, the dynamic influence relationship between the trajectories of surrounding obstacles and the driving trajectory of the ego vehicle can be effectively captured. A dimension attention mechanism for obstacles is designed to focus on key obstacles that cause the ego vehicle to accelerate, brake suddenly or change lanes, and to extract the influence of key surrounding obstacles and environmental information on the behavior of the ego vehicle, thereby assisting in determining whether the automatic driving system is operating correctly. Multi-view information is adaptively fused, the influence of different perspectives is considered, a perspective dimension attention mechanism is designed to weight and fuse multiple perspective information, the global representation of the driving scene is obtained, and the recognition accuracy is improved.

[0167] In one possible implementation, the feature vectors of multiple perspectives (such as the predicted trajectory interaction feature vector, the real trajectory interaction feature vector, the driving scene feature vector and the road environment feature vector) are first spliced and fused, and then the spliced feature vectors are input into a fusion network of a feature encoder and a perspective dimension attention mechanism to obtain the final global driving scene feature vector of the ego vehicle.

[0168] In step S504, based on the global driving scene feature vector, it is determined whether the target driving scene in which the ego vehicle is currently located and whether the automatic driving system fails in the target driving scene.

[0169] Through the foregoing steps, the global driving scene feature vector of the ego vehicle is obtained, and the target driving scene in which the ego vehicle is currently located and whether the automatic driving system fails in the target driving scene can be accurately identified.

[0170] For example, the global driving scene feature vector of the ego vehicle is taken as the input of a classifier, and the fault identification result of the automatic driving system of the ego vehicle is output, which includes the target driving scene in which the ego vehicle is currently located and whether the automatic driving system fails in the target driving scene. Whether the automatic driving system fails in the target driving scene means whether the automatic driving system can make correct operation and processing for the ego vehicle in the target driving scene, such as whether the automatic driving system can control the ego vehicle to brake correctly to avoid rear-end collision with the front vehicle when the ego vehicle is in a rear-end collision scene.

[0171] That is, the fault result of the automatic driving system identified in the embodiment of the present application includes the target driving scene and whether it fails, the target driving scene indicates the driving scene in which the ego vehicle is currently located, and whether it fails indicates whether the automatic driving system of the ego vehicle fails in the target driving scene.

[0172] The driving scene here is a more fine-grained driving scene. For example, if the current driving mode of the ego vehicle is AEB mode, the driving scene includes fine-grained driving scenes such as AEB false triggering, AEB late triggering, and AEB missing triggering. For another example, if the current driving mode of the ego vehicle is ACC mode, the driving scene includes fine-grained driving scenes such as driving off-center, collision with a following vehicle, cut-in collision, unexpected acceleration, unexpected deceleration, function failure to intervene, unreasonable obstacle avoidance, and function abnormality to push out. For another example, if the current driving mode of the ego vehicle is auto parking (APA) mode, the driving scene includes fine-grained driving scenes such as parking-in failure, parking-out failure, abnormal parking-out pose, abnormal parking-in pose, and poor human-vehicle interaction experience. For another example, if the current driving mode of the ego vehicle is NCA mode, the driving scene includes fine-grained driving scenes such as unreasonable lane changing, red light running at an intersection, unreasonable light flashing, and unreasonable obstacle avoidance.

[0173] It needs to be explained that the driving scene mentioned in the embodiments of the present application refers to a risk event scene that may occur, rather than a real scene that has occurred. For example, the rear-end collision scene refers to a driving scene in which the ego vehicle may collide with a front vehicle, and is not a rear-end collision event that has occurred, and thus can also be referred to as a rear-end collision risk driving scene. For example, there is a vehicle a 100 meters in front of the ego vehicle, and the driving scene in which the ego vehicle is located is determined to be a rear-end collision scene by comprehensively considering the speed of the ego vehicle and the speed of the vehicle a. That is, the embodiments of the present application aim to discover a fault of the automatic driving system before the vehicle accident caused by the fault of the automatic driving system occurs, and then repair the fault to ensure the reliability of the automatic driving system and ensure the safety of driving.

[0174] The method of the embodiments of the present application further includes, when it is identified that the automatic driving system fails in the target driving scene, and the fault is not solved (for example, the code or module causing the fault is solved after OTA upgrade), issuing a fault alarm information or a manual takeover request. The fault alarm information indicates that the current automatic driving system of the vehicle is in a fault state in the target driving scene, and is not suitable for starting automatic driving in the target driving scene. The manual takeover request can be used to request the user to manually control the vehicle to drive.

[0175] For example, it is identified by the embodiments of the present application that the automatic driving system fails in the AEB false triggering scene in the AEB mode, that is, the automatic driving system in the AEB mode will frequently trigger false triggering, that is, false emergency braking. Then, a fault alarm information is issued to the user to tell the user that the current automatic driving system of the vehicle in the AEB false triggering scene in the AEB mode will fail, and it is not suitable to use the secondary function in the driving scene in which AEB false triggering may occur. It is suggested to close the function and wait for the upgrade before using it again.

[0176] The following introduces one application implementation of the fault identification method of the automatic driving system provided by the embodiments of the present application.

[0177] FIG. 6 shows a system implementation architecture diagram of the fault identification method of the automatic driving system provided by the embodiments of the present application. As shown in FIG. 6, the input data required by the embodiments of the present application includes the ego vehicle driving data, the surrounding obstacle motion data and the road environment data, then the target obstacle is selected from the surrounding obstacles of the ego vehicle through target obstacle screening, and the motion data of the target obstacle is obtained, and then the driving scene information injection is used to fuse artificial experience to improve the richness of the scene information, and finally the ego vehicle driving data, the target obstacle motion data, the road environment data and the driving scene information injected by the driving scene information are taken as the input of the fault identification model, and the fault identification result is obtained through the inference of the fault identification model.

[0178] The specific acquisition method of the ego vehicle driving data, the surrounding obstacle motion data and the road environment data, the screening method of the target obstacle and the method of the driving scene information injection can be referred to the description above, and here, for the sake of brevity, will not be repeated.

[0179] In one example, the fault identification model includes a preprocessing module that preprocesses the ego vehicle driving data, target obstacle motion data, road environment data, and driving scene information. For example, the ego vehicle driving data includes the ego vehicle driving state data, the ego vehicle real driving trajectory data, and the ego vehicle predicted driving trajectory. The preprocessing of the ego vehicle driving data includes selecting the ego vehicle real driving trajectory data and the ego vehicle predicted driving trajectory from the ego vehicle driving data. For another example, the ego vehicle driving data only includes the ego vehicle driving state data. The preprocessing of the ego vehicle driving data includes calculating the ego vehicle real driving trajectory data and the ego vehicle predicted driving trajectory using the ego vehicle driving state data. For another example, the target obstacle motion data includes the target obstacle motion trajectory data and the target obstacle motion state data. The preprocessing of the target obstacle motion data includes selecting the target obstacle motion trajectory data from the target obstacle motion data. For another example, the target obstacle motion data only includes the target obstacle motion state data. The preprocessing of the target obstacle motion data includes calculating the target obstacle motion trajectory data using the target obstacle motion state data. For another example, the driving scene information is obtained by processing the ego vehicle driving data and the target obstacle motion data using artificial prior knowledge. For example, the ego vehicle predicted driving trajectory data is obtained by performing dynamic calculation on the ego vehicle driving state data using the specific rule determination result (e.g., the collision time tcc) and / or the dynamic equation according to the current driving mode. For another example, the driving scene information is obtained by extracting the ego vehicle driving data and the target obstacle motion data at the fault time using rules. For another example, the road environment information includes the road environment information such as the lane number, the lane line type, the distance to the intersection, and the distance to the traffic light of the road where the ego vehicle is located. The corresponding road environment information is selected from the multiple road environment information according to the current driving mode. For example, in the ACC mode, the lane number and the lane line type are selected as the road environment information.

[0180] The preprocessed data is then input into the fault identification model. The fault identification model processes the input data to predict the fault identification result of the automatic driving system.

[0181] FIG. 7 shows a schematic diagram of the processing of the input data by the fault identification model.

[0182] Taking the current driving model of the ego vehicle as the AEB mode as an example, the implementation process of the fault identification of the automatic driving system of the ego vehicle is introduced. In the AEB mode, the ttc collision time, the ego vehicle predicted trajectory, the obstacle duration, etc. are calculated as the injected driving scene information according to the running speed and coordinate change of the ego vehicle and the obstacle, so as to enrich the scene information. The road environment information includes the lane line type, the distance to the intersection, and the distance to the traffic light.

[0183] For the ego vehicle prediction trajectory, the time sequence representation and the trajectory interaction network are used to interact the prediction trajectory of the ego vehicle with the motion trajectory of the target obstacle in the corresponding time, extract the correlation between the two, and obtain the prediction trajectory interaction feature vector. For the posterior trajectory of the ego vehicle and the obstacle, the time sequence representation and the trajectory interaction network are used to extract the correlation between the two, and obtain the real trajectory interaction feature vector. The MLP is used to process the fault point time information and the scene feature to obtain the driving scene feature vector. The TCN network is used to obtain the road environment feature vector. The four feature vectors are spliced, and the global scene feature is obtained through the fusion of the feature encoder and the perspective dimension attention mechanism. Finally, the global scene feature is output by the classifier to obtain the classification result as the fault recognition result, i.e., the target driving scene of the ego vehicle in the current driving mode and whether the automatic driving system is faulty in the target driving scene.

[0184] The fault recognition method of the automatic driving system provided by the embodiments of the present application fuses artificial experience to improve the richness of scene information, captures the dynamic changes of the ego vehicle and surrounding obstacles, and the road environment from multiple perspectives, and adaptively fuses multi-perspective information to obtain comprehensive representation of the driving scene for fault recognition, which greatly improves the accuracy of fault recognition of the automatic driving system. It is convenient for operation and maintenance personnel to iteratively update the automatic driving system according to the recognized fault, improve the reliability of the automatic driving system, and ensure the driving safety of the ego vehicle.

[0185] The embodiments of the present application obtain the global driving scene feature vector through multi-perspective fusion, which also has other uses, such as user driving behavior analysis, construction of driving behavior portrait and vehicle owner portrait, and help in all-round driving risk assessment, such as traffic operation level accident risk assessment (whether dangerous driving), vehicle and vehicle owner driving level risk assessment (whether the vehicle owner is fatigued / drunk driving / is a novice driver), etc.

[0186] Therefore, the embodiments of the present application also provide a driving risk assessment method, which comprises obtaining the global driving scene feature vector of the ego vehicle, and determining the driving risk assessment result based on the global driving scene feature vector. Optionally, the driving risk assessment result comprises one or more of the traffic accident risk assessment result, the vehicle risk assessment result, and the driver driving risk assessment result.

[0187] Optionally, one implementation of determining the driving risk assessment result based on the global driving scene feature vector can be that the global driving scene feature vector is taken as the input of the classifier, and the driving risk assessment result is output. For example, the probability distribution of the output risk assessment result of the classifier is that the traffic accident risk probability is 0.8, the vehicle risk probability is 0.6, and the driver driving risk is 0.7, so the traffic accident risk assessment is high risk, the vehicle risk assessment is medium risk, and the driver driving risk assessment is medium-high risk.

[0188] In another example, the classifier can also output more detailed evaluation results based on the global driving scene feature vector, for example, in the driver driving risk evaluation, the classifier outputs a risk probability distribution of multiple categories, for example, the classifier outputs the risk of the driver driving fatigue as 0.8, the probability of drunk driving as 0.5, and the probability of the driver being a novice driver as 0.3, and the risk evaluation is that the driver is driving fatigue. Of course, the output of the classifier depends on the training label of the training data. The more detailed the training label of the training data is, the more detailed the output of the classifier is.

[0189] The method for obtaining the global driving scene feature vector of the ego vehicle is described above. For brevity, it will not be repeated here.

[0190] In one example, the driving risk evaluation method provided by the embodiment of the present application further includes, when the driving risk evaluation is currently detected to be high, warning the driver. For example, the driver has driven the vehicle for more than 5 hours, and through the driving risk evaluation method of the embodiment of the present application, it is evaluated that the driver matches high driving risk, then the driver is warned, and the warning includes but is not limited to one or more warning operations such as voice warning, light warning, seat belt tightening, etc.

[0191] Based on the same idea as the foregoing embodiment of the fault identification method of the automatic driving system, the embodiment of the present application also provides an automatic driving system fault identification device 800. The automatic driving system fault identification device 800 can be deployed in a terminal device or a cloud server to accurately identify the fault of the automatic driving system. The automatic driving system fault identification device 800 includes units or modules to implement each step of the automatic driving system fault identification method shown in FIGS. 5-7.

[0192] FIG. 8 is a structural schematic diagram of a fault identification device of an autonomous driving system according to an embodiment of the present application. As shown in FIG. 8, the fault identification device 800 of the autonomous driving system at least includes a first acquisition module 801, a first multi-view feature extraction module 802, a first feature fusion module 803, and a fault identification module 804. The first acquisition module 801 is configured to acquire driving related data of a host vehicle, wherein the driving related data of the host vehicle at least includes host vehicle driving data and target obstacle motion data, the target obstacle is an obstacle in the surroundings of the host vehicle that has an influence on the driving of the host vehicle; the first multi-view feature extraction module 802 is configured to perform feature extraction on the driving related data of the host vehicle based on a plurality of feature extraction networks respectively, to obtain a plurality of view feature vectors of the host vehicle, wherein the plurality of view feature vectors at least include a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector represents the correlation between the predicted driving trajectory of the host vehicle and the motion trajectory of the target obstacle, and the real trajectory interaction feature vector represents the correlation between the real driving trajectory of the host vehicle and the motion trajectory of the target obstacle; the first feature fusion module 803 is configured to fuse the plurality of view feature vectors to obtain a global driving scene feature vector of the host vehicle; and the fault identification module 804 is configured to determine the target driving scene in which the host vehicle is located in the current driving mode and whether the autonomous driving system is faulty in the target driving scene based on the global driving scene feature vector.

[0193] In one possible implementation, the host vehicle driving data includes predicted trajectory data of the host vehicle and real trajectory data of the host vehicle, the predicted trajectory data of the host vehicle indicates the predicted driving trajectory data of the host vehicle in a target time period, and the real trajectory data of the host vehicle indicates the real driving trajectory data of the host vehicle in the target time period; the target obstacle motion data includes motion trajectory data of the target obstacle, and the motion trajectory data of the target obstacle indicates the motion trajectory data of the target obstacle in the target time period; and the first multi-view feature extraction module 802 is specifically configured to: take the predicted trajectory data of the host vehicle and the motion trajectory data of the target obstacle as the input of a first feature extraction network, and output the predicted trajectory interaction feature vector; and take the real trajectory data of the host vehicle and the motion trajectory data of the target obstacle as the input of a second feature extraction network, and output the real trajectory interaction feature vector.

[0194] In another possible implementation, the predicted trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are taken as inputs of the first feature extraction network, and one specific implementation of outputting the predicted trajectory interaction feature vector is as follows: the predicted trajectory data of the ego vehicle is taken as input of the first time sequence representation network, and a predicted driving trajectory feature vector of the ego vehicle is output; the motion trajectory data of the target obstacle is taken as input of the second time sequence representation network, and a motion trajectory feature vector of the target obstacle is output; the predicted driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a first trajectory fusion feature vector; and the first trajectory fusion feature vector is taken as input of the first trajectory interaction network, and the predicted trajectory interaction feature vector is output.

[0195] In another possible implementation, the real trajectory data of the ego vehicle and the motion trajectory data of the target obstacle are taken as inputs of the second feature extraction network, and one specific implementation of outputting the real trajectory interaction feature vector is as follows: the real trajectory data of the ego vehicle is taken as input of the third time sequence representation network, and a real driving trajectory feature vector of the ego vehicle is output; the motion trajectory data of the target obstacle is taken as input of the fourth time sequence representation network, and a motion trajectory feature vector of the target obstacle is output; the real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; and the second trajectory fusion feature vector is taken as input of the second trajectory interaction network, and the real trajectory interaction feature vector is output.

[0196] Optionally, the first time sequence representation network, the second time sequence representation network, the third time sequence representation network, and the fourth time sequence representation network are all constructed based on a TCN; and / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on an MLP network.

[0197] In another possible implementation, the first feature fusion module 803 is specifically configured to: determine a first attention weight corresponding to the predicted trajectory interaction feature vector and a second attention weight corresponding to the real trajectory interaction feature vector; and perform weighted fusion on the predicted trajectory interaction feature vector and the real trajectory interaction feature vector based on the first attention weight and the second attention weight, to obtain the global driving scene feature vector of the ego vehicle.

[0198] In another possible implementation, the target obstacle is retrieved from a running log of the ego vehicle; or the target obstacle is selected from obstacles around the ego vehicle based on a preset rule.

[0199] In another possible implementation, the driving related data of the ego vehicle further includes road environment data, the road environment data indicating road environment information in which the ego vehicle drives; the feature vectors of the multiple perspectives further include a road environment feature vector, the global driving scene feature vector of the ego vehicle is further related to the road environment feature vector, and the road environment feature vector is obtained based on feature extraction on the road environment data.

[0200] Optionally, the road environment data comprises one or more of a lane number of a road where the ego vehicle is located, a lane line type, a distance to an intersection, and a distance to a traffic light.

[0201] In another possible implementation, the ego vehicle driving data further comprises one or more of an intelligent driving state, a speed, an acceleration, a position coordinate, an orientation angle, and a brake value of the ego vehicle; and / or, the target obstacle motion data further comprises one or more of a speed, an acceleration, and a position coordinate of the target obstacle.

[0202] In another possible implementation, the feature vectors of the multiple perspectives further comprise a driving scene feature vector; the driving scene feature vector is obtained based on feature extraction on first driving scene data and / or second driving scene data; wherein, the first driving scene data comprises result data calculated based on a dynamics equation and / or prior knowledge on the driving data and the target obstacle motion data, the result data being related to a driving mode of the ego vehicle; the second driving scene data comprises the ego vehicle driving data and the target obstacle motion data corresponding to a failure time point, the failure time point indicating a time point at which the driving of the ego vehicle is likely to be abnormal; and the global driving scene feature vector of the ego vehicle is further related to the driving scene feature vector.

[0203] In another possible implementation, the result data comprises one or more of a collision time of the ego vehicle and the target obstacle, a predicted driving trajectory of the ego vehicle, a duration of the target obstacle, a brake value, and a direction angle.

[0204] The fault identification apparatus 800 of the automatic driving system according to the embodiments of the present application can correspond to performing the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module in the fault identification apparatus 800 of the automatic driving system are respectively for realizing the corresponding processes of each method in FIGS. 5-7, and for brevity, will not be repeated here.

[0205] Based on the same idea as the foregoing embodiment of the driving risk assessment method, the present embodiment further provides a driving risk assessment apparatus 900, which can be deployed in a terminal device or a cloud server to realize accurate assessment of driving risk.

[0206] FIG. 9 is a structural schematic diagram of a driving risk assessment device according to an embodiment of the present application. As shown in FIG. 9, the driving risk assessment device 900 includes at least a second acquisition module 901, a second multi-view feature extraction module 902, a second feature fusion module 903, and a risk assessment module 904. The second acquisition module 901 is configured to acquire driving-related data of a host vehicle, wherein the driving-related data of the host vehicle includes at least driving data of the host vehicle and target obstacle motion data, and the target obstacle is an obstacle in the surroundings of the host vehicle that has an impact on driving of the host vehicle. The second multi-view feature extraction module 902 is configured to perform feature extraction on the driving-related data of the host vehicle from multiple views to obtain feature vectors of multiple views of the host vehicle, wherein the feature vectors of multiple views include at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector represents a correlation between a predicted driving trajectory of the host vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector represents a correlation between a real driving trajectory of the host vehicle and the motion trajectory of the target obstacle. The second feature fusion module 903 is configured to fuse the feature vectors of multiple views to obtain a global driving scene feature vector of the host vehicle. The risk assessment module 904 is configured to determine a driving risk assessment result based on the global driving scene feature vector.

[0207] Optionally, the driving risk assessment result includes one or more of a traffic accident risk assessment result, a vehicle risk assessment result, and a driver driving risk assessment result.

[0208] An embodiment of the present application provides a computer readable storage medium, which stores computer programs, and when the computer programs are executed by a processor, the above-mentioned method is implemented.

[0209] An embodiment of the present application provides a chip, which includes at least one processor and an interface, the at least one processor determines program instructions or data through the interface, and the at least one processor is configured to execute the program instructions to implement the above-mentioned method.

[0210] An embodiment of the present application provides a computer program or a computer program product, which includes instructions, and when the instructions are executed, the computer executes the above-mentioned method.

[0211] Those skilled in the art should further understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0212] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0213] The above detailed description of the specific implementation is further detailed for the purpose of the present application, technical solutions and beneficial effects. It should be understood that the above description is only a specific implementation of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A failure identification method of an autonomous driving system, characterized by, The method comprises: obtaining driving related data of a vehicle, wherein the driving related data of the vehicle at least comprises vehicle driving data and target obstacle motion data, the target obstacle being an obstacle around the vehicle that has an impact on driving of the vehicle; performing feature extraction on the driving related data of the vehicle based on a plurality of feature extraction networks respectively, to obtain a plurality of perspective feature vectors of the vehicle, wherein the plurality of perspective feature vectors at least comprises a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing a correlation between a predicted driving trajectory of the vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing a correlation between a real driving trajectory of the vehicle and the motion trajectory of the target obstacle; fusing the plurality of perspective feature vectors to obtain a global driving scene feature vector of the vehicle; based on the global driving scene feature vector, determining a target driving scene in which the vehicle is located in a current driving mode and whether an automatic driving system fails in the target driving scene.

2. The method of claim 1, wherein, The vehicle driving data comprises predicted trajectory data of the vehicle and real trajectory data of the vehicle, the predicted trajectory data of the vehicle indicating predicted driving trajectory data of the vehicle in a target time period, and the real trajectory data of the vehicle indicating real driving trajectory data of the vehicle in the target time period; The target obstacle motion data comprises motion trajectory data of the target obstacle, the motion trajectory data of the target obstacle indicating motion trajectory data of the target obstacle in the target time period; The performing feature extraction on the driving related data of the vehicle based on a plurality of feature extraction networks respectively, to obtain a plurality of perspective feature vectors of the vehicle, comprises: taking the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a first feature extraction network, and outputting the predicted trajectory interaction feature vector; taking the real trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a second feature extraction network, and outputting the real trajectory interaction feature vector.

3. The method of claim 2, wherein, The taking the predicted trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a first feature extraction network, and outputting the predicted trajectory interaction feature vector, comprises: taking the predicted trajectory data of the vehicle as an input of a first time sequence representation network, and outputting a predicted driving trajectory feature vector of the vehicle; taking the motion trajectory data of the target obstacle as an input of a second time sequence representation network, and outputting a motion trajectory feature vector of the target obstacle; fusing the predicted driving trajectory feature vector and the motion trajectory feature vector to obtain a first trajectory fusion feature vector; taking the first trajectory fusion feature vector as an input of a first trajectory interaction network, and outputting the predicted trajectory interaction feature vector.

4. The method according to claim 2 or 3, characterized in that, The taking the real trajectory data of the vehicle and the motion trajectory data of the target obstacle as inputs of a second feature extraction network, and outputting the real trajectory interaction feature vector, comprises: The real trajectory data of the ego vehicle is input into a third time sequence feature network, and a real driving trajectory feature vector of the ego vehicle is output; The motion trajectory data of the target obstacle is input into a fourth time sequence feature network, and a motion trajectory feature vector of the target obstacle is output; The real driving trajectory feature vector and the motion trajectory feature vector are fused to obtain a second trajectory fusion feature vector; The second trajectory fusion feature vector is input into a second trajectory interaction network, and a real trajectory interaction feature vector is output.

5. The method of claim 4, wherein, The first time sequence feature network, the second time sequence feature network, the third time sequence feature network, and the fourth time sequence feature network are all constructed based on a time domain convolution network; And / or, the first trajectory interaction network and the second trajectory interaction network are both constructed based on a multi-layer perception network.

6. The method according to any one of claims 1 to 5, characterized in that, The fusion of the feature vectors of the multiple perspectives to obtain the global driving scene feature vector of the ego vehicle comprises: determining a first attention weight corresponding to the predicted trajectory interaction feature vector and a second attention weight corresponding to the real trajectory interaction feature vector; based on the first attention weight and the second attention weight, the predicted trajectory interaction feature vector and the real trajectory interaction feature vector are weighted and fused to obtain the global driving scene feature vector of the ego vehicle.

7. The method according to any one of claims 1 to 6, characterized in that, The target obstacle is retrieved from the operation log of the ego vehicle; Alternatively, the target obstacle is selected from the obstacles around the ego vehicle based on a preset rule.

8. The method according to any one of claims 1 to 7, characterized in that, The driving related data of the ego vehicle also includes road environment data, which indicates the road environment information where the ego vehicle drives; The feature vectors of the multiple perspectives also include a road environment feature vector, and the global driving scene feature vector of the ego vehicle is also related to the road environment feature vector, which is obtained by feature extraction on the road environment data.

9. The method of claim 8, wherein, The road environment data includes one or more of the lane number, lane line type, distance to the intersection, and distance to the traffic light of the road where the ego vehicle is located.

10. The method according to any one of claims 1 to 9, characterized in that, The ego vehicle driving data also includes driving state data of the ego vehicle, which includes one or more of intelligent driving state, speed, acceleration, position coordinates, orientation angle, and brake value; And / or, the target obstacle motion data also includes motion state data of the target obstacle, which includes one or more of speed, acceleration, and position coordinates.

11. The method of claim 10, wherein, The feature vectors of the multiple perspectives also include a driving scene feature vector; The driving scene feature vector is obtained by feature extraction on first driving scene data and / or second driving scene data; The first driving scene data includes result data calculated based on a dynamic equation and / or prior knowledge on the driving state data and the motion state data, and the result data is related to the driving mode of the ego vehicle. The second driving scene data includes the ego vehicle driving data corresponding to a failure time and the target obstacle motion data, the failure time indicating a time when driving of the ego vehicle is likely to be abnormal; The global driving scene feature vector of the ego vehicle is also related to the driving scene feature vector vector.

12. The method of claim 11, wherein, The result data includes one or more of a collision time of the ego vehicle and the target obstacle, a predicted driving trajectory of the ego vehicle, a duration of the target obstacle, whether the target obstacle is a ghost obstacle, a brake value, whether autonomous braking, and a direction angle.

13. A driving risk assessment method characterized by, Comprising: Obtaining driving-related data of an ego vehicle, the driving-related data of the ego vehicle at least including ego vehicle driving data and target obstacle motion data, the target obstacle being an obstacle in the surroundings of the ego vehicle that has an impact on driving of the ego vehicle; Based on a plurality of feature extraction networks, respectively extracting features from the driving-related data of the ego vehicle to obtain a plurality of perspective feature vectors of the ego vehicle, the plurality of perspective feature vectors at least including a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing a correlation between a predicted driving trajectory of the ego vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing a correlation between a real driving trajectory of the ego vehicle and the motion trajectory of the target obstacle; Fusing the plurality of perspective feature vectors to obtain a global driving scene feature vector of the ego vehicle; Based on the global driving scene feature vector, determining a driving risk assessment result.

14. The method of claim 13, wherein, The driving risk assessment result includes one or more of a traffic accident risk assessment result, a vehicle risk assessment result, and a driver driving risk assessment result.

15. A failure identification device of an autonomous driving system, characterized by, Comprising: A first obtaining module for obtaining driving-related data of an ego vehicle, the driving-related data of the ego vehicle at least including ego vehicle driving data and target obstacle motion data, the target obstacle being an obstacle in the surroundings of the ego vehicle that has an impact on driving of the ego vehicle; A first multi-perspective feature extraction module for extracting features from the driving-related data of the ego vehicle based on a plurality of feature extraction networks to obtain a plurality of perspective feature vectors of the ego vehicle, the plurality of perspective feature vectors at least including a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing a correlation between a predicted driving trajectory of the ego vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing a correlation between a real driving trajectory of the ego vehicle and the motion trajectory of the target obstacle; A first feature fusion module for fusing the plurality of perspective feature vectors to obtain a global driving scene feature vector of the ego vehicle; A failure recognition module for determining, based on the global driving scene feature vector, a target driving scene in which the ego vehicle is currently located and whether the autonomous driving system is malfunctioning in the target driving scene.

16. A driving risk assessment apparatus characterized by comprising: Comprising: a second obtaining module, configured to obtain driving-related data of a vehicle, the driving-related data of the vehicle comprising at least the vehicle driving data and target obstacle motion data, the target obstacle being an obstacle around the vehicle that has an impact on driving of the vehicle; a second multi-view feature extraction module, configured to perform feature extraction on the driving-related data of the vehicle based on a plurality of feature extraction networks respectively, to obtain a plurality of view feature vectors of the vehicle, the plurality of view feature vectors comprising at least a predicted trajectory interaction feature vector and a real trajectory interaction feature vector, the predicted trajectory interaction feature vector representing a correlation between a predicted driving trajectory of the vehicle and a motion trajectory of the target obstacle, and the real trajectory interaction feature vector representing a correlation between a real driving trajectory of the vehicle and the motion trajectory of the target obstacle; a second feature fusion module, configured to fuse the plurality of view feature vectors to obtain a global driving scene feature vector of the vehicle; a risk assessment module, configured to determine a driving risk assessment result based on the global driving scene feature vector.

17. A computing device comprising a memory and a processor, wherein: The memory stores instructions, and when the instructions are executed by the processor, the method of any one of claims 1-14 is implemented.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, causes the method of any one of claims 1-14 to be implemented.

19. A computer program product, characterised in that, The computer program product comprises a computer program, and when the computer program is executed by the processor, the method of any one of claims 1-14 is implemented.

Images