Abnormality detection method and system and automatic driving device

Abstract

The anomaly detection method is applied to an automatic driving device, and comprises the following steps: acquiring data of various environmental states in a current driving state; fusing the data based on the various environment states to obtain target data; determining a driving decision output by the deep reinforcement learning model according to the target data, and obtaining a deep feature value of the deep reinforcement learning model; the driving decision indicates the next action of the automatic driving device; the actions comprise left turning, right turning, acceleration or braking; the depth feature value comprises a driving decision output by the deep reinforcement learning model, an activation value of an activation layer, and / or a feature value of one or more hidden layers; the classifier judges the current environment state according to the depth feature value, and outputs a judgment result that the environment state is abnormal under the condition that the depth feature value deviates from normal data; the anomalies are environmental states suffering from attacks. According to the invention, the automatic driving device in traffic can be prevented from being misguided by confronting security threats to make wrong driving decisions.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an anomaly detection method, system and autonomous driving device. Background Art

[0002] Since the progress of artificial intelligence innovation, the application of deep reinforcement learning (Deep Reinforcement Learning) to assisted driving / autonomous driving modules has been initially developed. The vehicle's own conditions and the state of the driving environment are input into the deep reinforcement learning model, and the model outputs a driving decision, such as turning left, turning right, or accelerating, braking, etc.

[0003] Deep reinforcement learning models face a variety of adversarial security threats, such as evasion attacks and adversarial policy attacks. Among them, evasion attacks are to find special inputs through adversarial algorithms such as FGSM and PGD, and use the blind spots and weaknesses of the learning algorithm to create internal errors, causing the model to make wrong decisions. Adversarial policy attacks are attackers who add an autonomous driving device (agent) to the environment and insert special algorithms into the autonomous driving device to affect the target of attack, causing the attacked target to make inappropriate or even dangerous actions. For example, an autonomous driving vehicle driving behind the main vehicle on the highway suddenly accelerates according to the inserted special algorithm, causing the attacked main vehicle to make an evasive action of turning left or right (a dangerous action), resulting in a scratch or collision with the vehicle passing on the left or right.

[0004] When applying deep reinforcement learning models to autonomous driving scenarios, these adversarial safety threats not only have a negative impact on the correctness of the model's prediction results, but are also more likely to threaten personal safety in traffic environments. From some early research work, adversarial policy attacks have been shown to be effective in multi-agent DRL driving scenarios. This means that by implementing adversarial policy attacks, target vehicles in traffic can be misled to make incorrect driving decisions, leading to unintentional harmful consequences. Summary of the invention

[0005] In order to solve the above problems, the embodiments of the present application provide a method, system and autonomous driving device for abnormality detection. Designed for the deep reinforcement learning model, the characteristic values ​​of the deep reinforcement learning model are adopted for analysis, and the detection attack capability range can include unique security attacks and adversarial strategy attacks against the deep reinforcement learning model, and can effectively defend against evasion attacks and adversarial strategy attacks.

[0006] In the first aspect, an embodiment of the present application provides a method for anomaly detection, which is applied to an autonomous driving device and includes: obtaining data of multiple environmental states under a current driving state; obtaining target data based on the data of multiple environmental states; determining a driving decision output by a deep reinforcement learning model based on the target data, and obtaining a deep feature value of the deep reinforcement learning model at the same time; the driving decision indicates the next action of the autonomous driving device; the action includes turning left, turning right, accelerating or braking; the deep feature value includes the driving decision output by the deep reinforcement learning model, the activation value of the activation layer, and / or the feature value of one or more hidden layers; the classifier judges the current environmental state based on the deep feature value, and when the deep feature value deviates from normal data, outputs a judgment result that the environmental state is abnormal; the abnormality is an environmental state that has been attacked.

[0007] Therefore, the embodiments of the present application are designed for the deep reinforcement learning model and adopt the characteristic values ​​of the deep reinforcement learning model for analysis. The range of attack detection capabilities can include unique security attacks and adversarial strategy attacks against the deep reinforcement learning model. It can effectively defend against dodge attacks and adversarial strategy attacks, thereby preventing the autonomous driving devices in traffic from being misled to make wrong driving decisions and causing unintentional harmful consequences.

[0008] In some feasible implementations, the method includes a training process of a deep reinforcement learning model: obtaining data of a normal environmental state of an autonomous driving device recorded in history to obtain a training data set; determining an action set corresponding to the training data set; training a deep reinforcement learning model using the data in the training data set as input and the actions in the action set as output targets to obtain a trained deep reinforcement learning model.

[0009] Therefore, the embodiment of the present application trains the deep reinforcement learning model according to the training data set under normal environmental conditions. The deep reinforcement learning model used in the driving scenario is trained in advance. The trained deep reinforcement learning model outputs driving decisions under normal environmental conditions. At the same time, the deep feature values ​​of the deep reinforcement learning model conform to the normal parameter distribution.

[0010] In some achievable implementations, the classifier is a machine learning binary classification algorithm, and the method includes a classifier training process: determining a set of deep feature values ​​of a deep reinforcement learning model under normal driving conditions, using the set of deep feature values ​​of the deep reinforcement learning model as input, using a normal judgment result as an output target, training the machine learning binary classification algorithm, and obtaining a trained classifier.

[0011] Therefore, the classifier obtained by training the embodiment of the present application according to the set of deep feature values ​​of the deep reinforcement learning model under normal environmental conditions can distinguish between normal and attacked environmental conditions.

[0012] In some achievable implementations, the feature value is a driving decision output by a deep reinforcement learning model, and the classifier judges the current environmental state based on the deep feature value. When the deep feature value deviates from normal data, the classifier outputs a judgment result that the environmental state is abnormal, including: the classifier judges the driving decisions output by the deep reinforcement learning model at multiple time points, and when the driving decisions output at multiple time points deviate from the normal data distribution, the judgment result output by the classifier is abnormal, and the multiple time points include the current moment; when the driving decisions output by the deep reinforcement learning model at multiple time points conform to the normal data distribution, the judgment result output by the classifier is normal.

[0013] In this way, the classifier of the embodiment of the present application adopts the driving decision output by the deep reinforcement learning model for analysis, and can identify the errors in the output layer feature values ​​of the model caused by adversarial strategy attacks, so as to provide indications of the adversarial security threats faced by the deep reinforcement learning model in the autonomous driving scenario.

[0014] In some feasible implementations, the depth feature value is the activation value of the model, and the classifier judges the current environmental state according to the depth feature value. When the depth feature value deviates from normal data, the classifier outputs a judgment result that the environmental state is abnormal, including: the classifier judges the environmental state at the current moment according to the activation value, and when the activation value is higher than a set threshold, the judgment result output by the classifier is abnormal; when the activation value meets the set threshold, the judgment result output by the classifier is normal.

[0015] In this way, the classifier of the embodiment of the present application adopts the activation value of the deep reinforcement learning model for analysis, and can identify the errors in the activation values ​​of multiple neurons within the model caused by various dodge attacks, so as to provide indications for the dodge attacks faced by the deep reinforcement learning model in the autonomous driving scenario.

[0016] In some achievable implementations, the deep feature value is a set of feature values ​​of one or more hidden layers of the deep reinforcement learning model, and the classifier judges the current environmental state based on the deep feature value, and outputs a judgment result that the environmental state is abnormal when the deep feature value deviates from normal data, including: the classifier judges based on the set of feature values ​​of one or more hidden layers, and when the set of feature values ​​of one or more hidden layers deviates from normal data, the judgment result output by the classifier is abnormal; when the feature values ​​of one or more hidden layers conform to normal data, the judgment result output by the classifier is normal.

[0017] In this way, the classifier of the embodiment of the present application adopts the deep feature values ​​of one or more hidden layers within the deep reinforcement learning model for analysis, and can identify errors in a certain area of ​​interest within the model caused by multiple adversarial safety threats, so as to provide indications of the adversarial safety threats faced in the autonomous driving scenario.

[0018] In some feasible implementations, the method further includes: determining whether to execute the action indicated by the driving decision based on a judgment result of whether the environmental state is normal or abnormal; when the environmental state is normal, executing the action indicated by the driving decision; when the environmental state is abnormal, refusing to execute the action indicated by the driving decision.

[0019] In this way, the autonomous driving device of the embodiment of the present application can effectively defend against security threats such as evasion attacks and adversarial strategy attacks, thereby preventing the autonomous driving device in traffic from making wrong driving decisions due to being misled by the adversarial security threats, resulting in unintentional harmful consequences.

[0020] In some feasible implementations, target data is obtained by fusing data based on multiple environmental states, including: converting data of multiple environmental states from analog signals to digital signals, and fusing multiple digital signals to obtain target data; the data of multiple environmental states also includes status data inside the vehicle.

[0021] In this way, the fusion of the embodiments of the present application can improve the accuracy of the signal, ensure the integrity and reliability of the signal, and obtain a higher detection probability and credibility.

[0022] In some practicable implementations, data of multiple environmental states under the current driving state are obtained, including: collecting data of multiple environmental states through one or a combination of GPS, camera components, radar components, lidar components, ultrasonic sensors.

[0023] In this way, the embodiments of the present application can sense data on various environmental states of the location of the autonomous driving device, and describe the real-time status of complex environments such as autonomous driving scenarios through data on various environmental states, thereby improving the authenticity and reliability of anomaly detection.

[0024] In the second aspect, the embodiment of the present application provides an abnormality detection system, which is applied to an automatic driving device. The system includes: a sensor component for obtaining data of multiple environmental states under the current driving state; a perception module for fusing data based on multiple environmental states to obtain target data; a decision module for determining the driving decision output by the deep reinforcement learning model according to the target data, and obtaining the deep feature value of the deep reinforcement learning model at the same time; the driving decision indicates the next action of the automatic driving device; the action includes turning left, turning right, accelerating or braking; the deep feature value includes the driving decision output by the deep reinforcement learning model, the activation value of the activation layer, and / or the feature value of one or more hidden layers; the classifier judges the current environmental state according to the feature value, and outputs the judgment result that the environmental state is abnormal when the action trajectory deviates from the normal data; the abnormality is the environmental state under attack. Its beneficial effects are as described in the first aspect, and will not be repeated here.

[0025] In some feasible implementations, the system also includes: a training module for a deep reinforcement learning model, which is used to obtain historical data of various environmental states of the autonomous driving device under normal environmental states to obtain a training data set; determine an action set corresponding to the training data set; and train the deep reinforcement learning model using the data in the training data set as input and the actions in the action set as output targets to obtain a trained deep reinforcement learning model.

[0026] In some feasible implementations, the classifier is a machine learning binary classification algorithm; the system also includes: a classifier training module, which is used to determine a set of deep feature values ​​of the deep reinforcement learning model under normal environmental conditions, and to train the machine learning binary classification algorithm with the set of deep feature values ​​of the deep reinforcement learning model as input and the normal judgment result as the output target to obtain a trained classifier.

[0027] In some feasible implementations, the feature values ​​include driving decisions output by the deep reinforcement learning model, and the classifier is used to make judgments based on the driving decisions output by the deep reinforcement learning model at multiple time points. When the driving decisions output by the deep reinforcement learning model at multiple time points deviate from the normal data distribution, the judgment result output by the classifier is abnormal; when the driving decisions output by the deep reinforcement learning model at multiple time points conform to the normal data distribution, the judgment result output by the classifier is normal, and the multiple time points include the current moment.

[0028] In some feasible implementations, the deep feature values ​​include a set of feature values ​​of one or more hidden layers of the deep reinforcement learning model, and the classifier is used to make judgments based on the set of feature values ​​of the one or more hidden layers. When the set of feature values ​​of one or more hidden layers deviates from normal data, the judgment result output by the classifier is abnormal; when the feature values ​​of one or more hidden layers conform to normal data distribution, the judgment result output by the classifier is normal.

[0029] In some feasible implementations, the deep feature value is the activation value of the model, and the classifier is used to judge the environmental state at the current moment according to the activation value. When the activation value is higher than the set threshold, the judgment result output by the classifier is abnormal; when the activation value meets the set threshold, the judgment result output by the classifier is normal.

[0030] In some feasible implementations, the system also includes: a control module, used to determine whether to execute the driving decision output by the deep reinforcement learning model based on the judgment result of whether the environmental state is normal or abnormal; when the environmental state is normal, execute the action output by the deep reinforcement learning model; when the environmental state is abnormal, refuse to execute the action output by the deep reinforcement learning model.

[0031] In some feasible implementations, the perception module is used to convert data of multiple environmental states from analog signals to digital signals, and fuse the multiple digital signals to obtain target data; the data of multiple environmental states also includes state data inside the vehicle.

[0032] In some possible implementations, the sensor component includes one or a combination of a GPS, a camera component, a radar component, a lidar component, and an ultrasonic sensor.

[0033] In a third aspect, an embodiment of the present application provides an automatic driving device, including an abnormality detection system as described in any one of the second aspects, and the system is used to implement an abnormality detection method as described in any one of the first aspects. Its beneficial effects are as described in the first aspect and will not be repeated here.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the method provided in the first aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0038] Figure 1 A schematic diagram of the deep reinforcement learning model framework in an autonomous driving device;

[0039] Figure 2 A schematic diagram of an example of determining the direction of movement through a deep reinforcement learning model;

[0040] Figure 3 A schematic diagram of the system architecture for anomaly detection provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a decision module provided in an embodiment of the present application;

[0042] Figure 5 An implementation flow chart of the anomaly detection system provided in the embodiment of the present application;

[0043] Figure 6a This is a training diagram for a deep reinforcement learning model;

[0044] Figure 6b This is a schematic diagram of the training of classifier C;

[0045] Figure 7 A flowchart of a method for abnormality detection provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of classifier C training provided in Example 2 of the present application;

[0047] Fig. 9 A schematic diagram of classifier C training provided in Example 3 of the present application;

[0048] Fig.10 A schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0050] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0051] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple terminals refers to two or more terminals.

[0052] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0053] In the description of the embodiments of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the description of the embodiments of the present application, the terms "first\second\third, etc." or module A, module B, module C, etc. are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0055] Relevant terms involved in the embodiments of this application:

[0056] Machine learning (ML): Machine learning is a science of artificial intelligence. The main research object of this field is artificial intelligence, especially how to improve the performance of specific algorithms through empirical learning.

[0057] Deep learning (DL): A type of machine learning technology based on deep neural network algorithms that uses multiple nonlinear transformations to process and analyze data. It is mainly used in perception, decision-making and other scenarios in the field of artificial intelligence, such as image and speech recognition, natural language translation, computer games, etc.

[0058] In the description of the embodiments of the present application, the numbers representing the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] Figure 1 Figure 1 is a schematic diagram of the deep reinforcement learning model framework in an autonomous driving device. Figure 1 As shown, the carrier of the deep reinforcement learning model is the automatic driving device 11, and the deep reinforcement learning model can be obtained through training. The training process of the model can be understood as a continuous optimization process of the deep reinforcement deep learning model parameter set. The interaction process between the deep learning model and the environment 12 can be defined using actions, states, and rewards. The training process of the deep reinforcement learning model is carried out in a normal simulated driving environment 12. The automatic driving device 11 learns through the reward feedback given by the simulated driving environment 12. After the automatic driving device 11 receives the state of the environment 12, it selects an action to execute, and then learns how to choose an action to maximize the reward value through the reward fed back by the environment 12.

[0061] Figure 2 This is a schematic diagram of an example of determining the direction of action through a deep reinforcement learning model. Figure 2As shown, in the autonomous driving scenario, the autonomous driving device 21 is loaded with a deep reinforcement learning model, and the deep reinforcement learning model receives the current state data St of the driving environment 22. The current state data St includes the speed of the autonomous driving device (ego car speed) and the information of neighboring devices (neighboring cars information). The autonomous driving device 21 selects to execute a travel direction At according to the current state data St. The travel direction At can be straight, right turn or left turn; after executing the travel direction At on the driving environment 22, the autonomous driving device 21 obtains the reward feedback Rt+1 given by the driving environment 12 at the next moment, including the optimized average speed (average speed) and the number of lane changes (count of changing); the autonomous driving device 21 determines the next travel direction according to the reward feedback Rt+1.

[0062] When deep reinforcement learning models face adversarial policy attacks, existing technologies analyze environmental information to determine whether anomalies occur. However, the way the environment is described is strongly related to the accuracy of anomaly detection, and the description of the environment affects the accuracy of anomaly detection. In some mission scenarios, the dynamic transformation of the environment can usually be described by Markov decision processes, but complex environments such as autonomous driving scenarios cannot be described by Markov decision processes, resulting in the inability of this technology to be transferred to autonomous driving scenarios.

[0063] From the perspective of abnormality detection technology for autonomous driving, existing patented technologies include technologies that use vehicle status and driving conditions for abnormality detection. However, this technology does not consider the use of artificial intelligence technology, such as the setting of deep reinforcement learning models. It is not clear whether this technology can be migrated to scenarios based on deep reinforcement learning models.

[0064] The embodiment of the present application provides an anomaly detection method, which detects the occurrence of adversarial attacks by analyzing the deep feature values ​​of the deep reinforcement learning model. The deep feature values ​​of the deep reinforcement learning model include: the driving decision (action) output by the model, the activation value (activation) of the model, and the deep features (deepfeatures) of other layers of the model. In the autonomous driving scenario, when the deep reinforcement learning model encounters an evasive attack, the activation value of the model will be abnormal, higher than the set threshold. When the model encounters an adversarial strategy attack, the decision action output by the model will deviate from the normal data distribution.

[0065] Figure 3A schematic diagram of the system architecture of anomaly detection provided in the embodiment of the present application. Figure 3 As shown, in an abnormality detection system, a sensor group 21, a perception module 22 and a decision module 23 are set.

[0066] The sensor group 21 includes a number of sensors, such as a global positioning system (GPS), a camera assembly, a radar assembly, a laser radar assembly, an ultrasonic sensor, etc. The sensor group is used to sense data of various environmental states of the location of the autonomous driving device. The environmental state data includes the location and speed (ego car speed) of the autonomous driving device, information of neighboring devices (neighbo ring cars information), and information of in-car conditions. The information of neighboring devices is used to describe environmental variables.

[0067] The Global Positioning System (GPS) is a sensor used to determine the geographic location of an autonomous driving device. The GPS module includes a transceiver, which determines the relative position of the autonomous driving device to the earth based on satellite positioning data and obtains GPS data.

[0068] A radar assembly for detecting characteristics of an object using radio waves, obtaining radar data such as the distance, altitude, direction or speed of the object. The radar assembly may be configured to transmit radio waves or microwave pulses, which may bounce off any object in the path of the wave. The object may return a portion of the energy of the wave to a receiver (e.g., a dish or antenna), which may also be part of the radar unit. The radar assembly may also be configured to perform digital signal processing on the received signal (bounced off the object), and may be configured to identify the object.

[0069] The LiDAR component is used to perform light detection and ranging using the visible light of the laser to obtain LiDAR data.

[0070] A camera assembly for acquiring image data of the environment in which the autonomous driving device is located, for example, a static camera, a video camera, and the like. To this end, the camera may be configured to detect visible light, or may be configured to detect light from other parts of the spectrum (such as infrared light or ultraviolet light). Other types of cameras are also possible. The camera may be a two-dimensional detector, or may have a three-dimensional spatial range. In some examples, the camera may be, for example, a distance detector configured to generate a two-dimensional image indicating the distance from the camera to several points in the environment. To this end, the camera may use one or more distance detection technologies. For example, the camera may be configured to use structured light technology, in which the autonomous driving device illuminates an object in the environment using a predetermined light pattern, such as a grid or checkerboard pattern, and uses a camera to detect reflections of a predetermined light pattern from the object. Based on the distortion in the reflected light pattern, the autonomous driving device may be configured to detect the distance of a point on the object. The predetermined light pattern may include infrared light or light of other wavelengths.

[0071] Ultrasonic sensors use ultrasonic waves to detect the characteristics of objects and obtain ultrasonic data, such as the distance, height, direction or speed of the object.

[0072] In some examples, the sensor group 21 also includes, for example, an inertial measurement unit (IMU), an accelerometer, and a gyroscope.

[0073] An IMU can be a device used to sense changes in position and orientation of an autonomous vehicle based on inertial acceleration and any combination thereof.

[0074] In some examples, the combination of sensors may include, for example, an accelerometer and a gyroscope. It is also possible that the sensor group 21 is a combination of other sensors.

[0075] The sensor group 21 obtains data of various environmental states from a real-time, changing environment. The data of various environmental states include radar data, lidar data, camera data, ultrasonic data and / or IMU data, etc.

[0076] The above radar data, lidar data, camera data, ultrasonic data and / or IMU data are examples of observation data of various environmental changes and are not exhaustive. Other data representing environmental changes and observation data of in-vehicle conditions are also within the scope of protection of this application.

[0077] The perception module 22 pre-processes and fuses the various status data collected by the sensor group 21 to obtain target data.

[0078] Preprocessing includes steps such as analog signal / digital signal conversion, geometric transformation, normalization, smoothing, restoration and enhancement for one or more of the data of various environmental states. In some embodiments, the preprocessing process includes filtering, windowing and synchronization for one or more of the data of various environmental states. Preprocessing eliminates irrelevant information in the data of various environmental states, restores useful real information, enhances the detectability of relevant information and simplifies data to improve the reliability of feature extraction, image segmentation, matching and recognition.

[0079] Fusion includes multi-channel signal fusion, image fusion and data fusion. Fusion can improve the accuracy of signals, ensure the integrity and reliability of signals, and obtain higher detection probability and credibility.

[0080] Among them, multi-channel signal fusion is to pre-process the multi-channel signals carrying the same information collected by the sensor group 21, such as windowing and synchronization, and quickly perform specific processing on the statistically independent multi-channel signals based on the field strength signal; the signals after specific processing are preferentially intercepted and combined into the most accurate received signal in a short time to reduce the impact of fading. The accuracy of the received signal is improved and the integrity and reliability of the signal are guaranteed.

[0081] Image fusion includes pixel-level image fusion, feature-level image fusion and decision-level image fusion. Pixel-level image fusion is to operate and process image data at the image pixel level. Feature-level image fusion is to extract the advantageous feature information of each image, such as edges, textures, etc., based on the existing imaging characteristics of each sensor; it mainly includes algorithms such as fuzzy clustering and support vector clustering. Decision-level image fusion belongs to the highest level of fusion. Compared with feature-level image fusion, it processes the source image after extracting the target features of the image, and then continues to perform feature recognition, decision classification and other processing, and then combines the decision information of each source image for joint reasoning to obtain the reasoning result.

[0082] Data fusion refers to multi-sensor fusion. Common data fusion methods include: data fusion based on the relationship between input data and input / output data types; defining the data hierarchy such as raw measurement, signal, feature or decision, and performing data fusion based on the defined hierarchy; performing data fusion based on the data structure type: centralized, decentralized or distributed. The purpose of multi-sensor fusion is to obtain a higher detection probability and credibility.

[0083] In some embodiments, the target data obtained after preprocessing and sensor fusion includes average vehicle speed and number of lane changes.

[0084] The decision module 23 includes a driving decision model 231 and a judgment unit 232. The driving decision model 231 outputs a driving decision based on the target data, and the driving decision indicates the next action of the automatic driving device, such as turning left, turning right, accelerating or braking, etc. The judgment unit 232 judges the environmental state according to the deep feature value of the driving decision model 231, and outputs a judgment result of whether the environmental state is normal or abnormal; normal is an environmental state that has not been attacked, and abnormal is an environmental state that has been attacked; the environmental state that has been attacked includes being attacked by evasion, adversarial strategy attack and / or other types of adversarial attacks.

[0085] In some autonomous driving scenarios, the input of the driving decision model 231 also includes map data and V2V / V2I data. The decision action model outputs the next driving decision based on the target data, map data and V2V / V2I data. The judgment unit 232 obtains the characteristic value of the deep reinforcement learning model, and makes a judgment based on the map data and V2V / V2I data to obtain a judgment result of whether the environmental state is normal or abnormal.

[0086] The abnormality detection system provided in the embodiment of the present application also includes a control module 24, which determines whether to execute the driving decision output by the deep reinforcement learning model according to the judgment result of whether the environmental state is normal or abnormal. When the environmental state is normal, the driving decision output by the deep reinforcement learning model is executed, such as turning left, turning right, accelerating or braking; when the environmental state is abnormal, the action output by the deep reinforcement learning model is refused to be executed.

[0087] For example, the main vehicle is traveling in a straight line on a highway, and an autonomous driving vehicle installed with a special algorithm behind it makes a sudden acceleration attack action according to the installed special algorithm. The sensor group 21 of the main vehicle obtains data of various environmental states from the real-time changing environment. Based on the data of various environmental states, the main vehicle, as the target of attack, makes a decision to turn right to avoid the attack. The judgment unit 232 judges the environmental state according to the deep feature value of the driving decision model 231, and outputs a judgment result that the environmental state is abnormal. The environmental state under attack may be an adversarial strategy attack. When the environmental state is abnormal, the control module 24 refuses to execute the right turn avoidance decision output by the deep reinforcement learning model, and keeps going straight to avoid dangerous actions that may cause scratches or collisions with vehicles passing on the right.

[0088] The automatic driving function of the automatic driving device is mainly realized by the control module 24. As a vehicle-mounted terminal device, the control module 24 is sometimes also called an electronic control unit (ECU), a domain control unit (DCU), or a mobile data center (MDC).

[0089] In some embodiments, the control module 24 receives the driving decision output by the decision module 23, such as turning left, turning right, accelerating or braking, performs task decomposition on the driving decision, obtains the next action, and assigns it to the power engine component of the vehicle body control to execute the action.

[0090] Figure 4 Schematic diagram of the decision module provided in the embodiment of the present application. Figure 4 As shown, the driving decision model 231 is a trained deep reinforcement learning model, and a classifier C is set in the judgment unit 232. The target data after preprocessing and fusion is input into the deep reinforcement learning model. The deep reinforcement learning model outputs a driving decision and transmits the driving decision to the control module 24. At the same time, the feature values ​​of the deep reinforcement learning model will be collected and transmitted to the classifier C, which will be judged by the classifier C. The classifier C outputs the judgment result of whether the environmental state is normal or abnormal; normal is the environmental state without attack; abnormal is the environmental state under evasion attack or adversarial strategy attack. The control module 24 decides whether to execute the driving decision based on the driving decision from the deep reinforcement learning model and the judgment result of the classifier C. If the environmental state judgment result is normal, the control module 24 executes the driving decision. If the environmental state judgment result is abnormal, the control module 24 does not execute the driving decision. Driving decisions include turning left, turning right, accelerating or braking.

[0091] The classifier C can be implemented by various available algorithms, such as a binary classification algorithm of machine learning, and the training data set of the classifier C is a set of observable features of the deep reinforcement learning model collected from the vehicle under normal environmental conditions. The binary classification algorithms of machine learning include but are not limited to: cluster-based local outlier factor; feature bagging; histogram-based outlier detection; K-nearest neighbors; isolation forest; one class support vector machines.

[0092] The above is an introduction to the abnormality prediction system of the automatic driving device provided in the embodiment of the present application. For ease of understanding, the implementation process of the system is described below.

[0093] Figure 5 This is a flowchart of the implementation of the anomaly detection system provided in the embodiment of the present application. Figure 5 As shown, in an automatic driving device 50, a sensor group 51 collects data of environmental states of multiple environmental states, and a perception module 52 performs preprocessing and fusion based on the data of environmental states of multiple environmental states to obtain target data. The preprocessing process is to convert the data of multiple environmental states from analog signals to digital signals, and the fusion is to fuse multiple sensor data; the target data is input into the deep reinforcement learning model of the decision module 53, and a driving decision is output; the deep reinforcement learning model is a model trained under the environmental state of normal driving; the driving decision is transmitted to the control module 54, and at the same time, the deep feature values ​​of the deep reinforcement learning model are collected and transmitted to the classifier C, and the deep feature values ​​include but are not limited to: the driving decision output by the output layer of the deep reinforcement learning model, the activation value (activation) of the activation layer of the model, and the deep features of any hidden layer in the model. The classifier C makes a judgment based on the deep feature values ​​of the deep reinforcement learning model and outputs a judgment result that the environmental state is normal or abnormal; the control module 54 decides whether to execute the driving decision based on the driving decision from the deep reinforcement learning model and the judgment result of the classifier C on the environmental state. If the result of the environmental state judgment is normal, the control module 54 executes the driving decision. If the result of the environmental state judgment is abnormal, the control module 54 does not execute the driving decision.

[0094] The automatic driving device 50 may be a car or may take the form of an intelligent vehicle. However, the exemplary system may also be implemented on other transport devices or in the form of other transport devices, such as intelligent transportation vehicles, intelligent agricultural equipment, intelligent construction equipment, and other transport devices with automatic driving functions. In addition, a robotic device may also be used to perform the methods and systems described herein, such as a sweeping robot, etc.

[0095] The embodiment of the present application checks the occurrence of attacks through the deep feature values ​​of the deep reinforcement learning model trained under normal driving environment conditions. When the model encounters an evasion attack, the activation value of the model will be higher than the threshold set under normal conditions. When the model encounters an adversarial strategy attack, the driving decision output by the model will deviate from the data distribution under normal conditions.

[0096] In the embodiment of the present application, the training process of the deep reinforcement learning model includes: Figure 6aAs shown, a data set of environmental states collected by a sensor component under normal driving conditions or experimental data in a simulated normal driving process is collected to obtain training set data; an action data set corresponding to the training data set is determined; a deep reinforcement learning model is trained using the data in the training data set as input and the actions in the action data set as output targets to obtain a trained deep reinforcement learning model.

[0097] The training process of classifier C includes: Figure 6b As shown, the set of deep feature values ​​of the deep reinforcement learning model under normal driving conditions is taken as input, and the judgment result that the environmental state is normal is taken as the output target, and the classifier C is trained to obtain the trained classifier C, where classifier C is a machine learning binary classification algorithm.

[0098] Next, based on the above description, the method for abnormality detection provided by the embodiment of the present application is introduced. The method is proposed based on the abnormality detection system described above, and part or all of the content of the method can refer to the relevant description above.

[0099] Figure 7 A flowchart of a method for abnormality detection provided in an embodiment of the present application. It is understood that the method can be executed by any autonomous driving device, equipment, or platform with computing and processing capabilities.

[0100] like Figure 7 As shown, the method for anomaly detection includes: S11, obtaining data of multiple environmental states under the current driving state; S12, fusing the data of multiple environmental states under the current driving state to obtain target data; S13, determining the driving decision output by the deep reinforcement learning model according to the target data, and obtaining the deep feature value of the deep reinforcement learning model at the same time; the driving decision indicates the next action of the automatic driving device; the action includes turning left, turning right, accelerating or braking; the deep feature value includes the driving decision output by the deep reinforcement learning model, the activation value of the activation layer, and / or the feature value of one or more hidden layers; S14, the classifier judges the current environmental state according to the feature value, and when the deep feature value deviates from normal data, outputs a judgment result that the environmental state is abnormal; the abnormality is an environmental state that has been attacked.

[0101] Next, each step of the embodiment of the present application is described.

[0102] S11, obtaining data of various environmental states under the current driving state.

[0103] Data on various environmental states can be collected through one or a combination of GPS, camera components, radar components, lidar components, ultrasonic sensors, and IMUs configured on the autonomous driving device. The data on various environmental states include radar data, lidar data, camera data, ultrasonic data, and / or IMU data as well as status data inside the vehicle.

[0104] S12, obtaining target data based on the data of various environmental states by fusing them.

[0105] The data of various environmental states are preprocessed and fused to obtain the target data. The preprocessing process includes the steps of analog signal / digital signal conversion, geometric transformation, normalization, smoothing, restoration and enhancement for one or more of the data of various environmental states.

[0106] In some implementations, one or more of the data of the various environmental states may be pre-processed by filtering, windowing, or synchronization.

[0107] In some embodiments, signal fusion, image fusion and / or data fusion may be performed on the preprocessed data of various environmental states to obtain target data, so as to improve the accuracy of the signal, ensure the integrity and reliability of the signal, and obtain a higher detection probability and credibility.

[0108] S13, determining the driving decision output by the deep reinforcement learning model according to the target data, and obtaining the deep feature value of the deep reinforcement learning model; the driving decision indicates the next action of the automatic driving device, and the action includes turning left, turning right, accelerating or braking.

[0109] The target data can be input into a trained deep reinforcement learning model, which outputs a driving decision that indicates the next action of the autonomous driving device, such as turning left, turning right, accelerating, or braking.

[0110] The deep feature values ​​of the deep reinforcement learning model include the driving decision output by the deep reinforcement learning model, the activation value of the activation layer, and / or the feature values ​​of one or more hidden layers.

[0111] S14, the classifier makes a judgment based on the deep feature value of the deep reinforcement learning model. When the deep feature value deviates from normal data, the classifier outputs a judgment result that the environmental state is abnormal; the abnormality is the environmental state that is under attack.

[0112] In some embodiments, a set of deep feature values ​​of a deep reinforcement learning model can be collected by a visualization method, and the set of feature values ​​can be input into a classifier C, which makes a judgment, and the classifier C outputs a judgment result of whether the current environmental state is normal or abnormal. Normal refers to an environmental state that has not been attacked; abnormal refers to an environmental state that has been attacked.

[0113] In some embodiments, the deep feature value is the driving decision output by the output layer of the deep reinforcement learning model, and the classifier makes a judgment based on the driving decision output by the deep reinforcement learning model at the current moment and the driving decisions output by the deep reinforcement learning model at multiple time points in history, and obtains a judgment result of whether the environmental state is normal or abnormal. In the case where the driving decision output by the deep reinforcement learning model at the current moment deviates from the normal data distribution, the judgment result output by the classifier is abnormal, and the deep reinforcement learning model may encounter adversarial strategy attacks. In the case where the driving decision conforms to the normal data distribution, the judgment result output by the classifier is normal, the deep reinforcement learning model has not encountered an attack, and is in a normal driving state.

[0114] In some embodiments, the deep feature value is the activation value of the activation layer of the deep reinforcement learning model, and the classifier makes a judgment based on the activation value to obtain a judgment result of whether the environmental state is normal or abnormal. When the activation value is higher than the set threshold, the judgment result output by the classifier is abnormal, and the deep reinforcement learning model may encounter an evasive attack. When the activation value meets the set threshold, the judgment result output by the classifier is normal, and the deep reinforcement learning model has not encountered an attack and is in a normal driving state.

[0115] In some embodiments, the deep feature value is a set of feature values ​​of one or more hidden layers of the deep reinforcement learning model, and the classifier makes a judgment based on the set of feature values ​​of one or more hidden layers to obtain a judgment result of whether the environmental state is normal or abnormal. In the case where the set of feature values ​​of one or more hidden layers deviates from normal data, the classifier judges that the result is abnormal, and the deep reinforcement learning model may encounter some kind of attack. In the case where the feature values ​​of one or more hidden layers conform to the normal data distribution, the classifier judges that the result is normal, the deep reinforcement learning model has not encountered an attack, and is in a normal driving state.

[0116] In some embodiments, the method for abnormality detection provided in the embodiment of the present application further includes step S15, in which it is determined whether to execute the driving decision output by the deep reinforcement learning model according to the judgment result of whether the environmental state is normal or abnormal. When the environmental state is normal, the action indicated by the driving decision output by the deep reinforcement learning model is executed; when the environmental state is abnormal, the action indicated by the driving decision output by the deep reinforcement learning model is refused to be executed.

[0117] Example 1

[0118] The abnormal attack detection method provided in Example 1 of the present application uses an ego-vehicle as the autonomous driving subject in an autonomous driving scenario with more than one vehicle, and the ego-vehicle uses a deep reinforcement learning model to implement driving decisions.

[0119] The training set data is obtained from the sensor data under normal environmental conditions in the field of historically recorded normal driving scenes or in the test scenes simulating the normal driving process; the action data set corresponding to the training data set is determined; the deep reinforcement learning model is trained with the data in the training data set as input and the actions in the action data set as output targets to obtain a trained deep reinforcement learning model. Among them, the normal driving scene refers to the autonomous driving scene without attack conditions.

[0120] Determine the deep feature values ​​of the trained deep reinforcement learning model in the host vehicle. The deep feature values ​​of the deep reinforcement learning model can be: action value, which is the driving decision of the output layer of the model; activation value of the activation layer of the model, and feature values ​​of one or more hidden layers.

[0121] A classifier C is established through a machine learning binary classification algorithm. The deep feature values ​​of the deep reinforcement learning model trained under normal environmental conditions are used as input, and the judgment result that the environmental state is normal is used as the output target. Classifier C is trained to obtain a trained classifier C.

[0122] The trained deep reinforcement learning model and classifier C are set in the decision module 53 of the host vehicle.

[0123] When the host vehicle is operating in the driving scenario, for a given time point, given the deep feature values ​​of the deep reinforcement learning model, the classifier C decides whether the current observation is normal.

[0124] The deep feature values ​​obtained by the deep reinforcement learning model training in Example 1 of the present application can be observable features obtained through visualization. One method of visualizing one feature is to visualize the weights of the model. In the visualization of weights, a commonly used method is the activation maximization method. The activation maximization method is to guess the features learned by the model in an indirect way. This activation maximization method is to find the input that can maximize the activation value of a certain neuron as a representative of the features learned by the neural network. A more direct method is the class activation map. The class activation map method is for a network containing a global average pooling layer, and the output results of each convolution kernel before the global average pooling layer are weighted summed based on the weights from the global average pooling layer to the classification layer, and then the weighted summed image is used as a heat map representing the feature map. The focus area of ​​the model can be detected by the visualization method of deep features.

[0125] refer to Figure 5 Given an anomaly detection system, the implementation process of Example 1 of the present application includes the following steps:

[0126] The sensor group 51 collects data of various environmental conditions.

[0127] The perception module 52 preprocesses and fuses the data of various environmental states to obtain target data. The preprocessing process is to convert the data of various environmental states from analog signals to digital signals, and the fusion is to fuse the data of multiple sensors.

[0128] The target data is input into the deep reinforcement learning model of the decision module 53, and a driving decision is output; the deep reinforcement learning model is a model trained under the normal driving environment state; the driving decision is transmitted to the control module 54, and at the same time, the observable feature values ​​of the deep reinforcement learning model will be collected and passed to the classifier C. The observable feature values ​​that can be collected include but are not limited to: the driving decision output by the output layer of the deep reinforcement learning model, the activation value (activation) of the activation layer of the model, and the features of any hidden layer in the model. Classifier C makes a judgment based on the feature values ​​of the deep reinforcement learning model and outputs a judgment result of whether the environment state is normal or abnormal.

[0129] In the implementation process of Example 1 of the present application, the decision module 53 monitors the observable features of the deep reinforcement learning model at each time slot. For each time slot, the decision module 53 feeds back the observable features of the deep reinforcement learning model to the classifier C. When the classifier C reports an abnormality, it is determined that there is an abnormality in the environmental state. When the decision module 53 detects an abnormality, a warning signal is sent to the control module 54.

[0130] The control module 54 determines whether to execute the driving decision based on the driving decision from the deep reinforcement learning model and the judgment result of the environment state by the classifier C. If the judgment result is normal, the control module 54 executes the driving decision; if the judgment result of the environment state is abnormal, the control module 54 does not execute the driving decision.

[0131] Example 2

[0132] The abnormal attack detection method provided in Example 2 of the present application is to determine an ego vehicle (ego-vehicle) in a driving scene with traffic, where there is at least one vehicle. The abnormal detection system of the vehicle uses the driving decision output by the deep reinforcement learning model for abnormal detection.

[0133] The action space A of the ego-vehicle is set to be A = {a1, a2, a3, ..., an}. a1, a2, a3, ..., an are n driving decisions that the ego-vehicle can choose, and each driving decision indicates an action taken by the ego-vehicle. The action trace is the action sequence taken by the ego-vehicle in the current driving scenario. For example, (s1, s2, s3, ..., st) is the action sequence of the ego-vehicle from time point 1 to t.

[0134] S201 , forming a motion data set according to data of a plurality of motion trajectories of a host vehicle.

[0135] The data of multiple motion trajectories of the host vehicle can be obtained from a historically recorded normal driving scene or a test scene simulating a normal driving process to form a motion trajectory data set, thereby obtaining a training data set.

[0136] S202, such as Figure 8 As shown, based on the action trajectory data set, an action-transi tion matrices M set of the ego-vehicle is established, and the action-transi tion matrices M set includes one or more action-transi tion matrices.

[0137] S203, taking the set of action-transition matrices M as input and the judgment result that the environment state is normal as the output target, training the classifier C to obtain the trained classifier C.

[0138] S204, configuring the trained classifier C in the abnormality detection system of the host vehicle.

[0139] S205 , when the host vehicle is operating in the current driving scenario, obtaining each driving decision taken by the host vehicle.

[0140] S206, monitoring the actions performed by the host vehicle at each time slot, where the actions are actions indicated by the driving decision.

[0141] S207, generating a real-time action transition matrix M' according to the driving decision at each time point including the current time point.

[0142] S208, input the real-time action transition matrix M' into the classifier C, and the classifier C determines whether M' is a statistical outlier (outliner); when the classifier determines that M' is a statistical outlier, it outputs a judgment result that the environmental state is abnormal; when the classifier determines that M' is a normal value, it outputs a judgment result that the environmental state is normal.

[0143] S209: When the classifier outputs an abnormal result, a warning signal is sent.

[0144] S210, the control module of the host vehicle determines whether to execute the driving decision based on the driving decision from the deep reinforcement learning model and the judgment result or warning signal of the environmental state of the classifier C. If the environmental state judgment result is normal, the control module 54 executes the driving decision; if the environmental state judgment result is abnormal, the control module does not execute the driving decision.

[0145] An experiment of the anomaly detection method provided in Example 2 of the present application was conducted in a simulator of a highway environment, where the goal of the host vehicle was to reach the destination safely as quickly as possible and avoid potential collisions.

[0146] In the experiment, four attacks were used to target the safety threats of the host vehicle, including three evasion attacks and one adversarial strategy attack.

[0147] Attack 1 is brake attacks, which tamper with the target data input into the deep reinforcement learning model so that the driving decision output by the model is braking.

[0148] The second attack is lane changing attacks, which tamper with the target data input into the deep reinforcement learning model so that the driving decision output by the model is lane change, such as left turn or right turn.

[0149] Attack three is speed attacks, which tamper with the target data input into the deep reinforcement learning model so that the driving decision output by the model is acceleration.

[0150] Attack 4 is an adversarial strategy attack, which adds a vehicle with an adversarial strategy attack algorithm to the traffic environment, causing the main vehicle to make abnormal action decisions.

[0151] Classifier C is implemented by different machine learning binary classification algorithms for observation and comparison.

[0152] In the experiments on braking and dodging attacks 1, 2, and 3, the K-nearest neighbors classifier performed better. The experimental results show that the K-nearest neighbors classifier can detect abnormal action transfer matrices, with a false positive rate of 0.05 and a true positive rate of 0.95.

[0153] In the experiment against adversarial strategy attacks, the Isolation Forest classifier performed better. The experimental results showed that the Isolation Forest classifier can detect abnormal action transfer matrices, with a false positive rate of 0.05 and a true positive rate of 0.8.

[0154] Example 3

[0155] The method for anomaly detection provided in Example 3 of the present application is to determine an ego vehicle (ego-vehicle) in a driving scene with traffic, where there is at least one vehicle. The anomaly detection system of the ego vehicle uses a deep reinforcement learning model for driving decision-making, and uses the activation value of the deep reinforcement learning model for anomaly detection.

[0156] The activation values ​​(activations) of the deep reinforcement learning model used in the ego-vehicle are used for judgment, where A = (a1, a2, a3, ..., an) represents the n activation values ​​of the n neurons in the activation layer of the deep reinforcement learning model.

[0157] S301, obtaining a set of n activation values ​​of n neurons of an activation layer of a deep reinforcement learning model in a host vehicle from a scene of a normal driving scene or a test scene simulating a normal driving process.

[0158] S302, such as Fig. 9 As shown, taking the set of activation values ​​as input and the judgment result that the environmental state is normal as the output target, the classifier C is trained to obtain the trained classifier C.

[0159] S303, configuring the trained classifier C in the abnormality detection system of the host vehicle.

[0160] S304 , when the host vehicle is operating in the current driving scenario, obtaining each driving decision taken by the host vehicle.

[0161] S305, monitoring n activation values ​​of n neurons of the deep reinforcement learning model.

[0162] S306, input the set of n activation values ​​of n neurons of the deep reinforcement learning model obtained by real-time monitoring into the classifier C, and the classifier C determines whether the set of activation values ​​of the deep reinforcement learning model at the current moment is a statistical outlier (outliner); when the classifier determines that the activation value set is an outlier, it outputs an abnormal judgment result, and when the classifier determines that the activation value set is a normal value, it outputs a normal judgment result. A statistical outlier refers to a value that is higher than the set threshold under the normal state.

[0163] S307, when the classifier outputs an abnormal result, a warning signal is sent.

[0164] S308, the control module of the host vehicle decides whether to execute the driving decision based on the driving decision from the deep reinforcement learning model and the judgment result of the environmental state of the classifier C. If the environmental state judgment result is normal, the control module 54 executes the driving decision; if the environmental state judgment result is abnormal, the control module does not execute the driving decision.

[0165] The activation value of the deep reinforcement learning model adopted in Example 3 of the present application is used for anomaly detection, which can make accurate judgments on dodge attacks.

[0166] An embodiment of the present application proposes an anomaly detection method, which detects and defends against malicious attacks on the deployed deep learning model based on a classifier, under the constraint that driving decisions are implemented by a deep reinforcement learning model.

[0167] The anomaly detection method proposed in the embodiment of the present application is designed for the deep reinforcement learning model and adopts the characteristic values ​​of the deep reinforcement learning model for analysis. The detection attack capability range can include unique security attacks and adversarial strategy attacks against the deep reinforcement learning model. It is the first defense technology solution against adversarial strategy attacks.

[0168] An embodiment of the present application proposes an autonomous driving device, comprising: a sensor component for collecting data on multi-channel environmental states; a perception module for obtaining target data based on the data on the multiple environmental states; an action decision module for determining an action output by a deep reinforcement learning model based on the target data, and obtaining a visualization feature value of the deep reinforcement learning model; an abnormality detection module for making a judgment based on the visualization feature value to obtain a judgment result of whether the environmental state is normal or abnormal; the normal state is an environmental state that has not been attacked; the abnormal state is an environmental state that has been attacked by an evasion attack or a confrontational strategy attack; and a control module for determining whether to execute the action output by the deep reinforcement learning model based on the judgment result of whether the environmental state is normal or abnormal.

[0169] Fig.10 A schematic diagram of an electronic device is provided for an embodiment of the present application. Fig.10 As shown, the electronic device 1000 includes: at least one memory 1001 for storing programs; at least one processor 1002 for executing the programs stored in the memory 1001. When the program stored in the memory is executed, the processor 1002 is used to execute the anomaly detection method as described in any one of the above embodiments.

[0170] The embodiment of the present application proposes a computer storage medium 1003, in which instructions are stored. When the instructions are executed on a computer, the computer executes the anomaly detection method as described in any one of the above embodiments.

[0171] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0172] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0174] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. An abnormality detection method, applied to an automatic driving device, It is characterized in that The method comprises: Obtain data on various environmental conditions under the current driving state; Obtaining target data by fusing the data of the multiple environmental states; Determine a driving decision output by a deep reinforcement learning model according to the target data, and simultaneously obtain a deep feature value of the deep reinforcement learning model; the driving decision indicates a next action of the autonomous driving device; the action includes turning left, turning right, accelerating or braking; the deep feature value includes the driving decision output by the deep reinforcement learning model, an activation value of an activation layer, and / or feature values ​​of one or more hidden layers; The classifier judges the current environmental state according to the depth feature value, and when the depth feature value deviates from normal data, outputs a judgment result that the environmental state is abnormal; the abnormality is the environmental state under attack.

2. The method according to claim 1, It is characterized in that The method includes a training process of the deep reinforcement learning model: Acquire historical data of the automatic driving device under a normal environmental state to obtain a training data set; Determining an action set corresponding to the training data set; The deep reinforcement learning model is trained using the data in the training data set as input and the actions in the action set as output targets to obtain the trained deep reinforcement learning model.

3. The method according to claim 1 or 2, It is characterized in that The classifier is a machine learning binary classification algorithm, and the method includes a training process of the classifier: Determine a set of deep feature values ​​of the deep reinforcement learning model under normal driving conditions, take the set of deep feature values ​​of the deep reinforcement learning model as input, take a normal judgment result as an output target, train the machine learning binary classification algorithm, and obtain a trained classifier.

4. The method according to any one of claims 1 to 3, It is characterized in that The feature value is a driving decision output by the deep reinforcement learning model, and the classifier judges the current environment state according to the deep feature value. When the deep feature value deviates from normal data, the judgment result that the environment state is abnormal is output, including: The classifier makes a judgment based on the driving decisions output by the deep reinforcement learning model at multiple time points. When the driving decisions output at the multiple time points deviate from the normal data distribution, the judgment result output by the classifier is abnormal, and the multiple time points include the current moment; When the driving decisions output by the deep reinforcement learning model at multiple time points conform to normal data distribution, the judgment result output by the classifier is normal.

5. The method according to any one of claims 1 to 3, It is characterized in that The depth feature value is the activation value of the model, and the classifier judges the current environment state according to the depth feature value. When the depth feature value deviates from normal data, the judgment result that the environment state is abnormal is output, including: The classifier judges the environmental status at the current moment according to the activation value. When the activation value is higher than a set threshold, the judgment result output by the classifier is abnormal; when the activation value meets the set threshold, the judgment result output by the classifier is normal.

6. The method according to any one of claims 1 to 3, It is characterized in that The deep feature value is a set of feature values ​​of one or more hidden layers of the deep reinforcement learning model, and the classifier judges the current environment state according to the deep feature value, and outputs a judgment result that the environment state is abnormal when the deep feature value deviates from normal data, including: The classifier makes a judgment based on the set of feature values ​​of the one or more hidden layers. When the set of feature values ​​of the one or more hidden layers deviates from normal data, the judgment result output by the classifier is abnormal; when the feature values ​​of the one or more hidden layers conform to normal data, the judgment result output by the classifier is normal.

7. The method according to any one of claims 1 to 6, It is characterized in that The method further comprises: Determining whether to execute the action indicated by the driving decision according to the result of judging whether the environmental state is normal or abnormal; When the environmental state is normal, executing the action indicated by the driving decision; When the environmental state is abnormal, the action of executing the driving decision instruction is refused.

8. The method according to any one of claims 1 to 7, It is characterized in that The step of fusing the data based on the multiple environmental states to obtain target data includes: The data of the various environmental states are converted from analog signals to digital signals, and the various digital signals are fused to obtain target data; the data of the various environmental states also include state data inside the vehicle.

9. The method according to any one of claims 1 to 8, It is characterized in that The data of various environmental conditions under the current driving state are obtained, including: The data of the various environmental conditions are collected through one or a combination of GPS, camera components, radar components, lidar components, and ultrasonic sensors.

10. An abnormality detection system, used in an automatic driving device, It is characterized in that The system comprises: A sensor component for acquiring data of various environmental conditions under a current driving condition; A perception module, used for fusing the data of the multiple environmental states to obtain target data; A decision-making module is used to determine the driving decision output by the deep reinforcement learning model according to the target data, and obtain the deep feature value of the deep reinforcement learning model at the same time; the driving decision indicates the next action of the automatic driving device; the action includes turning left, turning right, accelerating or braking; the deep feature value includes the driving decision output by the deep reinforcement learning model, the activation value (activation) of the activation layer, and / or the feature value of one or more hidden layers; the classifier judges the current environmental state according to the feature value, and when the action trajectory deviates from normal data, outputs a judgment result that the environmental state is abnormal; the abnormality is the environmental state under attack.

11. The system according to claim 10, It is characterized in that The system further comprises: The training module of the deep reinforcement learning model is used to obtain the data of various environmental states of the autonomous driving device under normal environmental states from historical records to obtain a training data set; determine an action set corresponding to the training data set; use the data in the training data set as input and the actions in the action set as output targets to train the deep reinforcement learning model, so as to obtain the trained deep reinforcement learning model.

12. The system according to claim 10, It is characterized in that The classifier is a machine learning binary classification algorithm; the system also includes: The training module of the classifier is used to determine a set of deep feature values ​​of the deep reinforcement learning model under normal environmental conditions, take the set of deep feature values ​​of the deep reinforcement learning model as input, take the normal judgment result as the output target, train the machine learning binary classification algorithm, and obtain a trained classifier.

13. The system according to any one of claims 10 to 12, It is characterized in that The feature value includes the driving decision output by the deep reinforcement learning model, and the classifier is used to make a judgment based on the driving decision output by the deep reinforcement learning model at multiple time points. When the driving decision output by the deep reinforcement learning model at multiple time points deviates from a normal data distribution, the judgment result output by the classifier is abnormal; When the driving decisions output by the deep reinforcement learning model at multiple time points conform to normal data distribution, the judgment result output by the classifier is normal, and the multiple time points include the current moment.

14. The system according to any one of claims 10 to 12, It is characterized in that The deep feature value includes a set of feature values ​​of one or more hidden layers of the deep reinforcement learning model, and the classifier is used to make a judgment based on the set of feature values ​​of the one or more hidden layers. When the set of feature values ​​of the one or more hidden layers deviates from normal data, the judgment result output by the classifier is abnormal; When the characteristic values ​​of the one or more hidden layers conform to normal data distribution, the judgment result output by the classifier is normal.

15. The system according to any one of claims 10 to 12, It is characterized in that The deep feature value is the activation value of the model, and the classifier is used to judge the current environmental state according to the activation value. When the activation value is higher than a set threshold, the judgment result output by the classifier is abnormal; When the activation value meets the set threshold, the judgment result output by the classifier is normal.

16. The system according to any one of claims 10 to 15, It is characterized in that The system further comprises: A control module is used to determine whether to execute the driving decision output by the deep reinforcement learning model according to the judgment result of whether the environmental state is normal or abnormal; when the environmental state is normal, execute the action output by the deep reinforcement learning model; when the environmental state is abnormal, refuse to execute the action output by the deep reinforcement learning model.

17. The system according to any one of claims 10 to 16, It is characterized in that The perception module converts the data of the multiple environmental states from analog signals to digital signals, and fuses the multiple digital signals to obtain target data; the data of the multiple environmental states also includes state data inside the vehicle.

18. The system according to any one of claims 10 to 16, It is characterized in that The sensor component includes one or a combination of GPS, camera component, radar component, lidar component, and ultrasonic sensor.

19. An automatic driving device, It is characterized in that It comprises an anomaly detection system as described in any one of claims 10-18, and the system is used to implement the anomaly detection method as described in any one of claims 1-9.

20. An electronic device, It is characterized in that include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 9.

21. A computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 9.

Images