Action poisoning attack system for autonomous driving model and action poisoning attack method for autonomous driving model

Abstract

An action poisoning attack system for an autonomous driving model trained on the basis of the actions of agents each determining the movement thereof, the agents virtually driving in a virtual space, may comprise: a target agent determination unit configured to determine a target agent to be attacked among the plurality of agents on the basis of location information of the agents in the virtual space; and a target action determination unit configured to manipulate action information output by the autonomous driving model with respect to the target agent to generate target action information, and allow the target agent to perform a target action, which is an action by the target action information, to interfere with training of the autonomous driving model.

Description

Action taint attack system for autonomous driving models and method for action taint attack for autonomous driving models

[0001] The present invention relates to an action contamination attack system for an autonomous driving model that is learned based on the actions of each agent that determine the movement of each agent driving virtually in a virtual space.

[0002] The present invention is derived from research conducted as part of the Ministry of Science and ICT's Blockchain Technology Development for Data Economy (Project Unique Number: 1711194405, Project Number: 2021-0-00565-003, Research Project Title: Development of User Identity Authentication and Management Technology for Self-Sovereign Identity Utilization, Project Management Agency: Information and Communications Technology Planning and Evaluation, Project Execution Agency: Garmin Information Systems Co., Ltd., Research Period: 2023.01.01 ~ 2023.12.31) and the Defense Acquisition Program Administration's Leading Technology Development (Project Number: KRIT-CT-21-037, Research Project Title: Cyber ​​Battlefield Management Artificial Intelligence Model Security Technology, Project Management Agency: Defense Technology Advancement Research Institute, Project Execution Agency: Soongsil University Industry-Academic Cooperation Foundation, Research Period: 2021.12.24. ~ 2026.12.23.). Meanwhile, the Korean government has no property interest in any aspect of the present invention.

[0003] A representative security vulnerability in deep reinforcement learning models is poisoning attacks. Poisoning attacks against deep reinforcement learning models can also be applied to multi-agent reinforcement learning models. Therefore, research on poisoning attacks against multi-agent reinforcement learning models is actively underway.

[0004] Reinforcement learning models are vulnerable to attacks that poison one or more of three factors: observations (the model's inputs), actions (the model's outputs), and rewards (the model's policy training). Because existing techniques focus on observation or reward poisoning attacks, they fail to consider the risk of action poisoning attacks, where the training of other agents is disrupted by the contaminated actions of other agents in multi-agent reinforcement learning models.

[0005] The present invention provides an action poisoning attack system capable of evaluating the safety of a multi-agent deep reinforcement learning-based autonomous driving model through a locality-based action poisoning attack, and a control method of the action poisoning attack system.

[0006] In addition, the present invention provides an action taint attack system and a control method of the action taint attack system that enable an attacker to perform an appropriate attack even in a situation where an attacker only has black box access rights to a multi-agent reinforcement learning-based autonomous driving model with a continuous action space.

[0007] In addition, the present invention provides an action contamination attack system and a control method of the action contamination attack system that can interfere with the training of an autonomous driving model (victim model) through a target action and cause it to converge to a non-optimal policy.

[0008] In addition, the present invention provides an action contamination attack system and a control method of the action contamination attack system, which enable developers of autonomous driving models to test the safety of autonomous driving models by testing action manipulation attacks during the training phase and to develop defense techniques against such attacks.

[0009] An action contamination attack system for an autonomous driving model learned based on the actions of each agent that determines the movement of each agent driving virtually in a virtual space according to an aspect of the disclosed invention includes: a target agent determination unit configured to determine, based on location information of the agents in the virtual space, a target agent that is a target of attack and that is intended to perform virtual driving based on manipulated action information rather than action information output by the autonomous driving model, among a plurality of agents; and a target action determination unit configured to generate target action information by manipulating the action information output by the autonomous driving model for the target agent, and to cause the target agent to perform a target action based on the target action information, thereby interfering with the learning of the autonomous driving model; wherein the autonomous driving model may include a machine learning model that learns by a machine learning method based on the actions of the agents while determining the action information of each agent based on the locations and actions of other agents in the virtual space.

[0010] In addition, the autonomous driving model may be configured to generate an action vector including a steering component related to steering of each agent and an acceleration component related to acceleration and deceleration of each agent as action information for each agent, and the target action determination unit may be configured to generate a target action vector including a manipulation steering component and a manipulation acceleration component as target action information for the target agent by changing at least one of the steering component and the acceleration component of the action vector output by the autonomous driving model.

[0011] In addition, the target action determination unit may be configured to: generate a target action vector including a manipulation steering component to cause the target agent to virtually drive in a direction opposite to the direction in which it would have steered based on an action by an action vector output by the autonomous driving model; and generate a target action vector including a manipulation acceleration component to cause the target agent to virtually drive in a direction opposite to the direction in which it would have accelerated and decelerated based on an action by an action vector output by the autonomous driving model.

[0012] In addition, the target agent determination unit may be configured to: determine, for each agent, the number of nearby agents, which are other agents located within a preset reference distance from each agent; and, among the plurality of agents, determine an agent whose number of nearby agents is greater than or equal to the preset number as a target agent.

[0013] In addition, the target action determination unit may be configured to: determine an average value of steering components of action vectors output by the autonomous driving model to the proximate agents as an average steering component; determine an average value of acceleration components of action vectors output by the autonomous driving model to the proximate agents as an average acceleration component; generate a target action vector including a manipulation steering component to virtually drive in a direction opposite to a direction in which steering would have been performed based on an action by the action vector including the average steering component; and generate a target action vector including a manipulation acceleration component to virtually drive in the opposite direction at a speed in which acceleration and deceleration would have been performed based on an action by the action vector including the average acceleration component.

[0014] In addition, the target action determination unit may be configured to: determine a weighted average value of the speeds of the adjacent agents driving virtually based on an action by an action vector output by the autonomous driving model as a weighted average speed; determine a similarity between the speed of the target agent driving virtually based on an action by an action vector output by the autonomous driving model and the weighted average speed; and determine the operation acceleration component based on the similarity between the speed of the target agent and the weighted average speed.

[0015] In addition, the target action determination unit may be configured to determine an operation acceleration component that causes virtual driving while accelerating more as the similarity between the speed of the target agent and the weighted average speed decreases.

[0016] In addition, the target action determination unit may be configured to determine a manipulation steering component intended to cause virtual driving by changing the virtual driving direction more significantly as the similarity between the speed of the target agent and the weighted average speed increases.

[0017] In addition, the target action determination unit may be configured to: determine one error randomly selected from among preset errors as the manipulation steering component; and determine one error randomly selected from among preset errors as the manipulation steering component.

[0018] A method for controlling an action contamination attack system for an autonomous driving model learned based on the actions of each agent that determines the movement of each agent driving virtually in a virtual space according to an aspect of the disclosed invention comprises the steps of: determining, by a target agent determination unit, a target agent that is an attack target for causing a plurality of agents to perform virtual driving based on manipulated action information rather than action information output by the autonomous driving model, based on location information of the agents in the virtual space; generating target action information by manipulating the action information output by the autonomous driving model for the target agent; and interfering with learning of the autonomous driving model by causing the target agent to perform a target action based on the target action information, by the target action determination unit, wherein the autonomous driving model: is learned by a machine learning method based on the actions of the agents while determining the action information of each agent based on the locations and actions of other agents in the virtual space; And a machine learning model configured to generate an action vector including a steering component related to the steering of each agent and an acceleration component related to the acceleration and deceleration of each agent as action information for each agent, wherein the step of determining the target agent comprises: a step of determining, by the target agent determination unit, the number of nearby agents, which are other agents located within a preset reference distance from each agent, for each agent;And, by the target agent determination unit, a step of determining, among the plurality of agents, an agent whose number of proximity agents is greater than or equal to a preset number as a target agent, and the step of generating the target action information may include a step of generating, by the target action determination unit, a target action vector including a manipulation steering component and a manipulation acceleration component by changing at least one or more of the steering component and the acceleration component of the action vector output by the autonomous driving model as target action information for the target agent.;

[0019] A computer-readable non-transitory recording medium according to one aspect of the disclosed invention can store a computer-readable computer program to execute a control method of an action contamination attack system for an autonomous driving model learned based on the actions of each agent that determine the movement of each agent driving virtually in a virtual space.

[0020] According to one aspect of the disclosed invention, the safety of a multi-agent deep reinforcement learning-based autonomous driving model can be evaluated through a locality-based action poisoning attack.

[0021] Furthermore, according to an embodiment of the present invention, an attacker can perform an appropriate attack even in a situation where he or she only has black box access to a multi-agent reinforcement learning-based autonomous driving model with a continuous action space.

[0022] Additionally, according to an embodiment of the present invention, training of an autonomous driving model (victim model) can be interrupted through a target action, causing convergence to a non-optimal policy.

[0023] In addition, according to an embodiment of the present invention, a developer of an autonomous driving model can test the safety of an autonomous driving model by testing an action manipulation attack during the training phase, and can prepare a defense technology against the attack.

[0024] Figure 1 is a configuration diagram of an action contamination attack system according to one embodiment.

[0025] FIG. 2 is a diagram illustrating a method for performing an action contamination attack on a target agent according to one embodiment.

[0026] FIG. 3 is a diagram illustrating a method for generating an action vector for a target agent according to one embodiment.

[0027] Figure 4 is a flowchart of an action contamination attack method according to one embodiment.

[0028] FIG. 5 is a diagram illustrating an experiment to verify an action contamination attack method on an intersection map according to one embodiment.

[0029] FIG. 6 is a diagram illustrating an experiment to verify an action contamination attack method on a roundabout map according to one embodiment.

[0030] FIG. 7 is a diagram illustrating an experiment to verify an action contamination attack method on a bottleneck road map according to one embodiment.

[0031] FIG. 8 is a graph illustrating the performance of an action contamination attack method on an intersection map according to one embodiment.

[0032] FIG. 9 is a graph illustrating the performance of an action contamination attack method in a roundabout map according to one embodiment.

[0033] FIG. 10 is a graph illustrating the performance of an action pollution attack method on a bottleneck road map according to one embodiment.

[0034] Figure 11 is a table illustrating the performance of an action contamination attack method according to one embodiment.

[0035] Throughout the specification, the same reference numerals denote the same components. This specification does not describe all elements of the embodiments, and any content that is general in the technical field to which the disclosed invention belongs or that overlaps between the embodiments is omitted. The term "~unit" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "~units" may be implemented as a single component, or a single "~unit" may include multiple components.

[0036] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0037] The term "unit" used in this specification refers to a unit that processes at least one function or operation, and may refer to, for example, software, an FPGA, or a hardware component. The function provided by the "unit" may be performed separately by multiple components, or may be integrated with other additional components. The "unit" in this specification is not necessarily limited to software or hardware, and may be configured to be located on an addressable storage medium, or may be configured to play back one or more processors.

[0038] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0039] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0040] The operating principle and embodiments of the disclosed invention are described below with reference to the attached drawings.

[0041] Figure 1 is a configuration diagram of an attack detection system according to one embodiment.

[0042] Referring to FIG. 1, an action contamination attack system (100) according to an embodiment of the present invention may include a target agent determination unit (110) and a target action determination unit (120). The action contamination attack system (100) may be provided in an attacker's terminal or server, but the location of the action contamination attack system (100) is not limited thereto.

[0043] The action contamination attack system (100) can interfere with the learning of an autonomous driving model (300) that is trained based on agents (200) driving virtually in a virtual space. Each agent (200) may be an object representing a virtual vehicle driving in a virtual space where a road is depicted. However, the object represented by the agent (200) is not necessarily limited to a vehicle.

[0044] Machine learning utilizes models composed of multiple parameters and can mean optimizing those parameters based on given data. Depending on the type of learning problem, machine learning can include supervised learning, unsupervised learning, and reinforcement learning. Supervised learning learns mappings between inputs and outputs and is applicable when input-output pairs are given as data. Unsupervised learning is applicable when there are only inputs and no outputs, and can identify patterns between inputs, etc.

[0045] The autonomous driving model (300) may be a machine learning model that learns based on the actions of each agent (200) that determines the movement of each agent (200) driving virtually in a virtual space. Specifically, the autonomous driving model (300) may be a machine learning model that learns based on the actions of the agents (200) using a deep reinforcement learning method while determining the action information of each agent (200) based on the positions and actions of other agents (200) in the virtual space.

[0046] Deep reinforcement learning used in training an autonomous driving model (300) may be an unsupervised learning model that trains policies, which are the behavioral patterns of agents (200) by interacting with a defined environment. At this time, each agent (200) takes an action appropriate for observations obtained from the environment, receives a reward for the action, and can select the next action. The autonomous driving model (300) may be an unsupervised learning model in which a policy is trained to maximize the expected value of the reward that the agent (200) can obtain using observation, action, and reward data obtained through interaction.

[0047] The learning method of the autonomous driving model (300) may be Multi-Agent Reinforcement Learning (MARL). This may be a learning method that trains multiple agents (200) to interact with the environment and perform a target task. This can be applied to swarm robots and drones, and autonomous driving technology, and can be used in learning where security is important, such as controlling swarm agents (200). For example, the learning method of the autonomous driving model (300) may be a MARL method that is characterized by important interaction between agents (200) and convergence instability, such as a POMDP (Partially Observable Markov Decision Process) that considers the possibility of partial observability, where each agent (200) can only observe a certain distance around them and not all agents (200) can know the entire situation.

[0048] Action information may include information on acceleration, deceleration, and steering, which serve as the basis for determining the movement of an agent (200) driving virtually in a virtual space. Each agent (200) can drive in the virtual space based on the action information data at each moment. This action information data may have different values ​​at each moment.

[0049] The action contamination attack system (100) can interfere with the learning of the autonomous driving model (300) by performing action contamination in a way that manipulates the action information that determines the movement of the agent (200) for the autonomous driving model (300).

[0050] The attack method of the action contamination attack system (100) may be an action poisoning attack. An action poisoning attack may be an attack method in which an attacker between the environment and the agent (200) manipulates the action of an agent (200) being trained, thereby disrupting policy training and causing convergence to a non-optimal policy. The action poisoning attack method may also be applied to a multi-agent reinforcement learning model. Since multi-agent reinforcement learning has multiple agents (200) sharing the same environment, when an attacker manipulates the action information of some agents (200), the observation values ​​of other agents (200) may also be disturbed, thereby disrupting convergence to an optimal policy.

[0051] The target agent decision unit (110) can receive location information of the agents (200) and action information of the agents (200) generated by the autonomous driving model (300).

[0052] The target agent determination unit (110) can determine, based on the location information of the agents (200) in the virtual space, the target agent (201) that is the target of an attack and that is intended to perform virtual driving based on manipulated action information rather than action information output by the autonomous driving model (300) among a plurality of agents (200). The target agent determination unit (110) can transmit information about the target agent (201) to the target action determination unit (120).

[0053] The target agent determination unit (110) can generate target action information by manipulating the action information output by the autonomous driving model (300) for the target agent (201). The target agent determination unit (110) can interfere with the learning of the autonomous driving model (300) by causing the target agent (201) to perform a target action based on the target action information.

[0054] FIG. 2 is a diagram illustrating a method for performing an action contamination attack on a target agent according to one embodiment.

[0055] Referring to Figure 2, the process of an action poisoning attack in a multi-agent reinforcement learning environment can be confirmed. In this case, the proposed action poisoning attack in a multi-agent reinforcement learning environment can assume a situation in which an attacker has only more limited black box access rights, as it does not require manipulation of the agent's (200) observation and reward functions.

[0056] The action contamination attack system (100) can manipulate the action of one of several target agents (201) with the authority of the attacker. At this time, black box access may be possible to request only the action information of the adjacent agent (202), which is a neighboring agent (200) during the observation of the target agent (201). At this time, under certain conditions, the attacker can manipulate the action (ai) of the target agent (201) into a tainted target action (ai*). For example, the attack may be performed when the number of adjacent adjacent agents (202) within a preset lidar radius is 4 or more, but the number of adjacent agents (202) that serve as a reference is not limited thereto. Due to such an attack, the adjacent agent (202) may be disturbed by the action and reward of the target agent (201), and the policy being trained may be prevented from being trained as an optimal policy.

[0057] When the actions of the agent (200) are discrete, the attacker can manipulate the agent (200) to select a non-optimal action. However, when the actions of the agent (200) are in a continuous space, such as in the autonomous driving model (300), there is a problem that it is difficult to determine an appropriate target action. The action poisoning attack system (100) can provide various target actions that the attacker can select in an action poisoning attack for safety evaluation of an autonomous driving model based on multi-agent reinforcement learning. From the attacker's perspective, since the target action is selected while having access to the black box, an action that is likely to interfere with the stable driving of the autonomous driving model must be set as the target action even without prior information.

[0058] The action contamination attack system (100) can provide target actions that violate the three principles of Reynolds' flocking algorithm, which simulates flocking of birds. The three principles described in Reynolds' flocking algorithm are cohesion, separation, and alignment, respectively. This theory states that individuals within a flock can safely form a flock by maintaining a sufficiently small distance from other individuals while maintaining a safe distance to avoid collisions and moving at a similar speed to adjacent individuals.

[0059] FIG. 3 is a diagram illustrating a method for generating an action vector for a target agent according to one embodiment.

[0060] Referring to FIGS. 1, 2, and 3, the target agent determination unit (110) can determine, for each agent (200), the number of nearby agents (202) located within a preset reference distance from each agent (200). The reference distance may be a distance in a virtual space corresponding to a distance at which a surrounding vehicle can be detected from an actual rider sensor's perspective.

[0061] The target agent decision unit (110) can determine, among a plurality of agents (200), an agent (200) having a number of proximity agents (202) greater than or equal to a preset number as the target agent (201).

[0062] For example, if the preset number is 4, the target agent determination unit (110) can determine an agent (200) with a number of other agents (200) located within a radius within the reference distance of 4 as the target agent (201).

[0063] The autonomous driving model (300) can generate an action vector as action information for each agent (200). The action vector can be a vector including a steering component related to the steering of each agent (200) and an acceleration component related to the acceleration and deceleration of each agent (200). The steering component can be any real number between -1 and 1. At this time, if the steering component is -1, the agent (200) that virtually drives based on the action vector including the steering component can perform a direction change by turning the steering wheel or the front tires completely to the left, and if the steering component is 1, the agent (200) that virtually drives based on the action vector including the steering component can perform a direction change by turning the steering wheel or the front tires completely to the right, but is not limited to this method. In addition, if the acceleration component is -1, an agent (200) driving virtually based on an action vector including the acceleration component can perform virtual driving by accelerating as much as possible in the backward direction, and if the steering component is 1, an agent (200) driving virtually based on an action vector including the acceleration component can perform virtual driving by accelerating as much as possible in the forward direction, but is not limited to this method.

[0064] The target action decision unit (120) can change at least one of the steering component and the acceleration component of the action vector output by the autonomous driving model (300) to generate a target action vector including the manipulation steering component and the manipulation acceleration component as target action information for the target agent (201). At this time, the target agent (201) can perform virtual driving of steering or acceleration / deceleration corresponding to the component value of the target action vector.

[0065] At this time, the method for generating the manipulation steering component and manipulation acceleration component for a target agent (201) may be one of the methods described below.

[0066] The target action decision unit (120) can generate a target action vector that includes a manipulation steering component intended to cause the target agent (201) to virtually drive in a direction opposite to the direction in which it would have steered, based on an action by an action vector output by the autonomous driving model (300).

[0067] The target action decision unit (120) can generate a target action vector that includes an operation acceleration component to virtually drive in the opposite direction at the speed at which the target agent (201) accelerated and decelerated based on the action by the action vector output by the autonomous driving model (300).

[0068] Meanwhile, according to the attack using the Anti-correlated Action method, the target agent (201) selected by the attacker can calculate the average steering and average acceleration of all the nearby agents (202) detected in the LiDAR sensor, and then use the average steering and average acceleration multiplied by -1 as the target action. At this time, the target agent (i target) (201) can perform an action opposite to the average action of the neighboring nearby agents ({i1,i2,...,iN}). Meanwhile, each action can be a vector (a real number between -1 and 1) composed of [steering i, acceleration i]. At this time, the steering and acceleration actions of all agents (200) can be real numbers between -1 and 1. The average action of the neighboring agents When expressed as , the action of the target agent (201) is can be expressed as

[0069] The target action determination unit (120) can determine the average value of the steering components of the action vector output by the autonomous driving model (300) to the proximity agents (202) as the average steering component. The target action determination unit (120) can determine the average value of the acceleration components of the action vector output by the autonomous driving model (300) to the proximity agents (202) as the average acceleration component.

[0070] The target action determination unit (120) can generate a target action vector including a manipulation steering component for the purpose of driving in a virtual direction opposite to the direction in which the vehicle would have been steered based on an action by an action vector including an average steering component. The target action determination unit (120) can generate a target action vector including a manipulation acceleration component for the purpose of driving in a virtual direction opposite to the speed in which the vehicle would have been accelerated and decelerated based on an action by an action vector including an average acceleration component.

[0071] Meanwhile, according to the Human-like Disruptive Action attack, the attack can be performed by reflecting the unstable actions of a human driver, such as the movements of a driver who suddenly accelerates rapidly or changes lanes abruptly. The target agent (201) selected by the attacker can calculate the weighted sum (v neighbor) of the average speeds of neighboring agents detected in the LiDAR sensor, and then calculate the cosine similarity with the speed (v target) of the target agent (201) as in [Equation 1] below. At this time, the result value can be the similarity between the speed of the target agent (201) and the weighted average speed of the speeds of nearby agents (202).

[0072]

[0073] [Equation 1]

[0074]

[0075]

[0076] The actions of the target agent (201), steering and acceleration, can be manipulated by the manipulation steering component and manipulation acceleration component calculated as in [Equation 2] below.

[0077]

[0078] [Equation 2]

[0079]

[0080]

[0081] This may reflect the fact that the unstable actions of human drivers are difficult for machine learning models to interpret and predict. That is, the target action can be set based on the cosine similarity of the speed (v target) of the target agent (i target) (201) and the weighted average (v neighbor) of the speeds of the nearby agents ({i1,i2,...,iN}) (202).

[0082] A situation with low cosine similarity may be one in which the target agent (201) accelerates significantly on its own, unlike the adjacent agent (202), such as when starting from a stationary position and rapidly accelerating or changing lanes. In this case, by manipulating the action to accelerate more significantly for the target agent (201) with low cosine similarity, the action can be more reliably contaminated.

[0083] On the other hand, if the action is manipulated to make a large change in direction for a target agent (201) with low cosine similarity, it may actually perform appropriate driving for nearby vehicles, and if the action is manipulated to make a larger change in direction for a target agent (201) with high cosine similarity, the action can be more reliably contaminated.

[0084] The target action determination unit (120) can determine the weighted average value of the speeds of the virtual driving proximity agents (202) based on the action by the action vector output by the autonomous driving model (300), as the weighted average speed. The target action determination unit (120) can determine the similarity between the speed of the virtual driving target agent (201) based on the action by the action vector output by the autonomous driving model (300) and the weighted average speed.

[0085] The target action determination unit (120) can determine the manipulation acceleration component based on the similarity between the speed of the target agent (201) and the weighted average speed. The target action determination unit (120) can determine the manipulation acceleration component with the intention of performing virtual driving with greater acceleration as the similarity between the speed of the target agent (201) and the weighted average speed is lower. The target action determination unit (120) can determine the manipulation steering component with the intention of performing virtual driving with greater change in the direction of virtual driving as the similarity between the speed of the target agent (201) and the weighted average speed is higher.

[0086] Meanwhile, according to the Random Action method of attack, any real number value can be used as the action of the target agent (201). For example, the range between -1 and 1, which is the range of steering and acceleration, which are the actions of the target agent (201), can be evenly divided into 500 points, and two pairs of real numbers randomly selected from among them can be used as the target action of the target agent (201). However, the range and number of real numbers that can be selected as the random real number value are not limited thereto.

[0087] The target action decision unit (120) can determine one real number randomly selected from among the preset real numbers as the manipulation steering component. The target action decision unit (120) can determine one real number randomly selected from among the preset real numbers as the manipulation steering component.

[0088] According to the attack methods described above, an attacker using an action contamination attack system (100) can manipulate the target model to take one of the target actions defined by the method described above when a preset number (e.g., 4) or more neighboring agents are detected within the LiDAR sensor radius of a randomly selected target agent (201) during the training process. By this manipulation, adversarial action data is added to the observations of nearby agents (202) of the target agent (201), and it may become difficult for the policy of the autonomous driving model (300) to converge to an optimal policy. In other words, when the attack is performed, the road passing rate of the agent (200) in the autonomous driving multi-agent reinforcement learning model, which is the target autonomous driving model (300), may decrease, and the collision and road departure accident rates may increase.

[0089] At least one component may be added or deleted in response to the performance of the components described above. Furthermore, those skilled in the art will readily understand that the relative positions of the components may be altered in response to the performance or structure of the system.

[0090] Figure 4 is a flowchart of an action contamination attack method according to one embodiment. This is merely a preferred embodiment for achieving the purpose of the present invention, and it is understood that certain components may be added or deleted as needed.

[0091] Referring to FIG. 4, the autonomous driving model (300) can determine action information of each agent (200) based on the location and actions of other agents (200) in the virtual space (1001).

[0092] The target agent determination unit (110) can determine the number of nearby agents (202), which are other agents (200) located within a preset standard distance from each agent (200), for each agent (200) (1002).

[0093] The target agent decision unit (110) can determine, among a plurality of agents (200), an agent (200) having a number of proximity agents (202) greater than or equal to a preset number as the target agent (201) (1003).

[0094] The target action decision unit (120) can calculate the steering component and the acceleration component by changing at least one of the steering component and the acceleration component of the action vector output by the autonomous driving model (300), and generate a target action vector including these as target action information for the target agent (201) (1004).

[0095] The target agent (201) may interfere with the learning of the autonomous driving model (300) by being controlled to perform a target action, which is an action based on target action information (1005).

[0096] The target agent determination unit (110) and the target action determination unit (120) may include any one of a plurality of processors included in the action contamination attack system (100). In addition, the action contamination attack method according to the embodiments of the present invention described so far and the embodiments to be described hereinbefore may be implemented in the form of a program that can be run by a processor.

[0097] Here, the program may include program commands, data files, and data structures, either singly or in combination. The program may be designed and produced using machine language code or high-level language code. The program may be specifically designed to implement the method for the above-described action contamination attack, or may be implemented using various functions or definitions that are known and available to those skilled in the art of computer software. The program for implementing the above-described action contamination attack method may be recorded on a recording medium readable by a processor. In this case, the recording medium may be a memory.

[0098] The memory can store a program that performs the operations described above and the operations described below, and the memory can execute the stored program. In the case where there are multiple processors and memories, they can be integrated into a single chip or provided in physically separate locations. The memory can include volatile memory such as static random access memory (S-RAM) and dynamic random access memory (DRAM) for temporarily storing data. In addition, the memory can include non-volatile memory such as read only memory (ROM), erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM) for long-term storage of control programs and control data.

[0099] The processor may include various logic circuits and arithmetic circuits, process data according to a program provided from memory, and generate control signals according to the processing results.

[0100] In order to verify the performance of the action contamination attack method according to an embodiment of the present invention, an experiment was conducted in which an attack was applied to the learning process of the continuous control system of an autonomous driving model (300) that is performed in a virtual space.

[0101] FIG. 5 is a diagram for explaining an experiment for verifying an action contamination attack method in an intersection map according to one embodiment, FIG. 6 is a diagram for explaining an experiment for verifying an action contamination attack method in a roundabout map according to one embodiment, and FIG. 7 is a diagram for explaining an experiment for verifying an action contamination attack method in a bottleneck road map according to one embodiment.

[0102] Referring to Figures 5, 6, and 7, experiments were conducted on three victim models based on Proximal Policy Optimization (PPO) for the intersection map, roundabout map, and bottleneck road map of the Metadrive simulator. Proximal Policy Optimization (PPO) can be a model that applies information on which actions are relatively good or bad to the objective function by introducing an advantage function. At this time, a clipping hyperparameter can be applied to prevent the policy update from being too abrupt. Modified IPPO (Independent PPO) is an algorithm that extends PPO to MARL. Since it only considers the reward of the agent itself, it can be a model that targets an algorithm with a modified objective function to maximize the sum of local rewards. Mean-Field PPO (MFPO) can be a model that applies Mean-Field MARL and evaluates the value of the average state of neighboring agents (centralized critic). CoPO (Coordinated PO) introduces a Local Coordination Factor (LCF) that determines the weighting of local and global rewards. Through joint training, the optimal LCF is determined. A higher LCF suggests compromise, while a lower LCF may reflect selfish behavior (such as interfering). All of the aforementioned models share the commonality that each agent uses information from neighboring agents during training.

[0103] FIG. 8 is a graph for explaining the performance of an action contamination attack method in an intersection map according to one embodiment, FIG. 9 is a graph for explaining the performance of an action contamination attack method in a roundabout map according to one embodiment, FIG. 10 is a graph for explaining the performance of an action contamination attack method in a bottleneck road map according to one embodiment, and FIG. 11 is a table for explaining the performance of an action contamination attack method according to one embodiment.

[0104] Referring to FIGS. 8, 9, 10 and 11, the performance of an attack detection method according to one embodiment can be confirmed through experimental results.

[0105] In each experiment, the evaluation metrics are success rate, crash rate, and out rate. The success rate is the number of agents that pass the map without crashing or falling out divided by the total number of agents. The crash rate is the number of agents that die due to collisions with other agents divided by the total number of agents. The out rate is the number of agents that die after leaving the road divided by the total number of agents.

[0106] Referring to Figures 8, 9, and 10, success rate graphs for each time step for the intersection, roundabout, and bottleneck experiments, respectively, can be seen. Here, compared to the unattacked experiment (victim model), the success rate is significantly lower when attacks (anti-correlated, disruptive, and random) are applied, and the collision and departure rates are significantly higher when the attack is applied.

[0107] Referring to Figure 11, it can be seen that all three actions, Disruptive, Anti-correlated, and Random, were successful in the attack, and the bottleneck road was the most vulnerable to the attack and was affected the most by the attack compared to other target models.

[0108] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. In an action contamination attack system for an autonomous driving model learned based on the actions of each agent that determine the movement of each agent driving virtually in a virtual space, A target agent determination unit configured to determine a target agent, which is an attack target, among a plurality of agents, based on the location information of the agents in the virtual space, and which is intended to perform virtual driving using manipulated action information rather than action information output by the autonomous driving model; and A target action determination unit is configured to generate target action information by manipulating the action information output by the autonomous driving model for the target agent, and to cause the target agent to perform a target action, which is an action based on the target action information, thereby interfering with the learning of the autonomous driving model. The above autonomous driving model is, An action contamination attack system, which is a machine learning model that learns based on the actions of the agents using a machine learning method, while determining the action information of each agent based on the location and actions of other agents in the virtual space.

2. In paragraph 1, The above autonomous driving model is, It is configured to generate an action vector including a steering component related to the steering of each agent and an acceleration component related to the acceleration and deceleration of each agent as action information for each agent. The above target action decision unit is, An action contamination attack system configured to generate a target action vector including a manipulation steering component and a manipulation acceleration component as target action information for the target agent by changing at least one of the steering component and the acceleration component of the action vector output by the autonomous driving model.

3. In paragraph 2, The above target action decision unit: Generate a target action vector including a manipulation steering component intended to drive virtually in a direction opposite to the direction in which the target agent would have steered based on an action by an action vector output by the autonomous driving model; and An action contamination attack system configured to generate a target action vector including an acceleration component for driving the target agent in a virtual direction at a speed at which the target agent would have accelerated or decelerated based on an action by an action vector output by the autonomous driving model.

4. In paragraph 2, The above target agent determination unit: The number of nearby agents, which are other agents located within a preset standard distance from each agent, is determined for each agent; and An action contamination attack system configured to determine, among a plurality of above agents, an agent having a number of adjacent agents greater than or equal to a preset number as a target agent.

5. In paragraph 4, The above target action decision unit: The autonomous driving model determines the average value of the steering components of the action vectors output to the proximity agents as the average steering component; The autonomous driving model determines the average value of the acceleration components of the action vector output for the proximity agents as the average acceleration component; Generating a target action vector including a manipulation steering component intended to drive in a virtual direction opposite to the direction in which the steering would have been made based on an action by an action vector including the above average steering component; and An action contamination attack system configured to generate a target action vector including a manipulation acceleration component intended to cause virtual driving in the opposite direction at a speed that would have been accelerated and decelerated based on an action by an action vector including the above average acceleration component.

6. In paragraph 4, The above target action decision unit: The weighted average value of the speeds of the adjacent agents driving virtually based on the action by the action vector output by the autonomous driving model is determined as the weighted average speed; Determine the similarity between the speed of the target agent driving virtually based on the action by the action vector output by the autonomous driving model and the weighted average speed; and An action contamination attack system configured to determine the operation acceleration component based on the similarity between the velocity of the target agent and the weighted average velocity.

7. In paragraph 6, The above target action decision unit: An action contamination attack system configured to determine an acceleration component of the operation to cause virtual driving while accelerating more significantly as the similarity between the speed of the target agent and the weighted average speed decreases.

8. In paragraph 7, The above target action decision unit: An action contamination attack system configured to determine a manipulation steering component intended to cause virtual driving by changing the virtual driving direction more significantly as the similarity between the speed of the target agent and the weighted average speed increases.

9. In paragraph 4, The above target action decision unit: determining one error selected at random from among the preset errors as the steering component of the operation; and An action contamination attack system configured to determine a randomly selected error from among a preset number of errors as the manipulation steering component.

10. A method of operation of an action contamination attack system for an autonomous driving model learned based on the actions of each agent that determine the movement of each agent driving virtually in a virtual space. A step of determining, by a target agent determination unit, a target agent that is a target of an attack and is intended to perform virtual driving using manipulated action information rather than action information output by the autonomous driving model, among a plurality of agents, based on location information of the agents in the virtual space; A step of generating target action information by manipulating the action information output by the autonomous driving model for the target agent by the target action decision unit; and By the target action determination unit, a step is included to prevent learning of the autonomous driving model by causing the target agent to perform a target action, which is an action based on the target action information. The above autonomous driving model: In the virtual space, the action information of each agent is determined based on the location and actions of other agents, and the actions of the agents are learned in a machine learning manner; and A machine learning model configured to generate an action vector including a steering component related to the steering of each agent and an acceleration component related to the acceleration and deceleration of each agent as action information for each agent. The steps for determining the above target agent are: A step of determining, for each agent, the number of nearby agents, which are other agents located within a preset standard distance from each agent, by the target agent determination unit; and A step of determining, by the target agent determination unit, an agent among the plurality of agents, in which the number of adjacent agents is greater than or equal to a preset number, as a target agent, The step of generating the above target action information is: A control method for an action contamination attack system, comprising a step of generating a target action vector including a manipulation steering component and a manipulation acceleration component as target action information for the target agent by changing at least one of a steering component and an acceleration component of an action vector output by the autonomous driving model by the target action determination unit.

11. A non-transitory recording medium storing a computer-readable computer program for executing the control method of the action contamination attack system of Article 10.

Images