A sample acquisition method, apparatus, device and computer readable storage medium

Abstract

This application provides a method, apparatus, device, and computer-readable storage medium for obtaining adversarial examples. The method includes: obtaining the i-th clean state corresponding to the i-th time step, where i is a positive integer; adversarially perturbing the i-th clean state based on the cumulative reward to obtain the i-th perturbed state, where the cumulative reward of the i-th perturbed state is minimized; making a decision based on the i-th perturbed state to obtain the i-th decision result; when the i-th decision result does not meet the termination condition, determining the (i+1)-th clean state based on the i-th clean state and the i-th decision result, and obtaining the (i+1)-th decision result corresponding to the (i+1)-th clean state, until the n-th decision result meets the termination condition, thereby obtaining an adversarial example including n perturbed states, where n is a positive integer greater than 1. This application improves the feasibility of obtaining adversarial examples in the field of artificial intelligence.

Description

Technical Field

[0001] This application relates to adversarial attack techniques in the field of artificial intelligence, and more particularly to a sample acquisition method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, Deep Reinforcement Learning (DRL) has been widely used in various fields (such as autonomous driving and intelligent transportation). However, DRL is susceptible to adversarial examples. Therefore, obtaining adversarial examples, using them to evaluate DRL models, or using them as training samples to improve DRL robustness has become increasingly important.

[0003] Generally, to obtain adversarial examples for DRL, gradient acquisition methods are typically used, such as the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). However, in the process of obtaining adversarial examples by acquiring gradients, the structure and parameters of the agent are not always available, resulting in low feasibility of obtaining adversarial examples. Summary of the Invention

[0004] This application provides a sample acquisition method, apparatus, device, and computer-readable storage medium, which can improve the feasibility of acquiring adversarial examples.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a sample acquisition method, including:

[0007] Obtain the i-th clean state corresponding to the i-th time step, where i is a positive integer;

[0008] The i-th clean state is subjected to adversarial perturbation based on the cumulative reward to obtain the i-th perturbed state, wherein the cumulative reward of the i-th perturbed state is the minimum.

[0009] A decision is made based on the i-th disturbance state to obtain the i-th decision result;

[0010] When the i-th decision result does not meet the termination condition, the (i+1)-th clean state is determined based on the i-th clean state and the i-th decision result, and the (i+1)-th decision result corresponding to the (i+1)-th clean state is obtained, until the n-th decision result obtained meets the termination condition, thus obtaining an adversarial sample including n perturbation states, where n is a positive integer greater than 1.

[0011] This application provides a sample acquisition device, including:

[0012] The state acquisition module is used to acquire the i-th clean state corresponding to the i-th time step, where i is a positive integer;

[0013] The state perturbation module is used to perform adversarial perturbation on the i-th clean state based on the cumulative reward to obtain the i-th perturbed state, wherein the cumulative reward of the i-th perturbed state is the minimum.

[0014] The state decision module is used to make a decision based on the i-th disturbance state and obtain the i-th decision result;

[0015] The sample acquisition module is used to determine the (i+1)th clean state based on the i-th clean state and the i-th decision result when the i-th decision result does not meet the termination condition, and to acquire the (i+1)th decision result corresponding to the (i+1)th clean state, until the n-th decision result satisfies the termination condition, thereby obtaining an adversarial sample including n perturbation states, where n is a positive integer greater than 1.

[0016] In this embodiment of the application, the state perturbation module is further configured to: employ the original adversarial perturbation model to perform adversarial perturbation on the j-th clean state sample corresponding to the j-th time step to obtain the j-th estimated perturbation state, where j is a positive integer; obtain the cumulative decision reward corresponding to the j-th estimated perturbation state; train the original adversarial perturbation model based on the cumulative decision reward to obtain the adversarial perturbation model; and perform adversarial perturbation on the i-th clean state through the adversarial perturbation model and based on the cumulative reward to obtain the i-th perturbation state.

[0017] In this embodiment of the application, the state perturbation module is further configured to: make a decision based on the j-th clean state sample to obtain the j-th decision result label; make a decision based on the j-th estimated perturbation state to obtain the j-th estimated decision result; obtain a misjudgment loss based on the distribution difference between the j-th clean state sample and the j-th estimated perturbation state; obtain an adversarial loss based on the difference between the j-th decision result label and the j-th estimated decision result; and obtain a hinge loss based on the j-th estimated perturbation state.

[0018] In this embodiment of the application, the state perturbation module is further used to train the original adversarial perturbation model by combining the adversarial loss, the misjudgment loss, the hinge loss and the cumulative decision reward, so as to obtain the adversarial perturbation model.

[0019] In this embodiment, the state perturbation module is further configured to: perform adversarial perturbation on the kth clean state sample corresponding to the kth time step to obtain the kth estimated perturbation state, where k is a positive integer; make a decision on the kth estimated perturbation state to obtain the kth estimated decision reward corresponding to the kth estimated decision result; use the original cumulative reward model to obtain the kth estimated cumulative reward corresponding to the kth estimated perturbation state; based on the kth clean state sample and the kth estimated decision result, determine the (k+1)th estimated perturbation state corresponding to the (k+1)th clean state sample, and use the original cumulative reward model to obtain the (k+1)th estimated cumulative reward corresponding to the (k+1)th estimated perturbation state; train the original cumulative reward model based on the kth estimated decision reward, the kth estimated cumulative reward, and the (k+1)th estimated cumulative reward to obtain the cumulative reward model; and obtain the cumulative decision reward corresponding to the jth estimated perturbation state through the cumulative reward model.

[0020] In this embodiment of the application, the state perturbation module is further configured to obtain the label distribution corresponding to the j-th clean state sample; obtain the estimated distribution corresponding to the j-th estimated perturbation state; obtain the sampling state between the j-th clean state sample and the j-th estimated perturbation state, and obtain the sampling distribution corresponding to the sampling state; and determine the initial distribution difference between the label distribution and the estimated distribution, and the distribution difference between the sampling distribution, as the misjudgment loss.

[0021] In this embodiment of the application, the state perturbation module is further configured to acquire state discrimination samples and state discrimination labels; use the original discrimination model to acquire the state prediction category of the state discrimination samples; train the original discrimination model based on the difference between the state prediction category and the discrimination label to obtain the discrimination model; and acquire the label distribution, the estimated distribution and the sampling distribution through the discrimination model.

[0022] In this embodiment of the application, the state perturbation module is further configured to obtain the initial loss corresponding to the misjudgment loss, the hinge loss, and the cumulative decision reward; obtain the difference between the initial loss and the adversarial loss; train the original adversarial perturbation model; and obtain the adversarial perturbation model.

[0023] In this embodiment of the application, the state decision module is further configured to make a decision based on the i-th perturbation state using a deep reinforcement model to obtain the i-th decision result.

[0024] In this embodiment of the application, the sample acquisition device further includes a sample application module, used to acquire n decision rewards corresponding to the n perturbation states; and to determine the evaluation result of the deep reinforcement model based on the comparison results of the n decision rewards and the reward threshold.

[0025] In this embodiment of the application, the sample application module is further configured to, when determining the deep reinforcement model as a model to be optimized based on the evaluation results, obtain n decision results corresponding to the n perturbation states; obtain n clean states based on the i-th clean state, determine the n decision actions corresponding to the n clean states as n decision result labels corresponding to the n perturbation states, wherein the n decision result labels correspond one-to-one with the n decision results; and optimize the deep reinforcement model based on the n decision difference between the n decision result labels and the n decision results.

[0026] In this embodiment of the application, the sample acquisition module is further configured to determine the i-th perturbation state as an anti-antibody when the i-th decision result satisfies the termination condition.

[0027] This application provides a sample acquisition device, including:

[0028] Memory, used to store executable instructions;

[0029] The processor, when executing executable instructions stored in the memory, implements the sample acquisition method provided in the embodiments of this application.

[0030] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the sample acquisition method provided in this application.

[0031] The embodiments of this application have at least the following beneficial effects: by treating the process of acquiring adversarial samples as a sequential decision-making process, and by perturbing each clean state using the method of minimizing the cumulative reward, adversarial samples including n perturbed states are obtained, making the acquisition of adversarial samples independent of the structure and parameters of the agent; thus, even when the structure and parameters of the agent cannot be obtained, adversarial samples can still be acquired, thereby improving the feasibility of acquiring antibody samples. Attached Figure Description

[0032] Figure 1 This is an optional architecture diagram of the sample acquisition system provided in the embodiments of this application;

[0033] Figure 2 This is another optional architecture diagram of the sample acquisition system provided in the embodiments of this application;

[0034] Figure 3This is provided by the embodiments of this application. Figure 1 A schematic diagram of the composition structure of a server;

[0035] Figure 4 This is an optional flowchart illustrating the sample acquisition method provided in an embodiment of this application;

[0036] Figure 5 This is another optional flowchart illustrating the sample acquisition method provided in the embodiments of this application;

[0037] Figure 6 This is another optional flowchart illustrating the sample acquisition method provided in the embodiments of this application;

[0038] Figure 7 This is a schematic diagram of an exemplary sample acquisition architecture provided in an embodiment of this application;

[0039] Figure 8 This is an exemplary comparative diagram of adversarial attacks provided in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of an exemplary structure for training a primitive adversarial perturbation model provided in an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of another exemplary training adversarial perturbation model provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of an exemplary environmental frame image provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0046] In the implementation of this application, the collection and processing of relevant data should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1) AI (Artificial Intelligence) is the theory, method, technology and application system that uses digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0049] 2) DL (Deep Learning) is a method in machine learning based on learning representations of data, including supervised learning and unsupervised learning. Deep Neural Networks (DNNs) are the foundation of deep learning.

[0050] 3) RL (Reinforcement Learning) is a fast, efficient and irreplaceable learning algorithm in machine learning that explores the unknown environment, builds an environment model, and learns an optimal policy.

[0051] 4) DRL combines the perceptual capabilities of deep learning (DL) with the decision-making capabilities of reinforcement learning (RL). Reinforcement learning defines the optimization objective, while deep learning provides the operating mechanism—the way to represent the problem and the way to solve it.

[0052] 5) Sequential decision-making refers to the process of arranging various decision results in chronological order; that is, sequential decision-making is a multi-stage decision-making process with a time sequence; each stage of the multi-stage decision-making process needs to make a decision in order to achieve the optimal state of the entire process; once the decision results of each stage in the multi-stage process are determined, they form a sequence of decision results for the problem.

[0053] 6) MDP (Markov Decision Process) is a formalized or mathematical model of sequential decision-making. It is usually constructed based on a set of interactive objects (agent and environment). Its elements include state, action, policy, reward, and return (i.e., cumulative reward). In MDP, the agent perceives the current state of the environment, performs actions on the environment according to the policy, thereby changing the state of the environment and obtaining a reward. The accumulation of the reward over time is called the return (i.e., cumulative reward).

[0054] 7) An agent is an agent that performs machine learning in an MDP. It can perceive the state of the external environment, make decisions, take actions in response to the environment, and adjust its decision-making strategy based on feedback from the environment. For example, the deep reinforcement model involved in the embodiments of this application.

[0055] 8) The environment is the collection of all things outside the agent in the MDP model. Its state will change due to the agent's actions, and the changes in the state of the environment can be fully or partially perceived by the agent. In addition, the environment will provide the agent with corresponding rewards after each decision.

[0056] 9) A state is a description of the environment. After an agent takes an action, the state changes, and the evolution of the state exhibits the Markov property. The set of all states in an MDP is the state space, which can be discrete or continuous. Furthermore, the Markov property states that given the current state and all past states, the conditional probability distribution of the future state of a stochastic process depends only on the current state; that is, given the current state, the future state is conditionally independent of the past states. In the embodiments of this application, the observed state, clean state, perturbed state, clean state sample, and estimated perturbed state are all states.

[0057] 10) An action is a description of the behavior of an agent and is the result of the agent's decision; the set of all possible actions in an MDP is the action space, which can be discrete or continuous.

[0058] 11) A policy, used for decision-making, is a process of determining the conditional probability distribution of actions based on the state. In the context of reinforcement learning, it is a stochastic decision. After the agent determines the action through the policy, the environment provides feedback to the agent.

[0059] 12) The reward is the scalar function value of the current state, action, and the state at the next time step; while the reward is the accumulation of the reward over time steps.

[0060] 13) GAN (Generative Adversarial Networks) is a deep learning model belonging to unsupervised learning. GANs generate better outputs through the game-like learning between (at least) two modules in the framework: a generative model (also known as a generator) and a discriminative model (also known as a discriminator). During training, the goal of the generative model in GAN is to generate information that is as realistic as possible to deceive the discriminative model, while the goal of the discriminative model in GAN is to distinguish the information generated by the generative model from real information as much as possible. Thus, the generative model and the discriminative model constitute a dynamic game process, so that in the end, the generative model can generate information realistic enough that it is difficult for the discriminative model to determine whether the information generated by the generative model is real or not. It should be noted that AdvGAN is an extension of GAN, which is a combination of GAN and intelligent agent. In AdvGAN, clean states are mapped to perturbed states through the generator of GAN, and the perturbed states are used as the input states of the discriminator. The discriminator is responsible for judging whether the input state is a perturbed state or a clean state.

[0061] 14) Adversarial examples refer to input samples formed by adding subtle perturbations to a dataset, that is, samples that can cause the model to give an incorrect output with high confidence; adversarial examples demonstrate the vulnerability of DNNs; the added subtle perturbations refer to the addition of imperceptible noise to the state; in the embodiments of this application, adversarial examples include at least one perturbed state.

[0062] 15) The loss function, also known as the cost function, is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of the random event; and the calculation result obtained by using the loss function is called the loss or the loss function value.

[0063] 16) Blockchain is an encrypted, chain-like transaction storage structure formed by blocks.

[0064] 17) A blockchain network is a collection of nodes that incorporate new blocks into a blockchain through consensus.

[0065] It's important to note that artificial intelligence (AI) is a comprehensive technology within computer science. It attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0066] Furthermore, artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, intelligent transportation, and virtual scene interaction. The application of AI in deep reinforcement learning discussed in this application's embodiments will be explained later.

[0067] It's also worth noting that with the rapid development of Reinforcement Learning (RL) and Deep Learning (DL), the DRL obtained through the combination of RL and DL can directly map from the original input (e.g., images) to the distribution of actions, making end-to-end reinforcement learning possible. However, with the application of DRL in virtual scene interaction, hyperparameter learning, and autonomous driving, DRL always requires DNNs to extract representative features from the original high-dimensional observation state. Therefore, DNNs are susceptible to adversarial examples. Thus, acquiring adversarial examples, using them to evaluate DRL models, or using adversarial examples as training samples for DRL to improve robustness is becoming increasingly important.

[0068] Generally, to obtain adversarial examples for DRL, gradient acquisition is typically employed, such as the Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD). However, in the process of acquiring adversarial examples through gradient acquisition, the agent's structure and parameters are not always available, and parameter backpropagation is required during gradient acquisition, which is time-consuming; thus, the feasibility and efficiency of acquiring adversarial examples are relatively low.

[0069] Furthermore, the process of obtaining adversarial examples utilizes the vulnerability of DNNs, directly transforming DRL attacks into DNN attacks. Since the purpose of attacking DNNs is to make them output a high-confidence incorrect prediction given the information, even if they can greedily induce the DRL agent to take incorrect actions at every step, they cannot obtain the minimum cumulative reward. Thus, the obtained adversarial examples are only locally optimal; therefore, the adversarial nature of the obtained adversarial examples is low.

[0070] Based on this, embodiments of this application provide a sample acquisition method, apparatus, device, and computer-readable storage medium, which can improve the efficiency and feasibility of acquiring adversarial samples, as well as enhance the adversarial nature of the adversarial samples. The exemplary application of the sample acquisition device provided in this application embodiment is described below. The sample acquisition device provided in this application embodiment can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, in-vehicle devices), or it can be implemented as a server. The exemplary application when the device is implemented as a server will be described below.

[0071] See Figure 1 , Figure 1 This is a schematic diagram of an optional architecture of the sample acquisition system provided in an embodiment of this application; as shown... Figure 1 As shown, to support a sample acquisition application, in the sample acquisition system 100, a terminal 400 (terminals 400-1 and 400-2 are shown as examples) connects to a server 200 (sample acquisition device) via a network 300. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the sample acquisition system 100 also includes a database 500, used to provide data support to the server 200 when the server 200 executes the sample acquisition method provided in this application, or when the server 200 provides computing services to the terminal 400 via the network 300.

[0072] Terminal 400 is used to send clean states (the i-th clean state, the (i+1)-th clean state, ..., the n-th clean state, for example, the relevant information of virtual objects in the virtual scene obtained by terminal 400-1, or the road conditions of the current road where the vehicle is located in the autonomous driving scenario obtained by terminal 400-2) to server 200 via network 300. It is also used to obtain the action instructions corresponding to the clean states via network 300 to execute the processing of the action instructions. Here, except for the initial clean state, the clean states sent by terminal 400 to server 200 via network 300 are all determined based on the previous clean state and the action instructions sent by server 200.

[0073] Server 200 is used to obtain the i-th clean state corresponding to the i-th time step from terminal 400 via network 300, where i is a positive integer; to perform adversarial perturbation on the i-th clean state based on the cumulative reward to obtain the i-th perturbed state, where the cumulative reward of the i-th perturbed state is the minimum; to make a decision based on the i-th perturbed state to obtain the i-th decision result, and to send the action instruction corresponding to the i-th decision result to terminal 400 via network 300, so that when the i-th decision result does not meet the termination condition, the server obtains the (i+1)-th clean state from terminal 400 via network 300 based on the i-th clean state and the i-th decision result, and obtains the (i+1)-th decision result corresponding to the (i+1)-th clean state, until the n-th decision result meets the termination condition, thus obtaining an adversarial sample including n perturbed states, where n is a positive integer greater than 1.

[0074] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle device, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0075] This application's embodiments can also be implemented using blockchain technology; see [link to relevant documentation]. Figure 2 , Figure 2 This is another optional architecture diagram of the sample acquisition system provided in the embodiments of this application. Figure 2 In the sample acquisition system 100 shown, sample acquisition is performed through server 200, and samples can also be acquired from multiple terminals through server 200. Figure 2 The example shows terminals 400-1 and 400-2 sending action commands corresponding to the clean state.

[0076] In some embodiments, server 200, terminal 400-1, and terminal 400-2 can join the blockchain network 600 and become nodes within it. The type of blockchain network 600 is flexible and diverse; for example, it can be any type of public blockchain, private blockchain, or consortium blockchain. Taking a public blockchain as an example, any electronic device of any business entity can access the blockchain network 600 without authorization to act as a consensus node. For instance, terminal 400-1 is mapped to consensus node 600-1 in the blockchain network 600, server 200 is mapped to consensus node 600-2, and terminal 400-2 is mapped to consensus node 600-3.

[0077] Taking blockchain network 600 as a consortium blockchain as an example, server 200, terminal 400-1, and terminal 400-2 can become nodes after obtaining authorization and accessing blockchain network 600. After server 200 obtains the clean state sent by terminal 400, it can make a decision on the clean state by executing a smart contract to obtain the action instruction corresponding to the clean state, and then send the action instruction corresponding to the clean state to blockchain network 600 for consensus. When the consensus is passed, the server then sends the action instruction corresponding to the clean state to terminals 400-1 and 400-2. It can be seen that by having multiple nodes in the blockchain network reach a consensus to confirm the action instruction before sending it to terminals 400-1 and 400-2, the reliability and accuracy of deep reinforcement learning applications can be improved.

[0078] See Figure 3 , Figure 3 This is provided by the embodiments of this application. Figure 1 A schematic diagram of the composition structure of a server. Figure 3 The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 240.

[0079] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0080] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0081] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0082] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0083] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0084] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0085] The network communication module 252 is used to reach other computer devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0086] In some embodiments, server 200 further includes a presentation module 253 for enabling the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with user interface 230 (e.g., a display screen, a speaker, etc.).

[0087] The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0088] In some embodiments, the sample acquisition device provided in this application can be implemented in software. Figure 3 A sample acquisition device 255 stored in memory 250 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a state acquisition module 2551, a state perturbation module 2552, a state decision module 2553, a sample acquisition module 2554, and a sample application module 2555. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0089] In other embodiments, the sample acquisition device provided in this application can be implemented in hardware. As an example, the sample acquisition device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the sample acquisition method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0090] The sample acquisition method provided in this application will be described below with reference to exemplary applications and implementations of the sample acquisition device provided in the embodiments of this application.

[0091] See Figure 4 , Figure 4 This is an optional flowchart illustrating the sample acquisition method provided in the embodiments of this application, which will be combined with... Figure 4 The steps shown are explained.

[0092] S401. Obtain the i-th clean state corresponding to the i-th time step.

[0093] In this embodiment of the application, when performing adversarial attacks or model optimization, the sample acquisition device acquires the observation state at the i-th time step, so as to obtain adversarial data for adversarial attacks or model optimization by adding imperceptible noise to the observation state at the i-th time step; wherein, the observation state at the i-th time step is the i-th clean state.

[0094] It should be noted that the observed state is the state captured in the environment. i is a positive integer: 1, 2, 3, ..., and when i is 1, the i-th clean state is the initial state captured in the environment.

[0095] S402. Based on the cumulative reward, perform adversarial perturbation on the i-th clean state to obtain the i-th perturbation state.

[0096] In this embodiment, the sample acquisition device acquires the perturbation observation state with the minimum cumulative reward corresponding to the i-th clean state, thus completing the adversarial perturbation of the i-th clean states; wherein, the perturbation observation state with the minimum cumulative reward corresponding to the i-th clean state is the i-th perturbation state. At this point, the sample acquisition device obtains adversarial examples based on the i-th perturbation state, thus obtaining adversarial examples that include at least the i-th perturbation state. Here, the cumulative reward of the i-th perturbation state is the minimum.

[0097] It should be noted that the sample acquisition device processes adversarial perturbations based on cumulative rewards, which is a forward propagation method and does not require access to the structure and parameters of the agent to be attacked or optimized. Furthermore, the cumulative reward is the actual payout.

[0098] For example, the cumulative reward can be represented by equation (1), which is:

[0099]

[0100] Among them, R t Let be the cumulative reward corresponding to the t-th adversarial perturbation at time step t, where i is t, t+1, ..., n, γ∈[0,1], and is a discount factor. i For the i-th counter-perturbation, a i For s i The corresponding decision result, r(s) i ,a i ) represents the reward corresponding to the i-th counter-perturbation.

[0101] Understandably, a larger cumulative reward indicates that the decision made by the agent to be attacked or the agent to be optimized is more correct. Here, the sample acquisition device performs adversarial perturbation on the i-th clean state based on minimizing the cumulative reward, making the i-th perturbed state more adversarial. Furthermore, the method based on minimizing the cumulative reward is a forward propagation method, which makes the feasibility and efficiency of adversarial perturbation both high.

[0102] S403. Make a decision based on the i-th perturbation state and obtain the i-th decision result.

[0103] In this embodiment of the application, after the sample acquisition device obtains the i-th perturbation state, it makes a decision based on the i-th perturbation state and obtains the action corresponding to the i-th time step. The action corresponding to the i-th time step is the i-th decision result.

[0104] It should be noted that the sample acquisition device makes a decision based on the i-th perturbation state and obtains the i-th decision result through a deep reinforcement model, which is the agent to be attacked or the agent to be optimized, and is a type of DRL.

[0105] S404. When the i-th decision result does not meet the termination condition, determine the (i+1)-th clean state based on the i-th clean state and the i-th decision result, and obtain the (i+1)-th decision result corresponding to the (i+1)-th clean state, until the n-th decision result obtained meets the termination condition, thus obtaining an adversarial sample including n perturbation states.

[0106] In this embodiment of the application, the sample acquisition device can acquire a termination condition, which is used to determine whether the current time step is the last time step based on the current decision result. For example, for the i-th time step, the termination condition is used to determine whether the i-th time step is the last time step based on the i-th decision result; and the termination condition is the rule corresponding to the action in the DRL, such as the virtual object in the virtual scene running out of energy, the autonomous driving ending, etc.

[0107] Here, the sample acquisition device determines whether the termination condition is met based on the obtained i-th decision result. If not, it indicates that the i-th time step is not the last time step. Therefore, the sample acquisition device continues to combine the i-th clean state and the i-th decision result to determine the (i+1)-th clean state of the environment at the (i+1)-th time step. Using the processing described in S402 and S403, it performs adversarial perturbation on the (i+1)-th clean state based on the cumulative reward, obtaining the (i+1)-th perturbation state with the minimum cumulative reward. A decision is then made based on the (i+1)-th perturbation state to obtain the (i+1)-th decision result. If the (i+1)-th decision result meets the termination condition, then the system obtains a result including the i-th perturbation state and the i-th... The adversarial example consists of +1 perturbation states. At this point, the n perturbation states include the i-th perturbation state and the (i+1)-th perturbation state, where n is 2. If the (i+1)-th decision result also does not meet the termination condition, the (i+2)-th clean state is determined based on the (i+1)-th clean state and the (i+1)-th decision result, and the (i+2)-th decision result corresponding to the (i+2)-th clean state is obtained, until the n-th decision result meets the termination condition. At this point, an adversarial example consisting of the i-th perturbation state, the (i+1)-th perturbation state, ..., the n-th perturbation state is obtained, where the n perturbation states include the i-th perturbation state, the (i+1)-th perturbation state, ..., the n-th perturbation state, where n is greater than 2.

[0108] It should be noted that n is a positive integer greater than 1: 2, 3, 4, ...

[0109] Understandably, by perturbing the clean state using the method of minimizing the cumulative reward, and obtaining the perturbed state as an antibody sample, the acquisition of antibody samples no longer requires access to the agent's structure and parameters, nor does it require gradient calculation. Furthermore, it can obtain adversarial samples with the lowest cumulative reward, thus improving feasibility, efficiency, and adversarial capability.

[0110] See Figure 5 , Figure 5 This is another optional flowchart illustrating the sample acquisition method provided in the embodiments of this application; as shown below. Figure 5 As shown, S403 is followed by S405; that is, after the sample acquisition device makes a decision based on the i-th disturbance state and obtains the i-th decision result, the sample acquisition method also includes S405, which will be explained below.

[0111] S405. When the i-th decision result satisfies the termination condition, the i-th perturbation state is determined as an adversarial sample.

[0112] It should be noted that the sample acquisition device determines whether the termination condition is met based on the obtained i-th decision result. If so, it indicates that the i-th time step is the last time step, and at this time, the i-th perturbation state is the adversarial sample.

[0113] In this embodiment of the application, the adversarial perturbation in S402 is implemented through an adversarial perturbation model. That is, the sample acquisition device uses the adversarial perturbation model to perform adversarial perturbation on the i-th clean state based on the cumulative reward to obtain the i-th perturbation state. The adversarial perturbation model is a network model used to obtain the perturbation state with the minimum cumulative reward, such as the generator in a generative adversarial network.

[0114] Here, the training process of the adversarial perturbation model includes steps S406 to S408, and each step is explained below.

[0115] S406. Using the original adversarial perturbation model, adversarial perturbation is applied to the j-th clean state sample corresponding to the j-th time step to obtain the j-th predicted perturbation state.

[0116] It should be noted that the original adversarial perturbation model is the network model to be trained to obtain the perturbation state with the minimum cumulative reward; the sample acquisition device initializes the parameters in the original adversarial perturbation model and uses the initialized original adversarial perturbation model to perform adversarial perturbation on the training samples; here, the training sample is the j-th clean state sample corresponding to the j-th time step, and the obtained adversarial perturbation result is the j-th predicted perturbation state corresponding to the j-th clean state sample. Where j is a positive integer: 1, 2, 3, ...

[0117] S407. Obtain the cumulative decision reward corresponding to the j-th predicted disturbance state.

[0118] In this embodiment of the application, the sample acquisition device acquires the cumulative reward corresponding to the j-th estimated disturbance state, and the acquired cumulative reward corresponding to the j-th estimated disturbance state is the cumulative decision reward.

[0119] S408. Based on cumulative decision rewards, train the original adversarial perturbation model to obtain the adversarial perturbation model.

[0120] In this embodiment of the application, the sample acquisition device adjusts the parameters of the original adversarial perturbation model based on minimizing the cumulative decision reward in order to train the original adversarial perturbation model. The trained original adversarial perturbation model is the adversarial perturbation model.

[0121] Understandably, because the process of training the original adversarial perturbation model using cumulative decision rewards integrates the feedback from all time steps in an event, rather than just considering the feedback from the current time step, it reflects the optimality of the entire decision sequence.

[0122] In this embodiment of the application, when the adversarial perturbation model is a generator in a generative adversarial network, steps S409 to S412 are included before step S408; that is, before the sample acquisition device trains the original adversarial perturbation model based on the cumulative decision reward and obtains the adversarial perturbation model, the sample acquisition method also includes steps S409 to S412. Each step is described below.

[0123] S409. Make a decision based on the j-th clean state sample to obtain the j-th decision result label, and make a decision based on the j-th predicted disturbance state to obtain the j-th predicted decision result.

[0124] In this embodiment, the sample acquisition device makes a decision based on the j-th clean state sample and obtains the action corresponding to the j-th clean state sample. The action corresponding to the j-th clean state sample is the j-th decision result label. Similarly, the sample acquisition device makes a decision based on the j-th estimated disturbance state and obtains the action corresponding to the j-th estimated disturbance state. The action corresponding to the j-th estimated disturbance state is the j-th estimated decision result.

[0125] It should be noted that the decision made based on the j-th clean state sample and the decision made based on the j-th predicted perturbation state can both be implemented by the agent to be attacked or the agent to be optimized.

[0126] S410. Based on the distribution difference between the j-th clean state sample and the j-th predicted perturbation state, obtain the misjudgment loss, and based on the difference between the j-th decision result label and the j-th predicted decision result, obtain the adversarial loss.

[0127] It should be noted that the misclassification loss obtained by the sample acquisition device is used to make the data distribution of the perturbed state approximate the data distribution of the clean state, that is, to minimize the classification categories between the perturbed state and the clean state. The adversarial loss obtained by the sample acquisition device is used to maximize the distance between the action obtained after adversarial perturbation and the action in the clean state, that is, to maximize the distance between the predicted value and the true value.

[0128] For example, the adversarial loss can be obtained through equation (2), which is:

[0129] L Adv-j =L(f(g(s) j )), t j (2)

[0130] Among them, L Adv-j To combat the losses, s j For the j-th clean state sample, g(s) j ) represents the j-th predicted disturbance state, f(g(s) j )) represents the j-th predicted decision outcome, t j Let L be the label of the j-th decision result, L() be the loss function, and g() be used for adversarial perturbation.

[0131] S411. Obtain the hinge loss based on the j-th predicted disturbance state.

[0132] In this embodiment of the application, the sample acquisition device also employs hinge loss to limit the degree of resistance to disturbances.

[0133] For example, the hinge loss can be obtained by equation (3), which is:

[0134] L hinge-j =max(0, ||g(s) j )‖2-ξ) (3)

[0135] Among them, L hinge-j ξ represents the hinge loss, and ξ is a preset parameter.

[0136] S412. Combine adversarial loss, misjudgment loss, hinge loss and cumulative decision reward to train the original adversarial perturbation model and obtain the adversarial perturbation model.

[0137] In this embodiment of the application, the sample acquisition device integrates adversarial loss, misjudgment loss, hinge loss and cumulative decision reward to train the original adversarial perturbation model in order to obtain the adversarial perturbation model.

[0138] In this embodiment, S407 is implemented using a cumulative reward model; that is, the sample acquisition device obtains the cumulative decision reward corresponding to the j-th estimated perturbation state through the cumulative reward model; wherein, the cumulative reward model is a model used to obtain the cumulative reward, such as a commentator network model. The training process of the cumulative reward model includes S413 to S417, and each step is described below.

[0139] S413. Perform adversarial perturbation on the kth clean state sample corresponding to the kth time step to obtain the kth predicted perturbation state.

[0140] It should be noted that the k-th predicted perturbation state is the k-th clean state sample after counteracting the perturbation, where k is a positive integer: 1, 2, 3, ...

[0141] S414. Make a decision on the k-th predicted disturbance state and obtain the reward corresponding to the k-th predicted decision result.

[0142] It should be noted that the sample acquisition device uses the agent to be attacked or the agent to be optimized to make a decision on the k-th predicted perturbation state, and the decision result obtained with respect to the k-th predicted perturbation state is the k-th predicted decision result; furthermore, the sample acquisition device can also obtain the reward corresponding to the k-th predicted decision result from the environment, and the reward corresponding to the k-th predicted decision result is the k-th predicted decision reward.

[0143] S415. Using the original cumulative reward model, obtain the k-th estimated cumulative reward corresponding to the k-th estimated disturbance state.

[0144] It should be noted that the original cumulative reward model is the model to be trained for obtaining the cumulative reward; the sample acquisition device initializes the parameters in the original cumulative reward model and uses the initialized cumulative reward model to obtain the cumulative reward corresponding to the k-th estimated perturbation state, which is the k-th estimated cumulative reward.

[0145] S416. Based on the kth clean state sample and the kth predicted decision result, determine the k+1th estimated perturbation state corresponding to the k+1th clean state sample, and use the original cumulative reward model to obtain the k+1th estimated cumulative reward corresponding to the k+1th predicted perturbation state.

[0146] In this embodiment, the sample acquisition device determines the (k+1)th clean state sample from the environment based on the kth clean state sample and the kth estimated decision result, and applies adversarial perturbation to the (k+1)th clean state sample. The (k+1)th clean state sample after adversarial perturbation is the (k+1)th estimated perturbation state. Here, the sample acquisition device describes the process of obtaining the kth estimated cumulative reward in S415 to obtain the (k+1)th estimated cumulative reward corresponding to the (k+1)th estimated perturbation state.

[0147] S417. Based on the k-th predicted decision reward, the k-th predicted cumulative reward, and the (k+1)-th predicted cumulative reward, train the original cumulative reward model to obtain the cumulative reward model.

[0148] In this embodiment, the sample acquisition device calculates a loss function value (loss) by combining the k-th estimated decision reward, the k-th estimated cumulative reward, and the (k+1)-th estimated cumulative reward. Based on the calculated loss function value, it performs backpropagation in the original cumulative reward model to train the original cumulative reward model. Here, the trained original cumulative reward model is the cumulative reward model. Furthermore, the sample acquisition device can use the difference between the combined result of the k-th estimated decision reward and the (k+1)-th estimated cumulative reward and the k-th estimated cumulative reward as the loss for training the original cumulative reward model.

[0149] For example, the loss function value used to train the original cumulative reward model can be achieved through equation (4), which is:

[0150] L TD-k =r k +γC(g(s k+1 ))-C(g(s k (4)

[0151] Among them, L TD-k r is the loss function value used to train the original cumulative reward model. k For the k-th predicted decision reward, C(g(s) k+1 )) represents the (k+1)th estimated cumulative reward, C(g(s) k )) represents the estimated reward for the k-th decision, and C() is used to obtain the cumulative reward.

[0152] In this embodiment of the application, S410 can be implemented by S4101 to S4104; that is, the sample acquisition device obtains the misjudgment loss based on the distribution difference between the j-th clean state sample and the j-th estimated disturbance state, including S4101 to S4104. Each step is described below.

[0153] S4101. Obtain the label distribution corresponding to the j-th clean state sample.

[0154] S4102. Obtain the predicted distribution corresponding to the j-th predicted disturbance state.

[0155] S4103. Obtain the sampling state between the j-th clean state sample and the j-th predicted disturbance state, and obtain the sampling distribution corresponding to the adopted state.

[0156] It should be noted that the label distribution is the data distribution corresponding to the j-th clean state sample obtained by the sample acquisition device, the prediction distribution is the data distribution corresponding to the j-th predicted perturbation state obtained by the sample acquisition device, and the adopted distribution is the data distribution corresponding to the sampling state between the j-th clean state sample and the j-th predicted perturbation state obtained by the sample acquisition device.

[0157] For example, the process by which the sample acquisition device acquires the sampling state between the j-th clean state sample and the j-th estimated perturbation state can be implemented by (5), where equation (5) is:

[0158]

[0159] in, This is the sampling state.

[0160] S4104. The initial distribution difference between the label distribution and the predicted distribution, and the distribution difference between the sampling distribution, are determined as the misjudgment loss.

[0161] In this embodiment, the sample acquisition device combines the label distribution, the estimated distribution, and the sampled distribution. The combined result of the label distribution, the estimated distribution, and the sampled distribution is the misclassification loss. Here, the sample acquisition device can first obtain the initial distribution difference between the label distribution and the estimated distribution, then obtain the distribution difference between the initial distribution difference and the sampled distribution, and determine the distribution difference between the initial distribution difference and the sampled distribution as the misclassification loss.

[0162] For example, the misjudgment loss can be obtained through equation (6), which is:

[0163]

[0164] Among them, L GAN-j To misjudge the loss, D(s) j ) represents the label distribution; D(g(s) j )) represents the estimated distribution; λ is the sampling distribution; λ is the gradient penalty coefficient.

[0165] In this embodiment, the label distribution, the predicted distribution, and the sampling distribution are all implemented through a discriminative model. The discriminative model is a classification model used to obtain the data distribution of the input data to determine the category based on the data distribution, such as D() in equation (6). The training process of the discriminative model includes: a sample acquisition device acquiring state discrimination samples and state discrimination labels; using the original discriminative model to obtain the state prediction category of the state discrimination samples; and training the original discriminative model based on the difference between the state prediction category and the discrimination label to obtain the discriminative model.

[0166] It should be noted that the state discrimination sample is the input training sample, the state discrimination label is the label of the state discrimination sample, which is the true category of the state discrimination sample; the state prediction category is the predicted category of the state discrimination sample; the original discrimination model is the classification model to be trained, and the trained original discrimination model is the discrimination model.

[0167] In this embodiment, S412 can be implemented through S4121 and S4122; that is, the sample acquisition device combines adversarial loss, misjudgment loss, hinge loss and cumulative decision reward to train the original adversarial perturbation model and obtain the adversarial perturbation model, including S4121 and S4122. Each step is described below.

[0168] S4121. Obtain the initial loss corresponding to the misjudgment loss, hinge loss, and cumulative decision reward.

[0169] S4122. Obtain the difference between the initial loss and the adversarial loss, train the original adversarial perturbation model, and obtain the adversarial perturbation model.

[0170] It should be noted that when the sample acquisition device combines adversarial loss, misjudgment loss, hinge loss, and cumulative decision reward to train the original adversarial perturbation model, the combination method of adversarial loss, misjudgment loss, hinge loss, and cumulative decision reward is as follows: first, the misjudgment loss, hinge loss, and cumulative decision reward are combined based on the corresponding weights to obtain the initial loss; then, the difference between the initial loss and the adversarial loss is obtained, and the difference between the initial loss and the adversarial loss is used as the loss function value for training the original adversarial perturbation model.

[0171] For example, adversarial loss, misjudgment loss, hinge loss, and cumulative decision reward can be combined using equation (7), which is:

[0172] L AdvGAN-MDP-j =αL GAN-j +βL hinge-j +L reward-j -L Adv-j (7)

[0173] Where α and β are weighting parameters, LAdvGAN-MDP-j L is the loss function value used to train the original adversarial perturbation model. reward-j To accumulate decision rewards, one can obtain them through a trained C(), i.e., C(g(s) j )).

[0174] See Figure 6 , Figure 6 This is another optional flowchart illustrating the sample acquisition method provided in the embodiments of this application; as shown below. Figure 6 As shown in the embodiment of this application, S404 is followed by S418 and S419; that is, after the sample acquisition device obtains adversarial samples including n perturbation states, the sample acquisition method further includes S418 and S419. Each step is described below.

[0175] S418. Obtain the n decision rewards corresponding to the n perturbation states.

[0176] It should be noted that the sample acquisition device obtains the reward corresponding to each of the n perturbation states from the environment. Specifically, it obtains the i-th decision reward corresponding to the i-th perturbation state, the (i+1)-th decision reward corresponding to the (i+1)-th perturbation state, and so on, until it obtains the n-th decision reward corresponding to the n-th perturbation state. Thus, the rewards corresponding to each of the n perturbation states constitute the n decision rewards. Here, the n decision rewards corresponding to the n perturbation states are in a one-to-one correspondence.

[0177] S419. Based on the comparison results of n decision reward and return thresholds, determine the evaluation results of the deep reinforcement model.

[0178] It should be noted that the sample acquisition device obtains the weighted sum of n decision rewards (which can be achieved through equation (1), except that r(s) in equation (1) is changed). i ,a i ) changed to r(g(s) i ),a i The weighted sum of the n decision rewards is the reward of the adversarial example. At this time, the sample acquisition device compares the weighted sum of the n decision rewards with the reward threshold. If the comparison result is that the weighted sum of the n decision rewards is greater than the reward threshold, the evaluation result of the deep reinforcement model is that the deep reinforcement model is robust; if the comparison result is that the weighted sum of the n decision rewards is less than or equal to the reward threshold, the evaluation result of the deep reinforcement model is that the deep reinforcement model has low robustness.

[0179] It is understood that, since the adversarial examples obtained in this application are highly adversarial, when the evaluation result is determined based on the adversarial examples to show that the deep reinforcement model is robust, the generalization ability of the deep reinforcement model is further ensured.

[0180] See also Figure 6 In this embodiment, the sample acquisition device can further optimize the deep reinforcement model through adversarial examples. Therefore, S420 to S422 are included after S419; that is, after the sample acquisition device determines the evaluation result of the deep reinforcement model based on the comparison results of n decision rewards and reward thresholds, the sample acquisition method further includes S420 to S422, and each step is described below.

[0181] S420. When the deep reinforcement model is determined to be the model to be optimized based on the evaluation results, obtain n decision results corresponding to n perturbation states.

[0182] It should be noted that when the sample acquisition device determines the deep reinforcement model as the model to be optimized based on the evaluation results, it obtains the action corresponding to each of the n perturbation states through the deep reinforcement model. That is, it obtains the i-th decision result corresponding to the i-th perturbation state, the (i+1)-th decision result corresponding to the (i+1)-th perturbation state, and so on, until it obtains the n-th decision result corresponding to the n-th perturbation state. Thus, the action corresponding to each of the n perturbation states constitutes n decision rewards. Here, the n decision results corresponding to the n perturbation states are in one-to-one correspondence.

[0183] S421. Based on the i-th clean state, obtain n clean states, and determine the n decision actions corresponding to the n clean states as the n decision result labels corresponding to the n perturbation states.

[0184] It should be noted that the sample acquisition device makes decisions and determines new states based on the i-th clean state, and can continuously acquire n clean states. It is easy to see that the n clean states include the i-th clean state. When the sample acquisition device makes decisions for each of the n clean states using a deep reinforcement model, it obtains n decision actions corresponding to the n clean states. Here, the n decision actions corresponding to the n clean states are labels for the n perturbation states. Therefore, the sample acquisition device determines the n decision actions as n decision result labels corresponding to the n perturbation states. In this system, the n clean states, n decision actions, n decision result labels, and n decision results correspond one-to-one.

[0185] S422. Optimize the deep reinforcement model based on n decision result labels and n decision differences corresponding to the n decision results.

[0186] In this embodiment, the sample acquisition device optimizes the decision-making strategy in the deep reinforcement model based on n decision result labels and n decision differences corresponding to the n decision results. Here, the n decision result labels, n decision results, and n decision differences correspond one-to-one.

[0187] It is understood that since the adversarial examples obtained in this application are highly adversarial, when the evaluation result based on the adversarial examples indicates that the robustness of the deep reinforcement model is low, optimizing the deep reinforcement model based on the adversarial examples can further improve the generalization ability of the deep reinforcement model.

[0188] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0189] See Figure 7 , Figure 7 This is a schematic diagram of an exemplary sample acquisition architecture provided in an embodiment of this application; as shown... Figure 7 As shown, in this exemplary sample acquisition architecture 7-1, firstly, the observation state 7-12 (clean state) is determined through the environment 7-11. Then, the generator (adversarial perturbation model) in the generative adversarial network 7-13 adversarially perturbs the observation state 7-12 to obtain the perturbation state 7-14. Then, the agent 7-15 (deep reinforcement model) makes a decision based on the perturbation state 7-14 to obtain the action 7-16 (decision result). The action 7-16 is used to act on the environment 7-11 and, combined with the observation state 7-12, determines a new observation state. At the same time, the environment 7-11 will give a reward 7-17 (decision reward) for the action 7-16.

[0190] It should be noted that at the current moment, the observation state s is determined through environmental 7-11. i (The i-th clean state), at a historical moment, the observed states determined by environment 7-11 include s1, s2…s i-1 After the adversarial perturbation processing of the generator in the adversarial generative network 7-13, the observed state s i g(s) is processed as a perturbation state 7-14 i (i) the i-th perturbation state, observed states s1, s2…s i-1 They are processed sequentially into the perturbation states 7-14: g(s1), g(s2), ... g(s). i-1 ); Disturbance states g(s1), g(s2)...g(s i-1 ), g(s i () Corresponds sequentially to actions a1, a2…a in actions 7-16. i-1 a i (the i-th decision result); and actions a1, a2…a… i-1 ai These correspond to rewards r1, r2...r in rewards 7-17, respectively. i-1 r i (Reward for the i-th decision), where the cumulative reward R is minimized. min =r1+r2…r i-1 +r i It can be obtained through equation (1).

[0191] It should also be noted that agent 7-15 and environment 7-11 constitute a new environment. The generator in the generative adversarial network 7-13 is an attacking agent that interacts with this new environment. The attacking agent takes the observed state 7-12 of agent 7-15 as input and outputs a perturbation state 7-14 as input to agent 7-15. Here, the attacking agent changes the observed state 7-12, not the state in the virtual engine. The state in the virtual engine refers to the facts within the virtual engine, while the observed state refers to the state obtained by capturing the state in the virtual engine through sensors.

[0192] in addition, Figure 7 The perturbation state g(s) i ) is obtained by minimizing the cumulative reward. Compared to a single-time-step attack, the perturbation state g(s) is obtained by minimizing the cumulative reward. i It has a high degree of competitiveness.

[0193] See Figure 8 , Figure 8 This is an exemplary comparative diagram of adversarial attacks provided in an embodiment of this application; as shown... Figure 8As shown, during a table tennis match, agent 7-15 (not shown) makes a decision regarding the clean state 8-11 and determines to perform the "down" operation 8-21. At this point, the next clean state 8-31, where the table tennis ball lands on the paddle, is obtained. Using a single-time-step attack (e.g., PGD), an adversarial perturbation is initiated against the clean state 8-11, resulting in a perturbed state 8-12. Agent 7-15 then makes a decision regarding the clean state 8-12 and determines to perform the "no-op" operation 8-22. At this point, the next clean state 8-32, where the table tennis ball lands on the paddle, is also obtained. Finally, using the generator (the adversarial perturbation model provided in this embodiment) in the generative adversarial network 7-13 (not shown), an adversarial perturbation is initiated against the clean state 8-11, resulting in a perturbed state 8-13. Agent 7-15 then makes a decision regarding the perturbed state 8-13 and determines to perform the "up" operation 8-23. At this point, the next clean state 8-33, where the table tennis ball leaves the paddle, is obtained. It is easy to see that since executing the "no-op" operation 8-22 still has the possibility of catching the ping-pong ball, while executing the "up" operation 8-23 will result in losing the game; therefore, a single time step attack only considers the action that causes agent 7-15 to make an incorrect decision at the current time step. Thus, even if the action that causes agent 7-15 to make an incorrect decision is attacked at every time step, it is difficult to obtain the minimum cumulative reward; while the adversarial perturbation model provided in this application embodiment considers the minimum cumulative reward; therefore, the adversarial perturbation model provided in this application embodiment can obtain an adversarial antibody with higher offensive capabilities.

[0194] In this embodiment, the process of attacking agent 7-15 is described as a sequential decision problem, which can be implemented using MDP. MDP is defined as (S, A, g, P, r, γ); where S is the set of clean states, A is the set of actions, g (g()) is the perturbation handling function used to adversarially perturb the clean states (a deterministic function), P is the state transition probability of S, and r is the reward. During the MDP-based attack, minimizing the cumulative reward can be achieved using the actor criticism method; see [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of an exemplary structure for training a primitive adversarial perturbation model provided in an embodiment of this application; as shown... Figure 9 As shown in Figure 9-1, in the training architecture of the original adversarial perturbation model, the adversarial perturbation network to be trained (9-11, the original adversarial perturbation model) is trained on a clean state s obtained from the environment (9-12). j (The j-th clean state sample) is subjected to adversarial perturbation to obtain the perturbation state g(s). j Next, the critic network model 9-13 (C() trained in equation (4), the cumulative reward model) obtains the perturbation state g(s) j The corresponding cumulative reward Lreward-j (C(g(s j Based on cumulative reward L reward-j Train the adversarial perturbation network 9-11 to obtain Figure 7 The generator in Generative Adversarial Networks 7-13.

[0195] See Figure 10 , Figure 10 This is a schematic diagram of another exemplary training adversarial perturbation model provided in an embodiment of this application; as shown... Figure 10 As shown in Figure 10-1, in the training architecture 10-1 of the original adversarial perturbation model, the generator to be trained 10-11 (the original adversarial perturbation model) is trained on the clean state s obtained from the environment 10-12. j Entering the counter-perturbation state, obtaining the perturbation state g(s) j Next, based on the commentator network model 10-13 (cumulative reward model), the perturbation state g(s) is obtained. j The corresponding cumulative reward L reward-j Based on the discriminator 10-15, obtain g(s) j The corresponding misjudgment loss L GAN-j Based on agent 10-14, obtain g(S) j The corresponding adversarial loss L Adv-j and obtain g(S) j The corresponding hinge loss L hinge-j Therefore, based on the cumulative reward L reward-j Misjudgment loss L GAN-j , countering losses L Adv-j and hinge loss L hinge-j Train the generator to be trained 10-11 to obtain Figure 7 The generator in Generative Adversarial Networks 7-13. The cumulative reward L... reward-j The misjudgment loss L is obtained by training the model based on equation (4). GAN-j Obtain and counteract the loss L through equation (6) Adv-j The hinge loss L is obtained through equation (2). hinge-j It can be obtained through equation (3).

[0196] In this embodiment, the agent is defined as f, the number of iterations is N, the generator is g, the replay experience is D, the environment is E, the batch size is m, and the number of iterations of the critic network model is n. cri The generator iterates n times. dis The generator's parameters are θ, the discriminator's parameters are ω, and the critic network model's parameters are ψ, with learning rates of λ1, λ2, and λ3. At this point, the generator is trained to obtain... Figure 7The generator process in Generative Adversarial Networks 7-13 can be implemented using the following pseudocode:

[0197]

[0198] It should be noted that lines 4 to 12 describe the process of training the discriminator; lines 13 (s' being the next clean state) and 14 describe the process of training the critic network model; therefore, lines 2 to 15 describe the process of training the critic network model based on the trained discriminator; lines 16 and 17 describe the process of training the generator; therefore, lines 1 to 18 describe the process of training the generator based on the trained discriminator and the trained critic network model. Where L... GAN With L GAN-j The acquisition method is consistent, L AdvGAN-MDP With L AdvGAN-MDP-j The acquisition method is the same.

[0199] In the embodiments of this application, see Figure 11 , Figure 11 This is a schematic diagram of an exemplary environmental frame image provided in an embodiment of this application; as shown... Figure 11 As shown, the exemplary environment frame image 11-1 includes frame image 11-11 corresponding to ping-pong, frame image 11-12 corresponding to speed racing, frame image 11-13 corresponding to maze, and frame image 11-14 corresponding to runner. Here, environments 1 to 4 are set to correspond to the ping-pong game environment in sequence (environment frame images are, for example, ping-pong game environment 11-12). Figure 11 Frame images 11-11 in the speed racing environment (environment frame images, for example) Figure 11 (Frame images 11-12 in the image), maze environment (environment frame image, for example, is...) Figure 11 Frame images 11-13) and the runner's environment (environmental frame images, for example, are shown in the image). Figure 11 Frame images 11-14 in the image; the perturbation budget is set to 1 / 255, 2 / 255 and 4 / 255; the attacking agents 1 to 3 are "DQN (vanilla)" agent, "DQN AdvTraining" agent and "SA-DQN (PGD)" agent respectively. The attack results obtained based on the adversarial perturbation methods 1 to 4 (reports from method 1 to method 4) are shown in Table 1.

[0200] Table 1

[0201]

[0202]

[0203] Among them, environments 1 to 4 correspond to the average cumulative reward of 50 episodes of the competition, with the reward of method 1 to method 4 being the cumulative reward of the clean state. The reward of method 1 is the cumulative reward of the clean state, the reward of method 2 is the cumulative reward of the perturbation and interference of 10 time steps based on PGD, the reward of method 3 is the cumulative reward based on generative adversarial network, and the reward of method 4 is the cumulative reward based on generative adversarial network based on MDP.

[0204] As can be seen from Table 1, the "SA-DQN(PGD)" agent, i.e., agent 3, is the most robust, while the "DQN(vanilla)" agent, i.e., agent 1, has the worst robustness. Furthermore, when the given perturbation budget is 4 / 255, i.e., the perturbation budget is relatively large, all adversarial perturbation methods can obtain adversarial examples with high adversarial strength. However, when the perturbation budget is very small, differences in adversarial strength appear between different adversarial perturbation methods. The perturbation adversarial method 4 provided in this application embodiment, compared to the other three perturbation adversarial methods, obtains lower rewards in all four games, especially when the perturbation budget is 1 / 255 and 2 / 255.

[0205] Here, in order to further demonstrate the effectiveness of the perturbation countermeasure method 4 provided in the embodiments of this application, the improved intelligent agent of intelligent 3, namely intelligent agent 4, is selected to carry out an adversarial attack. The attack results are shown in Table 2.

[0206] Table 2

[0207]

[0208]

[0209] As can be seen from Table 2, the cumulative reward of the adversarial perturbation method 4 provided in this application embodiment is 0 when the attack perturbation budget is 4 / 255 in the runaway environment (environment 4), while the rewards of adversarial perturbation method 2 and adversarial perturbation method 3 are 2894.0 and 11390.0, respectively; therefore, the adversarial sample obtained by the adversarial perturbation method provided in this application embodiment has a high adversarialness.

[0210] Understandably, combining Markov decision processes with generative adversarial networks (GANs) is gradient-free and does not require access to the structure and parameters of deep reinforcement learning models, thus improving the efficiency and feasibility of acquiring adversarial examples. Furthermore, by cumulatively rewarding clean states with adversarial perturbations, the feedback from all attack frames is integrated, leveraging not only the vulnerabilities of DNNs but also the vulnerabilities of deep reinforcement learning models' MDPs. This effectively generates adversarial examples with high adversarial strength.

[0211] In the embodiments of this application, "g" means "g".

[0212] The following description continues to illustrate the exemplary structure of the sample acquisition device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software module stored in the sample acquisition device 255 of the memory 250 may include:

[0213] The state acquisition module 2551 is used to acquire the i-th clean state corresponding to the i-th time step, where i is a positive integer;

[0214] The state perturbation module 2552 is used to perform adversarial perturbation on the i-th clean state based on the cumulative reward to obtain the i-th perturbed state, wherein the cumulative reward of the i-th perturbed state is the minimum.

[0215] State decision module 2553 is used to make a decision based on the i-th disturbance state and obtain the i-th decision result;

[0216] The sample acquisition module 2554 is used to determine the (i+1)th clean state based on the i-th clean state and the i-th decision result when the i-th decision result does not meet the termination condition, and to acquire the (i+1)th decision result corresponding to the (i+1)th clean state, until the n-th decision result satisfies the termination condition, thereby obtaining an adversarial sample including n perturbation states, where n is a positive integer greater than 1.

[0217] In this embodiment of the application, the state perturbation module 2552 is further configured to: employ the original adversarial perturbation model to perform adversarial perturbation on the j-th clean state sample corresponding to the j-th time step to obtain the j-th estimated perturbation state, where j is a positive integer; obtain the cumulative decision reward corresponding to the j-th estimated perturbation state; train the original adversarial perturbation model based on the cumulative decision reward to obtain the adversarial perturbation model; and perform adversarial perturbation on the i-th clean state through the adversarial perturbation model and based on the cumulative reward to obtain the i-th perturbation state.

[0218] In this embodiment of the application, the state perturbation module 2552 is further configured to make a decision based on the j-th clean state sample to obtain the j-th decision result label, and make a decision based on the j-th estimated perturbation state to obtain the j-th estimated decision result; obtain a misjudgment loss based on the distribution difference between the j-th clean state sample and the j-th estimated perturbation state, and obtain an adversarial loss based on the difference between the j-th decision result label and the j-th estimated decision result; and obtain a hinge loss based on the j-th estimated perturbation state.

[0219] In this embodiment of the application, the state perturbation module 2552 is further used to combine the adversarial loss, the misjudgment loss, the hinge loss and the cumulative decision reward to train the original adversarial perturbation model and obtain the adversarial perturbation model.

[0220] In this embodiment, the state perturbation module 2552 is further configured to: perform adversarial perturbation on the kth clean state sample corresponding to the kth time step to obtain the kth estimated perturbation state, where k is a positive integer; make a decision on the kth estimated perturbation state to obtain the kth estimated decision reward corresponding to the kth estimated decision result; use the original cumulative reward model to obtain the kth estimated cumulative reward corresponding to the kth estimated perturbation state; based on the kth clean state sample and the kth estimated decision result, determine the (k+1)th estimated perturbation state corresponding to the (k+1)th clean state sample, and use the original cumulative reward model to obtain the (k+1)th estimated cumulative reward corresponding to the (k+1)th estimated perturbation state; train the original cumulative reward model based on the kth estimated decision reward, the kth estimated cumulative reward, and the (k+1)th estimated cumulative reward to obtain the cumulative reward model; and obtain the cumulative decision reward corresponding to the jth estimated perturbation state through the cumulative reward model.

[0221] In this embodiment of the application, the state perturbation module 2552 is further configured to obtain the label distribution corresponding to the j-th clean state sample; obtain the estimated distribution corresponding to the j-th estimated perturbation state; obtain the sampling state between the j-th clean state sample and the j-th estimated perturbation state, and obtain the sampling distribution corresponding to the sampling state; and determine the initial distribution difference between the label distribution and the estimated distribution, and the distribution difference between the sampling distribution, as the misjudgment loss.

[0222] In this embodiment of the application, the state perturbation module 2552 is further configured to acquire state discrimination samples and state discrimination labels; use the original discrimination model to acquire the state prediction category of the state discrimination samples; train the original discrimination model based on the difference between the state prediction category and the discrimination label to obtain the discrimination model; and acquire the label distribution, the estimated distribution and the sampling distribution through the discrimination model.

[0223] In this embodiment of the application, the state perturbation module 2552 is further configured to obtain the initial loss corresponding to the misjudgment loss, the hinge loss, and the cumulative decision reward; obtain the difference between the initial loss and the adversarial loss; train the original adversarial perturbation model; and obtain the adversarial perturbation model.

[0224] In this embodiment of the application, the state decision module 2553 is further configured to make a decision based on the i-th perturbation state through a deep reinforcement model to obtain the i-th decision result.

[0225] In this embodiment of the application, the sample acquisition device 255 further includes a sample application module 2555, which is used to acquire n decision rewards corresponding to the n perturbation states; and determine the evaluation result of the deep reinforcement model based on the comparison result of the n decision rewards and the reward threshold.

[0226] In this embodiment of the application, the sample application module 2555 is further configured to, when determining the deep reinforcement model as a model to be optimized based on the evaluation results, obtain n decision results corresponding to the n perturbation states; obtain n clean states based on the i-th clean state, determine the n decision actions corresponding to the n clean states as n decision result labels corresponding to the n perturbation states, wherein the n decision result labels correspond one-to-one with the n decision results; and optimize the deep reinforcement model based on the n decision difference between the n decision result labels and the n decision results.

[0227] In this embodiment of the application, the sample acquisition module 2554 is further configured to determine the i-th perturbation state as an anti-antibody when the i-th decision result satisfies the termination condition.

[0228] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the sample acquisition method described above in this application.

[0229] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the sample acquisition method provided in this application, for example... Figure 4 The sample acquisition method is shown.

[0230] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0231] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0232] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0233] As an example, executable instructions can be deployed to execute on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network.

[0234] In summary, through the embodiments of this application, by treating the process of acquiring adversarial examples as a sequential decision-making process, and by perturbing each clean state using a method that minimizes the cumulative reward, adversarial examples comprising n perturbed states are obtained. This makes the acquisition of adversarial examples independent of the agent's structure and parameters. Thus, even when the agent's structure and parameters are unavailable, adversarial examples can still be acquired, thereby improving the feasibility of acquiring adversarial examples. Furthermore, the method for acquiring adversarial examples provided in this application is a forward propagation approach, eliminating the need to calculate gradients and improving the efficiency of adversarial example acquisition. Moreover, this application's embodiment perturbs the clean state using cumulative rewards, integrating feedback from all attack frames; that is, the method for acquiring adversarial examples provided in this application not only utilizes the vulnerabilities of DNNs but also the vulnerabilities of the deep reinforcement learning model's MDP, effectively generating adversarial examples and enhancing their adversarial nature.

[0235] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A sample acquisition method, characterized in that, A server used in autonomous driving scenarios, the server being connected to a terminal via a network, the method comprising: Receive the i-th clean state corresponding to the i-th time step sent by the terminal, where i is a positive integer, and the clean state includes the current road conditions of the vehicle in the autonomous driving scenario; The i-th clean state is subjected to adversarial perturbation based on the cumulative reward to obtain the i-th perturbed state, wherein the cumulative reward of the i-th perturbed state is the minimum. The i-th perturbation state is decided by a deep reinforcement model to obtain the i-th decision result, and the action command corresponding to the i-th decision result is sent to the terminal through the network. When the i-th decision result does not meet the termination condition, the (i+1)-th clean state is determined based on the i-th clean state and the i-th decision result, and the (i+1)-th decision result corresponding to the (i+1)-th clean state is obtained, until the n-th decision result obtained meets the termination condition, thus obtaining an adversarial sample including n perturbation states, where n is a positive integer greater than 1, and the termination condition is the end of the autonomous driving scenario.

2. The method according to claim 1, characterized in that, The process of applying adversarial perturbation to the i-th clean state based on cumulative rewards to obtain the i-th perturbation state is achieved through an adversarial perturbation model. The training process of the adversarial perturbation model includes: Using the original adversarial perturbation model, adversarial perturbation is applied to the j-th clean state sample corresponding to the j-th time step to obtain the j-th predicted perturbation state, where j is a positive integer; Obtain the cumulative decision reward corresponding to the j-th estimated disturbance state; Based on the cumulative decision reward, the original adversarial perturbation model is trained to obtain the adversarial perturbation model.

3. The method according to claim 2, characterized in that, Before training the original adversarial perturbation model based on the accumulated decision reward to obtain the adversarial perturbation model, the method further includes: Make a decision based on the j-th clean state sample to obtain the j-th decision result label, and make a decision based on the j-th predicted disturbance state to obtain the j-th predicted decision result; Based on the distribution difference between the j-th clean state sample and the j-th predicted perturbation state, the misjudgment loss is obtained, and based on the difference between the j-th decision result label and the j-th predicted decision result, the adversarial loss is obtained. The hinge loss is obtained based on the j-th estimated disturbance state; The step of training the original adversarial perturbation model based on the accumulated decision reward to obtain the adversarial perturbation model includes: The original adversarial perturbation model is trained by combining the adversarial loss, the misjudgment loss, the hinge loss, and the cumulative decision reward to obtain the adversarial perturbation model.

4. The method according to claim 2 or 3, characterized in that, The acquisition of the cumulative decision reward corresponding to the j-th estimated disturbance state is achieved through a cumulative reward model; The training process of the cumulative reward model includes: The kth clean state sample corresponding to the kth time step is subjected to adversarial perturbation to obtain the kth predicted perturbation state, where k is a positive integer; Make a decision on the kth predicted disturbance state and obtain the kth predicted decision reward corresponding to the kth predicted decision result; Using the original cumulative reward model, the kth estimated cumulative reward corresponding to the kth estimated disturbance state is obtained; Based on the kth clean state sample and the kth predicted decision result, the k+1th estimated perturbation state corresponding to the k+1th clean state sample is determined, and the original cumulative reward model is used to obtain the k+1th estimated cumulative reward corresponding to the k+1th predicted perturbation state. Based on the k-th estimated decision reward, the k-th estimated cumulative reward, and the (k+1)-th estimated cumulative reward, the original cumulative reward model is trained to obtain the cumulative reward model.

5. The method according to claim 3, characterized in that, The method of obtaining the misjudgment loss based on the distribution difference between the j-th clean state sample and the j-th predicted perturbation state includes: Obtain the label distribution corresponding to the j-th clean state sample; Obtain the estimated distribution corresponding to the j-th estimated disturbance state; Obtain the sampling state between the j-th clean state sample and the j-th predicted perturbation state, and obtain the sampling distribution corresponding to the sampling state; The initial distribution difference between the label distribution and the estimated distribution, and the distribution difference between the sampling distribution, are determined as the misjudgment loss.

6. The method according to claim 5, characterized in that, The label distribution, the estimated distribution, and the sampling distribution are obtained through a discriminant model; The training process of the discriminative model includes: Obtain state discrimination samples and state discrimination labels; The original discrimination model is used to obtain the state prediction category of the state discrimination sample; Based on the difference between the predicted state category and the discriminant label, the original discriminant model is trained to obtain the discriminant model.

7. The method according to claim 3, characterized in that, The process of training the original adversarial perturbation model by combining the adversarial loss, the misjudgment loss, the hinge loss, and the cumulative decision reward to obtain the adversarial perturbation model includes: Obtain the initial loss corresponding to the misjudgment loss, the hinge loss, and the cumulative decision reward; Obtain the difference between the initial loss and the adversarial loss, train the original adversarial perturbation model, and obtain the adversarial perturbation model.

8. The method according to claim 1, characterized in that, After obtaining the adversarial sample comprising n perturbation states, the method further includes: Obtain the n decision rewards corresponding to the n perturbation states; Based on the comparison results of the n decision reward and return thresholds, the evaluation result of the deep reinforcement model is determined.

9. The method according to claim 8, characterized in that, After determining the evaluation result of the deep reinforcement model based on the comparison results of the n decision reward and return thresholds, the method further includes: When the deep reinforcement model is determined to be the model to be optimized based on the evaluation results, n decision results corresponding to the n perturbation states are obtained; Based on the i-th clean state, n clean states are obtained, and the n decision actions corresponding to the n clean states are determined as n decision result labels corresponding to the n perturbation states, wherein the n decision result labels correspond one-to-one with the n decision results; The deep reinforcement model is optimized based on the n decision result labels and the n decision differences corresponding to the n decision results.

10. The method according to any one of claims 1 to 3, characterized in that, After making a decision based on the i-th perturbation state and obtaining the i-th decision result, the method further includes: When the i-th decision result satisfies the termination condition, the i-th perturbation state is determined as an anti-antibody.

11. A sample acquisition device, characterized in that, A server for use in autonomous driving scenarios, the server being connected to a terminal via a network, the device comprising: The state acquisition module is used to receive the i-th clean state corresponding to the i-th time step sent by the terminal, where i is a positive integer, and the clean state includes the road conditions of the vehicle on the road in the autonomous driving scenario. The state perturbation module is used to perform adversarial perturbation on the i-th clean state based on the cumulative reward to obtain the i-th perturbed state, wherein the cumulative reward of the i-th perturbed state is the minimum. The state decision module is used to make a decision on the i-th perturbation state through a deep reinforcement model, obtain the i-th decision result, and send the action command corresponding to the i-th decision result to the terminal through the network; The sample acquisition module is used to determine the (i+1)th clean state based on the i-th clean state and the i-th decision result when the i-th decision result does not meet the termination condition, and to acquire the (i+1)th decision result corresponding to the (i+1)th clean state, until the n-th decision result satisfies the termination condition, thereby obtaining an adversarial sample including n perturbation states, where n is a positive integer greater than 1, and the termination condition is the end of the autonomous driving scenario.

12. The apparatus according to claim 11, characterized in that, The state perturbation module is implemented through an adversarial perturbation model; The training process of the adversarial perturbation model includes: Using the original adversarial perturbation model, adversarial perturbation is applied to the j-th clean state sample corresponding to the j-th time step to obtain the j-th predicted perturbation state, where j is a positive integer; Obtain the cumulative decision reward corresponding to the j-th estimated disturbance state; Based on the cumulative decision reward, the original adversarial perturbation model is trained to obtain the adversarial perturbation model.

13. The apparatus according to claim 12, characterized in that, Before training the original adversarial perturbation model based on the accumulated decision reward to obtain the adversarial perturbation model, the state perturbation module is further used for: Make a decision based on the j-th clean state sample to obtain the j-th decision result label, and make a decision based on the j-th predicted disturbance state to obtain the j-th predicted decision result; Based on the distribution difference between the j-th clean state sample and the j-th predicted perturbation state, the misjudgment loss is obtained, and based on the difference between the j-th decision result label and the j-th predicted decision result, the adversarial loss is obtained. The hinge loss is obtained based on the j-th estimated disturbance state; The step of training the original adversarial perturbation model based on the accumulated decision reward to obtain the adversarial perturbation model includes: The original adversarial perturbation model is trained by combining the adversarial loss, the misjudgment loss, the hinge loss, and the cumulative decision reward to obtain the adversarial perturbation model.

14. A sample acquisition device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the sample acquisition method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, It stores executable instructions for use by a processor to implement the sample acquisition method according to any one of claims 1 to 10.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement a sample acquisition method according to any one of claims 1 to 10.

Images