A dynamic step length based variable length chromosome evolution method

Abstract

The application discloses a dynamic step length-based variable-length chromosome evolution method and belongs to the technical field of intelligent optimization. The application dynamically changes the step length of the variable-length chromosome evolution method, uses a larger step length to explore the whole search space in the first several generations, and uses a dynamic step length to develop the local search space in the middle and later periods. The application realizes a good balance between exploration and development on the search space, effectively increases the convergence speed of the algorithm, and is less likely to fall into the local optimal value of the problem, so that the deep neural network can obtain a solution with faster convergence speed and better performance for the adversarial attack problem.

Description

Technical Field

[0001] This invention belongs to the field of intelligent optimization technology and relates to a variable-length chromosome evolution method based on dynamic step size. Background Technology

[0002] Adversarial machine learning refers to the deliberate deception of machine learning models by introducing carefully generated input data. These attacks exploit vulnerabilities in the model's decision-making process, causing the machine learning model to produce incorrect classifications or outputs. Deep neural networks, the most important model in machine learning to date, have performed exceptionally well in many tasks. However, research has found that deep neural network models are vulnerable to generated attacks. Therefore, the security of artificial intelligence is a major concern. Researchers conducted several experiments in computer vision (CV) and found that even though some generated samples appear almost identical to the original samples to the naked eye, trained neural networks can still make prediction errors, such as misclassifying "temple" as "ostrich," resulting in incorrect classifications. Similar situations exist in the field of deep reinforcement learning (DRL). For example, for a trained agent, even minor modifications to its observations are enough to cause it to change its original action choice, resulting in incorrect outputs.

[0003] In the field of deep reinforcement learning, adversarial attacks refer to the application of imperceptible non-random noise to the input of an agent, thereby altering its predictions. For a deep reinforcement learning agent whose input is an image, the noise manifests as changes to the image's pixels. To put it simply, this attack targets the agent's "brain," aiming to confuse it and cause it to make incorrect decisions. The agent's brain is its well-trained deep neural network. Therefore, this adversarial attack is essentially an attack on the deep neural network. As mentioned earlier, adversarial machine learning studies attacks generated by machine learning algorithms and how to defend against such attacks. Similarly, how to generate adversarial attacks on deep neural networks is also a worthy research question.

[0004] These types of problems are typical optimization problems. A trained deep neural network, upon receiving a given input, generates a corresponding action probability distribution and then selects the action with the highest probability. The purpose of an adversarial attack is to mislead the agent into making incorrect judgments by adding tiny, imperceptible noise to its input. Evolutionary algorithms (EAs) have become a popular tool for solving complex problems across multiple domains and have been successfully applied to attack generation problems. Because the input to a deep neural network is a series of images, the attack is implemented by modifying the pixel values ​​of certain pixels in the image. The specific steps of the algorithm are: initializing and generating a first-generation population, where each individual encodes a set of attack content consisting of modified pixels. Next, applying evolutionary operators—crossover and mutation—to this population generates a next-generation population, where each individual also encodes an optimized set of attack content consisting of modified pixels. Then, evaluation is performed, and individuals with poor performance (those that failed to launch a successful attack) are removed. After several iterations, the desired solution (capable of launching a successful attack) is generated. Since modifying the number of pixels is also an optimization objective, the encoding length (chromosome length) of each individual also needs to change. Compared to traditional optimization methods, evolutionary algorithms are more flexible and adaptable, making them suitable for solving complex real-world problems. However, existing variable-length chromosome methods have shortcomings in how they change chromosome length, for example:

[0005] (1) Slow convergence speed: The length changes by 1 each time, which makes this type of method very slow when exploring a large search space.

[0006] (2) It is easy to get trapped in local optima: because the step size of its change is short and fixed, it is easy to get trapped in a local optimum and fail to find the global optimum. Summary of the Invention

[0007] To address the shortcomings of current methods in solving such problems, this invention proposes a variable-length chromosome evolution method to address the adversarial attack problem of deep neural networks, with the following specific improvements:

[0008] To address the slow convergence speed of the original algorithm, this invention increases the step size of chromosome changes in the first few generations of the population; the specific generations and step size will be described in the technical solution. To address the problem of the original algorithm easily getting trapped in local optima, this invention adopts a dynamic step size in subsequent generations to resolve the imbalance between exploration and development in the original algorithm.

[0009] Based on this, the present invention can be roughly divided into two stages:

[0010] (1) In the previous generations of the population, a larger step size is adopted to change the chromosome length.

[0011] (2) In the remaining generations of the population, dynamic step size is used to change the chromosome length.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A method for the evolution of variable-length chromosomes based on dynamic step size, the specific steps of which are as follows:

[0014] Step 1: Define the problem model, determine the task objective and fitness function.

[0015] 1.1) In a standard deep reinforcement learning scenario, at time step t, the agent determines the environmental state s based on the observed environmental state. t Generate an action probability distribution and take an action a t and obtain rewards from the environment. t Deep neural network attacks refer to attacks on the environmental state s observed by an agent. t Add a certain amount of noise δ t This causes the agent to make incorrect actions. Noise δ t It consists of two parts: the number of pixels attacked and the magnitude of the pixel change, that is, the size of the change in value for each pixel. For the solution to this problem, the fewer pixels attacked, the better, and the lower the magnitude of the pixel change, the better; in other words, it needs to generate inconspicuous noise. The noise is finally added to the target image, specifically, the target image is an 84x84 pixel grayscale matrix with a grayscale range of 0 to 255.

[0016] 1.2) Therefore, it can be concluded that the goal of a deep neural network attack is to maximize the difference in the probability of the agent's actions before and after the attack, as specifically stated below:

[0017]

[0018] Where, δ t This indicates the initial state s at time step t. t The added perturbation is also the solution to the current task. In this case, o and e represent the actions taken by the trained agent before and after the attack, respectively. π(·|s t ) e The probability distribution of the agent's actions is represented by π(·|s t The probability of choosing action e when π(·|s) is given. t ) represents when the agent observes state s t When this happens, the probability distribution of all candidate actions will be generated, which can be described as follows:

[0019]

[0020] in, Indicates the agent's choice of action The probability is determined by the agent based on the environmental state s. t The sum of the probabilities of all candidate actions is 1.

[0021] It is worth noting that when the function value f in formula (1) is greater than 0, the attack is successful, indicating that the agent has been deceived and has taken different actions. The following discussion will focus on optimizing formula (1).

[0022] Step 2: Determine the encoding method

[0023] Since the purpose of this invention is to generate noise that is as inconspicuous as possible, i.e., the fewer the number of attack pixels and the lower the magnitude of pixel changes, the better. Therefore, the encoding method is as follows: the x-coordinate, y-coordinate, and attack magnitude of the attack pixel are recorded at three consecutive points on the chromosome. The x-coordinate refers to the position of the attack pixel relative to the bottommost pixel, whose coordinate is 1. The y-coordinate refers to the position of the attack pixel relative to the leftmost pixel, whose coordinate is 1. The attack magnitude refers to the change in grayscale value for that pixel. A smaller grayscale value is negative, and a larger grayscale value is positive. The original encoding uses continuous encoding to generate decimals between 0 and 1, which are then scaled and rounded to map to the required range. For example, if a decimal of 0.12452 is generated at chromosome point 1, then scaling 0.12452 and multiplying it by 84 yields 10.45968, which is rounded to 10. Point 2 is similar; let's assume the final result is 28. A value of 0.56124 was generated at point 3. Scaling 0.56124 and multiplying it by 510 yields 286.2324. Subtracting 255 gives 31.2324, which is then rounded to 31. The combined effect of these three points is to add a grayscale value of 31 to the pixel at x-coordinate 10 and y-coordinate 28. For example, if the original pixel's grayscale value is 23, adding noise will change the grayscale value to 54.

[0024] Step 3: Initialize the population

[0025] First, we need to determine the current chromosome length *l* and the maximum chromosome length *ml*. Both can be calculated by multiplying the maximum number of pixels in the image by 3. Here, multiplying by 3 is used because in step 2, three points correspond to one pixel attack. The multiplication by 3 will be omitted below; that is, three points are considered a step size, so each step size corresponds to one pixel, which more intuitively illustrates the changes in pixel attacks. However, the actual chromosome length is still 3 times the number of attack pixels. For example, an 84x84 pixel matrix has 7056 pixels, so the longest chromosome can have 21168 points. Of course, both of these can be manually set. These parameters can be considered as adjusting the difficulty of the problem. For example, being able to attack all pixels is simpler than being able to attack only 100 pixels, because the former can be directly replaced with a different image. After determining the initial and maximum chromosome lengths, N initial offspring are randomly generated. Each point on the offspring chromosome is a random real number between 0 and 1. As for the maximum chromosome length (maximum number of pixel attacks), it can be any number from 1 to 7056. The lower the value, the more difficult the attack problem becomes (suggested range: 10 to 100). The current chromosome length (current number of pixel attacks) can be set to any value from 1 to the maximum chromosome length, which determines the current number of pixel attacks in the initial offspring (suggested range: 8 to 80).

[0026] Step 4: Generate offspring

[0027] Because older methods used a fixed-length approach to chromosome length changes during offspring generation, this invention dynamically changes chromosome length. Specifically, a larger step size is used in the first few generations, while a dynamic step size is used in the remaining generations, thus balancing exploration and development. The specific steps are as follows:

[0028] 4.1) When the population generation is less than or equal to When rounded, the step length is... (Rounding), this allows the population to essentially cover the entire search space. For example, when ml equals 100, The value is 10, meaning that in the first 10 generations, the step size of chromosome changes is 10. This means there is a 50% probability of increasing by 10 and a 50% probability of decreasing by 10. If the length exceeds the maximum, the maximum length is taken, and vice versa. In this way, in the first 10 generations, all populations will be distributed across a solution space of 1-100.

[0029] 4.2) When the population generation is greater than When rounded, the step length is... (Rounded to the nearest integer), where l represents the current number of pixel attacks, which is the current chromosome length. This allows subsequent generations to explore the local search space. For example, in the 11th generation, (Rounded to the nearest whole number) 9. A chromosome of length 80 has a 50% probability of increasing by 9 and a 50% probability of decreasing by 9. Thus, in later generations, shorter chromosomes will change by smaller steps, and vice versa. Consequently, the population will explore the local solution space of the search space.

[0030] 4.3) Changes in chromosome length (number of attack pixels) indicate changes in the solution space region explored by each individual. The two steps above use dynamic step size to change the chromosome length, ensuring that all individuals are distributed more evenly throughout the search space. At this point, evolutionary operators—crossover and mutation—are applied again to enable this generation of the population to continue exploring the search space and evolve towards making better attacks.

[0031] Step 5: Evaluate candidate solutions

[0032] The newly generated N candidate solutions are evaluated using formula (1). The evaluation results determine whether the candidate solution can achieve a successful attack, and serve as the basis for subsequent selection.

[0033] Step 6: Select the next generation of candidate solutions

[0034] After the newly generated candidate solutions are evaluated using formula (1), if their performance is better than the original candidate solutions, they are replaced; otherwise, they are discarded. Through this selection process, candidate solutions with poor performance are eliminated, while those with better performance are retained.

[0035] Step 7: Output the optimized solution

[0036] When the preset number of iterations is completed, the population retains the N best-performing candidate solutions that can achieve a successful attack.

[0037] Furthermore, in step 4, the process of generating new candidate solutions through crossover mutation is as follows:

[0038] In the encoding method of step 2, every two candidate solutions generate new candidate solutions through crossover and mutation operations, which promotes the target task to converge towards a successful attack: the crossover operation is a single-point crossover, which recombines and distributes the information at the same point of two candidate solutions to generate new candidate solutions; the mutation operation is a single-point mutation, which changes part of the information at a certain point in a single candidate solution to promote the diversification of candidate solutions; the final number of new candidate solutions generated is N.

[0039] The beneficial effects of this invention are as follows: This invention employs a novel variable-length chromosome evolution method based on dynamic step size in the processing of deep neural network attack problems. In the first few generations of population evolution, a large step size is used to fully explore the search space. In the remaining generations, a dynamic step size is adopted to further explore the remaining search space. Compared to previous methods, this invention achieves faster convergence and is less prone to getting trapped in local optima, thus achieving better optimization results. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the framework of the present invention.

[0041] Figure 2 This is a schematic diagram of adversarial attacks against deep reinforcement learning agents.

[0042] Figure 3(a) is a schematic diagram before the attack.

[0043] Figure 3(b) is a schematic diagram of a successful attack. Specific implementation examples:

[0044] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0045] The method of this invention can be used to achieve successful deep neural network adversarial attacks. The method flow of this invention is as follows: Figure 1 As shown.

[0046] A diagram illustrating attacks using deep reinforcement learning is shown below. Figure 2 Figure 3(a) and Figure 3(b) show the keyframes of the Atari game Pong used in this invention before and after the attack. The deep reinforcement learning-based agent uses keyframes of the Atari game Pong as input to perform actions and obtain rewards. The deep neural network attack adds noise to the keyframes of the game, thus affecting the above process. Figure 3(a) and Figure 3(b) show the schematic diagrams before and after the attack in the Atari game Pong. In this game scenario, the agent can perform two actions: "move up" and "move down," and obtain rewards, i.e., points, by hitting the ping-pong ball back. Figure 3(a) shows the normal situation before the attack; a properly trained agent on the right will move up to obtain points when facing an approaching ping-pong ball. Figure 3(b) shows the situation after the successful attack; noise is added in front of the agent, causing the agent to mistakenly believe the ping-pong ball is on the left and move down, resulting in no points being awarded.

[0047] The embodiments of the present invention will be described in detail below (e.g. Figure 1 As shown), the specific steps include:

[0048] (1) Establish Atari game model and objective function: Determine the scenario of an agent playing the Atari game Pong based on deep reinforcement learning as the attack environment. The goal of the attack is to make the agent change its original action, or make a misjudgment.

[0049] (2) Determine the encoding method: see the main text for details.

[0050] (3) Initialization process: For this deep neural network attack scenario, the first generation population is randomly generated.

[0051] (4) The step size of chromosome changes is determined based on the current generation number of the population. In the first few generations, a large step size is used to quickly explore the global search space. In later generations, a dynamic step size is adopted to continue exploring the local search space. After changing the chromosome step size, evolutionary operators such as crossover and mutation are applied.

[0052] (5) Candidate solution evaluation: The candidate solutions obtained in the above process are evaluated on the objective function of the target attack task.

[0053] (6) Select the next generation of candidate solutions: merge the old candidate solutions and the new candidate solutions, and select the next generation of population to enter the next cycle.

[0054] (7) Output the optimized solution: When the loop condition is not met, that is, when the loop reaches the preset number of times, a solution that performs well on the current target task is obtained, that is, the attack can be successfully implemented on the target task. Finally, the solution set of the deep neural network adversarial attack is output.

Claims

1. A method for the evolution of variable-length chromosomes based on dynamic step size, characterized in that the method includes the following steps: Step 1: Define the problem model, determine the task objective and fitness function. 1.1) In a standard deep reinforcement learning scenario, at time step At that time, the agent acts based on the observed environmental state. Generate action probability distribution and take action. and get rewards from the environment. Deep neural network attacks refer to attacks on the environmental state observed by an agent. Add some noise Noise causes the agent to make incorrect actions; It consists of two parts: the number of pixels attacked and the magnitude of the pixel change, that is, the value of each pixel being changed. For the solution to this problem, the fewer pixels attacked, the better, and the smaller the magnitude of the pixel change, the better. Noise needs to be added to the target image at the end. 1.2) This leads to the conclusion that the goal of a deep neural network attack is to maximize the difference in the probability of the agent's actions before and after the attack, as specifically stated below: in, Indicates the time step relative to the original state The added perturbation is also the solution to the current task; in this case, and These represent the actions taken by the trained agent before and after the attack, respectively. The probability distribution of the agent's actions is: Choose action The probability of; This represents when the agent observes the state. When, the probability distribution of all candidate actions is generated; When the formula ( function values ​​in ) When the value is greater than 0, the attack is successful, indicating that the agent has been deceived and has taken different actions. Step 2: Determine the encoding method The encoding method is as follows: the x-coordinate, y-coordinate, and attack amplitude of the attack pixel are recorded at three consecutive points on the chromosome; where the x-coordinate refers to the position of the attack pixel relative to the bottommost pixel, and the bottommost pixel coordinate is 1; the y-coordinate refers to the position of the attack pixel relative to the leftmost pixel, and the leftmost pixel coordinate is 1; the attack amplitude refers to the magnitude of the change in grayscale value for that pixel; a smaller grayscale value is negative, and a larger grayscale value is positive; the original encoding adopts continuous encoding to generate decimals between 0 and 1, which are then mapped to the required range through scaling and rounding; Step 3: Initialize the population First, determine the current chromosome length. l and maximum chromosome length ml According to the encoding method in step 2, l and ml All images are multiplied by 3 based on the maximum number of pixels. The multiplication by 3 will be omitted below. In other words, three points are considered as a step size. In this way, each step size corresponds to one pixel, which more intuitively illustrates the changes in pixel attacks. After determining the initial length and maximum length of the chromosome, N initial offspring are randomly generated. Each point on the offspring chromosome is a random real number between 0 and 1. Step 4: Generate offspring The chromosome length is changed by using a dynamic step size. The first few generations use a larger step size to explore the global search space. The remaining generations use a dynamic step size to further explore the local search space, thereby balancing exploration and development. After the chromosome length changes, evolutionary operators such as crossover and mutation are applied again to make the population evolve in the direction of making better attacks. The specific steps of step 4 are as follows: 4.1) When the population generation is less than or equal to At that time, the step length is This allows the population to essentially cover the entire search space; 4.2) When the population generation is greater than At that time, the step length is , The number of existing pixel attacks is the same as the existing chromosome length, which allows subsequent populations to explore the local search space. 4.3) Changes in chromosome length indicate changes in the solution space region explored by each individual. Steps 4.1) to 4.2) use dynamic step size to change chromosome length, so that all individuals are more evenly distributed throughout the search space. At this time, evolutionary operators such as crossover and mutation are applied again to enable this generation of the population to continue exploring the search space and evolve in the direction of making better attacks. In step 4.3), the process of generating new candidate solutions through crossover mutation is as follows: In the encoding method of step 2, every two candidate solutions generate new candidate solutions through crossover and mutation operations, prompting the target task to continuously converge towards a successful attack: the crossover operation is a single-point crossover, which recombines and redistributes the information at the same point in two candidate solutions to generate new candidate solutions; the mutation operation is a single-point mutation, which changes part of the information at a certain point in a single candidate solution, promoting the diversification of candidate solutions; the final number of new candidate solutions generated is... ; Step 5: Evaluate candidate solutions Newly generated The candidate solutions are given by the formula ( The evaluation results determine whether the candidate solution can achieve a successful attack and will be used as the basis for the next selection. Step 6: Select the next generation of candidate solutions The newly generated candidate solutions are in the formula ( After evaluation, if the solution performs better than the original candidate solution, it is replaced; otherwise, it is discarded. Step 7: Output the optimized solution When the preset number of iterations is completed, the best-performing cells are retained in the population. A candidate solution that can achieve a successful attack.

2. The method for evolution of variable-length chromosomes based on dynamic step size according to claim 1, characterized in that, In step 1.2), the The description is as follows: in, Indicates the agent's choice of action The probability is determined by the agent based on the environmental state. The sum of the probabilities of all candidate actions is 1.

3. The method for evolution of variable-length chromosomes based on dynamic step size according to claim 1, characterized in that, In step 1.2), the The description is as follows: in, Indicates the agent's choice of action The probability is determined by the agent based on the environmental state. The sum of the probabilities of all candidate actions is 1.

4. A method for the evolution of variable-length chromosomes based on dynamic step size according to any one of claims 1-3, characterized in that, In step 3, the maximum chromosome length ml Set to 10~100; Current chromosome length l Set it to 8~80.

5. The method for evolution of variable-length chromosomes based on dynamic step size according to claim 4, characterized in that, The target image mentioned in step 1.1) is an 84x84 pixel grayscale matrix with a grayscale range of 0 to 255.

Images