Feature manipulation-based attack and defense methods for continuous learning systems

Abstract

The present invention belongs to the field of pattern recognition, machine learning, multi-task learning, and adversarial attack technology, and specifically relates to an attack and defense method based on feature manipulation for a continuous learning system, aiming to solve the problem of poor security and robustness of existing intelligent systems based on continuous learning. The method of the present invention comprises: obtaining a clean sample of an image; extracting features of the clean sample; obtaining a target sample and extracting features as target anchor features; generating adversarial samples through an attack sample generation algorithm based on the clean sample features and in combination with the target anchor features; training an image classification model through a continuous learning algorithm, and counting the classification accuracy corresponding to the clean sample during C-type task classification learning; adding a first matrix as a training sample in a ratio of 1:n, and retraining; classifying images based on the trained image classification model. The present invention improves the security and robustness of existing intelligent systems based on continuous learning.

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition, machine learning, multi-task learning, and adversarial attack technology, and specifically relates to an attack and defense method, system, and device based on feature manipulation for a continuous learning capability system. Background Art

[0002] Deep artificial neural networks can extract high-level features from raw data and use them as a basis to achieve tasks such as pattern detection, recognition, and classification. They show very strong potential in learning complex mapping rules. However, this ability is a "static" ability, that is, once training is completed, the mapping is usually fixed. When learning new tasks, deep artificial neural networks often destroy the mapping established in previous tasks and do not have the ability to learn continuously. In the field of machine learning, this is often referred to as "catastrophic forgetting." Many application scenarios require deep artificial neural networks to be able to learn new information and adjust themselves, but "catastrophic forgetting" is undoubtedly a shortcoming. "Continuous learning algorithms" came into being, aiming to balance the knowledge of new and old tasks so that artificial intelligence systems have the ability to learn continuously. Such artificial intelligence systems are called "continuous learning systems."

[0003] Now there are many continuous learning algorithms and artificial intelligence systems that are used to overcome "catastrophic forgetting". Such a system can actively adapt to the environment in actual scenarios and greatly improve the work efficiency of human-computer collaboration. However, since the learning ability of the intelligent system built on the basis of neural networks is not closed, its learning ability is completely exposed to the actual scenario and is vulnerable to third-party intrusion. At present, there are relatively few studies on the continuous learning process of continuous attacks, and even fewer people are concerned about the corresponding defense algorithms. In fact, these contents are indispensable for the implementation of continuous learning algorithms. Based on this, the present invention proposes an attack and defense method based on feature manipulation for a continuous learning system (or intelligent system) with continuous learning ability, such as an image classification model based on continuous learning, which can covertly affect the learning process of the continuous learning system and manipulate the learning results of the system. Summary of the invention

[0004] In order to solve the above problems in the prior art, that is, to solve the problem that the existing intelligent system based on continuous learning is exposed to actual scenarios, is easily exploited, attacked and misled, resulting in poor security and robustness, the present invention proposes an attack and defense method based on feature manipulation for a continuous learning system, the method comprising:

[0005] Step S10, obtaining training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned;

[0006] Step S20, extracting the features of the clean sample through a pre-built feature extraction network as clean sample features;

[0007] Step S30, obtaining training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extracting features of the target samples through the feature extraction network as target anchor features;

[0008] Step S40, based on the clean sample features and in combination with the target anchor point features, an adversarial sample of the class B task to be classified and learned is generated through a preset attack sample generation algorithm;

[0009] Step S50, deleting clean samples from the image training sample set, adding adversarial samples to the image training sample set, and training the image classification model through a continuous learning algorithm, and counting the classification accuracy of the clean samples corresponding to the image classification model during the C-type task classification learning;

[0010] Step S60, if the classification accuracy is lower than the set threshold, then add a neuron in the linear classification layer of the image classification model to identify categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to step S70; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer;

[0011] Step S70: classify the image to be classified based on the trained image classification model.

[0012] In some preferred embodiments, the feature extraction network and the image classification model are both constructed based on a deep neural network; wherein the feature extraction network is constructed based on a deep neural network with a linear classification layer removed.

[0013] In some preferred embodiments, the feature-level loss function of the image classification model during continuous learning is a loss function constructed based on a distance function; the distance function includes Euclidean distance.

[0014] In some preferred implementations, the attack sample generation algorithm is:

[0015]

[0016]

[0017]

[0018] Clip X，∈ {X′}(x, y)=min{γ,X(x,y)+∈,max{0,X(x,y)-∈,X′(x,y)}}

[0019] Among them, X clean , X represent clean samples, represents the adversarial sample obtained in the Nth iteration, J(·,·) represents the loss function, and h s Represents the target anchor feature, Clip X，∈ {X′} represents the cropping function, (x, y) represents the pixel coordinates, ∈ represents the noise perturbation intensity, α and γ represent the preset weight parameters, and F represents the feature extraction network. represents the adversarial sample obtained in the untrimmed N+1th iteration, It means to find the gradient of the clean sample X.

[0020] In some preferred embodiments, the continuous learning algorithm is an OWM continuous learning algorithm.

[0021] In a second aspect of the present invention, a feature manipulation-based attack and defense system for a continuous learning capability system is proposed, wherein the continuous learning capability system is an image classification model, and the system comprises: a clean sample acquisition module, a clean sample feature extraction module, a target anchor feature extraction module, an adversarial sample generation method, a continuous learning module, a defense optimization module, and an image classification module;

[0022] The clean sample acquisition module is configured to acquire training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned;

[0023] The clean sample feature extraction module is configured to extract the features of the clean sample through a pre-built feature extraction network as clean sample features;

[0024] The target anchor feature extraction module is configured to obtain training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extract features of the target samples through the feature extraction network as target anchor features;

[0025] The adversarial sample generation method is configured to generate adversarial samples of the class B task to be classified and learned based on the clean sample features and in combination with the target anchor point features through a preset attack sample generation algorithm;

[0026] The continuous learning module is configured to delete clean samples from the image training sample set, add adversarial samples to the image training sample set, and train the image classification model through a continuous learning algorithm, and count the classification accuracy of the clean samples corresponding to the image classification model during the classification learning of the C-type task;

[0027] The defense optimization module is configured to add a neuron in the linear classification layer of the image classification model if the classification accuracy is lower than a set threshold, for identifying categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to the image classification module; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer;

[0028] The image classification module is configured to classify the image to be classified based on the trained image classification model.

[0029] According to the third aspect of the present invention, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned feature manipulation-based attack and defense method for a continuous learning capability system.

[0030] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned feature manipulation-based attack and defense method for the continuous learning capability system.

[0031] Beneficial effects of the present invention:

[0032] The present invention improves the security and robustness of existing intelligent systems based on continuous learning.

[0033] 1) This invention proposes a new neural network attack algorithm, which does not need to directly attack or modify system parameters, but only uses the system's own learning ability to manipulate the ability and learning results of the continuous learning system, mainly targeting the training process of the neural network, rather than a static neural network model like traditional neural network attack algorithms. In terms of algorithm design, it has the advantages of strong scalability, easy operation and strong concealment;

[0034] 2) This invention will be the first to systematically quantify and analyze the robustness of mainstream continuous learning algorithms. Currently, in the field of continuous learning, the focus is still on proposing new algorithms and refreshing the continuous learning performance of neural networks, while there is a lack of systematic research on the robustness of continuous learning algorithms. Therefore, this invention will not only be a useful supplement, but may also provide a new perspective for the design of continuous learning algorithms.

[0035] 3) The present invention reveals the potential risks of existing continuous learning algorithms when applied in actual scenarios, and also provides effective defense strategies. Whether it is the future research of new continuous learning algorithms or the implementation of existing continuous learning algorithms, the present invention has positive significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0037] Figure 1 It is a flowchart of an attack and defense method based on feature manipulation for a continuous learning capability system according to an embodiment of the present invention;

[0038] Figure 2 An example diagram of adversarial samples generated by an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of an adversarial attack process for continuous learning according to an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of the implementation effect of an image classification model of an embodiment of the present invention after being attacked; wherein (a) is a comparison diagram of the accuracy of each task of the attacked image classification model and the control group after all tasks are learned; (b) is a line graph of the change in the test accuracy of the attacked image classification model and the control group for clean 3 during the learning process;

[0041] Figure 5 : is a schematic diagram of the implementation effect of defense optimization of an embodiment of the present invention; wherein, (a) is a comparison chart of the accuracy of each task of the image classification model after defense and the image classification model without attack after all tasks are learned; (b) is a comparison chart of the accuracy of each task of the image classification model after attack and the image classification model after defense after all tasks are learned; (c) is a line chart of the test accuracy change of the control group, the attacked image classification model and the defended image classification model for clean 3 during the learning process;

[0042] Figure 6 It is a structural diagram of a computer system of an embodiment of the present invention that is suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0045] The first embodiment of the present invention is an attack and defense method based on feature manipulation for a continuous learning capability system, wherein the continuous learning capability system is an image classification model, such as Figure 1 As shown, the method comprises the following steps:

[0046] Step S10, obtaining training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned;

[0047] Step S20, extracting the features of the clean sample through a pre-built feature extraction network as clean sample features;

[0048] Step S30, obtaining training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extracting features of the target samples through the feature extraction network as target anchor features;

[0049] Step S40, based on the clean sample features and in combination with the target anchor point features, an adversarial sample of the class B task to be classified and learned is generated through a preset attack sample generation algorithm;

[0050] Step S50, deleting clean samples from the image training sample set, adding adversarial samples to the image training sample set, and training the image classification model through a continuous learning algorithm, and counting the classification accuracy of the clean samples corresponding to the image classification model during the C-type task classification learning;

[0051] Step S60, if the classification accuracy is lower than the set threshold, then add a neuron in the linear classification layer of the image classification model to identify categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to step S70; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer;

[0052] Step S70: classify the image to be classified based on the trained image classification model.

[0053] In order to more clearly illustrate the feature manipulation-based attack and defense method for a continuous learning capability system of the present invention, each step in an embodiment of the method of the present invention is described in detail below with reference to the accompanying drawings.

[0054] In this invention, a hidden and delayed attack is proposed for the artificial neural network continuous learning system. The damage to the system is not immediately manifested, but the target task performance is suddenly reduced in a specific continuous learning stage. This poses a major challenge to the current continuous learning system and also provides ideas for the robust design of continuous learning algorithms.

[0055] To achieve the above objectives, the present invention: 1) presets an attack target task; constructs a preset feature extraction network independent of the continuous learning system, which is used to extract the feature vector corresponding to the target task information type of the continuous learning system; 2) uses the feature extraction network to define a certain metric to measure the key features in the sample and determine the key features for the preset attack target task; 3) based on the key features of the preset target task, fine-tunes the features of the training samples of the preset task, thus completing the covert attack operation on the preset target task in the continuous learning system. The specific process is as follows:

[0056] Step S10, obtaining training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned;

[0057] In this embodiment, image training samples of the continuous learning system (i.e., image classification model) during classification learning are collected to construct an image sample training set. The image training sample set contains M types of tasks to be classified and learned. For example, in this embodiment, the MNIST training set is used as the image training set, which contains 60,000 images of 10 handwritten digits, that is, the image sample training set of the present invention contains 10 types of tasks to be classified and learned from 0 to 9.

[0058] The training samples corresponding to the class B tasks to be classified and learned in the image training sample set are obtained as clean samples. For example, in this embodiment, the number 3 is selected as a clean sample (ie, a sample to be attacked and learned), referred to as "clean 3".

[0059] Step S20, extracting the features of the clean sample through a pre-built feature extraction network as clean sample features;

[0060] In this embodiment, the feature extraction network is constructed based on a deep neural network. The feature extraction network of the present invention takes a fully connected neural network based on deep learning as an example, wherein the fully connected neural network based on deep learning is preferably a three-layer fully connected neural network with a structure of [784-800-10], and the network is trained on the entire MNIST training set. [784-800-10], that is, the first layer of the network is an input layer of 784 neurons, which matches the data dimension of the training sample; the second layer is a hidden layer containing 800 neurons, and the last layer is a classification layer containing 10 categories. We first use the traditional multi-output method to train the data, and train a feature vector extractor (i.e., feature extraction network) on the MNIST training set using the fully connected network based on deep learning, and then remove the last linear classification layer of the network, and use the output of the penultimate layer as the feature of the data. That is, the feature extraction network in the present invention is constructed based on a deep neural network with the linear classification layer removed.

[0061] When training the feature extraction network, the Adam algorithm is used, the learning rate is 0.1, the weight decay rate is 0.0001, and the size of each batch is 256.

[0062] Step S30, obtaining training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extracting features of the target samples through the feature extraction network as target anchor features;

[0063] In this embodiment, a certain spatial point or a certain subspace in the feature space corresponding to the image sample training set can be specified as the target feature. The selection of the target feature depends on the specific needs, and the sample feature in a task other than the attack target task can be selected as the target feature. For example, in this embodiment, the number 5 is preferably used as the target sample, and the feature of the target sample is extracted as the target anchor feature.

[0064] Step S40, based on the clean sample features and in combination with the target anchor point features, an adversarial sample of the class B task to be classified and learned is generated through a preset attack sample generation algorithm;

[0065] In this embodiment, all the numbers 3 in the image sample training set are taken out, and the Euclidean distance is preferably used as the loss function. In other embodiments, other distances can be used as the loss function according to actual needs. That is, the loss function of the feature level of the image classification model during continuous learning is constructed based on the distance function.

[0066] In the present invention, the loss function of the image classification model at the feature level during continuous learning is defined as J = ‖h clean -h adv ‖ 2 , where h clean is the original feature of the number 3, that is, the clean sample feature, h adv The target feature is obtained by referring to the feature of number 5, that is, the adversarial sample feature. The feature extraction network is fixed and the adversarial sample of the number is iteratively updated using the following method:

[0067]

[0068]

[0069]

[0070] Clip X，∈ {X′}(x,y)=min{γ,X(x,y)+∈,max{0,X(x,y)-∈,X′(x,y)}} (4)

[0071] Among them, X clean , X represent clean samples, represents the adversarial sample obtained in the Nth iteration, J(·,·) represents the loss function, and h s Represents the target anchor feature, Clip X，∈ {X′} represents the cropping function, (x, y) represents the pixel coordinates, ∈ represents the noise perturbation intensity, α and γ represent the preset weight parameters, and F represents the feature extraction network. represents the adversarial sample obtained in the untrimmed N+1th iteration, It means to find the gradient of the clean sample X.

[0072] For example, if α is 0 and γ is 255, then Clip X，∈ {X′}(x,y)=min{255,X(x,y)+∈,max{0,X(x,y)-∈,X′(x,y)}}.

[0073] After N rounds of iterations, we get the adversarial sample X adv , Figure 2 The generated adversarial samples and the original samples are very similar and very concealed.

[0074] Step S50, deleting clean samples from the image training sample set, adding adversarial samples to the image training sample set, and training the image classification model through a continuous learning algorithm, and counting the classification accuracy of the clean samples corresponding to the image classification model during the C-type task classification learning;

[0075] In this embodiment, a feedforward neural network capable of continuous learning is constructed. Preferably, a fully connected neural network based on deep learning of the network structure [784-800-10] is used as an example as an image classification model, that is, the image classification model in the present invention is constructed based on a deep neural network. During continuous learning, clean samples are deleted from the image training sample set, and adversarial samples are added to the image training sample set. The network is preferably trained using the OWM continuous learning algorithm in the order of 0 to 9. In other embodiments, other continuous learning algorithms can be selected to train the network according to actual needs. At this time, when the continuous learning system is learning the number 3, we replace 90% of the clean samples (or all clean samples) with attack samples. The desired attack effect will be triggered when the neural network actually learns task 5. The attack process is as follows Figure 3 As shown, a is the normal continuous learning process, b is the attacked process, class B tasks are the attacked tasks, and class C tasks are the trigger points of the attack. Figure 4 This is the attack effect of the method of the present invention. After the attack, the performance of Task 3 (number 3) dropped sharply compared to normal continuous learning ( Figure 4 (a)). During the process, Task 3 performed normally when it was just learned, but the performance dropped sharply after learning the number 5 ( Figure 4 (b)). In addition, Figure 4 as well as Figure 5 The “control” here refers to the classification accuracy of each digit of the image classification model after training (i.e., the test accuracy) without adding adversarial samples to the image training sample set of the image classification model.

[0076] Step S60, if the classification accuracy is lower than the set threshold, then add a neuron in the linear classification layer of the image classification model to identify categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to step S70; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer;

[0077] Experiments have shown that adversarial samples can be generated with any sample, even with random noise. And the generated adversarial samples are effective, that is, they can be 100% misleading to the network that generates them. This gives us inspiration: random noise fills the entire sample space, and the total amount and density are much larger than the MNIST dataset. But if the sample is forcibly input into the trained network, the network will forcibly classify it. If we can "teach" the network to recognize these noises (MNIST task-unnecessary samples), it is equivalent to "squeezing out" the possible distribution space of adversarial samples, thereby enhancing the robustness of the model.

[0078] In this embodiment, the first step is to expand the structure of the network and add one more neuron in the final classification layer to teach the system what is the "nothing" category, that is, the rejection category. The second step is to add some auxiliary samples to each task in addition to the samples of the original task during the training process, and merge them into the task to learn the task. In this way, the defense work is completed. The specific processing process is as follows:

[0079] If the classification accuracy is lower than the set threshold, a head (neuron) is added to the linear classification layer to identify random noise, and the network structure becomes [784-800-11]. For various tasks in the image training sample set containing adversarial samples, random noise images are generated at a ratio of 1:n (for example, "number 0" originally had 100 training images. According to the ratio of 1:n, if n is 6, 600 random noise images are generated, that is, now "number 0" has a total of 700 training images), and the 11th category label is marked and incorporated into the image training sample set containing adversarial samples. Use the 11-head network to train this data-enhanced image training sample set. The effect after defense optimization is as follows. Figure 5 As shown, the bar graph represents the test accuracy of each task after all tasks are learned, and the line graph represents the test accuracy of the clean number 3 as the learning process progresses.

[0080] Among them, after the attack, the accuracy of the number 3 dropped from 86.93% to 17.13%, and the performance loss was 69.8%; after defense, the accuracy dropped from 86.93% to 38.61%, and the performance loss was 48.32%, which is about 0.7 times the original loss of 69.8%, that is, the performance loss was reduced by about 30% after defense.

[0081] Step S70: classify the image to be classified based on the trained image classification model.

[0082] In this embodiment, an image to be classified is obtained, and the image classification model trained by a continuous learning algorithm (i.e., if the classification accuracy is lower than a set threshold, the image classification model trained in step S60 is used, otherwise the image classification model trained in step S50 is used) is used to classify the image to be classified, and the classification result is output.

[0083] In addition, the feature manipulation-based attack and defense methods for continuous learning system of the present invention can also be used for attack and defense of other intelligent systems, such as image detection and recognition, according to actual application scenarios and application needs in other embodiments, which will not be elaborated here one by one.

[0084] A feature manipulation-based attack and defense system for a continuous learning capability system according to a second embodiment of the present invention, wherein the continuous learning capability system is an image classification model, and the system comprises: a clean sample acquisition module, a clean sample feature extraction module, a target anchor feature extraction module, an adversarial sample generation method, a continuous learning module, a defense optimization module, and an image classification module;

[0085] The clean sample acquisition module is configured to acquire training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned;

[0086] The clean sample feature extraction module is configured to extract the features of the clean sample through a pre-built feature extraction network as clean sample features;

[0087] The target anchor feature extraction module is configured to obtain training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extract features of the target samples through the feature extraction network as target anchor features;

[0088] The adversarial sample generation method is configured to generate adversarial samples of the class B task to be classified and learned based on the clean sample features and in combination with the target anchor point features through a preset attack sample generation algorithm;

[0089] The continuous learning module is configured to delete clean samples from the image training sample set, add adversarial samples to the image training sample set, and train the image classification model through a continuous learning algorithm, and count the classification accuracy of the clean samples corresponding to the image classification model during the classification learning of the C-type task;

[0090] The defense optimization module is configured to add a neuron in the linear classification layer of the image classification model if the classification accuracy is lower than a set threshold, for identifying categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to the image classification module; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer;

[0091] The image classification module is configured to classify the image to be classified based on the trained image classification model.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0093] It should be noted that the feature manipulation-based attack and defense system for the continuous learning capability system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations of the present invention.

[0094] An electronic device according to the third embodiment of the present invention comprises at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the feature manipulation-based attack and defense method for a continuous learning capability system as claimed in the claim above.

[0095] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, wherein the computer instructions are used to be executed by the computer to implement the feature manipulation-based attack and defense method for a continuous learning capability system as claimed in the claims above.

[0096] Technicians in the technical field can clearly understand that for the convenience and simplicity not described, the specific working process and related instructions of the feature manipulation-based attack and defense device, electronic device, and computer-readable storage medium for the continuous learning capability system described above can refer to the corresponding process in the aforementioned method example and will not be repeated here.

[0097] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system of a server suitable for implementing the system, method, and device embodiments of the present application. Figure 6 The server shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0098] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0099] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube, a liquid crystal display, etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a local area network card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0100] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the CPU 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to: a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, etc., or any suitable combination of the above.

[0101] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as C language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network or a wide area network, or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The terms "first", "second", etc. are used to distinguish similar objects rather than to describe or indicate a particular order or sequence.

[0104] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus / device.

[0105] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A feature manipulation-based attack and defense method for a continuous learning capability system, wherein the continuous learning capability system is an image classification model, characterized in that: The method includes: Step S10, obtaining training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned; Step S20, extracting the features of the clean sample through a pre-built feature extraction network as clean sample features; Step S30, obtaining training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extracting features of the target samples through the feature extraction network as target anchor features; Step S40, based on the clean sample features and in combination with the target anchor point features, an adversarial sample of the class B task to be classified and learned is generated through a preset attack sample generation algorithm; Step S50, deleting clean samples from the image training sample set, adding adversarial samples to the image training sample set, and training the image classification model through a continuous learning algorithm, and counting the classification accuracy of the clean samples corresponding to the image classification model when the image classification model is learning the C-type task classification; Step S60, if the classification accuracy is lower than the set threshold, then add a neuron in the linear classification layer of the image classification model to identify categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to step S70; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer; Step S70: classify the image to be classified based on the trained image classification model.

2. The feature manipulation-based attack and defense method for a continuous learning capability system according to claim 1, characterized in that: The feature extraction network and the image classification model are both constructed based on deep neural networks; wherein the feature extraction network is constructed based on a deep neural network with a linear classification layer removed.

3. The feature manipulation-based attack and defense method for a continuous learning capability system according to claim 1, characterized in that: The loss function of the image classification model at the feature level during continuous learning is a loss function constructed based on a distance function; the distance function includes Euclidean distance.

4. The feature manipulation-based attack and defense method for a continuous learning capability system according to claim 3, characterized in that: The attack sample generation algorithm is: Clip X，∈ {X′}(X,y)=min{γ,X(X,y)+∈,max{0,X(x,y)-∈,X′(X,y)}} Among them, X clean , X represent clean samples, represents the adversarial sample obtained in the Nth iteration, J(·,·) represents the loss function, and h s Represents the target anchor feature, Clip X，∈ {X′} represents the cropping function, (x, y) represents the pixel coordinates, ∈ represents the noise perturbation intensity, α and γ represent the preset weight parameters, and F represents the feature extraction network. represents the adversarial sample obtained in the untrimmed N+1th iteration, It means to find the gradient of clean sample X.

5. The feature manipulation-based attack and defense method for a continuous learning capability system according to claim 1, characterized in that: The continuous learning algorithm is the OWM continuous learning algorithm.

6. A feature manipulation-based attack and defense system for a continuous learning capability system, wherein the continuous learning capability system is an image classification model, characterized in that: The system includes: a clean sample acquisition module, a clean sample feature extraction module, a target anchor feature extraction module, an adversarial sample generation method, a continuous learning module, a defense optimization module, and an image classification module; The clean sample acquisition module is configured to acquire training samples corresponding to the B-type tasks to be classified and learned in the image training sample set as clean samples; the image training sample set contains M types of tasks to be classified and learned; The clean sample feature extraction module is configured to extract the features of the clean sample through a pre-built feature extraction network as clean sample features; The target anchor feature extraction module is configured to obtain training samples corresponding to the C-type tasks to be classified and learned in the image training sample set as target samples, and extract features of the target samples through the feature extraction network as target anchor features; The adversarial sample generation method is configured to generate adversarial samples of the class B task to be classified and learned based on the clean sample features and in combination with the target anchor point features through a preset attack sample generation algorithm; The continuous learning module is configured to delete clean samples from the image training sample set, add adversarial samples to the image training sample set, and train the image classification model through a continuous learning algorithm, and count the classification accuracy of the clean samples corresponding to the image classification model during the classification learning of the C-type task; The defense optimization module is configured to add a neuron in the linear classification layer of the image classification model if the classification accuracy is lower than a set threshold, for identifying categories other than the M categories to be classified and learned; add the training samples of various tasks in the image training sample set containing adversarial samples and the first matrix to the image training sample set containing adversarial samples in a ratio of 1:n, and after the addition, train the image classification model with the added neurons until a trained image classification model is obtained; otherwise, jump to the image classification module; wherein the first matrix is ​​a pixel matrix constructed based on random noise; n is a positive integer; The image classification module is configured to classify the image to be classified based on the trained image classification model.

7. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to at least one of the processors; The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the feature manipulation-based attack and defense method for a continuous learning capability system as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the feature manipulation-based attack and defense method for a continuous learning capability system as described in any one of claims 1-5.

Images