Image risk concept based on feature orthogonality erasing method and system

Abstract

The application discloses an image risk concept continuous erasing method and system based on feature orthogonality. In the first erasing session, the sample features of the image classes to be forgotten and the image classes to be reserved are decomposed in subspaces by singular value decomposition, and the forgetting subspace and the reserved subspace are constructed respectively. In the forward propagation process of the image classification model, the sample features of the image classes to be forgotten are orthogonally projected into the forgetting subspace to weaken their discriminability, and the sample features of the image classes to be reserved are aligned to the reserved subspace to enhance their stability. In the next erasing session, a new data deletion request task is obtained, the forgetting subspace is expanded and the reserved subspace is contracted, the cumulative erasing among multiple tasks is realized, and the integrity of the reserved knowledge is maintained. The feature layer orthogonal constraint mechanism constructed by the application can effectively balance the irreversible forgetting and the stability of the reserved knowledge, and is suitable for the risk concept continuous erasing demand of continuous image data deletion requests in real scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, in particular to a risk concept continuous erasing method based on feature orthogonality, which is suitable for deep irreversible erasing of risk concepts by image classification pre-training model when facing continuous image data deletion requests. BACKGROUND

[0002] With the wide application of large-scale image classification pre-training model in image recognition, target detection, generative content understanding and other fields, the data scale and diversity relied on by the model are constantly expanding, which inevitably contains user privacy information, sensitive scenes or outdated and non-compliant risk concepts. In order to meet the requirements of data protection laws and regulations, the model provider must ensure that the model has completely removed the impact of the risk concept related to the deletion request after receiving the user's data deletion request, in order to prevent potential information leakage and misuse. In order to achieve this goal, risk concept erasure (machine forgetting) as an effective security patch mechanism emerges as the times require, aiming to quickly erase specified information without retraining the entire model. Existing erasing methods are mostly designed for single deletion request, usually adopting strategies such as suppressing target class weights, introducing adversarial perturbations or using teacher networks to generate confusion labels, so that the model loses the ability to distinguish the erased concept at the output layer. However, these methods often only act on the output port, ignoring the residual effect in the deep feature expression process, resulting in the erasing effect only staying on the surface, and the risk concept information can be partially recovered through feature inversion or inter-layer migration, thereby posing a threat to the privacy security of the model.

[0003] More attention should be paid to the fact that in real-world scenarios, user data deletion requests often occur continuously and dynamically, and the model needs to continuously respond to new erasing tasks at different time points, which has given rise to the more challenging research direction of continuous risk concept erasing. Existing methods often cause information interference between new and old tasks due to the lack of systematic constraints on the feature space in continuous erasing scenarios: on the one hand, the residual features of the erased concept may be mixed with the new concept, and the erasing effect will weaken with the accumulation of tasks; on the other hand, repeated erasing operations will cause excessive interference to the reserved classes, resulting in damage to the original knowledge and disorder of the model's discrimination boundary, and in severe cases, even causing the overall performance to decline. Therefore, there is an urgent need for a risk concept continuous erasing method that can impose orthogonality constraints on the feature space and support dynamic subspace adaptive adjustment, in order to achieve deep irreversible erasing in the multi-task, continuous deletion request scenario, while ensuring the integrity and stability of the preserved knowledge. SUMMARY

[0004] In order to solve the problems of incomplete risk concept erasing and interference of preserved knowledge in the prior art, the present application proposes an image risk concept continuous erasing method and system based on feature orthogonality.

[0005] The technical solutions specifically adopted in the present invention are as follows:

[0006] In a first aspect, the present invention provides a method for continuously erasing risk concepts based on feature orthogonality, comprising:

[0007] S1. Obtain training datasets constructed for each of a plurality of consecutive data deletion request tasks, wherein the forgotten image category set and the retained image category set in the training dataset corresponding to the next data deletion request task are both subsets of the retained image category set in the training dataset corresponding to the previous data deletion request task;

[0008] S2. Execute all data deletion request tasks in sequence through continuous erasure sessions with different risk concepts, and gradually erase the classification ability of the image classification pre-training model for all forgotten image categories. In each risk concept continuous erasure session, the image classification pre-training model performs a forward propagation process on each image sample in the training dataset corresponding to the current task. Each network layer needs to construct a forgotten feature matrix and a retained feature matrix based on the extracted image sample features. Then, the weighted total loss of the forgotten subspace orthogonal projection alignment loss, the retained subspace projection alignment loss, and the image sample classification cross entropy loss of the retained image category are calculated based on the existing forgotten subspace and retained subspace of the current network layer and used to update the current network layer. At the same time, the forgotten feature matrix is ​​orthogonally projected through the existing forgotten subspace to calculate the residual matrix. The extended part of the forgotten subspace is extracted by performing singular value decomposition on the residual matrix and merged with the existing forgotten subspace. At the same time, the retained feature matrix is ​​subjected to singular value decomposition to extract a new retained subspace for the next session.

[0009] As a preferred embodiment of the first aspect above, in the training data sets constructed for each of the consecutive data deletion request tasks, each training data set needs to include a forgotten image category data set and a retained image category data set, and the image categories of the two do not overlap. The retained image categories contained in the training data sets corresponding to the data deletion request tasks executed sequentially decrease in arithmetic progression, and the retained image categories contained in the training data set corresponding to the last data deletion request task are the image categories that are finally retained and not erased.

[0010] As a preferred embodiment of the first aspect, in each risk concept continuous erasure session, The image classification pre-training model needs to perform a forward propagation process on each image sample in the training data set corresponding to the current task. Each network layer outputs the image sample features, and then all image sample features belonging to the forgotten image category are combined into the forgotten feature matrix of the current network layer, and all image sample features belonging to the retained image category are combined into the retained feature matrix of the current network layer. The forgotten feature matrix is ​​projected using the projection matrix corresponding to the existing forgotten subspace of the current network layer, and the residual matrix between the original forgotten feature matrix and the projected matrix is ​​calculated. The residual matrix is ​​then subjected to singular value decomposition, and the basis vector set consisting of the main singular vectors is extracted from the decomposed left singular vector matrix as the extended part of the forgotten subspace. The extended part is merged with the existing forgotten subspace to complete the extended update of the forgotten subspace; at the same time, the retained feature matrix of the current network layer is subjected to singular value decomposition, and the basis vector set consisting of the main singular vectors is extracted from the decomposed left singular vector matrix as the new retained subspace after contraction; in the first risk concept continuous erasure session, there is no existing forgotten subspace and retained subspace, and the residual matrix of the current network layer is equivalent to the original forgotten feature matrix.

[0011] As a preferred embodiment of the first aspect, the projection matrix corresponding to the forgotten subspace is obtained by multiplying a basis matrix consisting of a basis vector set corresponding to the forgotten subspace and its transposed matrix.

[0012] As a preferred embodiment of the first aspect, in each risk concept continuous erasure session, the three loss items in the total loss are calculated as follows:

[0013] After each network layer obtains the image sample features of each forgotten image category, it uses the projection matrix corresponding to its own forgotten subspace to project the image sample features, and calculates the sum of the squares of the L2 norms of the projected features of all network layers as the forgotten subspace orthogonal projection alignment loss of the image samples of a single forgotten image category;

[0014] After each network layer obtains the image sample features of each retained image category, it uses the projection matrix corresponding to its own retained subspace to project the image sample features, calculates the difference matrix of the image sample features before and after the projection, and then calculates the sum of the squares of the L2 norms of the difference matrices of all network layers as the retained subspace projection alignment loss of the image samples of a single retained image category;

[0015] After the image classification pre-training model performs image classification on the image samples of each reserved image category in the training data set, the cross entropy is calculated between the image samples and the true category value as the image sample classification cross entropy loss of the reserved image category.

[0016] As a preferred of the first aspect, when updating the image classification pre-training model, a learnable low-rank matrix is added in advance to each network layer of the image classification pre-training model, and the weight matrix originally pre-trained by the network layer is kept frozen, and in each risk concept continuous erasing session, the low-rank matrix on each network layer is optimized based on the total loss calculated after forward propagation, to realize low-rank adaptation.

[0017] In a second aspect, the present application provides a risk concept continuous erasing system based on feature orthogonality, which comprises:

[0018] A data set acquisition module is configured to acquire training data sets respectively constructed for a plurality of continuous data deletion request tasks, wherein the set of forgotten image categories and the set of retained image categories in the training data set corresponding to a next data deletion request task are subsets of the set of retained image categories in the training data set corresponding to a previous data deletion request task.

[0019] An erasing session module is configured to sequentially execute all data deletion request tasks through different risk concept continuous erasing sessions to gradually erase the classification ability of the image classification pre-training model for all forgotten image categories; in each risk concept continuous erasing session, the image classification pre-training model performs a forward propagation process on each image sample in the training data set corresponding to the current task, each network layer needs to construct a forgotten feature matrix and a retained feature matrix based on the image sample features extracted by the network layer, and then calculate a weighted total loss of a forgotten subspace orthogonal projection alignment loss, a retained subspace projection alignment loss and a cross-entropy loss of image sample classification of retained image categories and use the total loss to update the current network layer, and at the same time, calculate a residual matrix after orthogonal projection of the forgotten feature matrix through the existing forgotten subspace, extract an extended part of the forgotten subspace by singular value decomposition of the residual matrix and merge the extended part with the existing forgotten subspace, and at the same time, extract a new retained subspace by singular value decomposition of the retained feature matrix for use in the next session.

[0020] In a third aspect, the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, can implement the risk concept continuous erasing method based on feature orthogonality according to any one of the first aspect.

[0021] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, can implement the risk concept continuous erasing method based on feature orthogonality according to any one of the first aspect.

[0022] In a fifth aspect, the present application provides a computer electronic device comprising a memory and a processor.

[0023] The memory is used to store computer programs;

[0024] The processor is configured to implement the risk concept continuous erasure method based on feature orthogonality as described in any one of the first aspects above when executing the computer program.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention constructs an orthogonal constraint mechanism for the feature space. Each time a new image data deletion request is received, the sample features of the image categories to be forgotten and the retained image categories are first decomposed into subspaces using singular value decomposition, thereby constructing a forgotten subspace and a retained subspace, respectively. During the forward propagation phase of the image classification pre-training model, the image sample features of the categories to be forgotten are orthogonally projected into the forgotten subspace, thereby weakening their discriminability; at the same time, the image sample features of the retained categories are aligned to the retained subspace to enhance the stability of feature expression. In response to the continuous occurrence of risk concept deletion requests, the present invention proposes a dynamic subspace adaptation mechanism. By incrementally expanding the forgotten subspace and synchronously shrinking the retained subspace, cumulative erasure across tasks is achieved while maximally maintaining the integrity of the retained knowledge and the stability of the model performance. Compared with traditional risk concept continuous erasure methods, the present invention can achieve irreversible deep forgetting of the features of the forgotten image categories when executing multiple data deletion requests in succession, while effectively maintaining the stability of the feature expression of the retained image categories. The method design is simple and efficient. Experimental results show that compared with existing methods, the present invention can more thoroughly erase sensitive information in multi-round risk concept continuous erasure tasks, while improving the classification accuracy of the retained classes and the overall robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the implementation process of the continuous erasing method of image risk concept based on feature orthogonality;

[0028] Figure 2 Schematic diagram of the continuous erasure method for image risk concept based on feature orthogonality;

[0029] Figure 3 This is a flowchart of a specific method implementation in an embodiment of the present invention;

[0030] Figure 4 This is a diagram showing the implementation effect of the method of the present invention on the CASIA-Face100 image dataset;

[0031] Figure 5 Schematic diagram of the module composition of the continuous erasure system of image risk concept based on feature orthogonality;

[0032] Figure 6It is a structural diagram of computer electronic equipment. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0034] like Figure 1 As shown, the present invention provides a method for continuous erasure of image risk concepts based on feature orthogonality, and its implementation method includes two parts: dataset acquisition and erasure session. The specific implementation methods of these two parts are described in detail below.

[0035] The dataset acquisition part of the first part is specifically implemented as follows: obtain the training datasets constructed for each of the consecutive data deletion request tasks, where the forgotten image category set and the retained image category set in the training dataset corresponding to the next data deletion request task are both subsets of the retained image category set in the training dataset corresponding to the previous data deletion request task.

[0036] It should be noted that the image classification pre-training model in the present invention is defined as a pre-training dataset in the image The trained neural network model The specific form of the image classification pre-training model is not limited. A ResNet-based pre-training model that has been pre-trained on image classification tasks or a Transformer-based pre-training model that has been pre-trained on image classification tasks can be used. In the embodiment of the present invention, a Transformer-based pre-training model is preferably used, such as Face-Transformer or Vision-Transformer.

[0037] It should be noted that the present invention needs to execute multiple data deletion request tasks continuously, and each data deletion request task needs to pre-build a corresponding training data set. The image categories contained in each training data set can be divided into forgotten image categories and retained image categories. All forgotten image categories constitute a forgotten image category set, and all retained image categories constitute a retained image category set. At the same time, all image samples belonging to the forgotten image category constitute a forgotten image category data set, and all image samples belonging to the retained image category constitute a retained image category data set. The forgotten image category set and the retained image category set in different data deletion request tasks are different. In the data deletion request tasks executed sequentially, the forgotten image category set continues to expand, while the retained image category set continues to shrink. Corresponding to the changes in image categories in the training data set, the present invention proposes a dynamic subspace adaptation mechanism, which realizes cumulative erasure across tasks by incrementally expanding the forgotten subspace and synchronously shrinking the retained subspace, while maximizing the integrity of the retained knowledge and the stability of the model performance.

[0038] In an embodiment of the present invention, in the training data sets constructed for each of the consecutive data deletion request tasks, each training data set needs to include a forgotten image category data set and a retained image category data set, and the image categories of the forgotten image category data set and the retained image category data set do not overlap. The retained image categories included in the training data sets corresponding to the data deletion request tasks executed in sequence decrease in an equivariant manner, and the retained image categories included in the training data set corresponding to the last data deletion request task are the image categories that are finally retained and not erased.

[0039] For example, suppose there are K non-overlapping image categories, of which K1 image categories need to be retained, and K-K1 image categories need to be forgotten through the data erasure task. There are N data deletion request tasks in total. Then, in the first data deletion request task, the image categories in the forgotten image category set are (K-K1) / N, and the image categories in the retained image category set are K-(K-K1) / N. In the second data deletion request task, the (K-K1) / N image categories originally in the retained image category set in the previous data deletion request task need to be used as the forgotten image category set of the current task. Therefore, the image categories in the forgotten image category set still number (K-K1) / N (but they do not overlap with the forgotten image category set of the previous task), and the image categories in the retained image category set number K-2(K-K1) / N. Similarly, in the Nth data deletion request task, the image categories in the forgotten image category set number (K-K1) / N, and the image categories in the retained image category set number K1. At this point, the K1 image categories in the retained image category set in the last data deletion request task are the image categories that the image classification pre-training model ultimately retains without being erased.

[0040] In practice, the image categories in the set of retained image categories in the final data deletion request task are determined based on the actual data erasure task requirements. Different task requirements correspond to different forgotten and retained image categories. It is important to ensure that the ultimately retained image categories that can be accurately classified correspond exactly to the set of retained image categories in the final data deletion request task.

[0041] The erasure session part of the second part is specifically implemented as follows: all data deletion request tasks are executed in sequence through different risk concept continuous erasure sessions, gradually erasing the classification ability of the image classification pre-trained model for all forgotten image categories; in each risk concept continuous erasure session, the image classification pre-trained model performs a forward propagation process on each image sample in the training dataset corresponding to the current task. Each network layer needs to construct a forgotten feature matrix and a retained feature matrix based on the extracted image sample features, and then combine the existing forgotten subspace and retained subspace of the current network layer to calculate the weighted total loss of the forgotten subspace orthogonal projection alignment loss, the retained subspace projection alignment loss, and the image sample classification cross entropy loss of the retained image category and use it to update the current network layer. At the same time, the forgotten feature matrix is ​​orthogonally projected through the existing forgotten subspace to calculate the residual matrix. The extended part of the forgotten subspace is extracted by performing singular value decomposition on the residual matrix and merged with the existing forgotten subspace. At the same time, the retained feature matrix is ​​subjected to singular value decomposition to extract a new retained subspace for the next session.

[0042] As mentioned above, the forgotten subspace of the present invention gradually expands as the data deletion request task is continuously executed. Conversely, the retained subspace gradually shrinks as the data deletion request task is continuously executed. Through this dynamic subspace adaptation mechanism, cumulative erasure across tasks can be achieved. In order to maintain generality, the following describes the update mechanism of the forgotten subspace and the retained subspace based on the continuous erasure session of a single risk concept.

[0043] In the above-mentioned erasure session part of the present invention, the goal of the risk concept continuous erasure method is defined as: in each risk concept continuous erasure session, the image classification pre-training model is updated so that the updated image classification pre-training model cannot accurately classify the image samples of the forgotten category in the current session, while continuing to accurately classify the image samples of the retained category in the current session, and maintains the incorrect classification state for all previously forgotten image samples to ensure that forgetting is irreversible.

[0044] In addition, the above risk concept of continuous erasure session is defined as follows:

[0045] The time interval from the arrival of data from the current data deletion task to the arrival of data from the next data deletion task is a risk concept continuous erasure session.

[0046] During each risk concept continuous erasure session, the pre-trained image classification model performs forward propagation on each image sample in the training dataset corresponding to the current task. Each network layer outputs the corresponding image sample features for each image sample in vector form. The features of all image samples belonging to the forgotten image category are then combined into the forgotten feature matrix of the current network layer. This combines the features of all image samples in the forgotten image category dataset into a matrix. Simultaneously, the features of all image samples belonging to the retained image category are combined into the retained feature matrix of the current network layer. This combines the features of all image samples in the retained image category dataset into a matrix. Each network layer has already constructed a forgotten subspace and a retained subspace in the previous risk concept continuous erasure session. These can be referred to as the existing forgotten subspace and the existing retained subspace. Both the forgotten subspace and the retained subspace are a set of basis vectors consisting of the principal singular vectors after singular value decomposition. The basis vectors of each subspace can be combined into a matrix-like basis matrix, which is used to construct the projection matrix that maps the matrix to its own subspace. The projection matrix corresponding to the forgotten subspace is obtained by multiplying the basis matrix consisting of the basis vector set corresponding to the forgotten subspace and its transpose matrix. Similarly, the projection matrix corresponding to the retained subspace is obtained by multiplying the basis matrix consisting of the basis vector set corresponding to the retained subspace and its transpose matrix. Therefore, in the current risk concept continuous erasure session, the projection matrix corresponding to the existing forgotten subspace of the current network layer can be used to project the forgotten feature matrix obtained in the current session (denoted as the original forgotten feature matrix). The residual matrix between the original forgotten feature matrix and the projected matrix is ​​then calculated. The residual matrix is ​​then subjected to singular value decomposition. The basis vector set consisting of the principal singular vectors (the first k vectors with the largest singular values, where k is an optimizable parameter) is extracted from the decomposed left singular vector matrix as the expanded portion of the forgotten subspace. This expanded portion is then merged with the existing forgotten subspace to complete the expanded update of the forgotten subspace. Simultaneously, the retained feature matrix of the current network layer is subjected to singular value decomposition, and the basis vector set consisting of the principal singular vectors is extracted from the decomposed left singular vector matrix as the new retained subspace after contraction. Therefore, the expanded forgotten subspace and the newly shrunk retained subspace in the current session can be used as the next risk concept to continuously erase the existing forgotten subspace and retained subspace in the session.

[0047] It is important to note that in the first risk concept continuous erasure session, there is actually no existing forgotten subspace or retained subspace. At this time, the projection matrices corresponding to the existing forgotten subspace and retained subspace can be considered all-zero matrices. Therefore, in the first risk concept continuous erasure session, under the all-zero matrix projection, for each network layer, the residual matrix of the current network layer is equivalent to the original forgotten feature matrix. In other words, the forgotten feature matrix obtained in the first session can be directly subjected to singular value decomposition. After extracting the basis vector set consisting of the main singular vectors from the decomposed left singular vector matrix, it can be used as the initial forgotten subspace. Similarly, the retained feature matrix obtained in the first session can be directly subjected to singular value decomposition. After extracting the basis vector set consisting of the main singular vectors from the decomposed left singular vector matrix, it can be used as the initial retained subspace.

[0048] See also Figure 2 As shown in the figure, the iterative principle of the above-mentioned risk concept continuous erasure method based on feature orthogonality of the present invention is demonstrated: first, in the Risk concept continues to erase the session, using the A pre-trained image classification model after forgetting obtained from a continuous erasure session of risk concepts The feature matrices of the image dataset of the forgotten category and the image dataset of the retained category in the current risk concept continuous erasure session are extracted respectively, and the singular value decomposition is performed on them to obtain the expanded forgotten category subspace and the shrunk retained category subspace; in the execution of When a data deletion request task is requested, during the forward propagation of the image classification pre-trained model, the parameters of the image classification pre-trained model itself need to be fixed and a learnable low-rank matrix needs to be added to each layer. The model with the added low-rank matrix is ​​then used to extract sample features of the forgotten category image data and the retained category image data respectively. The former is orthogonally projected into the orthogonal complement of the current forgotten subspace to weaken its discriminability, while the latter is projected and aligned into the current retained subspace to enhance its stability. This approach can achieve cumulative forgetting between multiple tasks while maintaining the integrity of the retained knowledge.

[0049] Furthermore, in the present invention, to improve the efficiency of parameter optimization for image classification pre-trained models and significantly reduce computing resource requirements while preserving pre-trained model performance, low-rank adaptation (LoRA) technology can be employed when updating image classification pre-trained models. Specifically, prior to model optimization, learnable low-rank matrices are pre-added to all network layers of the image classification pre-trained model, while the weight matrices obtained from the original pre-training of the network layers remain frozen. In each risk concept continuous erasure session, the low-rank matrices on each network layer are optimized based on the total loss calculated after forward propagation to achieve low-rank adaptation. LoRA technology itself is prior art, and specific implementation methods can be implemented by reference to prior art practices.

[0050] During low-rank adaptation in each risk concept continuous erasure session, the low-rank matrix parameter optimization needs to be calculated based on the total loss value calculated for the current session. As mentioned above, the total loss function includes three loss terms: the orthogonal projection alignment loss of the forgotten subspace, the projection alignment loss of the preserved subspace, and the cross-entropy loss of the image sample classification that preserves the image category. The calculation methods for each of these can be implemented as follows:

[0051] A) Forgotten Subspace Orthogonal Projection Alignment Loss: After each network layer obtains the image sample features of each forgotten image category, it projects the image sample features using the projection matrix corresponding to its own existing forgotten subspace. The sum of the squares of the L2 norms of the projected features of all network layers is calculated as the forgotten subspace orthogonal projection alignment loss for the image samples of a single forgotten image category.

[0052] B) Preserved Subspace Projection Alignment Loss: After each network layer obtains the image sample features of each retained image category, it projects the image sample features using the projection matrix corresponding to its own preserved subspace. The difference matrix of the image sample features before and after the projection is calculated, and the sum of the squares of the L2 norms of the difference matrices of all network layers is calculated as the preserved subspace projection alignment loss for the image samples of a single retained image category.

[0053] C) Image sample classification cross entropy loss of the retained image category: After the image classification pre-training model performs image classification on the image sample of each retained image category in the training dataset, the cross entropy is calculated with the true category value as the image sample classification cross entropy loss of the retained image category.

[0054] Therefore, based on the above-described method for continuous erasure of image risk concepts based on feature orthogonality, it can achieve deep irreversible forgetting of risk concepts for the image classification pre-training model in the scenario of continuous image data deletion requests, but still retain the ability to accurately classify non-risk concepts (i.e., image categories that need to be retained).

[0055] In order to enable those skilled in the art to better understand the specific implementation of the present invention, the specific implementation process of the above-mentioned image risk concept continuous erasing method based on feature orthogonality is demonstrated below through a preferred embodiment.

[0056] like Figure 3 As shown, in a preferred embodiment of the present invention, the above-mentioned image risk concept continuous erasing method based on feature orthogonality is implemented through the following steps S1 to S6.

[0057] S1. Get an image classification pre-training model and its corresponding image pre-training dataset , where the image pre-training dataset It is a dataset that includes multiple image categories and image samples. A series of training datasets for data deletion request tasks are divided according to image categories. Each training dataset includes a dataset of forgotten image categories and a dataset of retained image categories, and the image categories of the two do not overlap. The forgotten image category set and the retained image category set in the training dataset corresponding to the next data deletion request task are both subsets of the retained image category set in the training dataset corresponding to the previous data deletion request task. The retained image categories contained in the training datasets corresponding to the sequentially executed data deletion request tasks decrease in arithmetic progression. The retained image categories contained in the training dataset corresponding to the last data deletion request task are the image categories that are finally retained and not erased.

[0058] For ease of understanding, the above image pre-training dataset It is considered to be composed of training data sets corresponding to N data deletion request tasks. , among which The training dataset for the data deletion request task is , is the image training sample set, is a set of image category labels for image training samples, where each image training sample corresponds to a category label; Only in The risk concept is continuously erased in the session and used for the data deletion request task of the image classification pre-training model.

[0059] No. Training dataset for data deletion request tasks It contains two parts, namely and ,in For the forgotten image category dataset, For the retained image category dataset, the image categories of the forgotten dataset and the retained dataset do not overlap. Training dataset for data deletion request tasks middle, and All That is, it is necessary to further divide a portion of the retained image categories in the previous data deletion request task into the forgotten image categories in the next data deletion request task. In an embodiment of the present invention, the number of retained image categories in each data deletion request task is preferably kept in an arithmetic progression.

[0060] S2. Continuously receive multiple data deletion request tasks. Only one data deletion request task is processed in each risk concept continuous erasure session. The time interval from the arrival of data from the current data deletion task to the arrival of data from the next data deletion task is considered a risk concept continuous erasure session.

[0061] S3. In the first risk concept continuous erasure session, the image samples of the forgotten category and the image samples of the retained category are used to construct feature matrices representing their feature subspaces respectively, and the feature matrices are decomposed by singular value decomposition to obtain the forgotten subspace and the retained subspace; the feature subspace refers to the vector space spanned by the main features of the forgotten category or the retained category samples. The forgotten subspace is used to capture the features of the image category to be forgotten, and the retained subspace is used to capture the features of the image category to be retained.

[0062] Among them, the feature matrix composed of the image samples of the forgotten category refers to the The image classification pre-training model after the risk concept is continuously erased is used as the feature extractor. The image samples of the forgotten categories in the risk concept continuous erasure session are mapped to a feature set formed by a representation space, and the image sample features of all the forgotten categories are superimposed to form the feature matrix of the forgotten categories. , which is the aforementioned forgotten feature matrix.

[0063] The feature matrix composed of image samples of the retained category is the The image classification pre-training model after the risk concept is continuously erased is used as the feature extractor. The image samples of the retained category in the continuous erasure session of the risk concept are mapped to a feature set formed by a representation space, and the image sample features of all the image samples of the retained category are superimposed to form a feature matrix of the retained category. , that is, the aforementioned retained feature matrix.

[0064] For the forgetting class, the singular value decomposition is defined as follows:

[0065]

[0066] in represents the left singular vector matrix of the singular value decomposition, represents the right singular vector matrix of the singular value decomposition, represents the singular value diagonal matrix of the singular value decomposition, and the subscript f represents the forgotten category.

[0067] Similarly, for the retained class, the singular value decomposition is defined as follows:

[0068]

[0069] in represents the left singular vector matrix of the singular value decomposition, represents the right singular vector matrix of the singular value decomposition, represents the diagonal matrix of singular values ​​of the singular value decomposition, and the subscript r represents the retained category.

[0070] S4. During the forward propagation of the image classification pre-training model, the features of the image samples in the forgotten category are orthogonally projected onto the orthogonal complement of the forgotten subspace to calculate the forgotten subspace orthogonal projection alignment loss in order to weaken its discriminability. Simultaneously, the features of the image samples in the retained category are projected and aligned onto the retained subspace to enhance its stability by calculating the retained subspace projection alignment loss. The forward propagation process of the image classification pre-training model refers to the layer-by-layer feature extraction process of the input image samples through each layer of the neural network model; the features of the sample samples in the forgotten category and the features of the sample samples in the retained category refer to the intermediate feature representations of the image samples in each layer of the neural network model.

[0071] In the embodiment of the present invention, in order to avoid loss of generality, Represents the last risk concept to erase the session, Represents the current risk concept to continuously erase the session, Represents the next risk concept to continuously erase the session. The above S4 step specifically includes the following sub-steps:

[0072] S41, use Risk concepts are continuously erased after the session ends. Image classification pre-training model For any layer Extract features from image samples of forgotten categories , using the projection matrix defined as follows Perform an orthographic projection:

[0073]

[0074] in represents the basis matrix of the forgotten subspace, which is obtained by Before extraction The singular vectors constitute a basis vector set and are expressed in matrix form. The number k of principal singular vectors added to the basis vector set can be determined by the reconstruction error method and the allowed reconstruction error threshold. The principle of the reconstruction error method is to find a k as small as possible so that the original matrix reconstructed with the first k principal singular vectors (and their corresponding singular values ​​and right singular vectors) is The approximate matrix With the original matrix "Close enough" means the error between the two does not exceed a pre-set reconstruction error threshold. This error can be measured using the matrix's Frobenius norm. The reconstruction error threshold can be set based on actual needs; 1% is recommended.

[0075] In order to orthogonally project the features of the forgotten image samples to the orthogonal complement of the forgotten subspace to weaken its discriminability, the following optimization objectives are performed on the features of the forgotten image samples of this layer:

[0076]

[0077] in is the identity matrix, symbol " " means approaching.

[0078] The above optimization objective can be equivalent to minimizing the following forgotten subspace orthogonal projection alignment loss :

[0079]

[0080] in The number of network layers in the pre-trained model for image classification.

[0081] S42, use Risk concepts are continuously erased after the session ends. Image classification pre-training model For any layer Extract features from image samples of the reserved category , using the projection matrix defined as follows Perform an orthographic projection:

[0082]

[0083] in represents the basis matrix of the preserved subspace, which is obtained by Before extraction The singular vectors constitute a basis vector set and are expressed in matrix form. The number k of principal singular vectors of the basis vector set can be determined by the aforementioned reconstruction error method and the allowed reconstruction error threshold. The reconstruction error threshold can be set according to actual needs, and is recommended to be set to 1%.

[0084] In order to project and align the features of the image samples of the reserved class to the reserved subspace to enhance the stability thereof, the following optimization objective is performed on the features of the samples of the forgotten class of the layer:

[0085]

[0086] The above optimization objective can be equivalent to minimizing the reserved subspace projection alignment loss as follows :

[0087]

[0088] S43, using the first risk concept to continuously erase the image classification pre-training model after the erasing session The features of the image samples of the reserved class are extracted from the last layer, and are mapped to the classification space to calculate the classification cross-entropy loss of the image samples of the reserved image class with the image class label:

[0089]

[0090] wherein is a binary cross-entropy loss function, is the Logic vector directly output by the image classification pre-training model representing the image classification result, is the true value classification label of the image sample in One-hot form.

[0091] S44, parameter optimization is performed on the image classification pre-training model with the purpose of minimizing the loss function , wherein:

[0092]

[0093] wherein and are coefficients for balancing , and , is the number of image samples in the forgotten image class data set in the training data set of the current session, is the number of image samples in the reserved image class data set in the training data set of the current session.

[0094] S5. When there is a new data deletion request task, repeat step S3, use the training data set of the task to generate a new forgotten subspace and merge it with the previous forgotten subspace to achieve the expansion of the forgotten subspace, and at the same time generate a new retained subspace and update it with the previous retained subspace to achieve the contraction of the retained subspace, ensuring that the previously forgotten information is irreversibly forgotten and the retained information exists stably.

[0095] In an embodiment of the present invention, the above-mentioned step S5 specifically includes the following sub-steps:

[0096] S51. Obtain the training dataset corresponding to the new data deletion request task, including the new forgotten category image dataset and its image labels and a new reserved category image dataset and its image labels .

[0097] S52, use Risk concepts are continuously erased after the session ends. Image classification pre-training model For image datasets with forgotten categories and image datasets of the reserved categories Extract features and construct feature matrices for each forgotten category and the feature matrix of the retained category .

[0098] S53, use the projection matrix corresponding to the existing forgetting subspace of the current network layer to project the forgetting feature matrix, calculate the residual matrix of the original forgetting feature matrix and the projected matrix, and then perform singular value decomposition on the residual matrix. Specifically, it can be calculated as follows and The residual matrix of :

[0099]

[0100] in represents the projection matrix of the forgotten subspace, Represents the left singular vector matrix in the previous session Before The basis matrix corresponding to the basis vector set composed of singular vectors.

[0101] It should be noted that for the first session, In fact, it can be understood as a special form of the above residual matrix, that is, When regarded as an all-zero matrix, the above residual matrix Equivalent to the original forgotten feature matrix .

[0102] S54, singular value decomposition is performed on the residual matrix S54, singular value decomposition is performed on the residual matrix

[0103]

[0104] S54, singular value decomposition is performed on the residual matrix S54, singular value decomposition is performed on the residual matrix S54, singular value decomposition is performed on the residual matrix S54, singular value decomposition is performed on the residual matrix S54, singular value decomposition is performed on the residual matrix .

[0105] S55, singular value decomposition is performed on the feature matrix of the reserved category S55, singular value decomposition is performed on the feature matrix of the reserved category

[0106]

[0107] S55, singular value decomposition is performed on the feature matrix of the reserved category S55, singular value decomposition is performed on the feature matrix of the reserved category S55, singular value decomposition is performed on the feature matrix of the reserved category S55, singular value decomposition is performed on the feature matrix of the reserved category

[0108] S6, in the subsequent risk concept continuous erasing session, the steps of S3 to S5 are repeatedly executed, and the image classification pre-training model is continuously risk concept erased by using the training data set in the new data deletion request task.

[0109] It should be noted that the method steps shown in S1~S6 above can be realized in the form of computer programs or software function modules.

[0110] Therefore, based on the same inventive concept, as shown in Figure 5 The application also provides an image risk concept continuous erasing system based on feature orthogonality corresponding to the image risk concept continuous erasing method based on feature orthogonality provided by the above embodiment, which comprises:

[0111] A data set acquisition module is configured to acquire training data sets respectively constructed for a plurality of continuous data deletion request tasks, wherein the set of forgotten image categories and the set of reserved image categories in the training data set corresponding to the next data deletion request task are subsets of the set of reserved image categories in the training data set corresponding to the last data deletion request task.

[0112] The erasing session module is used for sequentially executing all data deletion request tasks in different risk concept continuous erasing sessions to gradually erase the classification ability of the image classification pre-training model for all forgotten image categories; in each risk concept continuous erasing session, the image classification pre-training model performs a forward propagation process on each image sample in a training data set corresponding to a current task, each network layer needs to construct a forgotten feature matrix and a reserved feature matrix based on the image sample features extracted by the network layer, and then calculate a forgotten subspace orthogonal projection alignment loss, a reserved subspace projection alignment loss and a weighted total loss of an image sample classification cross-entropy loss of a reserved image category, and use the total loss to update the current network layer, and calculate a residual matrix after the forgotten feature matrix is orthogonally projected through the existing forgotten subspace, extract an extended part of the forgotten subspace through singular value decomposition of the residual matrix and merge the extended part with the existing forgotten subspace, and extract a new reserved subspace through singular value decomposition of the reserved feature matrix for the next session.

[0113] In addition, based on the same inventive concept, as shown in Figure 6 The application further provides a computer electronic device corresponding to the image risk concept continuous erasing method based on feature orthogonality provided by the above-mentioned embodiment, which comprises a memory and a processor.

[0114] The memory is used for storing a computer program.

[0115] The processor is used for realizing the image risk concept continuous erasing method based on feature orthogonality as described above when the computer program is executed.

[0116] In addition, the logical instructions in the memory described above can be realized in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part of the prior art that contributes to the technical solutions or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the application.

[0117] Therefore, based on the same inventive concept, the application provides a computer readable storage medium corresponding to the image risk concept continuous erasing method based on feature orthogonality, and the storage medium stores a computer program. When the computer program is executed by a processor, the image risk concept continuous erasing method based on feature orthogonality as described above can be realized.

[0118] Therefore, based on the same inventive concept, the present application provides a computer program product comprising computer programs / instructions which, when executed by a processor, can implement the image risk concept continuous erasing method based on feature orthogonality as described above.

[0119] Specifically, in the computer readable storage medium of the above three embodiments, the stored computer programs are executed by the processor to perform the steps S1-S6 as described above.

[0120] It can be understood that the storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. various media that can store program codes.

[0121] It can be understood that the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0122] In addition, it should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method is only a logical functional division, and actual implementation can have another division mode, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0123] In order to better understand the technical effects that the method S1-S6 described above can achieve, the method described above will be applied to specific embodiments below, so that those skilled in the art can better understand the effects of the present application.

[0124] Embodiments

[0125] The following simulation experiments are conducted to verify the effectiveness of the risk concept continuous erasure method based on feature orthogonality on different tasks and datasets. The specific implementation method of this embodiment is as described in S1 to S6 above, and will not be repeated here. The performance advantages of the present invention are only demonstrated based on the experimental results.

[0126] This example uses the face recognition task dataset CASIA-Face100 and the image classification task dataset ImageNet-100 for verification. The specific image classification pre-training model used is Face-Transformer. On the CASIA-Face100 dataset, which contains a total of 100 image categories, four consecutive risk concept continuous erasure tasks (referred to as tasks 1 to 4) are set. Each task requires forgetting 20 categories, and the forgetting accuracy of the forgotten categories in the current risk concept continuous erasure task ( ), the retention accuracy of the retained categories in the current risk concept continuous erasure task ( ), the accuracy of the forgotten categories in the old tasks ( ) and comprehensive measurement indicators In this embodiment, and Set them to 1.0 and 0.2 respectively. Figure 4 The effect diagram of the implementation of the method of the present invention on the CASIA-Face100 image dataset is shown. In addition, as shown in Tables 1 and 2, compared with the current mainstream risk concept continuous erasure methods (such as L2, EWC, MAS, DER, SCRUB, etc.), the present invention achieves higher forgetting thoroughness and retention stability in each task stage, especially in Task 4, the forgetting accuracy is Reduced to 0.00%, retaining accuracy Maintained at 80.51%, comprehensive indicators It reaches 77.81%, while the forgetting accuracy of old tasks is maintained at 0.00%, which is significantly better than the comparison method.

[0127] Table 1 Implementation effect of the proposed method on the CASIA-Face100 dataset corresponding to tasks 1 and 2

[0128]

[0129] Table 2 Implementation effect of the proposed method on the CASIA-Face100 dataset corresponding to tasks 3 and 4

[0130]

[0131] At the same time, on the ImageNet-100 dataset, which contains 100 image categories, four consecutive risk concept erasure tasks (denoted as tasks 1 to 4) were set, each forgetting 20 categories, and the same evaluation indicators were used for comparison. The image classification pre-training model used on the ImageNet-100 dataset was Vision-Transformer. As shown in Tables 3 and 4, the comprehensive indicators of the proposed method in the four risk concept erasure tasks were Both are higher than the existing advanced methods, Task 4 stage is 0.10%, Maintained at 92.10%, comprehensive indicators The accuracy of forgetting the old tasks is maintained at 0.00%, which significantly shows that the present invention can maintain high-precision discrimination of the retained classes while ensuring thorough forgetting.

[0132] Table 3 Implementation results of the proposed method on the ImageNet-100 dataset corresponding to tasks 1 and 2

[0133]

[0134] Table 4 Implementation results of the proposed method on the ImageNet-100 dataset corresponding to tasks 3 and 4

[0135]

[0136] To sum up, compared with the existing technology, the embodiments of the present invention can achieve irreversible deep forgetting of forgotten categories in the scenario of continuous erasure of risk concepts of continuous image data deletion requests, and effectively ensure the integrity and stability of retained knowledge, verifying the superiority and practicality of the method of the present invention in multiple scenarios.

[0137] It should be understood that the present embodiment is only used to illustrate the present invention and is not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

Claims

1. A method for continuously erasing image risk concepts based on feature orthogonality, characterized in that: include: Obtaining training datasets constructed for each of a plurality of consecutive data deletion request tasks, wherein the forgotten image category set and the retained image category set in the training dataset corresponding to the next data deletion request task are both subsets of the retained image category set in the training dataset corresponding to the previous data deletion request task; All data deletion request tasks are executed sequentially through different risk concept continuous erasure sessions, gradually erasing the classification ability of the image classification pre-trained model for all forgotten image categories. In each risk concept continuous erasure session, the image classification pre-trained model performs a forward propagation process on each image sample in the training dataset corresponding to the current task. Each network layer needs to construct a forgotten feature matrix and a retained feature matrix based on the extracted image sample features. Then, the weighted total loss of the forgotten subspace orthogonal projection alignment loss, the retained subspace projection alignment loss, and the image sample classification cross entropy loss of the retained image category are calculated by combining the existing forgotten subspace and retained subspace of the current network layer and used to update the current network layer. At the same time, the forgotten feature matrix is ​​orthogonally projected through the existing forgotten subspace to calculate the residual matrix. The extended part of the forgotten subspace is extracted by singular value decomposition of the residual matrix and merged with the existing forgotten subspace. At the same time, the retained feature matrix is ​​subjected to singular value decomposition to extract a new retained subspace for the next session.

2. The method for continuously erasing image risk concepts based on feature orthogonality according to claim 1, characterized in that: In the training data sets constructed for multiple consecutive data deletion request tasks, each training data set needs to contain a forgotten image category data set and a retained image category data set, and the image categories of the two do not overlap. The retained image categories contained in the training data sets corresponding to the data deletion request tasks executed sequentially decrease in arithmetic progression, and the retained image categories contained in the training data set corresponding to the last data deletion request task are the image categories that are finally retained and not erased.

3. The method for continuously erasing image risk concepts based on feature orthogonality according to claim 1, wherein: In each risk concept continuous erasure session, the image classification pre-training model needs to perform a forward propagation process on each image sample in the training dataset corresponding to the current task. Each network layer outputs the image sample features. Then, the features of all image samples belonging to the forgotten image category are combined into the forgotten feature matrix of the current network layer, and the features of all image samples belonging to the retained image category are combined into the retained feature matrix of the current network layer. The forgotten feature matrix is ​​projected using the projection matrix corresponding to the existing forgotten subspace of the current network layer. The residual matrix between the original forgotten feature matrix and the projected matrix is ​​calculated. The residual matrix is ​​then subjected to singular value decomposition. The basis vector set consisting of the principal singular vectors is extracted from the decomposed left singular vector matrix as the extended part of the forgotten subspace. The extended part is merged with the existing forgotten subspace to complete the expansion and update of the forgotten subspace. Simultaneously, singular value decomposition is performed on the retained feature matrix of the current network layer, and the basis vector set consisting of the principal singular vectors is extracted from the decomposed left singular vector matrix as the new retained subspace after contraction. In the first risk concept continuous erasure session, there is no existing forgotten subspace or retained subspace. The residual matrix of the current network layer is equivalent to the original forgotten feature matrix.

4. The method for continuously erasing image risk concepts based on feature orthogonality according to claim 1, wherein: The projection matrix corresponding to the forgotten subspace is obtained by multiplying a basis matrix consisting of a basis vector set corresponding to the forgotten subspace and its transposed matrix.

5. The method for continuously erasing image risk concepts based on feature orthogonality according to claim 1, wherein: In each risk concept continuous erasure session, the three loss terms in the total loss are calculated as follows: After each network layer obtains the image sample features of each forgotten image category, it uses the projection matrix corresponding to its own forgotten subspace to project the image sample features, and calculates the sum of the squares of the L2 norms of the projected features of all network layers as the forgotten subspace orthogonal projection alignment loss of the image samples of a single forgotten image category; After each network layer obtains the image sample features of each retained image category, it uses the projection matrix corresponding to its own retained subspace to project the image sample features, calculates the difference matrix of the image sample features before and after the projection, and then calculates the sum of the squares of the L2 norms of the difference matrices of all network layers as the retained subspace projection alignment loss of the image samples of a single retained image category; After the image classification pre-training model performs image classification on the image samples of each reserved image category in the training data set, the cross entropy is calculated between the image samples and the true category value as the image sample classification cross entropy loss of the reserved image category.

6. The method for continuously erasing image risk concepts based on feature orthogonality according to claim 1, wherein: When updating the image classification pre-trained model, learnable low-rank matrices are added to all network layers of the image classification pre-trained model in advance, while the weight matrix obtained by the original pre-training of the network layer remains frozen. In each risk concept continuous erasure session, the low-rank matrix on each network layer is optimized based on the total loss calculated after forward propagation to achieve low-rank adaptation.

7. A system for continuously erasing image risk concepts based on feature orthogonality, characterized in that: include: a dataset acquisition module, configured to acquire training datasets constructed for each of a plurality of consecutive data deletion request tasks, wherein the forgotten image category set and the retained image category set in the training dataset corresponding to the next data deletion request task are both subsets of the retained image category set in the training dataset corresponding to the previous data deletion request task; The erasure session module is used to execute all data deletion request tasks in sequence through different risk concept continuous erasure sessions, and gradually erase the classification ability of the image classification pre-training model for all forgotten image categories; in each risk concept continuous erasure session, the image classification pre-training model performs a forward propagation process on each image sample in the training dataset corresponding to the current task. Each network layer needs to construct a forgotten feature matrix and a retained feature matrix based on the extracted image sample features, and then combine the existing forgotten subspace and retained subspace of the current network layer to calculate the weighted total loss of the forgotten subspace orthogonal projection alignment loss, the retained subspace projection alignment loss and the image sample classification cross entropy loss of the retained image category and use it to update the current network layer. At the same time, the forgotten feature matrix is ​​orthogonally projected through the existing forgotten subspace to calculate the residual matrix, and the extended part of the forgotten subspace is extracted by performing singular value decomposition on the residual matrix and merged with the existing forgotten subspace. At the same time, the retained feature matrix is ​​subjected to singular value decomposition to extract a new retained subspace for the next session.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for continuously erasing image risk concepts based on feature orthogonality as described in any one of claims 1 to 6 can be implemented.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for continuously erasing image risk concepts based on feature orthogonality as described in any one of claims 1 to 6 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the method for continuously erasing image risk concepts based on feature orthogonality as described in any one of claims 1 to 6 when executing the computer program.

Images