A knowledge reconstruction method of visual deep model for intelligent security

Abstract

A knowledge reorganization method for visual deep models for intelligent security. First, visual data from security edge measurements is collected, model tasks are clarified, data is collected for annotation, and the target task model is trained. Then, the attribution graph is used to calculate the transferable scores between the target model and different pre-trained models, and the normalized transferable scores are used as the probability of model selection, and the model set is formed by replacement sampling; based on the federated learning paradigm, the target model is distributed to different nodes, and the feature extraction ability of the pre-trained model is transferred by representation distillation. After the target model is learned and transferred, knowledge is reorganized in the central node by averaging, and the reorganized model is distributed to the selected model node for further iteration. Finally, the target data set is used to fine-tune the target model, so as to learn a visual deep model that performs well on the target task in the field of intelligent security, and protects data privacy and model privacy.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent security and federated learning, and relates to a visual depth model knowledge reorganization method for intelligent security. Background Art

[0002] Smart security is a field involving multiple technologies such as face recognition, big data analysis, and video structuring. With the rapid development of technologies such as deep learning, the functions of smart security products are becoming more and more diversified, and security technology has been successfully applied in various scenarios such as transportation and communities.

[0003] However, the success of deep learning technology depends on two major factors: a large amount of labeled data and the development of a practical form of deep learning model reuse: using a pre-trained and publicly verified network model by others, thereby greatly reducing the workload of network training. In the field of smart security, a large amount of labeled data requires a lot of manpower costs, so high-quality labeled data is scarce; and the growing form of model sharing also faces a problem: a network with good performance requires huge resources, and even if the researchers or institutions who trained the network intend to make the model public, they cannot share their models with the public due to privacy protection, security or intellectual property issues. Existing model reuse methods, such as knowledge distillation, transfer learning, and domain adaptation, usually require pre-trained models before they can be used, so they cannot solve this problem in this case. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a visual depth model knowledge reorganization method for intelligent security. The purpose is to train a student model with the help of several decentralized teacher models. This student model focuses on specific target tasks in the security field, such as depth estimation and pedestrian re-identification. In order to ensure that the model information is not leaked during the training process, the details of the pre-trained model (including pre-trained tasks and data) are always kept private, and other training participants cannot obtain any relevant information. Such a problem setting poses a huge challenge to the reuse of federated models, because it is difficult to find useful teacher models for the target tasks. At the same time, the training fields of different teacher models are different, and the training of student models must consider balancing the knowledge aggregation from different teachers.

[0005] To this end, the present invention proposes a visual deep model knowledge reorganization technology for intelligent security to solve such problems. The method customizes a student model by using the pre-trained knowledge of a private teacher model. The present invention does not train the student model by imitating the teacher model, but filters and absorbs useful knowledge from the teacher model through a feature-based training mode. Feature analysis is widely used in the literature on network interpretability and is also used to reveal the transferability of the network. Based on feature analysis, the present invention proposes an adaptive training mode. Useful teacher models are selected without directly accessing the private teacher model. Specifically, the present invention uses the labeled data of the target task to train the student model, and calculates the representation similarity between the student model and the private teacher model to select knowledge. And the probability of selection is increased according to the similarity. The selected teacher model transfers knowledge to the student model through representation distillation inside the local node, and reorganizes knowledge by averaging, and then fine-tunes according to the target task data.

[0006] The present invention considers the actual scenario of model training in the field of intelligent security. Limited annotated data makes it extremely difficult to directly train a model with good generalization performance. Under the condition that the pre-trained model and its training data cannot be directly obtained, the present invention draws on the existing federated learning paradigm and proposes a visual deep model knowledge reorganization method for intelligent security. The technical solution of the present invention is:

[0007] A visual depth model knowledge reorganization method for security, comprising the following steps:

[0008] 1. Collect and train intelligent security data;

[0009] The present invention first collects visual data from security edge devices (such as cameras), clarifies specific model tasks, such as pedestrian re-identification, and annotates a small portion of the collected data. This small portion of annotated data will serve as the target task dataset D T , which contains M data samples and labels. Due to the environment in which security equipment is located and the expensive labeling cost, the data collected by security equipment is usually insufficient, so it is difficult to obtain a well-performing deep model by directly using this part of data for training and learning. In order for the security deep model to have certain knowledge about the target task, it is first necessary to use the collected target task dataset to train the target task model.

[0010] 2. Select model knowledge;

[0011] Pre-trained model {M1,M2,…,M N} is trained on different training data and different tasks, so the pre-trained model contains knowledge from different fields. And the pre-trained model and data are stored on different nodes. For the specific target task model of intelligent security, the transferability of knowledge of different pre-trained models is different. The knowledge of pre-trained models with better transferability is more helpful for the target task. In order to select a suitable model from several pre-trained models, the present invention first calculates the attribution graph of different models on the target data set. in It represents the attribution map of the kth layer of the nth model for the jth input. The attribution map is obtained by calculating the back-propagation gradient of a specific input. The attribution map reflects the model's attention area for the input. Therefore, the distance between the attribution maps of different models can be used as a measure of the transferability of model knowledge. The transferability distance between different models is calculated using the target dataset as follows:

[0012]

[0013] in, Indicates the calculation of cosine similarity. After calculating the attribution graph distance between the target model and the pre-trained model, the transferable score between the models can be calculated based on the attribution graph distance.

[0014] The present invention normalizes the transferable fraction into a discrete probability distribution {P1, P2, ..., P N}, the probability value represents the probability of selecting a certain pre-trained model. The present invention will be based on the probability distribution {P1, P2, ..., P N} Sample k models with replacement to form a model set S M This model set will be used in subsequent federal knowledge reorganization.

[0015] 3. Reorganize federal knowledge;

[0016] Without direct access to pre-trained models and datasets, in order to integrate the model set S M The present invention divides the federated knowledge reorganization into two stages: local knowledge reorganization and central knowledge reorganization, according to whether the process occurs on the edge node of the pre-trained model. The division of the two stages can avoid direct contact with the pre-trained model, thereby protecting the privacy of the pre-trained model and data.

[0017] First, the present invention transforms the target model M t Parameters Copy to pre-trained model M nThe node where the target model is located. For most deep models, the model can be divided into an encoder part and a decoder part. The encoder encodes the original input into a vector in the feature space. The feature extraction part is task-independent and has good generalization. The decoder decodes the vector in the feature space into an output related to the task. In the process of local recombination, the present invention forces the feature output of the target model to be consistent with the feature output of the pre-training, and the copy of the target model will learn the feature extraction ability of the pre-training model. Since the pre-training model M n and the target model M t The network structure may be heterogeneous, and the output features of the target model and the local pre-trained model may have different output dimensions. Therefore, the present invention uses three 1×1 convolution operations to align the feature output dimensions of different models. The loss calculation in the local reorganization process is as follows:

[0018]

[0019] Among them, X represents the pre-trained model M n The training data set, Represents the pre-trained model M n Parameters, represents the parameters of the local target model, and They represent the feature outputs of the pre-trained model and the target model after alignment. The local reorganization process will update the parameters of the local target model based on the loss:

[0020]

[0021] Among them, η t represents the learning rate for local reorganization.

[0022] After the local knowledge reorganization is completed, the parameters of the local target model It will be uploaded to the central node. After collecting the target model parameters of different nodes, the central node will reorganize the knowledge of the central node, and the parameters of the target model will be updated as follows:

[0023]

[0024] After the knowledge of the central node is reorganized, the knowledge selection and local knowledge reorganization will be performed again. In the iterative learning process, the target model will screen and learn the representation extraction ability of the transferable teacher from multiple pre-trained models.

[0025] 4. Intelligent security task adaptation

[0026] The present invention takes into account the topological structure characteristics of the deep learning model. After the knowledge reorganization phase converges, it will continue to train on the target task data set of intelligent security to obtain more knowledge related to the target task. Specifically, this phase uses the target data set to train the decoder part of the target model. At this stage, the target model will learn the ability to map features to target labels. In turn, good performance can be obtained on tasks related to intelligent security, such as pedestrian re-identification.

[0027] The advantages of the present invention are: it can realize the transfer of model knowledge without directly accessing the private model, thereby achieving better performance on the target task in the field of intelligent security; the present invention is different from the traditional federated learning framework that only focuses on data privacy. The present invention not only takes into account data privacy, but also takes into account the privacy of the model. This provides a flexible and secure way for model sharing in the field of intelligent security. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the knowledge selection and federated knowledge reorganization of the present invention.

[0029] Figure 2 It is a schematic diagram of the local knowledge reorganization of the present invention.

[0030] Figure 3 It is a flowchart of the federal knowledge reorganization of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is explained and described clearly and completely below in conjunction with the accompanying drawings.

[0032] A visual depth model knowledge reorganization method for intelligent security includes the following steps:

[0033] 1. Collect and train intelligent security data;

[0034] The present invention first collects visual data from security edge devices (such as cameras), clarifies specific model tasks, such as pedestrian re-identification, and annotates a small portion of the collected data. This small portion of annotated data will serve as the target task dataset D T , which contains M data samples and labels. Due to the environment in which security equipment is located and the expensive labeling cost, the data collected by security equipment is usually insufficient, so it is difficult to obtain a well-performing deep model by directly using this part of data for training and learning. In order for the security deep model to have certain knowledge about the target task, it is first necessary to use the collected target task dataset to train the target task model.

[0035] 2. Select model knowledge;

[0036] Pre-trained model {M1,M2,…,M N} is trained on different training data and different tasks, so the pre-trained model contains knowledge from different fields. And the pre-trained model and data are stored on different nodes. For the specific target task model of intelligent security, the transferability of knowledge of different pre-trained models is different. The knowledge of pre-trained models with better transferability is more helpful for the target task. In order to select a suitable model from several pre-trained models, the present invention first calculates the attribution graph of different models on the target data set. in It represents the attribution map of the kth layer of the nth model for the jth input. The attribution map is obtained by calculating the back-propagation gradient of a specific input. The attribution map reflects the model's attention area for the input. Therefore, the distance between the attribution maps of different models can be used as a measure of the transferability of model knowledge. The transferability distance between different models is calculated using the target dataset as follows:

[0037]

[0038] in, Indicates the calculation of cosine similarity. After calculating the attribution graph distance between the target model and the pre-trained model, the transferable score between the models can be calculated based on the attribution graph distance.

[0039] The present invention normalizes the transferable fraction into a discrete probability distribution {P1, P2, ..., P N}, the probability value represents the probability of selecting a certain pre-trained model. The present invention will be based on the probability distribution {P1, P2, ..., P N} Sample k models with replacement to form a model set S M This model set will be used in the subsequent federal knowledge reorganization. Figure 1 )

[0040] 3. Reorganize federal knowledge;

[0041] Without direct access to pre-trained models and datasets, in order to integrate the model set S M The present invention divides the federated knowledge reorganization into two stages: local knowledge reorganization and central knowledge reorganization, according to whether the process occurs on the edge node of the pre-trained model. The division of the two stages can avoid direct contact with the pre-trained model, thereby protecting the privacy of the pre-trained model and data.

[0042] First, the present invention transforms the target model M t Parameters Copy to pre-trained model M nThe node where the target model is located. For most deep models, the model can be divided into an encoder part and a decoder part. The encoder encodes the original input into a vector in the feature space. The feature extraction part is task-independent and has good generalization. The decoder decodes the vector in the feature space into an output related to the task. In the process of local recombination, the present invention forces the feature output of the target model to be consistent with the feature output of the pre-training, and the copy of the target model will learn the feature extraction ability of the pre-training model. Since the pre-training model M n and the target model M t The network structure may be heterogeneous, and the output features of the target model and the local pre-trained model may have different output dimensions. Therefore, the present invention uses three 1×1 convolution operations to align the feature output dimensions of different models. The loss calculation in the local reorganization process is as follows:

[0043]

[0044] Among them, X represents the pre-trained model M n The training data set, Represents the pre-trained model M n Parameters, represents the parameters of the local target model, and They represent the feature outputs of the pre-trained model and the target model after alignment. The local reorganization process will update the parameters of the local target model based on the loss:

[0045]

[0046] Among them, η t represents the learning rate of local reorganization. ( Figure 2 )

[0047] After the local knowledge reorganization is completed, the parameters of the local target model It will be uploaded to the central node. After collecting the target model parameters of different nodes, the central node will reorganize the knowledge of the central node, and the parameters of the target model will be updated as follows:

[0048]

[0049] After the knowledge of the central node is reorganized, the knowledge selection and local knowledge reorganization will be carried out again. In the iterative learning process, the target model will screen and learn the representation extraction ability of the transferable teacher from multiple pre-trained models. ( Figure 3 )

[0050] 4. Adaptation to target tasks;

[0051] The present invention takes into account the topological structure characteristics of the deep learning model. After the knowledge reorganization phase converges, it will continue to train on the target task data set of intelligent security to obtain more knowledge related to the target task. Specifically, this phase uses the target data set to train the decoder part of the target model. At this stage, the target model will learn the ability to map features to target labels. In turn, good performance can be obtained on tasks related to intelligent security, such as pedestrian re-identification.

[0052] The present invention first collects visual data from security edge measurements (such as cameras), clarifies specific model tasks, such as pedestrian re-identification, annotates a small portion of the collected data, and uses the collected target task data set to train the target task model. Then, the attribution graph is used to calculate the transferable scores between the target model and different pre-trained models, and the normalized transferable scores are used as the probability of model selection, and the model set is formed by replacement sampling; on the basis of the federated learning paradigm, the target model is distributed to different nodes, and the feature extraction ability of the pre-trained model is characterized by distillation migration. After the target model learning and migration are completed, the knowledge is reorganized in the central node by averaging, and the reorganized model is distributed to the selected model node again for further iteration. Finally, the target data set is used to fine-tune the target model, so as to learn a visual depth model that performs well on the target task in the field of intelligent security, and protect data privacy and model privacy.

[0053] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A visual depth model knowledge reorganization method for intelligent security, comprising the following steps: Step 1: Collect and train smart security data; First, collect visual data from security edge measurements, clarify specific model tasks, and annotate a small portion of the collected data. The annotated data is used as the target task dataset D T , which contains M data samples and labels. Due to the environment in which security equipment is located and the expensive labeling cost, the data collected by security equipment is insufficient, so it is difficult to obtain a well-performing deep model by directly using this part of data for training and learning. In order for the deep model of security to have sufficient knowledge about the target task, it is first necessary to use the collected target task data set to train the target task model. Step 2: Select model knowledge; Pre-trained model {M1,M2,…,M N } is trained on different training data and different tasks, so the pre-trained model contains knowledge from different fields; and the pre-trained model and data are stored on different nodes; for the specific target task model of intelligent security, the transferability of knowledge of different pre-trained models is different; the knowledge of pre-trained models with better transferability is more helpful for the target task; in order to select a suitable model from several pre-trained models, first calculate the attribution graph of different models on the target data set in Represents the attribution map of the kth layer of the nth model for the jth input. The attribution map is obtained by calculating the back-propagation gradient of a specific input. The attribution map reflects the model's attention area for the input; Therefore, the distance between the attribution graphs of different models can be used as a measure of the transferability of model knowledge. The transferability distance between different models is calculated using the target data set as follows: in, Indicates the calculation of cosine similarity; After calculating the attribution graph distance between the target model and the pre-trained model, the transferable scores between the models can be calculated based on the attribution graph distance; Normalize the transferable scores to a discrete probability distribution {P1,P2,…,P N }, the probability value indicates the probability of selecting a pre-trained model; according to the probability distribution {P1, P2, …, P N } Sample k models with replacement to form a model set S M ;This model set will be used in the subsequent federal knowledge reorganization; Step 3: Reorganize federated knowledge; Without direct access to pre-trained models and datasets, in order to integrate the model set S M The federated knowledge is reorganized into the target model. According to whether the process occurs on the edge node of the pre-trained model, the federated knowledge reorganization is divided into two stages: local knowledge reorganization and central knowledge reorganization. The division of the two stages can avoid direct contact with the pre-trained model, thereby protecting the privacy of the pre-trained model and data. First, the target model M t Parameters Copy to pre-trained model M n The node where the deep model is located; the deep model can be divided into an encoder part and a decoder part. The encoder encodes the original input into a vector in the feature space. The feature extraction part is task-independent and has good generalization; the decoder decodes the vector in the feature space into an output related to the task; in the process of local recombination, the feature output of the target model is forced to be consistent with the feature output of the pre-training, and the copy of the target model will learn the feature extraction ability of the pre-training model; because the pre-training model M n and the target model M t The network structure may be heterogeneous, and the output features of the target model and the local pre-trained model may have different output dimensions. Therefore, three 1×1 convolution operations are used to align the feature output dimensions of different models. The loss calculation in the local reorganization process is as follows: Among them, X represents the pre-trained model M n The training data set, Represents the pre-trained model M n Parameters, represents the parameters of the local target model, and They represent the feature outputs after the pre-trained model and the target model are aligned; the local reorganization process will update the parameters of the local target model based on the loss: Among them, η t represents the learning rate of local reorganization; After the local knowledge reorganization is completed, the parameters of the local target model It will be uploaded to the central node; after collecting the target model parameters of different nodes, the central node will reorganize the knowledge of the central node, and the parameters of the target model will be updated as follows: After the knowledge reorganization of the central node is completed, knowledge selection and local knowledge reorganization will be carried out again; in the iterative learning process, the target model will screen and learn the representation extraction ability of transferable teachers from multiple pre-trained models; Step 4: Intelligent security task adaptation; Considering the topological structure characteristics of the deep learning model, after the convergence of the knowledge reorganization stage is completed, training will continue on the target task data set of intelligent security to obtain more knowledge related to the target task; specifically, the decoder part of the target model is trained using the target data set, and the target model will learn the ability to map features to target labels and achieve good performance on the model tasks of intelligent security.

2. The method for reorganizing visual depth model knowledge for intelligent security as claimed in claim 1, characterized in that: The security edge device described in step 1 is a camera, and the model task is pedestrian re-identification.

Images