Multi-source incremental learning and campus security monitoring method
Through the multi-source incremental learning method, combined with self-attention calculation and feature fusion, the problem of insufficient intelligent monitoring in university laboratory safety monitoring is solved, efficient safety analysis and model updating in a dynamic environment are achieved, and catastrophic forgetting is reduced.
Patent Information
- Application Number
- CN202410293477.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-14
- Publication Date
- 2025-09-16
AI Technical Summary
Existing technologies lack a systematic intelligent monitoring mechanism for university laboratory safety monitoring. The weight setting of multimodal image feature fusion is insufficient, resulting in insufficient data analysis capabilities in dynamic environments and catastrophic forgetting problems in incremental learning.
A multi-source incremental learning method is adopted to establish a campus security monitoring neural network by acquiring the main and secondary perspective images of the human face. BatchNormalization and data increment methods are used for training. Self-attention calculation and feature fusion modules are combined to optimize model parameters. Adaptive weighting and a new loss function are used to solve the model forgetting problem.
It achieves scientific analysis and accurate judgment of laboratory safety status in a dynamic environment, reduces the performance degradation of the model on old tasks after learning new tasks, and improves the robustness and interpretability of multimodal data.
Smart Images
Figure CN120656114A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of multimodal learning and incremental technology, and in particular to a multi-source incremental learning and campus security monitoring method. Background Art
[0002] Currently, many universities rely on manual inspections, laboratory personnel's voluntary adherence to laboratory rules, or semi-automated monitoring systems, without a systematic monitoring mechanism. The rapid development of intelligent AI-based monitoring offers a path forward in addressing laboratory safety monitoring issues.
[0003] In real-world data streams, as data volumes gradually increase, emerging incremental learning techniques can continuously process the continuous flow of information, absorbing new knowledge while retaining, integrating, and optimizing old knowledge. This approach addresses the common flaw in model training: catastrophic forgetting. However, because the weights used in most multimodal image feature fusions are constants pre-set based on experience, they lack the ability to represent and interpret weakly correlated multimodal images. Furthermore, existing big data intelligent analysis methods have limited research focused on dynamic environments. Summary of the Invention
[0004] This application provides a multi-source incremental learning and campus security monitoring method, which can be used to solve the technical problem that big data intelligent analysis methods have been less studied in dynamic environments.
[0005] This application provides a multi-source incremental learning and campus security monitoring method, the method comprising:
[0006] Step 1: Acquire facial data of a person and perform data preprocessing; wherein the facial data of the person includes a primary view image and a secondary view image of the person's face;
[0007] Step 2: Establish a campus security monitoring neural network;
[0008] The campus security monitoring neural network consists of three branches and a feature fusion module: the first branch includes a block splicing module for the main and secondary view images of facial data and a self-attention calculation module; the second branch includes a deep feature extraction module for the main view image of facial data; and the third branch includes a deep feature extraction module for the secondary view image of facial data.
[0009] Step 3: Use facial data to train the campus security monitoring neural network, use the BatchNormalization method to accelerate training, and use the data increment method to continuously update the model's learning parameters.
[0010] Furthermore, step 1, obtaining facial data of a person, includes:
[0011] Acquire facial data of people;
[0012] Use temporal relationships to align images; that is, ensure that a set of primary and secondary view images are captured at the same time. Capture images of the scene from both the primary and secondary viewpoints at a frequency of 200ms per frame, and store the captured images in the cloud.
[0013] At the same time interval, the primary view image P1 and the secondary view image P2 stored in the cloud are saved locally. P1 and P2 are convolved using a convolution kernel of size 3x3, step size 1, and no expansion to complete the preliminary feature extraction of P1 and P2 and obtain their preliminary feature maps F1 and F2:
[0014] F i =CNN 3*3 (P i ),i=1,2
[0015] Among them, CNN represents the processing of P i The convolution operation, F i Indicates the corresponding picture P i The initial feature map of .
[0016] Furthermore, the first branch of the campus security monitoring neural network includes a facial data segmentation and splicing module and a self-attention calculation module;
[0017] The block splicing module processes the preliminary feature maps F1 and F2 obtained from the primary and secondary view images in blocks, and performs embedding splicing on the linear embedding sequence of the image blocks and the corresponding position codes. The spliced result X is used as the input of the self-attention calculation module.
[0018] Furthermore, the self-attention calculation module constructs three linear transformation matrices W Q ,W K ,W V , and perform the following calculations:
[0019]
[0020]
[0021]
[0022] Among them, the three linear transformation matrices W Q ,W K ,W V Is a constant matrix used to transform the input X into Q, K, and V matrices; the Q, K, and V matrices are query matrix, key matrix, and value matrix, respectively;
[0023] Calculate each pair of eigenvectors x i and x j The attention score (Q i ,K j ), the attention score represents the attention to x i x j Attention level:
[0024]
[0025] Among them, x i and x j are any two eigenvectors in the input X; Q i It represents the corresponding eigenvector x i Query, K j It represents the corresponding eigenvector x j The key value, d k is the dimension of the vector;
[0026] Calculate the attention weight w ij , multiply the value vector of each eigenvector, i.e., the elements in the V matrix, by the corresponding attention weights and sum them up to get the final output z1, z2:
[0027] w ij =softmax(score(Q i ,K j ))
[0028] Among them, z1 and z2 are the weights of the main view image in the feature fusion part and the weight of the secondary view image in the third branch, respectively.
[0029] Furthermore, the structures of the second and third branches in the campus security monitoring neural network are as follows:
[0030] Starting from the input, it includes the initial convolution layer, the maximum pooling layer, the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the second pooling layer and the fully connected layer;
[0031] Among them, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer have the same structure;
[0032] Among them, the initial convolution layer is a 7*7 convolution layer with a convolution kernel size of 7*7, a stride of 2, a padding range of 3, and 64 output channels;
[0033] The maximum pooling layer uses maximum pooling, with a convolution kernel size of 3*3, a stride of 1, and a padding range of 1. The maximum pooling layer reduces the number of features from the upper convolution layer by half;
[0034] The first convolutional layer consists of four 3*3 sub-convolutional layers with the same structure, the sub-convolution kernel size is 3*3, the stride is 1, and the padding range is 1;
[0035] The second convolutional layer consists of a 1*1 sub-convolutional layer and four 3*3 sub-convolutional layers with the same structure. The 1*1 sub-convolutional layer is used to halve the image feature size but double the number of output channels.
[0036] The third and fourth convolutional layers are exactly the same as the second convolutional layer in structure and function;
[0037] The second pooling layer follows the fourth convolutional layer, and the pooling method used in this layer is average pooling;
[0038] The fully connected layer expands the result of the second pooling layer to obtain the deep image feature F d .
[0039] Furthermore, the second and third branches of the campus security monitoring network have a total of 20 layers, including convolutional layers, pooling layers, and linear layers. Starting from the maximum pooling layer, there is a fast connection between every two layers, allowing the output of a layer to skip one or more layers and connect to the input of the subsequent layer.
[0040] The representation of a quick connection is:
[0041] H(x)=F(x)+x
[0042] Among them, x represents the output feature of the upper layer of the network, H represents the ideal fitting function, and F represents the error function.
[0043] Furthermore, the feature fusion module uses the weights z1 and z2 calculated by the first branch and the depth image features F obtained by the second and third branches. d1 、F d2 According to the following method:
[0044] Y=z1*F d1 +z2*F d2
[0045] Perform feature fusion to obtain the fusion feature Y, and then use the softmax function to calculate the final classification result Y' of Y
[0046] Y'=softmax(Y).
[0047] Furthermore, step 3 uses facial data to train the campus security monitoring neural network, including:
[0048] Step 31: Use a public facial data set to train a campus security monitoring neural network;
[0049] The training parameters are: initial learning rate of 0.01, using Adam optimizer, training batch size of 200 times, batch size of 512; Batch Normalization method is used to accelerate training during the training process, and the loss function of the training objective is:
[0050]
[0051] The loss function includes the loss function of the old task Loss function for the new task and a regularization term Composition; among which Y o is the true label of the old task, is the predicted label of the old task, Y n is the true label of the new task, is the predicted label for the new task, is a shared parameter, is the parameter on the old task, is the parameter on the new task. The loss function of the old task adopts distillation loss, while the loss function of the new task adopts the form of cross entropy.
[0052] Step 32, using a data increment method to update the initial model using the newly acquired facial data of the person;
[0053] A new facial dataset is constructed based on data increment. The dataset consists of new data and old data. The new data refers to the data that conforms to the preset time period before the current time point, and the old data refers to the data that exceeds the preset time period after removing the data. The model is then retrained based on the new facial dataset, and the model parameters are updated to obtain a new model.
[0054] The present application notes that due to memory space limitations, it is impractical to require the model to store all features. Therefore, the model is required to store only new model features while storing some of the old model features. This will often lead to catastrophic forgetting when using the neural network model for incremental learning of new knowledge, that is, after learning a new task, the model's performance on the old task will catastrophically degrade. Based on the above problems, the present application designs a method to adaptively retrieve network structures that contribute more to historical tasks, expand the network to learn the current task, and adopts a new loss function as the optimization target of the incremental model. The main component is the sum of the loss function of the old task and the loss function of the new task, and introduces a regularization term. The loss function of the new task adopts the form of cross entropy, while the loss function of the old task adopts distillation loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 A flow chart of the method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0057] The following first introduces the embodiments of the present application with reference to the accompanying drawings.
[0058] To address the shortcomings of the existing technology, the present invention will use adaptive weighting to calculate weights, thereby scientifically analyzing and accurately judging the laboratory safety status. This application requires extracting multi-angle images through multimodal fusion detection and establishing a suitable incremental adaptive model for analysis. To address the above challenges, on the one hand, we focus on the robust representation of weakly correlated multi-source data, and on the other hand, we emphasize incremental adaptive modeling for dynamic environments.
[0059] This application provides a multi-source incremental learning and campus security monitoring method, which includes the following steps:
[0060] Step 1: Acquire facial data of a person and perform data preprocessing; wherein the facial data of the person includes a primary view image and a secondary view image of the person's face;
[0061] Step 2: Establish a campus security monitoring neural network;
[0062] The campus security monitoring neural network consists of three branches and a feature fusion module: the first branch includes a block splicing module for the main and secondary view images of facial data and a self-attention calculation module; the second branch includes a deep feature extraction module for the main view image of facial data; and the third branch includes a deep feature extraction module for the secondary view image of facial data.
[0063] Step 3: Use facial data to train the campus security monitoring neural network, use BatchNormalization to accelerate training, and use data increment to continuously update the model's learning parameters;
[0064] Specifically, step 1, obtaining facial data of a person, includes:
[0065] Acquire facial data. The scene is divided into primary and secondary perspectives. The primary perspective refers to the angle from which the character appears mostly from the front, while the secondary perspective refers to the angle from which the character appears mostly from the side. This division into primary and secondary perspectives is a priori determined based on surveys and statistics.
[0066] Then, using the pre-set custom skills in the multimedia terminal device, the images are aligned based on the time relationship. This ensures that a set of image data from the primary and secondary perspectives are captured at the same time. The scene is captured from the primary and secondary perspectives at a frequency of 200ms per frame, and the captured images are stored in the cloud, such as the Huawei Cloud OBS bucket.
[0067] At the same time interval, the primary view image P1 and the secondary view image P2 stored in the cloud are saved locally. P1 and P2 are convolved using a convolution kernel of size 3x3, step size 1, and no expansion to complete the preliminary feature extraction of P1 and P2 and obtain their preliminary feature maps F1 and F2:
[0068] F i =CNN 3*3 (P i ),i=1,2
[0069] Among them, CNN represents the processing of P i The convolution operation, F i Indicates the corresponding image P i The initial feature map of .
[0070] Step 2: Establish a campus security monitoring neural network;
[0071] Among them, the campus security monitoring neural network includes three branches and a feature fusion module. The first branch is mainly used to block and splice the main and secondary perspective image data and calculate the feature fusion weights; the second and third branches have the same structure, which are used to extract deep features of the main and secondary perspective images respectively; the feature fusion module uses the output of the first branch as the weight to fuse the output deep features of the second and third branches.
[0072] The first branch includes the facial data segmentation and splicing module and the self-attention calculation module;
[0073] The block splicing module processes the preliminary feature maps F1 and F2 obtained from the primary and secondary view images in blocks, and performs embedding splicing on the linear embedding sequence of the image blocks and the corresponding position codes. The spliced result X is used as the input of the self-attention calculation module.
[0074] Self-attention calculation module, constructing three linear transformation matrices W Q ,W K ,W V , and perform the following calculations:
[0075]
[0076]
[0077]
[0078] Among them, the three linear transformation matrices W Q ,W K ,W V Is a constant matrix used to transform the input X into Q, K, and V matrices; the Q, K, and V matrices are query matrix, key matrix, and value matrix, respectively;
[0079] Calculate each pair of eigenvectors x i and x j The attention score (Q i ,K j ), the attention score represents the attention to x i x j Attention level:
[0080]
[0081] Among them, x i and x j are any two eigenvectors in the input X; Q i It represents the corresponding eigenvector x i Query, K j It represents the corresponding eigenvector x j The key value, d k is the dimension of the vector;
[0082] Calculate the attention weight w ij , multiply the value vector of each eigenvector, i.e., the elements in the V matrix, by the corresponding attention weights and sum them up to get the final output z1, z2:
[0083] w ij =softmax(score(Q i ,K j ))
[0084] Among them, z1 and z2 are the weights of the main view image in the feature fusion part and the weight of the secondary view image in the third branch, respectively.
[0085] The structures of the second and third branches are as follows:
[0086] Starting from the input, it includes the initial convolution layer, the maximum pooling layer, the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the second pooling layer and the fully connected layer;
[0087] Among them, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer have the same structure;
[0088] Among them, the initial convolution layer is a 7*7 convolution layer with a convolution kernel size of 7*7, a stride of 2, a padding range of 3, and 64 output channels;
[0089] The maximum pooling layer uses maximum pooling, with a convolution kernel size of 3*3, a stride of 1, and a padding range of 1. The maximum pooling layer reduces the number of features from the upper convolution layer by half;
[0090] The first convolutional layer consists of four 3*3 sub-convolutional layers with the same structure, the sub-convolution kernel size is 3*3, the stride is 1, and the padding range is 1;
[0091] The second convolutional layer consists of a 1*1 sub-convolutional layer and four 3*3 sub-convolutional layers with the same structure. The 1*1 sub-convolutional layer is used to halve the image feature size but double the number of output channels.
[0092] The third and fourth convolutional layers are exactly the same as the second convolutional layer in structure and function;
[0093] The second pooling layer follows the fourth convolutional layer, and the pooling method used in this layer is average pooling;
[0094] The fully connected layer expands the result of the second pooling layer to obtain the deep image feature F d ;
[0095] The second and third branches of the campus security monitoring network consist of 20 layers, including convolutional layers, pooling layers, and linear layers. Starting from the maximum pooling layer, there is a fast connection between every two layers, allowing the output of a layer to skip one or more layers and connect to the input of the subsequent layer.
[0096] The representation of a quick connection is:
[0097] H(x)=F(x)+x
[0098] Among them, x represents the output feature of the upper layer of the network, H represents the ideal fitting function, and F represents the error function.
[0099] The feature fusion module uses the weights z1 and z2 calculated by the first branch and the depth image features F obtained by the second and third branches. d1 、F d2 According to the following method:
[0100] Y=z1*F d1 +z2*F d2
[0101] Perform feature fusion to obtain the fusion feature Y, and then use the softmax function to calculate the final classification result Y' of Y
[0102] Y'=softmax(Y).
[0103] Step 3: Use facial data to train the campus security monitoring neural network;
[0104] Step 31: Use a public facial data set to train a campus security monitoring neural network;
[0105] The training parameters are: initial learning rate of 0.01, using Adam optimizer, training batch size of 200 times, batch size of 512; Batch Normalization method is used to accelerate training during training, which can better extract image features. The loss function of the training objective is:
[0106]
[0107] The loss function includes the loss function of the old task Loss function for the new task and a regularization term Composition; among which Y o is the true label of the old task, is the predicted label of the old task, Y n is the true label of the new task, is the predicted label for the new task, is a shared parameter, is the parameter on the old task, is the parameter on the new task. The loss function of the old task adopts distillation loss, while the loss function of the new task adopts the form of cross entropy.
[0108] The training goal is to minimize the loss function. After training, the model weights that minimize the loss function are saved as the initial model.
[0109] Step 32, using a data increment method to update the initial model using the newly acquired facial data of the person;
[0110] Leveraging the ability of incremental learning to process continuous data streams in the real world, we introduce a data increment method. This method constructs a new facial dataset consisting of new data and old data. New data refers to data that falls within a preset time period before the current point in time, while old data is the remaining data after removing data that falls outside the preset time period. The model is then retrained based on the new facial dataset, updating the model parameters to produce a new model.
[0111] This application first runs the custom skills in the multimedia device to divide the scene into the main perspective and the secondary perspective. The main and secondary perspectives refer to the front perspective and the side perspective. The main and secondary perspective images are acquired at regular intervals, and the captured images are stored with the help of the cloud. Afterwards, the images stored in the cloud are pulled to the local computer at the same time intervals, and a simple convolution operation is performed on the images to complete the preliminary feature extraction. At this point, the data acquisition phase is completed, and then the data training phase is carried out. The main and secondary perspective images are respectively subjected to deep feature extraction through the second and third branches of the campus security monitoring network, and the main and secondary perspective images are adaptively weighted through the first branch of the network. Afterwards, the weights obtained from the first branch and the deep features of the main and secondary perspective images obtained from the second and third branches are fused, and the softmax function is used to complete the binary classification task of whether it is safe.
[0112] Because the process utilizes incremental learning, a pre-trained model is created using a public facial dataset before model training begins. This is followed by a dynamic update of model parameters by minimizing the loss function using incremental learning data increments, thus achieving the goal of streaming data processing.
[0113] The above-described embodiments of the present application do not constitute a limitation on the scope of protection of the present application.
Claims
1. A multi-source incremental learning and campus security monitoring method, characterized in that: The method comprises: Step 1: Acquire facial data of a person and perform data preprocessing; wherein the facial data of the person includes a primary view image and a secondary view image of the person's face; Step 2: Establish a campus security monitoring neural network; The campus security monitoring neural network consists of three branches and a feature fusion module: the first branch includes a block splicing module for the main and secondary view images of facial data and a self-attention calculation module; the second branch includes a deep feature extraction module for the main view image of facial data; and the third branch includes a deep feature extraction module for the secondary view image of facial data. Step 3: Use facial data to train the campus security monitoring neural network, use the BatchNormalization method to accelerate training, and use the data increment method to continuously update the model's learning parameters.
2. The method according to claim 1, characterized in that Step 1: Obtain facial data of the person, including: Acquire facial data of people; Use temporal relationships to align images; that is, ensure that a set of primary and secondary view images are captured at the same time. Capture images of the scene from both the primary and secondary viewpoints at a frequency of 200ms per frame, and store the captured images in the cloud. At the same time interval, the primary view image P1 and the secondary view image P2 stored in the cloud are saved locally. P1 and P2 are convolved using a convolution kernel of size 3x3, step size 1, and no expansion to complete the preliminary feature extraction of P1 and P2 and obtain their preliminary feature maps F1 and F2: F i =CNN 3*3 (P i ),i=1,2 Among them, CNN represents the processing of P i The convolution operation, F i Indicates the corresponding image P i The initial feature map of .
3. The method according to claim 1, characterized in that The first branch of the campus security monitoring neural network includes a facial data segmentation and splicing module and a self-attention calculation module; The block splicing module processes the preliminary feature maps F1 and F2 obtained from the primary and secondary view images in blocks, and performs embedding splicing on the linear embedding sequence of the image blocks and the corresponding position codes. The spliced result X is used as the input of the self-attention calculation module.
4. The method according to claim 3, characterized in that Self-attention calculation module, constructing three linear transformation matrices W Q ,W K ,W V , and perform the following calculations: Among them, the three linear transformation matrices W Q ,W K ,W V Is a constant matrix used to transform the input X into Q, K, and V matrices; the Q, K, and V matrices are query matrix, key matrix, and value matrix, respectively; Calculate each pair of eigenvectors x i and x j The attention score (Q i ,K j ), the attention score represents the attention to x i x j Attention level: Among them, x i and x j are any two eigenvectors in the input X; Q i It represents the corresponding eigenvector x i Query, K j It represents the corresponding eigenvector x j The key value, d k is the dimension of the vector; Calculate the attention weight w ij , multiply the value vector of each eigenvector, i.e., the elements in the V matrix, by the corresponding attention weights and sum them up to get the final output z1, z2: w ij =softmax(score(Q i ,K j )) Among them, z1 and z2 are the weights of the main view image in the feature fusion part and the weight of the secondary view image in the third branch, respectively.
5. The method according to claim 1, characterized in that The structures of the second and third branches in the campus security monitoring neural network are as follows: Starting from the input, it includes the initial convolution layer, the maximum pooling layer, the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the second pooling layer and the fully connected layer; Among them, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer have the same structure; Among them, the initial convolution layer is a 7*7 convolution layer with a convolution kernel size of 7*7, a stride of 2, a padding range of 3, and 64 output channels; The maximum pooling layer uses maximum pooling, with a convolution kernel size of 3*3, a stride of 1, and a padding range of 1. The maximum pooling layer reduces the number of features from the upper convolution layer by half; The first convolutional layer consists of four 3*3 sub-convolutional layers with the same structure, the sub-convolution kernel size is 3*3, the stride is 1, and the padding range is 1; The second convolutional layer consists of a 1*1 sub-convolutional layer and four 3*3 sub-convolutional layers with the same structure. The 1*1 sub-convolutional layer is used to halve the image feature size but double the number of output channels. The third and fourth convolutional layers are exactly the same as the second convolutional layer in structure and function; The second pooling layer follows the fourth convolutional layer, and the pooling method used in this layer is average pooling; The fully connected layer expands the result of the second pooling layer to obtain the deep image feature F d .
6. The method according to claim 5, characterized in that The second and third branches of the campus security monitoring network have a total of 20 layers, including convolutional layers, pooling layers, and linear layers. Starting from the maximum pooling layer, there is a fast connection between every two layers, allowing the output of a layer to skip one or more layers and connect to the input of the subsequent layer. The representation of a quick connection is: H(x)=F(x)+x Among them, x represents the output feature of the upper layer of the network, H represents the ideal fitting function, and F represents the error function.
7. The method according to claim 1, characterized in that The feature fusion module uses the weights z1 and z2 calculated by the first branch and the depth image features F obtained by the second and third branches. d1 、F d2 According to the following method: Y=z1*F d1 +z2*F d2 Perform feature fusion to obtain the fusion feature Y, and then use the softmax function to calculate the final classification result Y' of Y Y'=softmax(Y).
8. The method according to claim 1, characterized in that Step 3: Use facial data to train the campus security monitoring neural network, including: Step 31: Use a public facial data set to train a campus security monitoring neural network; The training parameters are: initial learning rate is 0.01, using Adam optimizer, training batch size is 200 times, batch size is 512; BatchNormalization method is used to accelerate training during training, and the loss function of the training objective is: The loss function includes the loss function of the old task Loss function for the new task and a regularization term Composition; among which Y o is the true label of the old task, is the predicted label of the old task, Y n is the true label of the new task, is the predicted label for the new task, is a shared parameter, is the parameter on the old task, is the parameter on the new task. The loss function of the old task adopts distillation loss, while the loss function of the new task adopts the form of cross entropy. Step 32, using a data increment method to update the initial model using the newly acquired facial data of the person; A new facial dataset is constructed based on data increment. The dataset consists of new data and old data. The new data refers to the data that conforms to the preset time period before the current time point, and the old data refers to the data that exceeds the preset time period after removing the data. The model is then retrained based on the new facial dataset, and the model parameters are updated to obtain a new model.