Low-sample fine tuning frame construction method and system for video anomaly detection, terminal and medium

Abstract

The invention provides a few-sample fine-tuning frame construction method for video anomaly detection. The method comprises the following steps: constructing a few-sample scene video anomaly detection fine-tuning frame based on an auxiliary task, segmenting a target which is easy to generate an anomaly in a video, and predicting an optical flow graph at the same time. Through two auxiliary tasks of segmentation and optical flow estimation, rich spatial and temporal features are extracted to promote an anomaly detection model to be finely adjusted to a new scene; a three-stage training strategy is adopted, for three parts of an auxiliary task network, namely a segmentation branch network, an optical flow branch network and a segmentation and optical flow information fusion module, one part is trained in each of three training stages, and parameters of the other two parts are frozen, so that the instability of the training process is reduced. According to the few-sample fine tuning frame construction method and construction system for video anomaly detection, the segmentation target of the video and the optical flow information are effectively combined, and the method and system have the advantages of being high in fine tuning efficiency, low in cost, easy to implement, flexible in deployment and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and specifically to a method, system, terminal and medium for constructing a few-sample fine-tuning framework for video anomaly detection. Background Art

[0002] Video anomaly detection is an important and complex computer vision task that aims to identify abnormal behaviors or events appearing in videos. This task has broad application prospects in the field of public safety, especially in the analysis and processing of surveillance videos. In recent years, with the popularization of intelligent monitoring systems and the sharp increase in the amount of video data, the research on video anomaly detection has received widespread attention. However, this task is highly challenging. First, the definition of "abnormal" behavior is fuzzy and unbounded, and any event that deviates from the normal behavior pattern can be regarded as an anomaly. Second, the probability of abnormal events in the real world is much lower than that of normal events, which leads to the scarcity of abnormal samples. This not only makes the production of datasets difficult, but also poses a huge challenge to the training of models.

[0003] Therefore, most research works use unsupervised learning methods for video anomaly detection. These methods only use normal data during training and use proxy tasks such as frame reconstruction and future frame prediction to model normal events. Specifically, by training the model to reconstruct video frames or predict future frames, if there is a significant difference between the predicted results and the actual results during testing, the frame can be considered to contain abnormal behavior. The advantage of this method is that it does not require abnormal samples for training, which reduces the difficulty of data preparation. However, this method has a significant defect, that is, the training and test samples must come from the same scene, otherwise the performance of the model will be significantly reduced. This limits the generalization ability of the model in practical applications, because in practical applications, cameras often need to be deployed in different locations, and the scenes monitored by these cameras are different from the scenes in the training set used by the model. In addition, it is costly to collect data and train anomaly detection models specifically for each camera, which is not conducive to deploying anomaly detection models on a large number of cameras. A feasible solution is to first train the model on a benchmark anomaly detection dataset. When it needs to be deployed on a new camera, only a small amount of video data of the camera needs to be collected for model fine-tuning.

[0004] The standard fine-tuning method is to directly train the model using data from the new scene. However, since the new scene can only provide a small amount of data, and standard fine-tuning requires sufficient data to adapt well to the new scene, when the amount of data is small, this method is prone to overfitting the model, resulting in a decrease in detection performance. In recent years, meta-learning methods have gradually emerged in the field of video anomaly detection. Meta-learning aims to enable the model to quickly adapt to new scenes through training with a small number of samples, which is of great significance in cross-scene detection. Through meta-learning, the model can effectively learn with limited data, thereby solving the problem of inconsistent scenes between the training set and the test set. This type of method has improved the practical application value of the model to a certain extent. However, meta-learning methods usually require special design during the training phase, or require the model architecture to be adjusted, which limits its versatility in different tasks and applications. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a method, system, terminal and medium for constructing a few-sample fine-tuning framework for video anomaly detection.

[0006] (II) Technical solution To achieve the above object, according to one aspect of the present invention, a method for constructing a video anomaly detection model fine-tuning framework is provided, comprising: S1: Construct a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks, generate corresponding segmentation maps and optical flow maps for real frames and model prediction frames respectively, calculate segmentation loss, optical flow loss, and prediction loss between predicted frames and real frames, and use these three losses to fine-tune the anomaly detection model. The segmentation process extracts the shape features of the target in the video, and the optical flow estimation process extracts the motion features of the object in the time dimension. By using the above three losses at the same time, the fine-tuning framework can fine-tune the pre-trained model with only a small number of samples to adapt to new scenes; S2: A three-stage training strategy is adopted. For the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network, and the segmentation and optical flow information fusion module, one part is trained in each of the three training stages, and the parameters of the other two parts are frozen, thereby reducing the instability of the training process; S3: Using the existing video anomaly detection dataset, complete the training of the video anomaly detection model. Then complete the training of the auxiliary task network; S4: Use the few-shot fine-tuning framework for video anomaly detection to fine-tune the model on new scenes and test the model's detection capabilities when facing new scenes. The final anomaly detection model few-shot scene fine-tuning framework is constructed.

[0007] Preferably, the S1: constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks, comprises: S101: Construct an anomaly detection model for outputting prediction frames. The model can output future prediction frames based on multiple continuous input video frames. By comparing the difference between the image and the real future frame, it can be determined whether this frame has an anomaly.

[0008] S102: Construct a segmentation branch network to generate a segmentation map of the image. The network can extract shape information of the target in the image in the spatial dimension, such as a binary segmentation map of other objects such as a person or a vehicle.

[0009] S103: Construct an optical flow branch network to generate an optical flow map of the image, which can extract the motion information of the object in the image in the time dimension.

[0010] S104: Construct a segmentation and optical flow information fusion module to adaptively fuse global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information.

[0011] Preferably, the S101 includes that the anomaly detection model is composed of a network similar to a U-Net structure, specifically, the network contains multiple convolutional layers, batch normalization layers and pooling layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; In the encoding stage, a pooling layer is connected after every two layers of convolution, using a 3x3 convolution kernel and a 2x2 maximum pooling layer. The convolution layer extracts features, and the batch normalization layer is used to accelerate training and stabilize the network. The pooling layer is used to reduce the size of the feature map and increase the receptive field. The purpose of the encoding stage is to extract high-level features of the image while gradually reducing spatial information. After the encoding is completed, it will go through several more convolutions and then enter the decoding stage. The decoding part of the network consists of multiple deconvolution layers and convolution layers. The deconvolution layer is used to gradually restore the spatial size of the feature map. Each deconvolution step is followed by two convolution layers, and jump connections are made to the feature maps of the corresponding layers of the encoder, combined with high-resolution features from the encoder, which helps to restore detail information. After decoding, the predicted image is output through a convolution layer.

[0012] The S102 includes: constructing a segmentation branch network, which is composed of a lightweight U-Net-like structure network. First, the features of the input image are extracted through three downsampling operations, and the high-dimensional input data is mapped to a low-dimensional space. This low-dimensional representation contains important features required for the segmentation task, while retaining semantic information of different scales. In the decoding stage, the network gradually expands the low-dimensional representation to the same size as the input image through three deconvolution operations, and finally generates a segmentation map. The deconvolution process also uses a jump connection, which directly transfers the high-resolution features of the shallow convolution to the decoding stage across layers, providing rich detail information for the decoding process, and finally outputs the segmentation map through the convolution layer; The S103 includes constructing an optical flow branch network, which is composed of two encoders and a decoder. The encoder contains multiple convolution layers, batch normalization layers and pooling layers. Each convolution layer is followed by a batch normalization layer and a ReLU activation function. A pooling layer is connected after every two convolution layers, using a 3x3 convolution kernel and a 2x2 maximum pooling layer. For two consecutive frames of input images, they are respectively input into two decoders with the same structure. The parameters of the initial two layers of convolution of the two decoders are shared to achieve feature extraction and generalization capabilities. Then, the two decoders are subjected to two downsampling operations to gradually extract features of different scales and fuse these features. Next, the decoder part is subjected to two upsampling operations, through a deconvolution layer, a convolution layer, a batch normalization layer, a ReLU activation function and a pooling layer, to restore the fused features to the same spatial resolution as the input image. In the upsampling process, a jump connection is used to transmit the information of the two encoders to the upsampling layers corresponding to the decoder, respectively, and combined with the high-resolution information from the shallow network, the optical flow map is finally output through the convolution layer; S104, constructing a segmentation and optical flow information fusion module, first extracting the shallow features of the next frame image after the first two layers of convolution of the encoder from the optical flow branch, recorded as , extract the high-level features before the last convolution layer from the segmentation branch, denoted as , for the feature , respectively, with the spatial range of and The pooling kernel encodes each channel, so the output of the height h and width w of the cth channel can be written as follows:

[0013]

[0014] Then, connect along the spatial dimension and And send it to 1×1 convolution, and then use the nonlinear activation function to calculate , and then along the spatial dimension It is divided into two parts: one along the height direction is recorded as , and the other along the width direction is , next, and Perform a 1×1 convolution transformation, and finally use the sigmoid function to obtain two new weight matrices: and , these weight matrices represent the attention weights of the optical flow features in the height and width dimensions; The same steps are also applied to the features of the split branch , and get two corresponding attention weight matrices and After obtaining these attention weights, apply the coordinate attention weights of the optical flow branch to the features of the segmentation branch Specifically, for a specific location , whose output It is expressed as:

[0015] Similarly, the coordinate attention weights of the segmentation branch are applied to the features of the optical flow branch On, its output It is expressed as:

[0016] In this way, two new feature representations are obtained and , which fuse the attention weights from the segmentation and optical flow branches respectively. Next, these two features are added element by element to obtain a new fused feature Finally, the original segmentation features and fusion features The concatenation is performed in the channel dimension to form a new feature representation, which is processed by a convolutional layer to output the final segmentation result. .

[0017] Preferably, S2 adopts a three-stage training strategy, for the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network and the segmentation and optical flow information fusion module, one of the parts is trained in three training stages respectively, and the parameters of the remaining two parts are frozen, thereby reducing the instability of the training process, including: S201, in the first training stage, only the optical flow branch network is trained, and the parameters of the rest of the auxiliary task network are frozen, using two consecutive frames of images and As input, the label of the optical flow map is recorded as , the loss used during training is recorded as ,in Represents the parameters of the optical flow branch network; it is calculated using the Endpoint error (EPE) commonly used in the optical flow field , that is, calculating the predicted optical flow map and The Euclidean distance of S202, in the second training stage, the segmentation branch network is trained, and the parameters of the rest of the auxiliary task network are frozen, using a single frame image As input, the label of the segmentation map is recorded as , the output of the segmentation branch network is recorded as The loss used during training is denoted by ,in Represents the parameters of the optical flow branch network. Here we use the intensity loss With gradient loss Add together to get the segmentation loss .

[0018] S203, in the third training stage, we only train the OSF block, and the rest of the parameters of the RGB2SegFlow network are also frozen. We use two consecutive frames of images and Input to the RGB2SegFlow network, the optical flow features and segmentation features obtained from the two branch networks are used as the input of the OSF block, using the above segmentation map As labels, the output is recorded as , the loss function is the same as in the second training stage.

[0019] Preferably, the S301: using an existing video anomaly detection data set to complete the training of the video anomaly detection model; and then completing the training of the auxiliary task network, includes: The anomaly detection model is pre-trained using the public video anomaly detection dataset Shanghai Tech, and a segmentation map is generated on the Shanghai Tech dataset as a real segmentation label using the pre-trained model of the open source algorithm YOLOv8 and MobileSAM; an optical flow map is generated on the Shanghai Tech dataset as a real optical flow label using the pre-trained model provided by the open source algorithm GMA, including: Obtain the public video anomaly detection dataset Shanghai Tech, and take every five consecutive frames in the training set of Shanghai Tech as a group, with the first four frames as input and the fifth frame as the true value image. The goal of the training model is to predict the future frames given the past four frames to be close to the true value image, so that the model can learn the ability to predict normal videos. Obtain the pre-trained models of the open source algorithms YOLOv8, MobileSAM, and GMA; use the YOLOv8 model to detect targets such as people, cars, and items carried by people in the Shanghai Tech training set, input the detected target boxes and the corresponding original images into the MobileSAM model, and output the binary segmentation map of the target as the true value map of the segmentation task; input every two frames in the ShanghaiTech training set as a group into the GMA model, and the model estimates the optical flow map of the next frame of the image as the true value map of the optical flow estimation task; use the segmentation and optical flow true value maps generated above as labels, and train the auxiliary task network on the Shanghai Tech training set in an end-to-end manner, so that the network learns the mapping relationship between the original image and the segmentation map and optical flow map; The setting of initialization parameters of the anomaly detection model and the auxiliary task network includes: The initial parameters of the anomaly detection model are randomly initialized in the pre-training stage, and the initial parameters of the auxiliary task network are randomly initialized; Determining the loss function of the anomaly detection model and the auxiliary task network includes: The loss function of the anomaly detection model adopts a weighted linear combination of intensity loss and gradient loss; the auxiliary task network adopts a weighted linear combination of intensity loss and gradient loss as the loss function of the segmentation task, and adopts a weighted linear combination of intensity loss and EPE (Endpoint Error) as the loss function of the segmentation task; Determining the hyper parameters related to the anomaly detection model and the auxiliary task network includes: The stochastic gradient descent algorithm is used in the pre-training process of the anomaly detection model, with a learning rate of 0.0002 and a batch size of 16. The pre-training process requires 13 epochs to iterate until the model converges. The stochastic gradient descent algorithm is used in the training process of the auxiliary task network, with a learning rate of 0.0002 and a batch size of 8. The first training stage requires 30,000 iterations until the model converges, the second training stage requires 8,000 iterations until the model converges, and the third training stage requires 6,000 iterations until the model converges.

[0020] Preferably, the few-sample fine-tuning framework for video anomaly detection is used to fine-tune the model on a new scene with a few samples, and to test the detection capability of the model when facing the new scene, including: S401, use the test set part of the public video anomaly detection dataset UCSD Ped2 to perform a few-sample fine-tuning on the pre-trained anomaly detection model; obtain the public dataset UCSD Ped2, randomly extract 10 groups of data from the test set part, each group of data contains 5 consecutive video frames; these data are used as the few-sample fine-tuning data on the UCSD Ped2 dataset; use the fine-tuning framework to train on a small amount of fine-tuning data, send the current predicted frame generated by the model and the previous real frame to the auxiliary task network to generate a predicted segmentation map and optical flow map, and send the corresponding current real frame and the previous real frame to the auxiliary task network again to generate a real segmentation map and optical flow map as the true value map of the segmentation and optical flow estimation tasks; during the training process, the parameters of the auxiliary task network remain unchanged, and only the parameters of the anomaly detection model are updated, so that the model can adapt to new scenes in the UCSD Ped2 dataset; in the new scene, the fine-tuned model can predict normal frames well, but generates a large prediction error when predicting abnormal frames, and the size of the prediction error can be used to distinguish whether the image has an abnormality; S402, setting initialization parameters of the pre-trained anomaly detection model, where the pre-trained anomaly detection model uses parameters obtained by pre-training on the Shanghai Tech training set as initialization parameters; S403, determining the loss function of the pre-trained anomaly detection model; the loss function of the anomaly detection model adopts a weighted linear combination of the intensity loss and gradient loss of the prediction graph, the loss function of the segmentation task, and the loss function of the optical flow estimation task. Specifically, Loss function for prediction graph , the loss function for the segmentation task ,in Refers to the segmentation map generated by the auxiliary task network using the real frame, Refers to the segmentation maps generated by the auxiliary task network using the frames predicted by the anomaly detection model. is the strength loss of the segmentation map, is the gradient loss of the segmentation map. The loss function of the optical flow estimation task is ,in Refers to the optical flow map generated by the auxiliary task network using the real past frame and the current frame. It refers to the optical flow map generated by the auxiliary task network using the real past frames and the current frame predicted by the anomaly detection model. is the strength loss of the segmentation map, isEndpointerror(EPE); S404, determining the relevant hyper parameters in the pre-trained anomaly detection model, the learning rate is 0.0015, the batch size is 3, and the fine-tuning process needs to iterate 870 epochs until the model converges.

[0021] S405, test the fine-tuned anomaly detection model on the test set of UCSD Ped2 to evaluate the performance of the pre-trained anomaly detection model on new scenes after fine-tuning using the fine-tuning framework; use the fine-tuned anomaly detection model on the test set of UCSD Ped2 to calculate the mean square error between the predicted image and the true image of each frame, and frames with large differences are considered abnormal frames. Then, the AUC score (%) is obtained by calculating the area under the ROC (receiver operating characteristic curve). The AUC is used to reflect the model's ability to detect anomalies. The larger the AUC, the stronger the model's ability to distinguish whether it is abnormal. The final fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks is constructed.

[0022] According to the third aspect of the present invention, a method for fine-tuning video anomaly detection in a few-sample scene based on an auxiliary task is provided, wherein a fine-tuned anomaly detection model is constructed by using any of the above-mentioned construction methods or construction systems for fine-tuning a video anomaly detection model in a few-sample scene, and a small amount of data samples in any new scene are used as fine-tuning data for the fine-tuning framework to obtain an anomaly detection model that can adapt to the new scene.

[0023] According to a fourth aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above methods when executing the program.

[0024] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which can be used for any of the methods described above when executed by a processor.

[0025] (III) Beneficial effects Compared with the prior art, the present invention provides a method for constructing a few-sample fine-tuning framework for video anomaly detection, which has the following beneficial effects: 1. The method for constructing a few-sample fine-tuning framework for video anomaly detection uses the object shape information in the spatial dimension and the motion information in the temporal dimension to help the fine-tuning process. It achieves excellent performance on the video anomaly detection dataset UCSD Ped2 by fine-tuning the pre-trained model with only a small number of samples. Through segmentation tasks and optical flow estimation tasks, more information that is beneficial to the main task of anomaly detection can be mined from limited samples, thereby increasing the constraints in the model parameter update process and improving the fine-tuning effect.

[0026] 2. The method for constructing a few-sample fine-tuning framework for video anomaly detection is efficient, low-cost, easy to implement, and very flexible. It can be applied to various anomaly detection models using different backbone networks according to actual needs in terms of efficiency and performance. As long as the output of the anomaly detection model is an image, the few-sample fine-tuning framework for video anomaly detection provided by the present invention can be applied. After fine-tuning with a small amount of new scene data, the anomaly detection model can adapt to the new scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a workflow diagram of a method for fine-tuning a few-sample scene in a preferred embodiment of the present invention.

[0028] Figure 2 Schematic diagram of the components of a system for fine-tuning a small number of sample scenes in a preferred embodiment of the present invention.

[0029] Figure 3 These are some subjective experimental results obtained by the few-sample scene fine-tuning method and system provided in a preferred embodiment of the present invention on public datasets in the field of video anomaly detection. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. An embodiment of the present invention provides a method for constructing a few-sample scene fine-tuning framework. The method aims at the characteristics of segmentation and optical flow information and realizes the construction of a few-sample scene video anomaly detection fine-tuning framework based on auxiliary tasks.

[0031] like Figure 1 As shown, the method, system, terminal and medium for constructing a few-sample scenario fine-tuning framework provided in this embodiment may include: S1, build a fine-tuning framework for video anomaly detection in few-shot scenarios based on auxiliary tasks; S2 adopts a three-stage training strategy. For the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network, and the segmentation and optical flow information fusion module, one part is trained in each of the three training stages, and the parameters of the other two parts are frozen, thereby reducing the instability of the training process; S3, using the existing video anomaly detection dataset to complete the training of the video anomaly detection model. Then complete the training of the auxiliary task network; S4, using the auxiliary task-based few-shot scene video anomaly detection fine-tuning framework, perform few-shot fine-tuning on other video anomaly detection datasets with different scenes from the training dataset, and test the model's anomaly detection ability when facing new scenes. The final auxiliary task-based few-shot scene video anomaly detection fine-tuning framework is constructed.

[0032] In a preferred embodiment of S1, constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks may include: A framework for fine-tuning video anomaly detection in a few-sample scene based on auxiliary tasks is constructed. The framework mainly includes an anomaly detection model and an auxiliary task network. The auxiliary task network consists of a segmentation branch network, an optical flow branch network, and a segmentation and optical flow information fusion module. The framework can be used to fine-tune the anomaly detection model. S101, build an anomaly detection model. The anomaly detection model consists of a network similar to the U-Net structure. Specifically, the network contains multiple convolutional layers, batch normalization layers, and pooling layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. In the encoding stage, a pooling layer is connected after every two convolutions. Using a 3x3 convolution kernel and a 2x2 maximum pooling layer, the convolution layer extracts features, the batch normalization layer is used to accelerate training and stabilize the network, and the pooling layer is used to reduce the size of the feature map while increasing the receptive field. The purpose of the encoding stage is to extract high-level features of the image while gradually reducing spatial information. After the encoding is completed, it will go through several more convolutions and then enter the decoding stage. The decoding part of the network consists of multiple deconvolutional layers and convolutional layers. The deconvolutional layer is used to gradually restore the spatial size of the feature map. Each deconvolution step is followed by two convolutional layers, and jump connections are made with the feature maps of the corresponding layers of the encoder. Combined with the high-resolution features from the encoder, it helps to restore detail information. After decoding, the predicted image is output through a convolutional layer.

[0033] S102, construct a segmentation branch network. The segmentation branch network consists of a lightweight U-Net-like structure network. First, the features of the input image are extracted through three downsampling operations, and the high-dimensional input data is mapped to a low-dimensional space. This low-dimensional representation contains the important features required for the segmentation task while retaining semantic information at different scales. In the decoding stage, the network gradually expands the low-dimensional representation to the same size as the input image through three deconvolution operations, and finally generates a segmentation map. Skip connections are also used in the deconvolution process, which directly transfers the high-resolution features of the shallow convolution across layers to the decoding stage, providing rich detail information for the decoding process. Finally, the segmentation map is output through the convolution layer.

[0034] S103, construct an optical flow branch network. The optical flow branch network consists of two encoders and a decoder. The encoder contains multiple convolutional layers, batch normalization layers, and pooling layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. A pooling layer is connected after every two convolution layers. A 3x3 convolution kernel and a 2x2 maximum pooling layer are used. For two consecutive frames of input images, they are input into two decoders with the same structure respectively. The parameters of the first two layers of convolution of the two decoders are shared, so as to achieve more efficient feature extraction and stronger generalization ability. Then the two decoders are downsampled twice to gradually extract features of different scales, and then these features are fused. Next, the decoder part is upsampled twice, and the fused features are restored to the same spatial resolution as the input image through the deconvolution layer, convolution layer, batch normalization layer, ReLU activation function and pooling layer. In the upsampling process, a jump connection is used to pass the information of the two encoders to the corresponding upsampling layer of the decoder, and combine the high-resolution information from the shallow network. Finally, the optical flow map is output through the convolution layer.

[0035] S104, construct the segmentation and optical flow information fusion module. First, extract the shallow features of the next frame image after the first two layers of convolution of the encoder from the optical flow branch, denoted as , extract the high-level features before the last convolution layer from the segmentation branch, denoted as For the feature , respectively, with the spatial range of and The pooling kernel encodes each channel. So the output of the height h and width w of the cth channel can be written as the following two formulas:

[0036]

[0037] Then, connect along the spatial dimension and And send it to 1×1 convolution, and then use the nonlinear activation function to calculate , and then along the spatial dimension It is divided into two parts: one along the height direction is recorded as , and the other along the width direction is Next, and Perform a 1×1 convolution transformation, and finally use the sigmoid function to obtain two new weight matrices: and , these weight matrices represent the attention weights of the optical flow features in the height and width dimensions.

[0038] Similar steps are also applied to the features of the split branch , and get two corresponding attention weight matrices and After obtaining these attention weights, apply the coordinate attention weights of the optical flow branch to the features of the segmentation branch Specifically, for a specific location , whose output It is expressed as:

[0039] Similarly, the coordinate attention weights of the segmentation branch are applied to the features of the optical flow branch On, its output It is expressed as:

[0040] In this way, two new feature representations are obtained and , which fuse the attention weights from the segmentation and optical flow branches respectively. Next, these two features are added element by element to obtain a new fused feature Finally, the original segmentation features and fusion features The concatenation is performed in the channel dimension to form a new feature representation. This new feature representation is processed by a convolution layer and the final segmentation result is output. .

[0041] In a preferred embodiment of S2, a three-stage training strategy is adopted. For the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network, and the segmentation and optical flow information fusion module, one of the parts is trained in three training stages respectively, and the parameters of the remaining two parts are frozen, so as to reduce the instability of the training process, which may include: In the first training stage, only the optical flow branch network is trained, the network parameters of the remaining part of the auxiliary task network are frozen, and the real optical flow map is used as the label. In the second training stage, only the segmentation branch network is trained, the other parameters of the auxiliary task network are frozen, and the real segmentation map is used as the label. In the third training stage, only the segmentation and optical flow information fusion module is trained, the network parameters of the segmentation branch and the optical flow branch are frozen, and the real segmentation map is used as the label. This staged learning method can ensure that each module converges stably under relatively simple conditions. Among them: S201, in the first training stage, only the optical flow branch network is trained, and the parameters of the rest of the auxiliary task network are frozen. and As input, the label of the optical flow map is recorded as , the loss used during training is recorded as ,in Represents the parameters of the optical flow branch network. Endpoint error (EPE) commonly used in the field of optical flow is used to calculate , that is, calculating the predicted optical flow map and The Euclidean distance of S202, in the second training stage, the segmentation branch network is trained, and the parameters of the rest of the auxiliary task network are frozen. As input, the label of the segmentation map is recorded as , the output of the segmentation branch network is recorded as The loss used during training is denoted by ,in Represents the parameters of the optical flow branch network. Here we use the intensity loss With gradient loss Add together to get the segmentation loss .

[0042] S203, in the third training stage, we only train the OSF block, and the rest of the parameters of the RGB2SegFlow network are also frozen. We use two consecutive frames of images and Input to the RGB2SegFlow network, and the optical flow features and segmentation features obtained from the two branch networks are used as the input of the OSF block. As labels, the output is recorded as , the loss function is the same as in the second training stage.

[0043] In a preferred embodiment of S3, an existing video anomaly detection dataset is used to complete the training of the video anomaly detection model. Then, the training of the auxiliary task network may include: S301, use the public video anomaly detection dataset Shanghai Tech to pre-train the anomaly detection model, use the pre-trained models of the open source algorithms YOLOv8 and MobileSAM to generate segmentation maps on the Shanghai Tech dataset as real segmentation labels. Use the pre-trained model provided by the open source algorithm GMA to generate optical flow maps on the Shanghai Tech dataset as real optical flow labels. Then use the generated labels to train the auxiliary task network. Among them: We obtained the public video anomaly detection dataset Shanghai Tech, and took every five consecutive frames in the training set of Shanghai Tech as a group, with the first four frames as input and the fifth frame as the ground truth image. The goal of the training model is to predict the future frames given the past four frames to be close to the ground truth image, so that the model can learn the ability to predict normal videos.

[0044] Obtain the pre-trained models of the open source algorithms YOLOv8, MobileSAM, and GMA. Use the YOLOv8 model to detect targets such as people, cars, and items carried by people in the Shanghai Tech training set, input the detected target boxes and the corresponding original images into the MobileSAM model, and output the binary segmentation map of the target as the true value map of the segmentation task. Input every two frames in the ShanghaiTech training set as a group into the GMA model, which estimates the optical flow map of the next frame of the image as the true value map of the optical flow estimation task. Use the segmentation and optical flow truth maps generated above as labels, and train the auxiliary task network on the Shanghai Tech training set in an end-to-end manner, so that the network learns the mapping relationship between the original image and the segmentation map and optical flow map. Specifically: The Shanghai Tech dataset consists of a training set and a test set, with 13 different scenes. The training set contains only 274,515 normal images, and the test set contains 42,883 images, including 130 abnormal events. The UCSD Ped2 dataset also consists of a training set and a test set, with only one scene. The training set contains only 2,550 normal images, and the test set contains 2,010 images, including 12 abnormal events. The normal situation in the training video is pedestrians walking on the sidewalk, and the abnormal situations in the test video include: people trampling on the lawn, people skateboarding on the sidewalk, and cars driving on the sidewalk.

[0045] S302, setting initialization parameters of the anomaly detection model and the auxiliary task network. The initial parameters of the anomaly detection model are randomly initialized in the pre-training stage, and the initial parameters of the auxiliary task network are randomly initialized; S303, determine the loss function of the anomaly detection model and the auxiliary task network. The auxiliary task network uses a weighted linear combination of intensity loss and gradient loss as the loss function of the segmentation task, and uses a weighted linear combination of intensity loss and EPE (Endpoint Error) as the loss function of the segmentation task. The specific details of the loss function have been explained in the three-stage training strategy and will not be repeated here.

[0046] S304, determine the relevant hyperparameters in the anomaly detection model and the auxiliary task network. The stochastic gradient descent algorithm is used in the pre-training process of the anomaly detection model, the learning rate is 0.0002, the batch size is 16, and the pre-training process needs to iterate 13 epochs until the model converges. The stochastic gradient descent algorithm is used in the training process of the auxiliary task network, the learning rate is 0.0002, the batch size is 8, and the first training stage needs 30,000 iterations until the model converges, the second training stage needs 8,000 iterations until the model converges, and the third training stage needs 6,000 iterations until the model converges.

[0047] In a preferred embodiment of S4, a fine-tuning framework for video anomaly detection in a few-sample scene based on an auxiliary task is used to perform fine-tuning on other video anomaly detection datasets with different scenes from the training dataset, and the anomaly detection ability of the model when facing new scenes is tested. The final fine-tuning framework for video anomaly detection in a few-sample scene based on an auxiliary task may include: S401, using the test set of the public video anomaly detection dataset UCSD Ped2, fine-tune the pre-trained anomaly detection model with a few samples. The public dataset UCSD Ped2 was obtained, and 10 groups of data were randomly selected from the test set, each containing 5 consecutive video frames. These data were used as the few-sample fine-tuning data on the UCSD Ped2 dataset. The fine-tuning framework was used to train on a small amount of fine-tuning data, and the current predicted frame generated by the model and the previous true frame were sent to the auxiliary task network to generate the predicted segmentation map and optical flow map. The corresponding current true frame and the previous true frame were sent to the auxiliary task network again to generate the real segmentation map and optical flow map as the true value map of the segmentation and optical flow estimation tasks. During the training process, the parameters of the auxiliary task network remained unchanged, and only the parameters of the anomaly detection model were updated, so that the model could adapt to the new scenes in the UCSD Ped2 dataset. In the new scene, the fine-tuned model can predict the normal frame well, but it produces a large prediction error when predicting the abnormal frame. The size of the prediction error can be used to distinguish whether the image has an abnormality.

[0048] S402, setting initialization parameters of fine-tuning anomaly detection model. The pre-trained anomaly detection model uses the parameters obtained by pre-training on the Shanghai Tech training set as initialization parameters.

[0049] S403, determining the loss function of the pre-trained anomaly detection model. The loss function of the anomaly detection model adopts a weighted linear combination of the intensity loss and gradient loss of the prediction graph, the loss function of the segmentation task, and the loss function of the optical flow estimation task. Specifically: Loss function for prediction graph , the loss function for the segmentation task ,in Refers to the segmentation map generated by the auxiliary task network using the real frame, Refers to the segmentation maps generated by the auxiliary task network using the frames predicted by the anomaly detection model. is the strength loss of the segmentation map, is the gradient loss of the segmentation map. The loss function of the optical flow estimation task is ,in Refers to the optical flow map generated by the auxiliary task network using the real past frame and the current frame. It refers to the optical flow map generated by the auxiliary task network using the real past frames and the current frame predicted by the anomaly detection model. is the strength loss of the segmentation map, It is Endpointerror(EPE).

[0050] S404, determine the relevant hyper parameters in the pre-trained anomaly detection model. The learning rate is 0.0015, the batch size is 3, and the fine-tuning process needs to iterate 870 epochs until the model converges.

[0051] S405, testing the fine-tuned anomaly detection model on the UCSD Ped2 test set, so as to evaluate the performance of the pre-trained anomaly detection model after fine-tuning using the fine-tuning framework on a new scenario. The fine-tuned anomaly detection model is used on the UCSD Ped2 test set to calculate the mean square error between the predicted image and the true image of each frame. Frames with large differences are considered abnormal frames. Then the AUC score (%) is obtained by calculating the area under the ROC (receiver operating characteristic curve). The AUC is used to reflect the model's ability to detect anomalies. The larger the AUC, the stronger the model's ability to distinguish whether it is abnormal. The final fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks is constructed.

[0052] An embodiment of the present invention provides a system for constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks, including: A module for building a fine-tuning framework for few-shot scenes, which is used to build a fine-tuning framework for video anomaly detection in few-shot scenes based on auxiliary tasks; Anomaly detection model and auxiliary task network training module, which is used to pre-train the anomaly detection model using the training set of the Shanghai Tech dataset, and use the pre-trained models of the open source algorithms YOLOv8, MobileSAM and GMA to generate corresponding segmentation maps and optical flow maps on the Shanghai Tech dataset as the true value maps of the segmentation task and optical flow estimation task, which are used to train the auxiliary task network.

[0053] The anomaly detection model fine-tuning module is used to use the fine-tuning framework to fine-tune the anomaly detection model using a small number of samples in the new scene, so that the model can adapt to the new scene. Different from the scene of the Shanghai Tech dataset in the pre-training process, a small number of samples from the test set of UCSD Ped2 are used to train the anomaly detection model during fine-tuning.

[0054] The anomaly detection model testing module is used to test the fine-tuned anomaly detection model using known test images and evaluate the performance of the model fine-tuned using the few-sample fine-tuning framework in new scenarios.

[0055] In a preferred embodiment, the fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks may include: Anomaly detection model, used to output predicted frames. This model can output future predicted frames based on multiple consecutive input video frames. By comparing the difference between the image and the real future frame, it can be determined whether this frame has an abnormality.

[0056] Auxiliary task network, which consists of a segmentation branch network, an optical flow branch network, and a segmentation and optical flow information fusion module, where: The segmentation branch network is used to generate the segmentation map of the image. This network can extract the shape information of the target in the image in the spatial dimension, such as the binary segmentation map of other objects such as people or vehicles.

[0057] The optical flow branch network is used to generate the optical flow map of the image. It can extract the motion information of the objects in the image in the time dimension.

[0058] The segmentation and optical flow information fusion module adaptively fuses global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information.

[0059] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0060] An embodiment of the present invention provides a method for constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on an auxiliary task. The fine-tuning framework for a few-sample scene is constructed by using the construction method or construction system of the fine-tuning framework for a few-sample scene in any of the above embodiments. A small number of video frames of any new scene are used as input of the fine-tuning framework to fine-tune an anomaly detection model that can adapt to the new scene.

[0061] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-mentioned methods or systems when executing the program.

[0062] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviated: RAM), such as static random access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0063] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0064] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0065] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0066] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can be used to execute the method or system of any one of the above embodiments of the present invention.

[0067] The method for constructing a few-sample fine-tuning framework for video anomaly detection provided by the above-mentioned embodiments of the present invention and the construction system, system, terminal and medium adopt a few-sample scene video anomaly detection fine-tuning framework based on auxiliary tasks, fully consider the help of object shape information in the spatial dimension and motion information in the temporal dimension for the fine-tuning process, and obtain excellent performance by fine-tuning the pre-trained model with only a small number of samples on the video anomaly detection dataset UCSD Ped2. The performance achieved after fine-tuning in this cross-dataset setting mainly depends on making full use of the difference between the model prediction map and the real map, the difference between the predicted segmentation map and the segmentation truth map, and the difference between the predicted optical flow map and the optical flow truth map to fine-tune the model. Through the segmentation task and the optical flow estimation task, more information that is beneficial to the main task of anomaly detection is mined from limited samples, thereby increasing the constraints in the model parameter update process and improving the fine-tuning effect.

[0068] The method and system for constructing a few-sample fine-tuning framework for video anomaly detection provided by the above-mentioned embodiment of the present invention, the system, the terminal and the medium, use an auxiliary task network composed of a segmentation branch network, an optical flow branch network and a segmentation and optical flow information fusion module to perform two auxiliary tasks of segmentation and optical flow estimation, effectively extracting the target shape information in the spatial dimension of the video frame and the target motion information in the temporal dimension. And the auxiliary task network is trained through a three-stage training strategy to ensure that each part of the network can converge stably under relatively simple conditions, which is conducive to the auxiliary task network learning to predict accurate segmentation maps and optical flow maps, thereby better mining segmentation and optical flow information and improving the fine-tuning effect.

[0069] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a few-sample fine-tuning framework for video anomaly detection, characterized in that: include: S1: Build a fine-tuning framework for video anomaly detection in few-shot scenes based on auxiliary tasks, generate corresponding segmentation maps and optical flow maps for real frames and model prediction frames, calculate segmentation loss, optical flow loss, and prediction loss between predicted frames and real frames, and use these three losses to fine-tune the anomaly detection model; S2: A three-stage training strategy is adopted. For the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network, and the segmentation and optical flow information fusion module, one part is trained in each of the three training stages, and the parameters of the other two parts are frozen, thereby reducing the instability of the training process; S3: using an existing video anomaly detection dataset to complete the training of the video anomaly detection model, and then completing the training of the auxiliary task network; S4: Use the few-shot fine-tuning framework for video anomaly detection to fine-tune the model on new scenes, test the model’s detection capabilities when facing new scenes, and construct the final anomaly detection model few-shot scene fine-tuning framework.

2. The method for constructing a few-sample fine-tuning framework for video anomaly detection according to claim 1, characterized in that: S1: constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks, including: S101: construct an anomaly detection model for outputting a predicted frame. The model can output a future predicted frame based on a plurality of continuous input video frames, and can determine whether this frame has an anomaly by comparing the difference between the image and the real future frame; S102: constructing a segmentation branch network to generate a segmentation map of the image; the network can extract shape information of the target in the image in the spatial dimension, such as a binary segmentation map of other objects such as a person or a vehicle; S103: construct an optical flow branch network to generate an optical flow map of the image; capable of extracting motion information of objects in the image in the time dimension; S104: Construct a segmentation and optical flow information fusion module to adaptively fuse global frequency information and local spatial information based on the difference and complementarity between frequency information and spatial information.

3. The method for constructing a few-sample fine-tuning framework for video anomaly detection according to claim 2, further comprising any one or more of the following: S101 comprises that the anomaly detection model is composed of a network similar to a U-Net structure, specifically, the network contains multiple convolutional layers, batch normalization layers and pooling layers, and each convolutional layer is followed by a batch normalization layer and a ReLU activation function; In the encoding stage, a pooling layer is connected after every two convolution layers, using a 3x3 convolution kernel and a 2x2 maximum pooling layer. The convolution layer extracts features, and the batch normalization layer is used to accelerate training and stabilize the network. The S102 includes: constructing a segmentation branch network, which is composed of a lightweight U-Net-like structure network. First, the features of the input image are extracted through three downsampling operations, and the high-dimensional input data is mapped to a low-dimensional space. The low-dimensional representation contains important features required for the segmentation task, while retaining semantic information of different scales. In the decoding stage, the network gradually expands the low-dimensional representation to the same size as the input image through three deconvolution operations, and finally generates a segmentation map. The deconvolution process also uses a jump connection, which directly transfers the high-resolution features of the shallow convolution to the decoding stage across layers, providing rich detail information for the decoding process, and finally outputs the segmentation map through the convolution layer; The S103 includes: constructing an optical flow branch network, which is composed of two encoders and a decoder. The encoder contains multiple convolution layers, batch normalization layers and pooling layers. Each convolution layer is followed by a batch normalization layer and a ReLU activation function. A pooling layer is connected after every two convolution layers, using a 3x3 convolution kernel and a 2x2 maximum pooling layer. For two consecutive frames of input pictures, they are respectively input into two decoders with the same structure. The parameters of the initial two layers of convolution of the two decoders are shared to achieve feature extraction and generalization capabilities. Then, the two decoders are subjected to two downsampling operations to gradually extract features of different scales and fuse these features. Next, the decoder part is subjected to two upsampling operations, through a deconvolution layer, a convolution layer, a batch normalization layer, a ReLU activation function and a pooling layer, to restore the fused features to the same spatial resolution as the input image. In the upsampling process, a jump connection is used to transmit the information of the two encoders to the upsampling layers corresponding to the decoder, respectively, and the high-resolution information from the shallow network is combined to finally output the optical flow map through the convolution layer. S104, constructing a segmentation and optical flow information fusion module, first extracting the shallow features of the next frame image after the first two layers of convolution of the encoder from the optical flow branch, recorded as , extract the high-level features before the last convolution layer from the segmentation branch, denoted as , for the feature , respectively, with the spatial range of and The pooling kernel encodes each channel, so the output of the height h and width w of the cth channel can be written as follows: Then, connect along the spatial dimension and And send it to 1×1 convolution, and then use the nonlinear activation function to calculate , and then along the spatial dimension It is divided into two parts: one along the height direction is recorded as , and the other along the width direction is , next, and Perform a 1×1 convolution transformation, and finally use the sigmoid function to get two new weight matrices: and , these weight matrices represent the attention weights of the optical flow features in the height and width dimensions; The same steps are also applied to the features of the split branch , and get two corresponding attention weight matrices and After obtaining these attention weights, apply the coordinate attention weights of the optical flow branch to the features of the segmentation branch Specifically, for a specific location , whose output It is expressed as: Similarly, the coordinate attention weights of the segmentation branch are applied to the features of the optical flow branch On, its output It is expressed as: In this way, two new feature representations are obtained and , which fuse the attention weights from the segmentation and optical flow branches respectively. Next, these two features are added element by element to obtain a new fused feature Finally, the original segmentation features and fusion features The concatenation is performed in the channel dimension to form a new feature representation, which is processed by a convolutional layer to output the final segmentation result. .

4. The method and system for constructing a few-sample fine-tuning framework for video anomaly detection according to claim 1, characterized in that: The S2 adopts a three-stage training strategy. For the three parts of the auxiliary task network: the segmentation branch network, the optical flow branch network, and the segmentation and optical flow information fusion module, one of the parts is trained in three training stages respectively, and the parameters of the remaining two parts are frozen, thereby reducing the instability of the training process, including: S201, in the first training stage, only the optical flow branch network is trained, and the parameters of the rest of the auxiliary task network are frozen, using two consecutive frames of images and As input, the label of the optical flow map is recorded as , the loss used during training is recorded as ,in Represents the parameters of the optical flow branch network; it is calculated using the Endpoint error (EPE) commonly used in the optical flow field , that is, calculating the predicted optical flow map and The Euclidean distance of S202, in the second training stage, the segmentation branch network is trained, and the parameters of the rest of the auxiliary task network are frozen, using a single frame image As input, the label of the segmentation map is recorded as , the output of the segmentation branch network is recorded as , the loss used during training is recorded as ,in Represents the parameters of the optical flow branch network, where intensity loss is used With gradient loss Add together to get the segmentation loss ; S203, in the third training stage, we only train the OSF block, and the rest of the parameters of the RGB2SegFlow network are also frozen. We use two consecutive frames of images and Input to the RGB2SegFlow network, the optical flow features and segmentation features obtained from the two branch networks are used as the input of the OSF block, using the above segmentation map As labels, the output is recorded as , the loss function is the same as in the second training stage.

5. The method and system for constructing a few-sample fine-tuning framework for video anomaly detection according to claim 1, characterized in that: S301: using an existing video anomaly detection dataset to complete the training of the video anomaly detection model; Then the training of the auxiliary task network is completed, including: The anomaly detection model is pre-trained using the public video anomaly detection dataset Shanghai Tech, and a segmentation map is generated on the Shanghai Tech dataset as a real segmentation label using the pre-trained model of the open source algorithm YOLOv8 and MobileSAM; an optical flow map is generated on the Shanghai Tech dataset as a real optical flow label using the pre-trained model provided by the open source algorithm GMA, including: Obtain the public video anomaly detection dataset Shanghai Tech, and take every five consecutive frames in the training set of Shanghai Tech as a group, with the first four frames as input and the fifth frame as the true value image. The goal of the training model is to predict the future frames given the past four frames to be close to the true value image, so that the model can learn the ability to predict normal videos. Obtain the pre-trained models of the open source algorithms YOLOv8, MobileSAM, and GMA; use the YOLOv8 model to detect targets such as people, cars, and items carried by people in the Shanghai Tech training set, input the detected target boxes and the corresponding original images into the MobileSAM model, and output the binary segmentation map of the target as the true value map of the segmentation task; input every two frames in the ShanghaiTech training set as a group into the GMA model, and train the auxiliary task network on the Shanghai Tech training set in an end-to-end manner, so that the network can learn the mapping relationship between the original image and the segmentation map and optical flow map; The setting of initialization parameters of the anomaly detection model and the auxiliary task network includes: The initial parameters of the anomaly detection model are randomly initialized in the pre-training stage, and the initial parameters of the auxiliary task network are randomly initialized; Determining the loss function of the anomaly detection model and the auxiliary task network includes: The loss function of the anomaly detection model adopts a weighted linear combination of intensity loss and gradient loss; the auxiliary task network adopts a weighted linear combination of intensity loss and gradient loss as the loss function of the segmentation task, and adopts a weighted linear combination of intensity loss and EPE (Endpoint Error) as the loss function of the segmentation task; Determining the hyper parameters related to the anomaly detection model and the auxiliary task network includes: The stochastic gradient descent algorithm is used in the pre-training process of the anomaly detection model, with a learning rate of 0.0002 and a batch size of 16. The pre-training process requires 13 epochs to iterate until the model converges. The stochastic gradient descent algorithm is used in the training process of the auxiliary task network, with a learning rate of 0.0002 and a batch size of 8. The first training stage requires 30,000 iterations until the model converges, the second training stage requires 8,000 iterations until the model converges, and the third training stage requires 6,000 iterations until the model converges.

6. The method and system for constructing a few-sample fine-tuning framework for video anomaly detection according to claim 1, characterized in that: The few-sample fine-tuning framework for video anomaly detection is used to fine-tune the model on new scenes and test the detection ability of the model when facing new scenes, including: S401, using the test set of the public video anomaly detection dataset UCSD Ped2, fine-tuning the pre-trained anomaly detection model with a few samples; S402, setting initialization parameters of the pre-trained anomaly detection model, where the pre-trained anomaly detection model uses parameters obtained by pre-training on the Shanghai Tech training set as initialization parameters; S403, determining the loss function of the pre-trained anomaly detection model; the loss function of the anomaly detection model adopts a weighted linear combination of the intensity loss and gradient loss of the prediction graph, the loss function of the segmentation task, and the loss function of the optical flow estimation task. Specifically, Loss function for prediction graph , the loss function for the segmentation task ,in Refers to the segmentation map generated by the auxiliary task network using the real frame, refers to the segmentation map generated by the auxiliary task network using the frames predicted by the anomaly detection model, is the strength loss of the segmentation map, is the gradient loss of the segmentation map, the loss function of the optical flow estimation task ,in Refers to the optical flow map generated by the auxiliary task network using the real past frame and the current frame. It refers to the optical flow map generated by the auxiliary task network using the real past frames and the current frame predicted by the anomaly detection model. is the strength loss of the segmentation map, isEndpointerror(EPE); S404, determining the relevant hyper parameters in the pre-trained anomaly detection model, the learning rate is 0.0015, the batch size is 3, and the fine-tuning process needs to iterate 870 epochs until the model converges; S405, test the fine-tuned anomaly detection model on the test set of UCSD Ped2, so as to evaluate the performance of the pre-trained anomaly detection model after fine-tuning using the fine-tuning framework on the new scene; use the fine-tuned anomaly detection model on the test set of UCSD Ped2 to calculate the mean square error between the predicted image and the true image of each frame of the image, and the frames with large differences are regarded as abnormal frames; then calculate the area under the ROC (receiver operating characteristic curve) to obtain the AUC score (%), and use the AUC to reflect the ability of the model to detect anomalies. The larger the AUC, the stronger the ability of the model to distinguish whether it is abnormal.

7. A system for constructing a fine-tuning framework for video anomaly detection in a few-sample scene based on auxiliary tasks, characterized in that: include: A module for building a fine-tuning framework for few-shot scenes, which is used to build a fine-tuning framework for video anomaly detection in few-shot scenes based on auxiliary tasks; Anomaly detection model and auxiliary task network training module, which is used to pre-train the anomaly detection model using the training set of the Shanghai Tech dataset, and use the pre-trained models of the open source algorithms YOLOv8, MobileSAM, and GMA to generate corresponding segmentation maps and optical flow maps on the Shanghai Tech dataset as the truth maps of the segmentation task and the optical flow estimation task, so as to train the auxiliary task network; The anomaly detection model fine-tuning module is used to use the fine-tuning framework to fine-tune the anomaly detection model using a small number of samples in the new scenario, so that the model can adapt to the new scenario. Different from the scenario of the Shanghai Tech dataset in the pre-training process, a small number of samples from the test set of UCSD Ped2 are used to train the anomaly detection model during fine-tuning. The anomaly detection model testing module is used to test the fine-tuned anomaly detection model using known test images and evaluate the performance of the model fine-tuned using the few-sample fine-tuning framework in new scenarios.

8. A method for fine-tuning video anomaly detection in a few-sample scene based on auxiliary tasks, characterized in that: A few-sample fine-tuning framework for video anomaly detection is constructed by using the method and system for constructing a few-sample fine-tuning framework for video anomaly detection described in any one of claims 1-6 or the system for constructing a few-sample fine-tuning framework for video anomaly detection described in claim 7. A small amount of data samples in any new scene is used as fine-tuning data for the fine-tuning framework, and fine-tuning is performed to obtain an anomaly detection model that can adapt to the new scene.

9. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method described in any one of claims 1 to 6 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 6 or 8.

Images