A small sample video abnormal behavior classification method, system, device and medium

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a small sample video abnormal behavior classification method, system, device and medium, comprising: collecting Kinetics dataset and various abnormal behavior video dataset; establishing a self-supervised graph neural network model composed of a feature extraction network and a graph generation module; pre-training the self-supervised graph neural network model on the Kinetics dataset, and fine-tuning the training through various abnormal behavior video dataset to obtain a fine-tuned self-supervised graph neural network model. The present application realizes small sample learning through self-supervised learning and graph neural network, solves the problem that the existing small sample recognition method cannot fully utilize the time sequence information of the video for modeling, and cannot fully mine the hidden information of the data to improve the performance of the model, and the present application can not only learn the long-term time sequence information and spatial information of the video, but also is not easily affected by small data volume and noise.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device and medium for classifying abnormal behaviors in few-sample videos based on a self-supervised graph neural network. Background Technology

[0002] In recent years, with the emergence of powerful computing platforms, large-scale datasets, advanced models, and convolutional neural network (CNN) algorithms, video anomaly detection based on CNNs has begun to replace traditional methods and has been deployed in real-world scenarios. However, traditional neural networks rely on large amounts of labeled data. When using a model to identify a category not seen in the training set, it requires collecting tens of thousands of samples as training data. This work is tedious, labor-intensive, and costly. This phenomenon is even more pronounced in video behavior classification, where video collection and labeling are significantly more difficult than image collection. Furthermore, some anomaly behavior categories are not commonly seen, leading to data scarcity. Therefore, few-shot learning, which involves learning from a small number of samples, is increasingly crucial. Increasingly attracting attention, humans possess the ability to learn and generalize rapidly from a limited number of samples. For example, humans can learn features from a few image samples and quickly distinguish the category of the sample. Typically, humans only need to see one or two images of a certain animal to recognize the species in different scenarios because humans have learned a lot of experiential knowledge. Using this experiential knowledge, they can quickly identify new targets. However, due to the limited number of training samples for few-shot learning, how to learn more knowledge from limited samples and more accurately classify video behaviors to promote the application and popularization of video abnormal behavior classification methods is a major challenge facing few-shot learning.

[0003] Few-shot video behavior classification is a behavior classification technique developed based on few-shot learning models. It can learn on a small amount of dataset and achieve good generalization. It has been widely promoted and applied in real-world scenarios. Video abnormal behavior classification is one such application, such as detecting abnormal behavior of people in public places, recognizing abnormal behavior of pedestrians, and detecting abnormal behavior of passengers. These abnormal behaviors pose a huge threat to the lives and property of the public. Existing few-shot abnormal behavior classification uses traditional deep convolutional neural networks for classification. However, the collection of labeled data for abnormal video behaviors is difficult, and the amount is small and the types are numerous. In this case, traditional deep learning networks often suffer from low recognition rates and insufficient generalization ability due to insufficient labeled data. Secondly, humans often need information from several frames to recognize an action. How to learn temporal information is also a huge challenge for the promotion and application of video abnormal behavior classification.

[0004] Currently, although some researchers have provided different solutions to the problem of learning temporal information in small sample abnormal behavior classification data, the problem of reduced model performance due to sample scarcity is still not solved. Moreover, video is different from images, as it adds a time dimension. How to efficiently learn the temporal information of video and improve the classification performance of the model has always been an unsolved problem. Summary of the Invention

[0005] This invention provides a method, system, device, and medium for classifying abnormal behavior in small sample videos. The technical problem it solves is that existing small sample recognition methods cannot fully utilize the temporal information of videos for modeling and cannot fully mine the hidden information in the data to improve model performance.

[0006] To address the above technical problems, this invention provides a method, system, device, and medium for classifying abnormal behaviors in few-sample videos based on a self-supervised graph neural network.

[0007] In a first aspect, the present invention provides a method for classifying abnormal behavior in small-sample videos, the method comprising the following steps:

[0008] Collect Kinetics datasets and various abnormal behavior video datasets;

[0009] A self-supervised graph neural network model consisting of a feature extraction network and a graph generation module is established.

[0010] The feature extraction network is pre-trained on the Kinetics dataset to obtain a pre-trained feature extraction network.

[0011] Based on the pre-trained feature extraction network, the graph generation module is pre-trained using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model;

[0012] Various abnormal behavior video datasets are input into a pre-trained self-supervised graph neural network model for fine-tuning training, resulting in a fine-tuned self-supervised graph neural network model.

[0013] In a further implementation, the step of pre-training the feature extraction network on the Kinetics dataset to obtain the pre-trained feature extraction network includes:

[0014] Sparse sampling is performed on the Kinetics dataset to obtain video images with a preset number of sampled frames;

[0015] The video image is randomly flipped to obtain rotated images at different angles;

[0016] A category label and a rotation angle label are set on the rotated image to obtain a labeled image;

[0017] Using the rotation angle of the label image as a supervision signal, the feature extraction network is subjected to self-supervised learning to obtain a pre-trained feature extraction network.

[0018] In a further implementation, the step of pre-training the graph generation module based on the pre-trained feature extraction network and using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model includes:

[0019] Sparse sampling is performed on the Kinetics dataset to divide each video data in the Kinetics dataset into several video segments;

[0020] Extract images from each video segment to obtain several video segment images corresponding to each video data;

[0021] The pre-trained feature extraction network is used to extract features from the video segment image to obtain a set of video feature nodes;

[0022] A set of node relationships is obtained based on the set of video feature nodes and the pre-constructed convolutional neural network;

[0023] Construct a video graph network based on the set of video feature nodes and the set of node relationships;

[0024] Graph convolution is performed on the video graph network to update node features and edge information;

[0025] By aggregating nodes, each video graph network is aggregated into a single node, resulting in an aggregated node.

[0026] The Softmax classifier is used to classify and identify the pool nodes to obtain a pre-trained graph generation module, thereby realizing the pre-training of the self-supervised graph neural network model.

[0027] In a further implementation, the step of obtaining the node relationship set based on the video feature node set and the pre-built convolutional neural network includes:

[0028] Calculate the square of the feature difference between any two nodes in the video feature node set to obtain the node difference feature;

[0029] The node difference features are input into a pre-constructed convolutional neural network, and the features of a single edge are obtained through a sigmoid layer.

[0030] The node relationship set is calculated based on the single edge features and the number of video frames;

[0031] The convolutional neural network includes a convolutional layer, a BatchNorm layer, another convolutional layer, a BatchNorm layer, and a ReLU layer connected in sequence.

[0032] In a further implementation, the formula for performing graph convolution on the video graph network to update node features and edge information is as follows:

[0033]

[0034]

[0035] In the formula, f (k+1) (x k ) represents the graph network after (K+1) graph convolutions, σ represents the activation function, and D (k) This represents the degree matrix of the k-th graph convolution. Let W represent the adjacency matrix of the k-th graph convolution. edge Represents learning weights, Let the edge between any two frame nodes i and j be an MLP. edge Describes a convolutional neural network, f(z) i f(z) represents the feature of frame node i. j ) represents the characteristics of frame node j.

[0036] In a further embodiment, the feature extraction network includes a RestNet-50 convolutional network.

[0037] In a further implementation, the loss function used by the feature extraction network is:

[0038]

[0039] In the formula, c n y represents the category label of the nth video. m p represents the rotation angle label of the m-th video. n p represents the category probability value predicted by the feature extraction network. m The value represents the probability of the rotation angle predicted by the feature extraction network, γ represents the contribution of the self-supervised loss to the total loss, and N represents the total number of videos.

[0040] Secondly, the present invention provides a small-sample video abnormal behavior classification system, the system comprising:

[0041] The data acquisition module is used to acquire Kinetics datasets and various abnormal behavior video datasets;

[0042] The model building module is used to build a self-supervised graph neural network model consisting of a feature extraction network and a graph generation module;

[0043] The model pre-training module is used to pre-train the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network; it is also used to pre-train the graph generation module based on the pre-trained feature extraction network using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model.

[0044] The model fine-tuning module is used to input various abnormal behavior video datasets into a pre-trained self-supervised graph neural network model for fine-tuning training, resulting in a fine-tuned self-supervised graph neural network model.

[0045] Thirdly, the present invention also provides a computer device, including a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the computer device performs the steps of implementing the above-described method.

[0046] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0047] This invention provides a method, system, device, and medium for classifying anomalous behaviors in small-sample videos based on a self-supervised graph neural network. The method utilizes the Kinetics dataset for self-supervised learning, encoding each frame of the image into graph nodes through feature extraction. Each video constitutes a graph network, and temporal information is captured by node aggregation, thereby learning the long-term temporal information of the video. Finally, the graph network is aggregated into a single node through node aggregation, and the classification result of the action is obtained after passing through a softmax layer. Video frames with different anomalous behaviors are then used to fine-tune the pre-trained self-supervised graph neural network model. Compared with existing technologies, this method not only utilizes self-supervised learning to mine potential supervisory information in the data and improves the model's feature extraction capability, but also breaks away from traditional video behavior classification methods by using graph convolutional neural networks, fully utilizing the temporal information of the video, and obtaining an accurate classification model with a small number of samples. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a method for classifying abnormal behaviors in small-sample videos provided by an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the self-supervised graph neural network model architecture provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of a video graph network provided in an embodiment of the present invention;

[0051] Figure 4 This is a block diagram of a small-sample video abnormal behavior classification system provided in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0054] refer to Figure 1 This invention provides a method for classifying abnormal behaviors in small sample videos, such as... Figure 1 As shown, the method includes the following steps:

[0055] S1. Collect Kinetics dataset and various abnormal behavior video datasets.

[0056] Before training the model, this embodiment needs to collect the Kinetics dataset (a video behavior classification dataset) for pre-training the self-supervised graph neural network model and various abnormal behavior video datasets for fine-tuning the self-supervised graph neural network model. The various abnormal behavior video datasets include 9 categories of abnormal behavior video datasets, such as arson, shooting, traffic accidents, burglary, fighting, robbery, explosion, outdoor theft, and vandalism. Each category of abnormal behavior video dataset includes 100 video samples.

[0057] S2. Establish a self-supervised graph neural network model consisting of a feature extraction network and a graph generation module.

[0058] This invention combines self-supervised learning to solve the problem of classifying abnormal behaviors in videos with few samples. Self-supervised learning has become a popular research area in recent years. It aims to improve the feature extraction ability of unlabeled data by designing proxy tasks to mine the data's own representational characteristics as supervision information (the supervision information obtained here does not refer to the original task labels faced by self-supervised learning, but the constructed proxy task labels). Video abnormal behavior classification involves many different types of behavior. For some categories, it is difficult to collect a large number of effective samples. Self-supervised learning is an effective method to solve this problem.

[0059] Meanwhile, this invention introduces graph neural networks to address the problem that models cannot learn long-term temporal information from videos. Unlike images, videos contain a time dimension. Typically, humans need to combine information from several video frames to determine the category of an action. Temporal changes are related to the accuracy of behavior classification. We use graph convolutional networks to solve this problem. We represent each video frame as a graph node, enabling the model to learn the spatial information of the video. The edges between nodes represent the relationships between frames. By establishing these relationships, the model can learn not only information from adjacent frames but also information from distant frames, thereby achieving the learning of long-term temporal information. Video abnormal behavior classification data usually has few samples. Therefore, we use self-supervised learning to mine potential supervisory information from the data and combine it with few-sample learning to solve the problem of data scarcity.

[0060] S3. Pre-train the feature extraction network on the Kinetics dataset to obtain the pre-trained feature extraction network.

[0061] In one embodiment, the step of pre-training the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network includes:

[0062] The Kinetics dataset is sparsely sampled to obtain video images with a preset number of sampling frames; in this embodiment, the preset number of sampling frames is preferably set to 8 frames, but those skilled in the art can adjust it according to the specific implementation situation, and it is not limited to the embodiment of the present invention.

[0063] The video image is randomly flipped to obtain rotated images at different angles;

[0064] A category label and a rotation angle label are set on the rotated image to obtain a labeled image;

[0065] Using the rotation angle of the label image as a supervision signal, the feature extraction network is subjected to self-supervised learning to obtain a pre-trained feature extraction network; in this embodiment, the feature extraction network includes a RestNet-50 convolutional network.

[0066] The loss function used by the feature extraction network is:

[0067]

[0068] In the formula, the loss function used by the feature extraction network consists of two parts: the first term is the classification loss, and the second term is the self-supervised loss. m and n represent the video sequence numbers, where m, n ∈ [0, N]; c n y represents the category label of the nth video. m p represents the rotation angle label of the m-th video. np represents the category probability value predicted by the feature extraction network. m The value represents the probability of the rotation angle predicted by the feature extraction network, γ represents the contribution of the self-supervised loss to the total loss, and N represents the total number of videos.

[0069] Specifically, such as Figure 2 As shown, in this embodiment, 8 frames of images are sparsely sampled from each frame of video input to the feature extraction network, and then randomly flipped to obtain rotated images at different angles. The flip angles are any angles in the set A = {0°, 90°, 180°, 270°}. Then, category labels and rotation angle labels are assigned to the rotated images to obtain labeled images. Next, the rotation angle of the labeled images is used as a supervision signal for self-supervised learning, so that the feature extraction network not only learns to predict the category of each frame of image, but also learns to predict the rotation angle of the image. In this embodiment, this task is called pretext (ancillary task). Compared with the single task of predicting category labels, the feature extraction network trained in this embodiment can learn stronger feature extraction capabilities, and can more accurately predict the category and rotation angle of the image. Because the image is rotated, the robustness of the model is also improved to some extent.

[0070] This embodiment uses the Kinetics dataset to pre-train the feature extraction network, which enables the network to learn to predict the action category and rotation angle of each frame of video, thereby improving the network's feature extraction ability and robustness. At the same time, the network parameters are updated through a loss function.

[0071] S4. Based on the pre-trained feature extraction network, the graph generation module is pre-trained using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model.

[0072] In one embodiment, the step of pre-training the graph generation module based on the pre-trained feature extraction network and using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model includes:

[0073] Sparse sampling is performed on the Kinetics dataset to divide each video data in the Kinetics dataset into several video segments;

[0074] Extract images from each video segment to obtain several video segment images corresponding to each video data;

[0075] The pre-trained feature extraction network is used to extract features from the video segment image to obtain a set of video feature nodes;

[0076] A set of node relationships is obtained based on the set of video feature nodes and the pre-constructed convolutional neural network;

[0077] Construct a video graph network based on the set of video feature nodes and the set of node relationships;

[0078] Graph convolution is performed on the video graph network to update node features and edge information;

[0079] By aggregating nodes, each video graph network is aggregated into a single node, resulting in an aggregated node.

[0080] The Softmax classifier is used to classify and identify the pool nodes to obtain a pre-trained graph generation module, thereby realizing the pre-training of the self-supervised graph neural network model.

[0081] In one embodiment, the step of obtaining the node relationship set based on the video feature node set and the pre-built convolutional neural network includes:

[0082] Calculate the square of the feature difference between any two nodes in the video feature node set to obtain the node difference feature;

[0083] The node difference features are input into a pre-constructed convolutional neural network, and the features of a single edge are obtained through a sigmoid layer.

[0084] The node relationship set is calculated based on the single edge features and the number of video frames;

[0085] The convolutional neural network includes a convolutional layer, a BatchNorm layer (batch normalization layer), a convolutional layer, a BatchNorm layer, and a ReLU layer connected in sequence.

[0086] In one embodiment, the formula for performing graph convolution on the video graph network to update node features and edge information is as follows:

[0087]

[0088]

[0089] In the formula, f (k+1) (x k ) represents the graph network after (K+1) graph convolutions, σ represents the activation function, and D (k) This represents the degree matrix of the k-th graph convolution. Let W represent the adjacency matrix of the k-th graph convolution. edge Represents learning weights, Let the edge between any two frame nodes i and j be an MLP. edge Describes a convolutional neural network, f(z) if(z) represents the feature of frame node i. j ) represents the characteristics of frame node j.

[0090] Specifically, this embodiment first performs sparse sampling on each video in the Kinetics dataset to obtain several video segments. This embodiment preferentially divides each video data into 8 video segments, extracts one frame from each segment, and selects 8 video segment images as the feature representation of the video. That is, 8 video segment images represent one video, which allows each video to be represented by a fixed-length segment, while also enabling long-distance temporal modeling. Simultaneously, a pre-trained feature extraction network is used to extract the features of the 8 video segment images corresponding to each video (here, the input video segment images are not flipped to maintain the temporal and spatial continuity of the video action) to obtain the set β of video feature nodes representing the features of each video. m,t ={ε1,ε2,…,ε t}, where m represents the m-th video in the dataset, and t represents the frame number of the m-th video; calculate the square of the difference between any two nodes in the video feature node set corresponding to each video to obtain the node difference feature. The formula for calculating the node difference feature is:

[0091] E m,jj′ =(f(z) j )-f(z j′ )) 2

[0092] In the formula, E m,jj′ f(z) represents the edge between the node in the j-th frame and the node in the j′-th frame in the m-th video. j f(z) represents the feature of the j-th frame node. j′ ) represents the features of the node in frame j′.

[0093] Then, the node difference features are input into a convolutional neural network, and the features of a single edge are obtained through a sigmoid layer, followed by t 2 The calculation yields the node relationship set E. m,jj′ , of which E m,jj′ ={e m,12 e m,21, , ..., e m,jj′ , ..., e m,tt}, the set of node relationships E m,jj′ This indicates the strength of the relationship between nodes, that is, the strength of the relationship between frames, such as e. m,12 This represents the strength of the relationship between the first and second frames of a video segment in the m-th video. For example... Figure 3As shown, this embodiment constructs a video graph corresponding to the video based on the obtained set of video feature nodes and the set of node relationships. Thus, a video constitutes a video graph G(β). m,t E m,jj′ Each node represents a feature of a video frame, and the edges represent the strength of the relationship between video frames.

[0094] Next, graph convolution is performed on the video graph to update node features and edge information. After several graph convolutions, each node will contain information from other nodes, thus enabling the graph generation module to learn the temporal and spatial information of the video.

[0095] Finally, through node aggregation, each video graph network is aggregated into a single node, resulting in an aggregated node. This aggregated node is then fed into a Sofmax layer to obtain the node category, which represents the category of abnormal video behavior. This category is then further processed using the node loss function Loss. node The graph generation module is optimized, and the node loss is reduced. node The formula is:

[0096]

[0097] In the formula, q n p represents the category of the nth node. n This represents the nth node category predicted by the graph generation model, where N represents the total number of video nodes.

[0098] This embodiment pre-trains the graph generation module, enabling it to learn the spatiotemporal information of the video from the existing dataset. This allows for final fine-tuning of the self-supervised graph neural network model in subsequent stages to adapt to the few-shot learning task.

[0099] S5. Input various abnormal behavior video datasets into the pre-trained self-supervised graph neural network model for fine-tuning training to obtain a fine-tuned self-supervised graph neural network model.

[0100] This embodiment uses various collected abnormal behavior video datasets to perform few-sample learning and fine-tuning on a pre-trained self-supervised graph neural network model, enabling it to be applied to various abnormal behavior classification scenarios with scarce data. The method for fine-tuning the training process is the same as the processing method in steps S3 to S4 above, and will not be repeated here. When fine-tuning using various abnormal behavior video datasets, this embodiment can use graph neural networks to represent the small amount of collected video abnormal behavior data as a graph structure, and use node aggregation technology to learn temporal and spatial information to achieve accurate classification of abnormal video behaviors.

[0101] After the complete training described above, the self-supervised graph neural network model is constructed. Finally, the model is deployed as a computer program that can be read by a computer-readable storage medium. The collected video dataset is input into the constructed self-supervised graph neural network model to classify abnormal behaviors.

[0102] This invention provides a method for classifying abnormal video behavior using a few samples. It improves the model's feature extraction capability and robustness through self-supervised learning, and introduces a graph neural network to enable the model to better learn temporal information between frames and differences between videos, thus enhancing the model's ability to discriminate features. Compared with existing technologies, this embodiment utilizes the advantages of self-supervised learning and graph neural networks for few-sample learning, solving the problem of model degradation caused by sample scarcity, improving the accuracy and efficiency of abnormal video behavior classification, and broadening the application areas of abnormal video behavior classification.

[0103] It should be noted that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] In one embodiment, such as Figure 4 As shown, this embodiment of the invention provides a small-sample video abnormal behavior classification system, the system comprising:

[0105] Data acquisition module 101 is used to acquire Kinetics datasets and various abnormal behavior video datasets;

[0106] Model building module 102 is used to build a self-supervised graph neural network model consisting of a feature extraction network and a graph generation module;

[0107] The model pre-training module 103 is used to pre-train the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network; it is also used to pre-train the graph generation module based on the pre-trained feature extraction network using the Kinetics dataset to achieve the pre-training of the self-supervised graph neural network model.

[0108] The model fine-tuning module 104 is used to input various abnormal behavior video datasets into the pre-trained self-supervised graph neural network model for fine-tuning training, so as to obtain the fine-tuned self-supervised graph neural network model.

[0109] For specific limitations regarding a small-sample video abnormal behavior classification system, please refer to the above-described limitations regarding a small-sample video abnormal behavior classification method, which will not be repeated here. Those skilled in the art will recognize that the various modules and steps described in conjunction with the embodiments disclosed in this application can be implemented in hardware, software, or a combination of both. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] This invention provides a few-shot video abnormal behavior classification system. The system collects existing datasets and various abnormal behavior video datasets through a data acquisition module; it builds a self-supervised graph neural network model consisting of a feature extraction network and a graph generation module through a model building module; simultaneously, a model pre-training module pre-trains the graph network using existing datasets to enhance the model's ability to extract temporal and spatial information; and a model fine-tuning module fine-tunes the pre-trained model using real-time collected abnormal behavior video datasets to achieve few-shot learning. Compared with existing technologies, this few-shot classification method based on self-supervised learning and graph neural networks can not only mine useful supervisory information through self-supervised learning and improve the model's feature extraction ability, but also effectively learn the temporal and spatial information of videos using graph neural networks, enabling long-term temporal modeling.

[0111] Figure 5 This invention provides a computer device including a memory, a processor, and a transceiver, which are connected to each other via a bus. The memory is used to store a set of computer program instructions and data, and can transmit the stored data to the processor. The processor can execute the program instructions stored in the memory to perform the steps of the above method.

[0112] The memory may include volatile memory or non-volatile memory, or both; the processor may be a central processing unit, a microprocessor, an application-specific integrated circuit, a programmable logic device, or a combination thereof. By way of example, but not limitation, the programmable logic device described above may be a complex programmable logic device, a field-programmable gate array, a general-purpose array logic, or any combination thereof.

[0113] In addition, memory can be a physically independent unit or integrated with the processor.

[0114] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0115] In one embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0116] This invention provides a method, system, device, and medium for classifying anomalous behaviors in few-shot videos based on a self-supervised graph neural network. The method pre-trains the feature extraction model using a self-supervised approach, effectively improving the model's robustness. Simultaneously, pre-training the graph neural network using the existing Kinetics video behavior dataset enables the model to learn the spatiotemporal features of the video. This pre-training process effectively helps the model obtain more effective spatiotemporal information during the few-shot learning stage, improving the model's classification accuracy. Furthermore, this application utilizes various collected video anomalous behavior datasets for network fine-tuning to achieve the classification of video anomalous behaviors.

[0117] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed, it can include the processes of the embodiments of the above methods.

[0119] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. A small sample video anomaly behavior classification method, characterized in that, The method comprises the following steps: Collecting a Kinetics dataset and a video dataset of various abnormal behaviors; Building a self-supervised graph neural network model composed of a feature extraction network and a graph generation module; Pre-training the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network; Pre-training the graph generation module based on the pre-trained feature extraction network using the Kinetics dataset to realize pre-training of the self-supervised graph neural network model; Inputting the video dataset of various abnormal behaviors into the pre-trained self-supervised graph neural network model for fine-tuning training to obtain a fine-tuned self-supervised graph neural network model; The step of pre-training the graph generation module based on the pre-trained feature extraction network using the Kinetics dataset to realize pre-training of the self-supervised graph neural network model comprises: Sparse sampling the Kinetics dataset, and dividing each video data in the Kinetics dataset into a plurality of video clips; Extracting images in each video clip to obtain a plurality of video clip images corresponding to each video data; Extracting features of the video clip images using the pre-trained feature extraction network to obtain a video feature node set; Obtaining a node relationship set according to the video feature node set and a pre-constructed convolutional neural network; Constructing a video graph network according to the video feature node set and the node relationship set; Performing graph convolution on the video graph network to update node features and edge information; Converging each video graph network into a single node by node convergence to obtain a converged node; Classifying and identifying the converged node using a Softmax classifier to obtain a pre-trained graph generation module to realize pre-training of the self-supervised graph neural network model.

2. The small sample video anomaly behavior classification method of claim 1, wherein, The step of pre-training the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network comprises: Sparse sampling the Kinetics dataset to obtain video images with a preset number of sampling frames; Randomly flipping the video images to obtain rotated images with different angles; Setting class labels and rotation angle labels for the rotated images to obtain labeled images; Performing self-supervised learning on the feature extraction network using the rotation angles of the labeled images as supervision signals to obtain a pre-trained feature extraction network.

3. The small sample video anomaly behavior classification method of claim 1, wherein, The step of obtaining a node relationship set according to the video feature node set and a pre-constructed convolutional neural network comprises: Calculating the square of the difference between any two node features in the video feature node set to obtain a node difference feature; Inputting the node difference feature into a pre-constructed convolutional neural network and obtaining a single edge feature through a sigmoid layer; Obtaining a node relationship set according to the single edge feature and the number of video frames; The convolutional neural network comprises, in sequence, a convolutional layer, a BatchNorm layer, a convolutional layer, a BatchNorm layer, and a Relu layer.

4. The small sample video anomaly behavior classification method of claim 1, wherein, The calculation formula for performing graph convolution on the video graph network to update node features and edge information is: In the formula, f (k+1) (x k ) represents the graph network after (K+1) times of graph convolution, σ represents an activation function, D (k) represents the degree matrix of the kth graph convolution, represents the adjacency matrix of the kth graph convolution, W edge represents a learning weight, represents an edge between any two frame nodes i and j, MLP edge represents a convolutional neural network, f(z i ) represents the feature of the frame node i, and f(z j ) represents the feature of the frame node j.

5. The small sample video anomaly behavior classification method of claim 1, wherein: The feature extraction network comprises a RestNet-50 convolution network.

6. The small sample video anomaly behavior classification method of claim 1, wherein, The loss function adopted by the feature extraction network is: wherein c n represents the class label of the nth video, y m represents the rotation angle label of the mth video, p n represents the class probability value predicted by the feature extraction network, p m represents the rotation angle probability value predicted by the feature extraction network, γ represents the contribution degree of the self-supervised loss to the total loss, and N represents the total number of videos.

7. A small sample video anomaly behavior classification system characterized by, The system comprises: a data acquisition module configured to acquire a Kinetics dataset and various abnormal behavior video datasets; a model establishment module configured to establish a self-supervised graph neural network model comprising a feature extraction network and a graph generation module; a model pre-training module configured to pre-train the feature extraction network on the Kinetics dataset to obtain a pre-trained feature extraction network, and further configured to pre-train the graph generation module based on the pre-trained feature extraction network and using the Kinetics dataset to realize pre-training of the self-supervised graph neural network model; a model fine-tuning module configured to input the various abnormal behavior video datasets into the pre-trained self-supervised graph neural network model for fine-tuning training to obtain a fine-tuned self-supervised graph neural network model; the pre-training of the graph generation module based on the pre-trained feature extraction network and using the Kinetics dataset to realize the pre-training of the self-supervised graph neural network model comprises: performing sparse sampling on the Kinetics dataset, and dividing each video data in the Kinetics dataset into a plurality of video clips; extracting images in each video clip to obtain a plurality of video clip images corresponding to each video data; extracting features of the video clip images using the pre-trained feature extraction network to obtain a video feature node set; obtaining a node relationship set according to the video feature node set and a pre-constructed convolutional neural network; constructing a video graph network according to the video feature node set and the node relationship set; performing graph convolution on the video graph network to update node features and edge information; aggregating each video graph network into a single node by node aggregation to obtain an aggregated node; classifying and identifying the aggregated node using a Softmax classifier to obtain a pre-trained graph generation module to realize the pre-training of the self-supervised graph neural network model.

8. A computer device, comprising: The computer device comprises a processor and a memory, the processor is connected with the memory, the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory to enable the computer device to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 6 is realized.

Images