Self-supervised action process anomaly detection method and system based on comparative learning

Abstract

The invention provides a self-supervised action process anomaly detection method and system based on comparative learning, relates to the field of operation process action anomaly detection in industrial control, and is used for solving the problems of dependence on a large amount of labeled data and insufficient model generalization ability in an existing method. Through an innovative video representation learning model and a comparative learning method, the system can generate pseudo-annotation data by using an unannotated action process video, so that the annotation cost is remarkably reduced. The core technology comprises a video representation model based on ResNet, a time convolution layer and a Transform encoder, and an action label generation and mapping technology combined with a dynamic time warping algorithm. The main functions of the system are as follows: capturing camera stream data in real time, identifying and comparing action processes through a trained action detection model, and marking as abnormal if action missing, sequence error or redundant actions are detected. The method is suitable for industrial production, video monitoring, medical diagnosis and other scenes.

Description

Technical Field

[0001] The present invention relates to the field of abnormal detection of operation process actions in industrial control, and particularly to a self-supervised action process abnormal detection method based on contrast learning. Background Art

[0002] With the rapid development of intelligent manufacturing, automated production, and intelligent monitoring technologies, action process monitoring and abnormal detection have received increasing attention in multiple fields such as industrial production, video surveillance, and medical diagnosis. In these fields, timely and effective detection of abnormal situations in the process can not only improve production efficiency but also reduce potential safety hazards and resource waste. Traditional abnormal detection methods usually rely on supervised learning and require a large amount of labeled abnormal data for training. However, in actual scenarios, abnormal data is often difficult to obtain or the annotation cost is extremely high, especially in complex action processes, where the cost of manual annotation is even more expensive and time-consuming. Therefore, the application of past learning methods in abnormal detection has been somewhat limited. Existing industrial control operation process abnormal recognition methods often require a large amount of manually labeled data for model training and cannot quickly identify abnormal situations such as missing actions, incorrect sequences, or redundant actions in actual scenarios, thus unable to ensure the efficiency and safety of the process.

[0003] The existing patent document with the literature number CN 118470014 A (the invention patent has been authorized) discloses an industrial abnormal detection method and system, which includes: obtaining an image to be abnormally detected; based on the obtained image and a pre-trained abnormal detection model, detecting and locating the abnormality; the abnormal detection model includes: a backbone network, a pooling layer, a cascade flow, a twin flow, and a constant flow; the cascade flow includes several sequentially arranged flow blocks. This invention makes the difference between abnormal data features and normal data features more obvious, thereby improving the effect of locating abnormalities. The object of comparison is a single picture, and this existing technology does not mention the abnormal detection of operation processes or work processes in industrial control.

[0004] The existing patent document with the document number CN118115935A discloses a method and device for identifying abnormal behaviors, which includes: obtaining M target video segments with detection areas from N video segments containing the complete operation process of an operator, where N and M are positive integers greater than 1, and M is less than or equal to N; obtaining the action behavior list corresponding to the complete operation process of the operator based on the M target video segments; and identifying the abnormal action behaviors of the operator based on the action behavior list and the standard operation process. The device executes the method. This prior art identifies abnormal behaviors in video segments containing the complete operation process of an operator, standardizes the action behaviors of the operator, ensures the personal safety of the operator, avoids irreparable losses to the factory, and improves the work efficiency of the operator. In this prior art, it is necessary to collect the complete videos, then decompose them and perform model training separately. The model for abnormal judgment of video segments requires a large amount of video annotation during training.

[0005] Therefore, it is imperative to provide a method that can reduce the dependence on large-scale manually labeled data, save data annotation costs, and quickly identify abnormalities in industrial control processes. Summary of the Invention

[0006] The technical problem to be solved by the present invention is:

[0007] The present invention discloses a self-supervised action process anomaly detection method and system based on contrast learning, aiming to solve the problems of the existing action process anomaly detection methods relying on large-scale labeled data and insufficient model generalization ability.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] In the first aspect, a self-supervised action process anomaly detection method based on contrast learning is provided. Through an innovative video representation algorithm, the method can automatically generate pseudo-labels from a large number of unlabeled videos, and then use these data to train an efficient action detection framework to construct a robust action process anomaly detection system. That is, the method generates pseudo-labels by training a video representation learning model; uses the pseudo-labels to train an action detection model to achieve action process anomaly detection. The method mainly includes the following steps:

[0010] S1: Collect video sequences of normal action processes;

[0011] S2: Use the video representation learning model to extract features and perform fine-grained learning on the obtained videos to identify the action features of each process;

[0012] S3: Manually annotate the sub-action boundaries of a standard action process video to determine the time interval of each sub-action stage;

[0013] S4: Apply the dynamic time warping algorithm to find the optimal time alignment path P between the unlabeled video and the labeled standard process video;

[0014] S5: According to the time alignment path, map the labels of the standard process video to the unlabeled video to generate pseudo-labels, and then obtain a series of labeled videos;

[0015] S6: Use the labeled videos as a dataset to train an action detection model so that the action detection model can predict the category to which each frame of action belongs;

[0016] S7: Real-time capture the camera stream data, use the trained action detection model for detection, and compare the detected action sequence with the standard action process template. If the sequence completely matches, it is determined to be a normal process; if there are missing actions, incorrect order, or redundant actions, it is determined to be an abnormal process.

[0017] Combined with the first aspect, further, the functional implementation process of the video representation learning model described in step 2 is as follows:

[0018] View generation and enhancement: For the same video, randomly crop a certain number of frames to generate two videos. Perform data augmentation on the two obtained videos, including: normalization and size adjustment, brightness, contrast adjustment, color jitter, and horizontal flipping, etc., to enhance the robustness of the model to different scenarios, and generate two different views V1 and V2 of the same video;

[0019] Video encoder: Encode the preprocessed video data, and convert the input multiple time frames into an embedded space representation through the video encoder to effectively capture the temporal and spatial information in the video;

[0020] Frame embedding generation: Each frame in the two views is passed into the encoder for feature extraction to generate a 128-dimensional frame-level feature representation;

[0021] Frame-frame contrastive learning: Construct positive and negative sample pairs through the time distance of frames. Frames with a relatively close time distance are used as positive samples, and frames with a relatively far time distance are used as negative samples; maximize the similarity of positive sample frame embeddings in different views, and at the same time minimize the similarity of negative sample frame embeddings, and construct the loss function as follows:

[0022]

[0023] Among them, w(i,j) represents the allocation weight of the positive sample pair (i,j), exp represents the exponential function, t i and t j respectively represent the time steps of frames i and j, L frameIt represents the frame contrast loss. K represents the number of frames contained in a single view. log represents the natural logarithm function with base e. Σ represents the summation function. corr(i,j) represents the normalized similarity of the positive sample pair (i,j). τ is the temperature parameter, and ε is a very small number to prevent division by zero. It represents the cosine similarity between the feature vector of the i-th frame of view V1 and the feature vector of the j-th frame of view V2.

[0024] Segment-segment contrastive learning: Divide the video sequence into non-overlapping temporal segments, and each segment is obtained by averaging the embeddings of a fixed number of frames; normalize the generated segment features. Similar to frame-frame contrast, segments with a closer temporal distance are used as positive samples, and segments with a farther distance are used as negative samples; maximize the frame embedding similarity of positive sample segments while minimizing the embedding similarity of negative sample segments.

[0025] Backward optimization of the encoder: By calculating the loss functions in frame-frame and segment-segment contrastive learning, perform backward gradient propagation to update the weight values of the encoder to better focus on the fine-grained features of actions in the video.

[0026] Combined with the first aspect, further, the encoder of the video representation learning model described in step 2 includes:

[0027] Spatial feature extraction module: Based on the backbone network of ResNet50, it is used to extract the spatial features of each frame. Adaptive max pooling is used to reduce the feature dimension, and each frame is represented as a feature vector of a fixed size.

[0028] Temporal dynamics capture module: Use a one-dimensional temporal convolutional layer to process the sequence of frame-level feature vectors; capture the short-term temporal dependencies between frames and output a feature sequence containing temporal dynamic information.

[0029] Long-term dependence modeling module: It includes two Transformer encoder layers, and each layer contains a multi-head self-attention mechanism and a feed-forward network; this module is used to capture the long-term temporal dependencies in the video and perform global modeling on the temporal information.

[0030] Feature mapping and normalization module: Use a multi-layer perceptron (MLP) to map the features of the long video to a fixed 128-dimensional vector space; ensure the stability of the features and the applicability of contrastive learning through L2 normalization.

[0031] Combined with the first aspect, further, the ResNet50 backbone network includes:

[0032] Multiple convolutional layers, used to extract spatial features from the input video frames;

[0033] A specific Conv3c layer, used to extract high-level features;

[0034] An adaptive max pooling layer for reducing the dimension of the feature map;

[0035] A flattening operation to convert the pooled features into a feature vector for each frame.

[0036] In combination with the first aspect, further, the one-dimensional temporal convolutional layer includes:

[0037] Multiple one-dimensional convolutional kernels for performing convolutional operations on the sequence of inter-frame feature vectors;

[0038] An activation function for introducing non-linearity;

[0039] A normalization layer for stabilizing the training process.

[0040] In combination with the first aspect, further, the Transformer encoder includes:

[0041] A multi-head self-attention layer for calculating the dependencies between different positions in the input sequence;

[0042] A feed-forward network for further processing the output of the self-attention layer;

[0043] Residual connections and layer normalization for enhancing gradient flow and stabilizing training.

[0044] In combination with the first aspect, further, the multi-layer perceptron (MLP) includes:

[0045] Multiple fully-connected layers for mapping the features output by the Transformer encoder to a 128-dimensional space;

[0046] An activation function for introducing non-linearity;

[0047] An L2 normalization layer for normalizing the output features for use in contrast learning.

[0048] In combination with the first aspect, further, the process of labeling a standard video in step 3 is as follows: Using the LabelVideo Tools software, label the time intervals of each action phase to generate the label sequence of the standard process video where indicates that the i-th frame belongs to the j-th category.

[0049] In combination with the first aspect, further, the process of finding the alignment path P in step 4 is as follows:

[0050] Use the encoder to extract the features of each frame of the labeled video and the unlabeled video to obtain the sequences and

[0051] Calculate the frame feature similarity matrix between the target video and the standard process video:

[0052] Wherein, sim(·) represents the similarity of the frame embedding vectors.

[0053] Apply the dynamic time warping algorithm to find the optimal time alignment path P = {(i 1 , j 1 ), (i 2 , j 1 ),...,(i k , j k )}.

[0054] Wherein, (i m , j n ) means that the m-th frame of the unlabeled video should be aligned with the n-th frame of the labeled video.

[0055] Combined with the first aspect, further, the implementation process of generating pseudo-labels by the label mapping described in step 5 is: according to the path P, map the labels of the labeled video to the target video

[0056] In the formula, L target [i] represents the label of the i-th frame of the target video, and L standard [j] represents the label of the j-th frame of the labeled video.

[0057] Combined with the first aspect, further, the implementation process of training the action detection model described in step 6 is:

[0058] Use the two-dimensional spatial backbone network ResNet50 to extract feature vectors from multiple frames of the video;

[0059] Use the encoder Longformer network based on temporal attention to encode the feature vectors, wherein the encoder based on temporal attention processes the feature vectors sequentially to capture the temporal dependencies between frames;

[0060] Combine the encoded feature vectors with the position encoding information;

[0061] Use a multi-layer perceptron to generate action classification predictions based on the combined encoded feature vectors and position encoding information.

[0062] The classification loss is calculated as follows:

[0063] Where C is the number of action process categories, is the one-hot encoding of the true label, and is the probability predicted by the model.

[0064] Calculate the classification loss value, and optimize the weight value of the encoder by backpropagation of the reverse gradient.

[0065] In a second aspect, a self-supervised action process anomaly detection system based on contrast learning is provided. The system has program modules corresponding to the steps, and when running, it executes the steps in the self-supervised action process anomaly detection method based on contrast learning, including:

[0066] A data input module for receiving normal video sequences and real-time camera stream data;

[0067] A video feature learning module for extracting spatio-temporal features of videos;

[0068] A dynamic alignment and pseudo-labeling module for generating pseudo-labeled data;

[0069] An action detection model for predicting action categories of video sequences;

[0070] An anomaly detection module for comparing action sequences with standard action templates and marking anomalies.

[0071] A computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the self-supervised action process anomaly detection method based on contrast learning when called by a processor.

[0072] A computer device includes a memory and a processor. A computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the steps of the self-supervised action process anomaly detection method based on contrast learning.

[0073] The advantages of the present invention are as follows:

[0074] The self-supervised action process anomaly detection method based on contrast learning proposed by the present invention can automatically generate pseudo-labels from a large number of unlabeled videos through an innovative video feature algorithm, and then train an efficient action detection framework with these data to construct a robust action process anomaly detection system. It solves the problems of dependence on a large number of labeled data and insufficient model generalization ability in existing methods, solves the limitation of the application of past learning methods in anomaly detection, and solves the problem that the past action recognition models in anomaly detection rely on a large number of manually labeled video data and are difficult to be popularized and used. The present invention only needs to label the sub-action boundaries of an action process and a certain amount of unlabeled normal action process videos, and then can train an efficient and robust action process anomaly detection model to quickly identify anomalies such as missing actions, incorrect sequences, or redundant actions in actual scenarios.

[0075] Through an innovative self-supervised learning method, the present invention can generate pseudo-labels using a large number of unlabeled videos, greatly reducing the dependence on large-scale manually labeled data, thereby significantly saving the data annotation cost. The constructed real-time action process anomaly detection system can quickly capture and analyze camera stream data, and can quickly identify anomalies such as missing actions, incorrect sequences, or redundant actions in actual scenarios, ensuring the efficiency and safety of the industrial control process.

[0076] The present invention proposes a self-supervised action process anomaly detection system based on contrastive learning, which can generate pseudo-label data using unlabeled action process videos, thereby significantly reducing the annotation cost. The core technologies include a video representation model based on an encoder composed of ResNet50, temporal convolutional layers, and Transformer, and an action annotation generation and mapping technology combined with the dynamic time warping algorithm. The main functions of the system are: real-time capturing of camera stream data, identifying and comparing action processes through a trained action detection model, and marking as anomalies if missing actions, incorrect sequences, or redundant actions are detected. This method is applicable to scenarios such as industrial production, video surveillance, and medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is the overall process framework diagram (flowchart of the self-supervised action process anomaly detection method based on contrastive learning of the present invention);

[0078] Figure 2 is a schematic diagram of the video representation learning process (schematic diagram of the functional composition and process of the video representation learning model);

[0079] Figure 3 is the architecture diagram of the video representation learning model (schematic diagram of the composition and process of the video encoder in the video representation learning model);

[0080] Figure 4 is a schematic diagram of action annotation generation (schematic diagram of the process of mapping the labels of the standard process video to the unlabeled video by the time alignment path); In the figure: Figures (a) and (b) show an example of an alignment path, Figure (c) shows an example of label mapping, and different colors represent different action labels;

[0081] Figure 5 is the architecture diagram of the action anomaly detection model (schematic diagram of the composition and process of the action video encoder);

[0082] The English captions in the drawings are well-known meanings in the art. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will be combined with the accompanying drawings in the embodiments of the present invention Figures 1-5, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments.

[0084] Embodiment 1. Refer to Figure 1 Describe this embodiment. The present invention proposes a self-supervised action process anomaly detection method based on contrast learning. The core of this method is to train a video representation learning model to generate pseudo-labels, thereby meeting the training needs of the action detection model. By combining these two models, the system can perform real-time analysis on the surveillance video stream in the scene, so as to effectively judge whether the action process is abnormal. Specifically, as Figure 1 shown, it includes the following steps:

[0085] S1: Collect video sequences of normal action processes: The system first collects videos of normal action processes through high-definition cameras installed at key positions in the scene, providing raw data for subsequent model training;

[0086] S2: Use the video representation learning model to extract features and perform fine-grained learning on the obtained videos, identify the action features of each process, and lay a foundation for generating pseudo-labels;

[0087] S3: Manually annotate the sub-action boundaries of a standard action process video to determine the time interval of each sub-action stage;

[0088] S4: Apply the dynamic time warping algorithm to find the optimal time alignment path P between the unannotated video and the annotated standard process video;

[0089] S5: According to the time alignment path, map the labels of the standard process video to the unannotated video to generate pseudo-labels, and then obtain a series of labeled videos;

[0090] S6: Use the labeled videos as a dataset to train the action detection model, so that the action detection model can predict the category to which each frame of action belongs;

[0091] S7: Real-time capture the camera stream data, use the trained action detection model for detection, compare the detected action sequence with the standard action process template. If the sequence completely matches, it is determined to be a normal process; if there are missing actions, incorrect order, or redundant actions, it is determined to be an abnormal process.

[0092] Embodiment 2. Refer to Figure 2 Describe this embodiment. In the self-supervised action process anomaly detection method based on contrast learning described in this embodiment, the specific implementation process of the video representation learning model based on contrast learning is as follows:

[0093] View Generation and Enhancement: For the same video, randomly crop a certain number of frames to generate two videos. Perform data augmentation on the two obtained videos, including: normalization and size adjustment, brightness and contrast adjustment, color jitter, and horizontal flipping, etc., to enhance the robustness of the model to different scenarios and generate two different views V1 and V2 of the same video;

[0094] Video Encoder: Encode the preprocessed video data, and convert multiple input time frames into an embedded space representation through the video encoder to effectively capture the temporal and spatial information in the video;

[0095] Frame Embedding Generation: Each frame in the two views is fed into the encoder for feature extraction to generate a 128-dimensional frame-level feature representation;

[0096] Frame-Frame Contrastive Learning: Construct positive and negative sample pairs through the time distance between frames. Frames with a closer time distance are used as positive samples, and frames with a farther distance are used as negative samples; Maximize the similarity of positive sample frame embeddings in different views, while minimizing the similarity of negative sample frame embeddings, and construct the loss function as follows:

[0097]

[0098] where w(i,j) represents the assignment weight of the positive sample pair (i,j), exp represents the exponential function, t i and t j represent the time steps of frames i and j respectively, L frame represents the frame contrast loss, K represents the number of frames included in a single view, log represents the logarithmic function with the natural base e, Σ represents the summation function, corr(i,j) represents the normalized similarity of the positive sample pair (i,j), τ is the temperature parameter, and ε is a very small number to prevent division by zero, represents the cosine similarity between the feature vector of the i-th frame of view V1 and the feature vector of the j-th frame of view V2.

[0099] Segment-Segment Contrastive Learning: Divide the video sequence into non-overlapping time segments, and each segment is obtained by averaging the embeddings of a fixed number of frames; Normalize the generated segment features. Similar to frame-frame contrast, segments with a closer time distance are used as positive samples, and segments with a farther distance are used as negative samples; Maximize the similarity of frame embeddings of positive sample segments, while minimizing the similarity of negative sample segment embeddings;

[0100] Reverse Optimization of the Encoder: Calculate the loss functions in frame-frame and segment-segment contrastive learning, and perform backpropagation of gradients to update the weight values of the encoder to better focus on the fine-grained features of actions in the video.

[0101] Embodiment 3. This embodiment further limits the encoder described in Embodiment 2. The encoder consists of the following parts:

[0102] Spatial Feature Extraction Module:

[0103] Based on the ResNet50 backbone network, it is used to extract the spatial features of each frame. It includes:

[0104] Multiple convolutional layers, which are used to extract spatial features from the input video frames;

[0105] A specific Conv3c layer, which is used to extract high-level features;

[0106] Adaptive max pooling layer, which is used to reduce the dimension of the feature map;

[0107] Flatten operation, which converts the pooled features into feature vectors for each frame.

[0108] Temporal Dynamics Capture Module:

[0109] Adopts a one-dimensional temporal convolutional layer to process the sequence of frame-level feature vectors. Captures the short-term temporal dependencies between frames and outputs a feature sequence containing temporal dynamic information. It includes:

[0110] Multiple one-dimensional convolutional kernels, which are used to perform convolutional operations on the sequence of inter-frame feature vectors;

[0111] Activation function, which is used to introduce non-linearity;

[0112] Normalization layer, which is used to stabilize the training process.

[0113] Long-Term Dependency Modeling Module:

[0114] This module is used to capture the long-term temporal dependencies in the video and perform global modeling on the temporal information. It includes:

[0115] Multi-Head Self-Attention Layer, which is used to calculate the dependencies between different positions in the input sequence;

[0116] Feed-Forward Network, which is used to further process the output of the self-attention layer;

[0117] Residual Connection and Layer Normalization, which are used to enhance the gradient flow and stabilize the training.

[0118] Feature Mapping and Normalization Module:

[0119] Maps the features of the long video to a fixed 128-dimensional vector space and ensures the stability of the features and the applicability of contrastive learning. It includes:

[0120] Multiple fully connected layers, which are used to map the features output by the Transformer encoder to the 128-dimensional space;

[0121] Activation function, used to introduce non-linearity;

[0122] L2 normalization layer, used to normalize the output features for contrast learning.

[0123] Embodiment 4. Refer to Figure 4 To describe this embodiment, the specific process of generating pseudo-labels in this embodiment is as follows:

[0124] Using the Label Video Tools software, manually divide the action segments of a standard action process video to label the time intervals of each action stage, generating the label sequence of the standard process video

[0125] where l i j indicates that the i-th frame belongs to the j-th category.

[0126] Contains the labels of the standard process video from the first frame to the last frame.

[0127] Feature extraction of the standard video and the unlabeled video:

[0128] Use the encoder to extract the features of each frame of the labeled video and the unlabeled video, obtaining the sequence

[0129]

[0130] Calculate the frame feature similarity matrix between the target video and the standard process video:

[0131] where sim(·) represents the similarity of the frame embedding vectors.

[0132] Apply the dynamic time warping algorithm to find the optimal time alignment path P = {(i 1 , j 1 ), (i 2 , j 1 ),..., (i k , j k )}.

[0133] where (i m , j n ) indicates that the m-th frame of the unlabeled video should be aligned with the n-th frame of the labeled video.

[0134] According to the path P, map the labels of the labeled video to the target video

[0135] where L target [i] represents the label of the i-th frame of the target video, Lstandard [j] represents the label of the j-th frame of the labeled video.

[0136] Embodiment 5. Refer to Figure 5 To describe this embodiment, the implementation process of training the action detection model in this embodiment is as follows:

[0137] Use a two-dimensional spatial backbone network: the ResNet50 convolutional neural network to extract feature vectors from multiple frames of the video;

[0138] Use a time-attention-based encoder: the Longformer network to encode the feature vectors, where the time-attention-based encoder processes the feature vectors sequentially to capture the temporal dependencies between frames;

[0139] Combine the encoded feature vectors with the position encoding information;

[0140] Use a multi-layer perceptron to generate action classification predictions based on the combined encoded feature vectors and position encoding information.

[0141] The classification loss is calculated as follows:

[0142] where C is the number of action process categories, is the one-hot encoding of the true label, and is the probability predicted by the model.

[0143] Calculate the classification loss value and optimize the weight value of the encoder through backpropagation of the gradient.

[0144] Embodiment 6. This embodiment further limits the judgment of whether an abnormal action process occurs as described in Embodiment 1.

[0145] The real-time action process anomaly detection process is as follows:

[0146] Real-time capture the camera stream data, input it into the trained action detection model, and compare the detected action sequence with the standard action process template. If the sequence matches exactly, it is determined as a normal process; if there are missing actions, incorrect order, or redundant actions, it is marked as abnormal and an alarm record is made, so as to standardize the operator's action behavior in a timely manner when the operator has the above abnormal behaviors..

[0147] Embodiment 7. A computer device described in this embodiment includes a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of Embodiments 1 to 6.

[0148] Embodiment 8. A computer-readable storage medium described in this embodiment is used to store a computer program, and the computer program executes the method described in any one of Embodiments 1 to 6.

[0149] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0150] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than limit the scope of its protection. Although the present disclosure has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that after reading the present disclosure, various changes, modifications or equivalent substitutions can still be made to the specific implementation manners of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending for publication.

Claims

1. A self-supervised action flow anomaly detection method based on contrastive learning, characterized in that: The method generates pseudo labels by training a video representation learning model; and uses the pseudo labels to train an action detection model to achieve action process anomaly detection, including the following steps: S1: Collect video sequences of normal action processes; S2: Use the video representation learning model to extract features and perform fine-grained learning on the acquired video to identify the characteristics of each process action; S3: Manually annotate the sub-action boundaries of a standard action flow video and determine the time interval of each sub-action stage; S4: Apply the dynamic time warping algorithm to find the optimal time alignment path P between the unlabeled video and the labeled standard process video; S5: According to the time alignment path, the labels of the standard process video are mapped to the unlabeled video to generate pseudo labels, and then a series of labeled videos are obtained; S6: Using the labeled video as a data set to train an action detection model, so that the action detection model can predict the category to which each frame of action belongs; S7: Capture camera stream data in real time, use the trained motion detection model for detection, and compare the detected motion sequence with the standard motion process template. If the sequence matches completely, it is judged as a normal process; if the action is missing, the order is wrong, or there are redundant actions, it is judged as an abnormal process.

2. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 1, characterized in that: The functions of the video representation learning model described in step 2 include: View generation and enhancement: For the same video, a certain number of frames are randomly cropped to generate two videos; data enhancement is performed on the two videos, including: standardization and resizing, brightness and contrast adjustment, color jittering, and horizontal flipping to enhance the robustness of the model to different scenes, and generate two different views V1 and V2 of the same video; Video encoder: Encode the preprocessed video data and convert the input multiple time frames into embedded spatial representations through the video encoder to effectively capture the temporal and spatial information in the video; Frame embedding generation: Each frame in the two views is passed into the video encoder for feature extraction to generate a 128-dimensional frame-level feature representation; Frame-to-frame contrastive learning: construct positive and negative sample pairs through the temporal distance of frames, with frames with closer temporal distance as positive samples and frames with farther temporal distance as negative samples; maximize the similarity of positive sample frame embeddings of different views, while minimizing the similarity of negative sample frame embeddings, and construct the loss function as follows: Among them, w(i,j) represents the weight assigned to the positive sample pair (i,j), exp represents the exponential function, and t i and t j denote the time steps of frames i and j respectively, L frame represents the frame contrast loss, K represents the number of frames contained in a single view, log represents the logarithmic function with the natural base e as the base, Σ represents the summation function, corr(i,j) represents the normalized similarity of the positive sample pair (i,j), τ is the temperature parameter, and ε is a very small number to prevent division by zero. represents the cosine similarity between the feature vector of the i-th frame of view V1 and the feature vector of the j-th frame of view V2; Clip-clip contrast learning: The video sequence is divided into non-overlapping time clips, each of which is obtained by averaging the embeddings of a fixed number of frames. The generated clip features are normalized, similar to frame-frame contrast, with clips with closer temporal distance as positive samples and clips with farther temporal distance as negative samples. The similarity of frame embeddings of positive sample clips is maximized, while the similarity of embeddings of negative sample clips is minimized. Reverse Optimize Encoder: By calculating the loss function of frame-frame and clip-clip contrastive learning, backward gradient propagation is performed to update the encoder weight values ​​to better focus on the fine-grained features of the action in the video.

3. According to the self-supervised action flow anomaly detection method based on contrastive learning in claim 2, the video encoder in the video representation learning model comprises: Spatial feature extraction module: Based on the ResNet50 backbone network, it is used to extract the spatial features of each frame; Use adaptive max pooling to reduce feature dimensions and represent each frame as a fixed-size feature vector; Temporal dynamics capture module: uses a one-dimensional temporal convolutional layer to process the frame-level feature vector sequence; captures the short-term temporal dependencies between frames and outputs a feature sequence containing temporal dynamic information; Long-term dependency modeling module: It includes two Transformer encoder layers, each of which contains a multi-head self-attention mechanism and a feedforward network. This module is used to capture long-term temporal dependencies in videos and globally model temporal information. Feature mapping and normalization module: Use multi-layer perceptron MLP to map the features of long videos to a fixed 128-dimensional vector space; L2 normalization is used to ensure the stability of features and the applicability of contrastive learning.

4. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 3 is characterized in that: The ResNet50 backbone network in the spatial feature extraction module includes: Multiple convolutional layers to extract spatial features from input video frames; A specific Conv3c layer for extracting high-level features; Adaptive max pooling layer to reduce the dimensionality of feature maps; Flattening operation converts the pooled features into feature vectors for each frame; The one-dimensional temporal convolution layer in the temporal dynamic capture module includes: Multiple one-dimensional convolution kernels are used to perform convolution operations on the feature vector sequence between frames; Activation function, used to introduce nonlinearity; Normalization layer, used to stabilize the training process; The Transformer encoder described in the long-term dependency modeling module includes: A multi-head self-attention layer that computes dependencies between different positions in the input sequence; A feed-forward network for further processing the output of the self-attention layer; Residual connections and layer normalization to enhance gradient flow and stabilize training; The multi-layer perceptron (MLP) in the feature mapping and normalization module includes: Multiple fully connected layers are used to map the features output by the Transformer encoder into a 128-dimensional space; Activation function, used to introduce nonlinearity; The L2 normalization layer is used to normalize the output features for contrastive learning.

5. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 4 is characterized in that: The process of labeling a standard video in step 3 is as follows: Use the Label Video Tools software to label the time interval of each action stage and generate a label sequence for the standard process video. In the formula, l i j Indicates that the i-th frame belongs to the j-th category.

6. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 5, characterized in that: The implementation process of finding the alignment path P described in step 4 is as follows: Use the encoder to extract the features of each frame of the labeled video and the unlabeled video to obtain the sequence and Calculate the frame feature similarity matrix between the target video and the standard process video: Among them, sim(·) represents the similarity of frame embedding vectors; Apply the dynamic time warping algorithm to find the optimal time alignment path P = {(i1, j1), (i2, j1), ..., (i k ,j k )}; Among them, (i m ,j n ) indicates that the mth frame of the unlabeled video should be aligned with the nth frame of the labeled video.

7. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 6, characterized in that: The process of generating pseudo labels by label mapping in step 5 is as follows: according to path P, the labels of the annotated video are mapped to the target video. Where, L target [i] represents the label of the i-th frame of the target video, L standard [j] represents the j-th frame label of the annotated video.

8. The self-supervised action flow anomaly detection method based on contrastive learning according to claim 7, characterized in that: The action detection model training implementation process described in step 6 is as follows: Extract feature vectors from multiple frames of a video using a 2D spatial backbone network; encoding the feature vector using a temporal attention-based action video encoder, wherein the temporal attention-based action video encoder sequentially processes the feature vector to capture temporal dependencies between frames; Combine the encoded feature vector with the position encoding information; Use a multi-layer perceptron to generate action classification predictions based on the combined encoded feature vector and positional encoding information; The classification loss is calculated as follows: Where C is the number of action flow categories, is the unique hot encoding of the true label, and is the probability predicted by the model; Calculate the classification loss value and use the reverse gradient propagation to optimize the encoder weights; The two-dimensional spatial backbone network of the action video encoder is: ResNet50 convolutional neural network; The temporal network adopted by the temporal attention-based action video encoder is: Longformer network.

9. A self-supervised action process anomaly detection system based on contrastive learning, characterized in that: The system has a program module corresponding to the steps of any one of claims 1 to 8 above, and executes the steps in the self-supervised action process anomaly detection method based on contrastive learning when running, which includes: A data input module, used for receiving normal video sequences and real-time camera stream data; Video representation learning module, used to extract spatiotemporal features of videos; Dynamic alignment and pseudo-annotation module, used to generate pseudo-annotation data; Action detection model, used to predict action categories in video sequences; The anomaly detection module is used to compare the action sequence with the standard action template and mark the anomalies.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the self-supervised action process anomaly detection method based on contrastive learning described in any one of claims 1 to 8 when called by a processor.

11. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the steps of the self-supervised action process anomaly detection method based on contrastive learning as described in any one of claims 1 to 8.

Images