Video abnormal behavior detection method based on semantic embedding learning
Through a method based on semantic embedding learning, the cosine similarity of the video embedding vector and the video description embedding vector are used to detect video abnormal behavior, which solves the shortcomings of obtaining label data and real-time detection in the prior art, and realizes accurate detection under video-level labeling and high generalization capabilities in different scenarios.
Patent Information
- Application Number
- CN202510067992.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-16
- Publication Date
- 2025-04-25
AI Technical Summary
The existing video anomaly behavior detection methods are insufficient in obtaining a large amount of anomaly category annotation data and real-time detection, resulting in low generalization ability and practicality in actual detection scenarios.
Using a video anomaly behavior detection method based on semantic embedding learning, by comparing the cosine similarity of the video embedding vector and the video description embedding vector in the embedding space, the video clip with the highest anomaly score is detected, realizing accurate detection under video-level annotation and high generalization ability in different scenarios.
This method significantly improves the supervised learning problem caused by the difficulty of obtaining abnormal behavior labeling data, realizes accurate detection under video-level labeling, and improves the generalization ability of abnormal data sets in different scenarios.
Smart Images
Figure CN119888575A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a method for detecting abnormal behavior in videos based on semantic embedding learning. Background Art
[0002] Abnormal behavior detection in videos aims to automatically identify situations that are significantly different from normal behavior patterns by analyzing the behavior, action patterns, and scene changes of people in the video. These abnormal behaviors are not limited to violent behaviors such as fighting and robbery, but also include social problems such as traffic accidents and fires. In recent years, with the rapid development of deep learning technology, especially the widespread application of convolutional neural networks (CNN), long short-term memory networks (LSTM), and graph neural networks (GNN), abnormal behavior detection technology in videos has made significant progress, and most of them focus on abnormal behaviors in surveillance videos. With the development of network technology, people are not only focusing on abnormal behaviors in surveillance videos, but also on abnormal behaviors in streaming videos. Traditional network models based on single-label supervised learning can no longer meet the needs of current work, because in a dynamically changing environment, new abnormal behavior categories will continue to appear, which makes it impossible to obtain a large amount of labeled data of new categories in a short time, and thus cannot effectively perform supervised learning. In addition, in some special scenarios, such as live video, the detection of abnormal behavior requires a certain degree of real-time performance, so new efficient and real-time detection models are urgently needed. How to design efficient algorithms that can capture complex dynamic patterns and adapt to abnormal behavior recognition in different scenarios and tasks is still a key issue in the research of video abnormal behavior detection. In general, the generalization ability and practicality of existing video abnormal behavior detection methods in real detection scenarios need to be improved. Summary of the invention
[0003] To solve the above problems, the present invention proposes a novel method for detecting abnormal behavior in videos. This method can solve the problem of difficulty in obtaining a large amount of abnormal category annotation data and low practicality of abnormal behavior detection in videos. By comparing the cosine similarity of the video embedding vector and the video description embedding vector in the embedding space, the video clip with the highest abnormal score is detected. This achieves accurate detection of abnormal behavior in videos under video-level annotation and high generalization ability for abnormal data sets in different scenarios. At the same time, video abnormal behavior data sets in new scenarios are collected to allow the model to better learn the semantic features of the data in different scenarios.
[0004] In order to achieve the above technical objectives, the specific technical solutions are as follows:
[0005] A method for detecting abnormal behavior in a video based on semantic embedding learning, the method comprising the following steps:
[0006] S1: Collect various video data containing abnormal behaviors, and then
[0007] Annotate the video with a corresponding video description;
[0008] S2: Build a semantic embedding network model suitable for abnormal behavior detection in videos;
[0009] S3: In the semantic embedding network model established by S2, various video data containing abnormal behaviors collected and labeled by S1 are trained to obtain a trained semantic embedding encoder;
[0010] S4: Use the partial video data containing various abnormal behaviors collected and labeled by S1, input them into the semantic embedding network model, and use the semantic embedding encoder obtained by S3 to detect abnormal behaviors in the video.
[0011] Furthermore, various video data containing abnormal behaviors are collected. The videos mainly come from short videos and live videos containing abnormal behaviors that appear on Internet video platforms.
[0012] Further, when labeling the video with a corresponding video description according to the abnormal behavior contained in the video, the abnormal behavior in the video is analyzed, and then a video description is labeled on the video according to the abnormal behavior to describe the video content.
[0013] Furthermore, the semantic embedding network model includes a feature extraction module and a semantic embedding learning module;
[0014] Among them, the feature extraction module is used to extract the image features of the input video frame and the text features of the video description; the semantic embedding learning module is responsible for calculating the similarity distance between image features and text features, and continuously reducing the distance between the matching image features and text features.
[0015] Furthermore, the process of extracting the image features of the input video frame and the text features of the video description by the feature extraction module includes:
[0016] Extract 16 frames from the video in equal steps, and then encode these 16 frames through Vision Transformer. The encoding process includes:
[0017]
[0018] Among them, x t,n represents the original feature vector of the frame expansion, W emb is the linear projection matrix, x cls is [CLS]token, Represents the feature sequence of the frame;
[0019] Select the feature with the highest attention score in the feature sequence of each frame. The process includes:
[0020]
[0021] Among them, LN(.) means normalization of input, MHSA(.) means multi-head self-attention operation, represents the updated frame features;
[0022] Aggregate the selected features with the highest attention scores to obtain the global features of the video. The aggregation process includes:
[0023] v=AvgPool([z 1 ,...,z t ])
[0024] Among them, AvgPool(.) represents the aggregation function, and v represents the global feature of the video after the feature with the highest attention score of each frame is aggregated;
[0025] The text description is encoded through a text encoder. The encoding process includes:
[0026] d = f θ (c)
[0027] Where c represents the original output video description, f θ (.) represents the text encoder, and d represents the features obtained after encoding.
[0028] Furthermore, the process of the semantic embedding learning module includes:
[0029] dis=f c (v,d)
[0030] Among them, f c (v, d) is the contrast loss function; v is the global feature of the video, d is the feature of the video description. The contrast loss function is used to iteratively shorten the distance between the matching video and the video description, lengthen the distance between the mismatching video and the video description, and amplify the inter-class distance between positive and negative samples.
[0031] Furthermore, in the semantic embedding network model established in S2, various video data containing abnormal behaviors collected and labeled by S1 are trained to obtain a trained semantic embedding encoder. The main process includes:
[0032] First, the training video is divided into 16 frames with equal step length, and then encoded and summarized by Vision Transformer to obtain the global features of the video; the video description is encoded by the text encoder to obtain features;
[0033] Then, the video description corresponding to each video is regarded as a positive sample, and the non-corresponding one is a negative sample. The positive and negative samples are embedded in the constructed semantic network model.
[0034] Finally, after training the semantic embedding network, an embedding space with deep semantic embedding is obtained, which is specifically represented by the trained image encoder and text encoder.
[0035] The present invention has at least the following beneficial effects
[0036] Compared with some existing abnormal behaviors in videos, the method provided by the present invention significantly improves the problem of being unable to conduct effective supervised learning due to the difficulty in obtaining abnormal behavior annotation data. The method detects the video clip with the highest abnormal score by comparing the cosine similarity of the video embedding features and the video description embedding features in the embedding space. This achieves accurate detection of abnormal behaviors in videos under video-level annotation and high generalization capabilities for abnormal data sets in different scenarios.
[0037] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0039] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0041] See also Figure 1 The present invention provides a method for detecting abnormal behavior in a video based on semantic embedding learning, which specifically comprises the following steps:
[0042] (1): Collect various video data containing abnormal behaviors, and then annotate the video with corresponding video descriptions based on the abnormal behaviors contained in the video;
[0043] (2): Construct a semantic embedding network model suitable for abnormal behavior detection in videos;
[0044] (3): In the semantic embedding network model established in step (2), the various video data containing abnormal behaviors collected and labeled in step (1) are trained to obtain a trained semantic embedding encoder;
[0045] (4): Use the partial video data containing various abnormal behaviors collected and labeled in step (1) to input the semantic embedding network model, and use the semantic embedding encoder obtained in step (3) to detect abnormal behaviors in the video.
[0046] As a preferred embodiment of the present invention, in step (1), the steps of collecting various types of video data containing abnormal behaviors and then labeling the video with corresponding video descriptions according to the abnormal behaviors contained in the video are as follows:
[0047] Step (1-1), collecting various video data containing abnormal behaviors, the videos mainly come from short videos and live videos containing abnormal behaviors that appear on Internet video platforms;
[0048] Step (1-2), when labeling the video with a corresponding video description according to the abnormal behavior contained in the video, the abnormal behavior in the video is analyzed, and then a video description is labeled to describe the video content according to the abnormal behavior.
[0049] As a preferred embodiment of the present invention, in step (2), the steps of constructing a semantic embedding network model suitable for video abnormal behavior detection are as follows:
[0050] Step (2-1) extracts input video features, extracts 16 frames from the video in equal steps, and then encodes these 16 frames through Vision Transformer. The encoding process includes:
[0051]
[0052] Among them, x t,n represents the original feature vector of the frame expansion, W emb is the linear projection matrix, x cls is [CLS]token; Represents the feature sequence of the frame,
[0053] Select the feature with the highest attention score in the feature sequence of each frame. The process includes:
[0054]
[0055] Among them, LN(.) means normalization of input, MHSA(.) means multi-head self-attention operation, represents the updated frame features;
[0056] Aggregate the selected features with the highest attention scores to obtain the global features of the video. The aggregation process includes:
[0057] v=AvgPool([z 1 ,...,z t ])
[0058] Among them, AvgPool(.) represents the aggregation function, and v represents the global feature of the video after the feature with the highest attention score of each frame is aggregated;
[0059] Step (2-2), extracting video description features, encoding the video description through a text encoder, the encoding process includes:
[0060] d = f θ (c)
[0061] Where c represents the original output video description, f θ (.) represents the text encoder, and d represents the features obtained after encoding;
[0062] Step (2-3), the process of embedding the learning module includes:
[0063] dis=f c (v,d)
[0064] Among them, f c (v, d) is the contrast loss function, v is the global feature of the video, and d is the feature of the video description. The contrast loss function is used to iteratively shorten the distance between the matching video and the video description, lengthen the distance between the mismatching video and the video description, and amplify the inter-class distance between positive and negative samples.
[0065] As a preferred embodiment of the present invention, in step (3), the steps of obtaining a trained semantic embedding encoder from the various types of video data containing abnormal behaviors collected and labeled in the training step (1) are as follows:
[0066] Step (3-1), divide the training video into 16 frames with equal step length, and then encode and summarize it through Vision Transformer to obtain the video global feature v; encode the video description through the text encoder to obtain the feature d;
[0067] Step (3-2), treating each video description corresponding to the video as a positive sample, and the non-corresponding video description as a negative sample, and embedding the positive and negative samples into the semantic embedding network model constructed in claims 1 and S2;
[0068] Step (3-3), after training the semantic embedding network, an embedding space with deep semantic embedding is obtained. This space is specifically manifested as the trained image encoder and text encoder.
[0069] As a preferred embodiment of the present invention, in step (4), the various types of partial video data containing abnormal behaviors collected and annotated in step (1) are input into the semantic embedding network model, and the semantic embedding encoder obtained in step (3) is used to detect abnormal behaviors in the video as follows:
[0070] Step (4-1) is to infer the abnormal behavior in the video by comparing the cosine similarity of the video global features and the video description features in the semantic embedding space. The mathematical expression is as follows:
[0071]
[0072] Where sim(·) represents the cosine similarity function, v represents the global feature of the video, and d represents the video description feature. Figure 1 The figure is a flow chart of the method of the present invention.
[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A video abnormal behavior detection method based on semantic embedding learning, characterized in that: The method mainly includes the following steps: S1: Collect various video data containing abnormal behaviors, and then annotate the video with corresponding video descriptions according to the abnormal behaviors contained in the video; S2: Build a semantic embedding network model suitable for abnormal behavior detection in videos; S3: In the semantic embedding network model established by S2, various video data containing abnormal behaviors collected and labeled by S1 are trained to obtain a trained semantic embedding encoder; S4: Use the various video data containing abnormal behaviors collected and labeled by S1, input them into the semantic embedding network model, and use the semantic embedding encoder obtained by S3 to detect abnormal behaviors in the video.
2. According to the method of claim 1, the method is characterized in that: In S1, various video data containing abnormal behaviors are collected. The videos mainly come from short videos and live videos containing abnormal behaviors that appear on Internet video platforms.
3. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 1, characterized in that: In S1, when labeling a corresponding video description to the video according to the abnormal behavior contained in the video, the abnormal behavior in the video is analyzed, and then a video description is labeled to the video according to the abnormal behavior to describe the video content.
4. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 1, characterized in that: In S2, the semantic embedding network model includes a feature extraction module and a semantic embedding learning module; Among them, the feature extraction module is used to extract the image features of the input video frame and the text features of the video description; the semantic embedding learning module is responsible for calculating the similarity distance between image features and text features, and continuously reducing the distance between the matching image features and text features.
5. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 4, characterized in that: The process of extracting the image features of the input video frame and the text features of the video description by the feature extraction module includes: Extract 16 frames from the video in equal steps, and then encode these 16 frames through Vision Transformer. The encoding process includes: Among them, x t,n represents the original feature vector of the frame expansion, W emb is the linear projection matrix, x cls is [CLS]token, Represents the feature sequence of the frame; Select the feature with the highest attention score in the feature sequence of each frame. The process includes: Among them, LN(·) represents the normalization of the input, MHSA(.) represents the multi-head self-attention operation, represents the updated frame features; Aggregate the features with the highest attention scores selected from each frame to obtain the global features of the video. The aggregation process includes: v=AvgPool([z1,...,z t ]) Among them, AvgPool(·) represents the aggregation function, which is aggregated by average pooling, and v represents the global feature of the video after the feature with the highest attention score of each frame is aggregated; Encode the text description through a text encoder. The encoding process includes: d=f θ (c) Among them, c represents the original output video description, f θ (·) represents the text encoder, and d represents the feature obtained after encoding.
6. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 4, characterized in that: The learning process of the semantic embedding learning module includes: dis=f c (v,d) Among them, f c (v, d) is the contrast loss function, v is the global feature of the video, and d is the feature of the video description. The contrast loss function is used to iteratively shorten the distance between the matching video and the video description, lengthen the distance between the mismatching video and the video description, and amplify the inter-class distance between positive and negative samples.
7. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 1, characterized in that: In S3, the semantic embedding network model established in S2 is used to train various video data containing abnormal behaviors collected and labeled in S1 to obtain a trained semantic embedding encoder. The main process includes: S3-1, divide the training video into 16 frames with equal step length, and then encode and summarize it through Vision Transformer to obtain the global features of the video; encode the video description through the text encoder to obtain the features; S3-2, regard the video description corresponding to each video as a positive sample, and the non-corresponding video description as a negative sample, and embed the positive and negative samples into the semantic embedding network model constructed in claim 1 and S2; S3-3, after training the semantic embedding network, an embedding space with deep semantic embedding is obtained. This space is specifically manifested as the trained image encoder and text encoder.
8. The method for detecting abnormal behavior in videos based on semantic embedding learning according to claim 1, characterized in that: In S4, the partial video data containing various abnormal behaviors collected and annotated by S1 are input into the semantic embedding network model, and the semantic embedding encoder obtained by S3 is used to detect abnormal behaviors in the video; The semantic embedding space is obtained through training according to claim 7, and the abnormal behavior in the video is inferred by comparing the cosine similarity of the global features of the video and the abnormal video description features in the semantic embedding space. The mathematical expression is as follows: Where sim(·) represents the cosine similarity function, v represents the global feature of the video, and d′ represents the abnormal video description feature.