Video anomaly feature detection method and device, computer device, and storage medium

Abstract

The application relates to the technical field of video anomaly detection, and discloses a video anomaly feature detection method and device, computer equipment and a storage medium, the video anomaly feature detection method comprises the following steps: obtaining target video data to be detected; performing first feature extraction on the target video data based on a text feature dimension to obtain semantic prior features; performing second feature extraction on the target video data based on a video feature dimension to obtain fusion visual features; wherein the fusion visual features are obtained by fusing normal visual features and abnormal visual features of the target video data; performing feature fusion based on the semantic prior features and the fusion visual features to obtain final features of the target video data; and performing abnormal feature detection based on the final features to obtain an abnormal feature detection result; the application can meet the demand of performing abnormal detection on large-scale video data, and can simultaneously detect multiple abnormal visual features during the abnormal feature detection.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method, apparatus, computer device, and storage medium for detecting abnormal features in videos. Background Technology

[0002] Anomaly detection primarily refers to the detection of anomalous data that deviates significantly from normal data within expected normal data sets. Anomaly detection has been applied in many important fields, such as video surveillance, network intrusion detection, healthcare, and education.

[0003] In the field of video anomaly detection, a large amount of abnormal video data has emerged due to the varying parameters of acquisition equipment, the complexity of scenarios, and the lack of standardization in the acquisition process. Traditional detection methods mainly rely on manual screening for the detection of such abnormal video data. However, relying on manual screening is insufficient to meet the needs of anomaly detection for large-scale video data. Summary of the Invention

[0004] In view of this, the present invention provides a video anomaly feature detection method, apparatus, computer equipment and storage medium to solve the problem that existing video anomaly feature detection methods are difficult to meet the needs of anomaly detection for large-scale video data.

[0005] In a first aspect, the present invention provides a video anomaly feature detection method, which includes: acquiring target video data to be detected; performing a first feature extraction on the target video data based on the text feature dimension to obtain semantic prior features; performing a second feature extraction on the target video data based on the video feature dimension to obtain fused visual features; wherein the fused visual features are obtained by fusing normal visual features and abnormal visual features of the target video data; performing feature fusion based on the semantic prior features and the fused visual features to obtain the final features of the target video data; and performing anomaly feature detection based on the final features to obtain anomaly feature detection results.

[0006] In one optional implementation, a second feature extraction is performed on the target video data based on the video feature dimension to obtain fused visual features, including: extracting video features from the target video data to obtain visual features to be classified from the target video data; inputting the visual features to be classified into a pre-trained visual feature detection model, and using the visual feature detection model to detect based on the similarity between the visual features to be classified and preset video features to obtain normal visual features and abnormal visual features; wherein, the visual feature detection model is trained from historical video data and corresponding historical visual feature classification results; and fusing the normal visual features and abnormal visual features to obtain fused visual features.

[0007] In one optional implementation, the visual feature detection model includes a first feature extraction network composed of a first memory term and a second feature extraction network composed of a second memory term; detection is performed based on the similarity between the visual feature to be classified and preset video features, including: detecting the visual feature to be classified based on the first memory term through the first feature extraction network to obtain normal visual features; and detecting the visual feature to be classified based on the second memory term through the second feature extraction network to obtain abnormal visual features.

[0008] In one optional implementation, the video anomaly feature detection method further includes: calculating the memory item similarity between the first memory item and the second memory item of the visual features to be classified; determining the most relevant target memory item based on the memory item similarity; and updating the target memory item based on the visual features to be classified.

[0009] In one optional implementation, the video anomaly feature detection method further includes: normalizing the similarity of memory items to obtain attention values ​​corresponding to the first and second memory items; and fusing the attention values ​​and the corresponding memory items to obtain fused visual features.

[0010] In one optional implementation, a first feature extraction is performed on the target video data based on the text feature dimension to obtain semantic prior features, including: extracting semantic features of the target video data to obtain general prompt information and status prompt information; and fusing the general prompt information and status prompt information to obtain semantic prior features.

[0011] In one optional implementation, feature fusion is performed based on semantic prior features and fused visual features, including: obtaining a preset set of semantic features; extracting semantic features from the preset set of semantic features based on semantic prior features; and performing feature fusion based on the similarity between semantic features and fused visual features to obtain the final features.

[0012] Secondly, the present invention provides a video anomaly detection device, comprising: an acquisition module for acquiring target video data to be detected; a first feature extraction module for performing first feature extraction on the target video data based on text feature dimensions to obtain semantic prior features; a second feature extraction module for performing second feature extraction on the target video data based on video feature dimensions to obtain fused visual features; wherein the fused visual features are obtained by fusing normal visual features and abnormal visual features of the target video data; a feature fusion module for performing feature fusion based on semantic prior features and fused visual features to obtain final features of the target video data; and an anomaly scoring module for performing anomaly scoring based on the final features to obtain an anomaly detection result.

[0013] Thirdly, the present invention provides a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the video anomaly feature detection method described in the first aspect or any corresponding embodiment thereof.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the video anomaly feature detection method described in the first aspect or any corresponding embodiment thereof.

[0015] The video anomaly detection method provided by this invention performs a first feature extraction on the target video data based on the text feature dimension to obtain semantic prior features, thereby extracting multiple anomalous visual features that may be contained in the target video data, thus enhancing the understanding of anomalous visual features. Furthermore, it performs a second feature extraction on the target video data based on the video feature dimension, which can consider the impact of normal visual features and anomalous visual features on the anomaly detection of the target video data at the same time. This solves the problem that existing video anomaly detection methods are difficult to meet the needs of anomaly detection for large-scale video data, and can detect multiple anomalous visual features simultaneously during anomaly detection. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a video anomaly feature detection method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart illustrating another video anomaly feature detection method according to an embodiment of the present invention;

[0019] Figure 3 This is a flowchart illustrating another video anomaly feature detection method according to an embodiment of the present invention;

[0020] Figure 4 This is a flowchart illustrating another video anomaly feature detection method according to an embodiment of the present invention;

[0021] Figure 5 This is a structural diagram of a video anomaly feature detection model according to an embodiment of the present invention;

[0022] Figure 6 This is a structural block diagram of a video anomaly detection device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] For sports activity video data in educational settings, a large number of abnormal sports activity video data have emerged due to the varying parameters of the acquisition equipment, the complexity of the scenarios, and the lack of standardization in the acquisition process. Traditional video anomaly detection methods mainly rely on educational experts to investigate these abnormal sports activity video data. However, with the gradual expansion of the scale of educational data construction, traditional video anomaly detection methods are difficult to meet the needs of anomaly detection for large-scale sports activity video data.

[0025] To address the aforementioned issues, with the development of artificial intelligence technology, anomaly detection in videos based on data mining and intelligent optimization algorithms has become a trend. Among related technologies, deep learning-based video anomaly detection methods are mainly divided into semi-supervised and weakly supervised methods. Research by the inventors has revealed that semi-supervised anomaly detection methods only train on normal data, limiting the model's detection performance and making it difficult to meet the high-precision requirements of the education field. Weakly supervised video anomaly detection methods, on the other hand, only provide video-level labels and train on both normal and anomaly data. Compared to unsupervised methods, this method introduces a small amount of prior information, significantly improving the model's detection performance. However, this method is mainly suitable for surveillance videos with relatively simple anomaly types and is difficult to directly apply to sports activity videos with a greater variety of anomalies. Therefore, there is an urgent need to propose a new video anomaly detection method to solve the aforementioned problems in related technologies.

[0026] To address the aforementioned problems, according to an embodiment of the present invention, a video anomaly feature detection method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This embodiment provides a method for detecting abnormal features in videos. Figure 1 This is a flowchart of a video anomaly feature detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0028] Step S101: Obtain the target video data to be detected.

[0029] For example, the target video data is captured by a shooting device, and the target video data is video data of the target subject engaging in sports activities. The target subject can be a student, an adult, an elderly person, etc.

[0030] For example, the target video data contains attribute information based on the target video data. The target video data is annotated with text features of different attribute dimensions to obtain semantic features of the target video data. Specifically, when annotating the text dimensions, the aspects considered include at least the action type dimension representing the action type of the target video data and the state type dimension representing the state of the target video data. Among them, the attribute information is used to represent the sports activity attribute of the target video, and the state type dimension includes normal state and abnormal state. This invention defines the normal state as the collection standard of the national smart platform, that is, it stipulates that the student is in the center of the camera, facing the camera, fully visible in the frame, and only the student is included.

[0031] Therefore, this invention defines videos that do not meet the above requirements as abnormal. Among the abnormal videos, this invention filters out 7 common abnormal rope skipping state videos that exist in large-scale rope skipping data, as shown in Table 1.

[0032] Table 1 Summary of Abnormal Types

[0033]

[0034] For example, the sports activity video data can be rope skipping activity video data; after obtaining the rope skipping video data, the rope skipping video data is annotated with text in the dimensions of action type and state type.

[0035] Step S102: Extract the first feature from the target video data based on the text feature dimension to obtain semantic prior features.

[0036] Since the target video data may contain a variety of abnormal visual features, in order to enhance the understanding of abnormal visual features from the text dimension, this embodiment extracts features from the pre-annotated semantic features based on the text feature dimension, so that the abnormal visual features of the target video data can be considered from the text dimension in this invention.

[0037] Step S103: Extract the second feature from the target video data based on the video feature dimension to obtain the fused visual feature; wherein, the fused visual feature is obtained by fusing the normal visual feature and abnormal visual feature of the target video data.

[0038] Since the target video data may contain both video data with abnormal visual features and video data with normal visual features, for example, a second feature extraction is performed on the target video data based on the video feature dimension to obtain fused visual features. This approach simultaneously considers the impact of normal and abnormal visual features on the detection of abnormal features in the target video data.

[0039] Step S104: Perform feature fusion based on semantic prior features and fused visual features to obtain the final features of the target video data.

[0040] To ensure that the detection of anomalies in target video data is considered in light of both normal and abnormal visual features, and to account for the impact of multiple abnormal visual features on anomaly detection, this embodiment performs feature fusion based on semantic prior features and fused visual features to obtain the final features of the target video data. During the fusion process, multiple abnormal visual features that the target video data may possess, as well as video data containing both abnormal visual features and normal visual features, are retained, so that the above-mentioned features can be considered when performing anomaly detection.

[0041] Step S105: Perform anomaly detection based on the final features to obtain the anomaly detection results.

[0042] The video anomaly detection method provided in this embodiment performs a first feature extraction on the target video data based on the text feature dimension to obtain semantic prior features, thereby extracting multiple abnormal visual features that may be contained in the target video data, thus enhancing the understanding of abnormal visual features. Furthermore, it performs a second feature extraction on the target video data based on the video feature dimension, which can consider the impact of normal visual features and abnormal visual features on the anomaly detection of the target video data at the same time. This solves the problem that existing video anomaly detection methods are difficult to meet the needs of anomaly detection for large-scale video data, and can detect multiple abnormal visual features simultaneously during anomaly detection.

[0043] This embodiment provides a method for detecting abnormal features in videos. Figure 2 This is a flowchart of a video anomaly feature detection method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0044] Step S201: Obtain the target video data to be detected. For details, please refer to [link to relevant documentation]. Figure 1Step S101 of the illustrated embodiment will not be described again here.

[0045] Step S202: Extract the first feature from the target video data based on the text feature dimension to obtain semantic prior features. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0046] Step S203: Extract the second feature from the target video data based on the video feature dimension to obtain the fused visual features.

[0047] Specifically, step S203 includes:

[0048] Step S2031: Extract video features from the target video data to obtain the visual features to be classified from the target video data.

[0049] Step S2032: Input the visual features to be classified into a pre-trained visual feature detection model. The visual feature detection model detects the visual features to be classified based on the similarity between the visual features to be classified and the preset video features to obtain normal visual features and abnormal visual features.

[0050] For example, the visual feature detection model is trained using historical video data and corresponding historical visual feature classification results. Specifically, historical video data is input into the visual feature detection model, which then classifies the visual features to be classified to obtain classification results. During model training, the model parameters are continuously adjusted. On the one hand, this enables the model to store typical normal and abnormal visual features from historical visual features. On the other hand, it enables the model to learn the correspondence between different visual features to be classified and feature types.

[0051] For example, after the model is trained, the visual features to be classified are input into the pre-trained visual feature detection model. The similarity between the visual feature detection model and the preset video features is used to detect normal visual features and abnormal visual features.

[0052] Step S2033: Fuse normal visual features and abnormal visual features to obtain fused visual features.

[0053] In related technologies, methods for visual feature detection using memory networks mainly use a single memory network to store and enhance the features of normal videos. However, the inventors have found that when the similarity between normal and abnormal videos is high, a single memory network is difficult to distinguish between normal and abnormal visual features. Based on this, in some optional implementations, the visual feature detection model includes a first feature extraction network composed of a first memory term and a second feature extraction network composed of a second memory term.

[0054] In this embodiment, the visual feature detection model can be a dual-memory network model; wherein, a single memory network can store typical video features, which can be used for feature enhancement of normal videos; while the design of the dual-memory network model can increase the distance between normal video features and abnormal video features when performing visual feature detection.

[0055] For example, in a dual-memory network, each memory network consists of m memory items with dimension c. Using a video feature zj as the query, the most relevant memory item is retrieved. The memory item is the aforementioned preset video feature. For example, the preset video feature includes preset normal visual sub-features and preset abnormal visual sub-features. Based on this, in some optional implementations, the visual feature detection model includes two memory networks, each consisting of multiple memory items. The first memory network is used to memorize normal visual features based on the first memory items and consists of m first memory items with dimension c. The second memory network is used to memorize abnormal visual features based on the second memory items and consists of m second memory items with dimension c. The first memory items represent the preset normal visual sub-features, and the second memory items represent the preset abnormal visual sub-features.

[0056] The above step S2032 includes:

[0057] Step a1: The first feature extraction network detects the visual features to be classified based on the first memory term to obtain normal visual features;

[0058] Step a2: The second feature extraction network detects the visual features to be classified based on the second memory term to obtain abnormal visual features.

[0059] In this embodiment, in order to enable the visual feature detection model to further learn the difference information between normal video features and abnormal video features, a self-supervised approach is used to update the first and second memory items. Therefore, in some optional implementations, the video abnormal feature detection method further includes: calculating the memory item similarity between the first and second memory items of the visual features to be classified; determining the most relevant target memory item based on the memory item similarity; and updating the target memory item based on the visual features to be classified.

[0060] Specifically, the similarity between the first and second memory items of the visual feature to be classified can be calculated by calculating the Euclidean distance between them. Alternatively, the similarity between the first and second memory items can be calculated by calculating the cosine similarity between them. For example, this embodiment will be described using cosine similarity as an example.

[0061] For example, firstly, the cosine similarity between the i-th memory item in the memory network and the j-th visual feature to be classified is calculated using the following formula:

[0062]

[0063] In formula (1), e i Z represents the i-th memory item. j This represents the j-th visual feature to be classified. This represents the visual features z to be classified. j The resulting matrix is ​​transposed.

[0064] Next, the memory term most relevant to the video feature is retrieved and denoted as k. j Specifically, the argmax function is used for calculation, and the formula is as follows:

[0065] k j =arg i maxs ij (2)

[0066] Where, arg i Maxs represents the i-th independent variable corresponding to the i-th memory item. ij This represents the set of memory items with the highest cosine similarity.

[0067] Finally, the most relevant memories are updated, specifically using the following update formula:

[0068]

[0069] Where τ is the attenuation rate, τ∈[0,1], j is the j-th video feature, and k (j) This is the index of the memory item most relevant to the j-th video feature.

[0070] After updating the first and second memory items individually or simultaneously, it is necessary to update the attention corresponding to the memory items to enhance the visual features. Therefore, in some optional implementations, the video anomaly feature detection method further includes: normalizing the similarity of the memory items to obtain the attention values ​​corresponding to the first and second memory items; and fusing the attention values ​​and the corresponding memory items to obtain fused visual features.

[0071] For example, after updating the memory network, video features are enhanced. First, the calculated cosine similarity is normalized to obtain the attention a. ij .

[0072]

[0073] Where exp(·) represents the normalization process for the variable, s ij denoted by formula (1), denoted by the similarity calculated between the i-th memory item and the j-th video feature, and m represents the number of memory items.

[0074] Secondly, enhanced video features are obtained by aggregating the features of memory items based on attention.

[0075]

[0076] Step S204 involves feature fusion based on semantic prior features and fused visual features to obtain the final features of the target video data. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0077] Step S205: Perform anomaly detection based on the final features to obtain the anomaly detection results. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0078] The video anomaly feature detection method provided in this embodiment stores and enhances normal visual features and abnormal visual features in each memory network, fully considering the degree of difference between normal visual features and abnormal visual features, and can effectively distinguish between normal visual features and abnormal visual features.

[0079] By analyzing existing jump rope video data, this invention reveals that there are numerous types of abnormal states in the existing jump rope data, making it difficult for models to understand and learn each one. Therefore, this invention introduces a semantic guidance module, which enhances the understanding of various jump rope anomaly types by transferring knowledge contained in the CLIP model pre-trained on large-scale image and text data. Based on this, this embodiment provides a video anomaly feature detection method. Figure 3 This is a flowchart of a video anomaly feature detection method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0080] Step S301: Obtain the target video data to be detected. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0081] Step S302: Extract the first feature from the target video data based on the text feature dimension to obtain semantic prior features.

[0082] Specifically, step S302 includes:

[0083] Step S3021: Extract semantic features from the target video data to obtain general prompt information and status prompt information.

[0084] In this invention, a two-level prompt is designed for the target video data based on semantic features, converting the action type and state type of the target video data into text descriptions. Specifically, the text description templates for action types include "{type} jump rope video," "a {type} jump rope video," and "{type} jump rope video for anomaly detection," etc. The prompt templates for normal video states are "no tiptoeing," "with rope," "full-body shot," "front view," and "upright video." The prompt templates for abnormal video states are "non-jump rope video," "tiptoeing," "completely sideways," "not full-body shot," "back view," and "horizontal video." It should be understood that when the target video data contains normal visual features, it corresponds to the above-mentioned normal video states; when the target video data contains abnormal visual features, it corresponds to the above-mentioned abnormal video states.

[0085] Step S3022: The general prompt information and the status prompt information are fused to obtain semantic prior features.

[0086] For example, given the action type as one-legged jump and the status type as back view, the first-level prompt first fills it into the general template "{type} jump rope video" to get the action type description of one-legged jump rope video. The second-level prompt adds the status prompt "back view" to get the final text description "a type of one-legged jump rope video with back view".

[0087] For example, for target video data of rope skipping, the semantic features extracted from the target video data result in the following: the general prompt information is "single-leg jump", and the status prompt information is "back view". In the fusion process, the general prompt information is first filled into the general template "rope skipping video of type {type}" to obtain the action type description "rope skipping video of single-leg jump". Based on the status prompt information, the status-level prompt "back view" is added to obtain the text description "a type of single-leg rope skipping video of back view".

[0088] For example, after obtaining the text description through the above prompt template, the present invention uses a pre-trained text encoder to obtain the semantic feature set t = {t1,...,t} of the text description. n}

[0089] Step S303: Extract the second feature from the target video data based on the video feature dimension. For details, please refer to [link / reference]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0090] Step S304 involves feature fusion based on semantic prior features and fused visual features to obtain the final features of the target video data. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0091] Step S305: Perform anomaly detection based on the final features to obtain the anomaly detection results. For details, please refer to [link to relevant documentation]. Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0092] In this embodiment, semantic features of the target video data are extracted to obtain general prompt information and status prompt information. The general prompt information and status prompt information are fused to obtain semantic prior features, which are then introduced into the semantic guidance module so that the impact of various rope skipping anomaly types on anomaly feature detection is considered when performing anomaly feature detection.

[0093] To ensure that the detection of anomalies in target video data considers both normal and anomalous visual features, and to account for the influence of multiple anomalous visual features on anomaly detection, in some optional implementations, Figure 4 This is a flowchart of a video anomaly feature detection method according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0094] Step S401: Obtain the target video data to be detected. For details, please refer to [link to relevant documentation]. Figure 3 Step S301 of the illustrated embodiment will not be described again here.

[0095] Step S402: Extract the first feature from the target video data based on the text feature dimension to obtain semantic prior features; for details, please refer to [link to relevant documentation]. Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0096] Step S403: Extract a second feature from the target video data based on the video feature dimension to obtain fused visual features; for details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.

[0097] Step S404: Based on semantic prior features and fused visual features, feature fusion is performed to obtain the final features of the target video data.

[0098] Specifically, step S404 includes:

[0099] Step S4041: Obtain a preset semantic feature set.

[0100] In this embodiment, the preset semantic feature set is the semantic feature set t = {t1,...,t...} obtained by extracting historical visual features using a pre-trained text encoder to describe the text. n}; where historical visual features include historical normal visual sub-features and historical abnormal visual sub-features, and the pre-trained text encoder is a large-scale image and text data pre-training model (Contrastive Language-Image Pre-training, CLIP).

[0101] Step S4042: Extract semantic features from a preset semantic feature set based on semantic prior features.

[0102] Step S4043: Perform feature fusion based on the similarity between semantic features and fused visual features to obtain the final features.

[0103] In this embodiment, feature fusion can be performed using the concat method. The concat method can concatenate semantic features and fused visual features according to the channel dimension to obtain the final features. The above fusion method considers the influence of multiple abnormal visual features on abnormal feature detection in semantic features, and considers the influence of normal visual features and abnormal visual features on the abnormal feature detection of target video data at the same time. When performing abnormal state detection, it can consider the influence of normal visual features and abnormal visual features on the abnormal feature detection of target video data at the same time. Moreover, considering the influence of multiple abnormal visual features on abnormal feature detection, the detection can be more accurate.

[0104] Step S405: Perform anomaly detection based on the final features to obtain the anomaly detection results. For details, please refer to [link to relevant documentation]. Figure 3 Step S304 of the illustrated embodiment will not be described again here.

[0105] The video anomaly feature detection method provided in this embodiment can consider the influence of both normal and abnormal visual features on the anomaly feature detection of the target video data when performing anomaly state detection. Furthermore, by considering the influence of multiple abnormal visual features on anomaly feature detection, the detection can be more accurate.

[0106] As one or more specific application embodiments of the present invention, the method shown in the above embodiments can be implemented using a pre-trained video anomaly feature detection model; Figure 5 This is a structural diagram of a video anomaly feature detection model according to an embodiment of the present invention. In this embodiment, it will be combined with... Figure 5 The technical solution of the present invention will be described.

[0107] For example, such as Figure 5As shown, the video anomaly detection model includes a semantic feature extraction sub-model, a visual feature extraction sub-model, and an anomaly scoring sub-model. Specifically, the target video data is processed by the semantic feature extraction sub-model to obtain text features, and the target video data is processed by the visual feature extraction sub-model to obtain visual features. The visual features and text features are then scored by the anomaly scoring sub-model to obtain the anomaly detection result.

[0108] Specifically, after acquiring the target video data to be detected, the semantic feature extraction sub-model first extracts state prompt information and general prompt information based on the prompt information library, then fuses the state prompt information and general prompt information, and inputs the fused prompt information into the CLIP text encoder, and obtains semantic features through the pooling layer.

[0109] After acquiring the target video data to be detected, the visual feature extraction sub-model first extracts features from the target video data through the CLIP image encoder and the temporal attention network in sequence to obtain the visual features to be detected. Then, the visual features are input into the abnormal video memory network and the normal video memory network respectively to obtain abnormal visual features and normal visual features. Based on the visual features to be detected, the neural network is updated in a self-supervised update manner. Finally, the normal visual features and abnormal visual features are fused to obtain the fused visual features.

[0110] After obtaining the fused visual and textual features, the anomaly scoring sub-model first fuses the fused visual and semantic features to obtain the final features, and then performs anomaly scoring based on the final features.

[0111] For example, when training a video anomaly detection model, the objective function of the model training consists of reconstruction loss and cross-entropy loss; the reconstruction-based method mainly encodes and decodes the input video, and trains the neural network with the goal of reducing reconstruction error. In this process, the commonly used loss function is mean square error, which measures the quality of image reconstruction by the squared mean of the differences between the pixel values ​​of all pixels before and after image reconstruction; the calculation of mean square error is shown in formula (6):

[0112]

[0113] Among them, z t For input features, The output features for reconstruction.

[0114] Under the supervision of video-level tags, the model is optimized using cross-entropy loss. The formula for calculating cross-entropy is shown in formula (7):

[0115]

[0116] Where y iThis represents the predicted result, y′ i Tags indicating video content;

[0117] The loss of the final model is shown in (8), where λ is a hyperparameter.

[0118] loss = loss rec +λloss entro (8)

[0119] For example, in order to comprehensively evaluate the performance of the anomaly detection method of the present invention, the inventors used the area under the curve (AUC) of the receiver operating characteristic (ROC) curve as a measure of model performance. AUC can quantitatively describe the performance of the model, representing the probability that the predicted positive probability value is greater than the predicted negative probability value, that is, the probability that the algorithm will rank a randomly selected positive sample before a negative sample based on the score value.

[0120] To further verify the effectiveness of the functional modules contained in different sub-models of the proposed Video Anomaly Detection Model (CSC-ME), this invention conducts ablation experiments on a self-built jump rope dataset to compare the performance of different modules. Specifically, the impact of the dual-memory network and semantic guidance module on the results is explored; the AUC results of the ablation experiment are shown in Table 2. Here, w / o All indicates the removal of all new modules designed in this paper; Table 2 is the ablation experiment results of this invention, see Table 2:

[0121] Table 2

[0122]

[0123] As can be clearly seen from the results in Table 2, the technical solution of introducing the semantic guidance module and the dual-memory network in this invention significantly improves the network performance. The addition of the semantic guidance module and the dual-memory network plays a crucial role in the abnormal rope skipping data classification task and improves model performance in several aspects. First, the addition of the semantic guidance module has a positive impact on network performance. This module utilizes the knowledge from large-scale pre-trained models to introduce semantic information about abnormal types into the model, thereby enhancing the model's understanding of abnormal types. This step plays a key role in better distinguishing different abnormal rope skipping behaviors. By introducing the semantic guidance module, the model can more accurately identify and classify abnormal rope skipping data, thus improving AUC performance. Second, the improvement of the dual-memory network further enhances the model's performance. The design of this network allows the model to retain feature representations of both normal and abnormal data simultaneously, not just normal data, avoiding the limitation of a single memory network that can only represent normal data. Through the dual-memory network, the model can better adapt to the differences between normal and abnormal data, improving the model's generalization and performance. Finally, when the semantic guidance module and the dual-memory network are combined, the network performance is further improved. This combination leverages semantic information guidance and dual feature preservation to further enhance the model's classification and recognition capabilities. This indicates a synergistic effect between the two modules, improving the model's overall performance.

[0124] (II) Comparative Experiment

[0125] To compare the performance of the proposed algorithm horizontally, the AUC metric was compared with that of classic algorithms on a self-built dataset. Table 3 shows the comparative experimental results. See Table 3:

[0126] Table 3

[0127]

[0128] Table 3 illustrates the performance differences of different models when processing jump rope data. In the absence of prior information, the AUC values ​​of unsupervised methods are generally lower than those of weakly supervised methods. For example, the AUC values ​​of the MPN model and the GCL model are 56.03% and 78.01%, respectively. Weakly supervised methods provide video-level labels, introducing a small amount of prior information, thus significantly improving the AUC value. Specifically, the IBL model employs a multi-instance learning strategy for anomaly detection in jump rope videos and introduces an inner bag loss to constrain the feature spaces of normal and abnormal videos, achieving an AUC value of 82.69%. To reduce data correlation between training batches, the CLAWS model uses a random batch training method, achieving an AUC value of 87.28%. However, both the IBL and CLAWS models only consider the information of the current jump rope video frame, ignoring the semantics of the temporal context. Therefore, the CRFD model uses a causal temporal relationship module to capture local temporal dependencies between video frames, performing temporal enhancement on the features, achieving an AUC value of 89.24%. In the early stages of training anomaly detection models, insufficient training accuracy often leads to the selection of incorrect anomaly segments. To reduce the probability of incorrect selection, the MSL model uses sequences of multiple video segments as optimization units and introduces a self-training strategy to progressively learn and adjust anomaly scores, achieving an AUC of 90.01%. To more comprehensively understand and distinguish between normal and abnormal jump rope behavior, the UR-DMU model introduces a memory network to store information on normal and abnormal jump rope data separately, training normal and abnormal jump rope data in pairs, achieving an AUC of 91.43%. However, the memory network proposed in this method uses a backpropagation strategy to update memory terms, which can easily lead to the model getting trapped in local minima, resulting in slow convergence and increased training costs.

[0129] This invention further proposes the CSC-ME model, which incorporates linguistic prior information about abnormal rope skipping data cues and employs a self-supervised update strategy to improve the memory network of UR-DMU. These improvements increase the AUC to 94.14%, achieving a 2.71% performance improvement compared to the previous state-of-the-art model, UR-DMU. This means that the CSC-ME model has achieved satisfactory results in the abnormal rope skipping data detection task, significantly improving the model's performance.

[0130] This embodiment also provides a video anomaly detection device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0131] This embodiment provides a video anomaly detection device, such as... Figure 6 As shown, it includes:

[0132] The data acquisition module 501 is used to acquire the target video data to be detected.

[0133] The first feature extraction module 502 is used to extract the first feature from the target video data based on the text feature dimension to obtain semantic prior features.

[0134] The second feature extraction module 503 is used to extract second features from the target video data based on the video feature dimension to obtain fused visual features.

[0135] The feature fusion module 504 is used to perform feature fusion based on semantic prior features and fused visual features to obtain the final features of the target video data.

[0136] The anomaly detection module 505 is used to perform anomaly detection based on the final features and obtain the anomaly detection results.

[0137] In this embodiment of the video anomaly detection device, the first feature extraction module 502 performs first feature extraction on the target video data based on the text feature dimension to obtain semantic prior features, thereby extracting multiple abnormal visual features that may be contained in the target video data, thus enhancing the understanding of abnormal visual features; and the second feature extraction module 503 performs second feature extraction on the target video data based on the video feature dimension, which can consider the impact of normal visual features and abnormal visual features on the anomaly feature detection of the target video data at the same time, solving the problem that existing video anomaly feature detection methods are difficult to meet the needs of anomaly detection for large-scale video data, and can detect multiple abnormal visual features at the same time during anomaly feature detection.

[0138] In some optional implementations, the second feature extraction module 503 includes:

[0139] The visual feature extraction unit is used to extract video features from the target video data to obtain the visual features to be classified from the target video data.

[0140] The visual feature detection unit is used to input the visual features to be classified into a pre-trained visual feature detection model. The visual feature detection model detects the visual features to be classified based on the similarity between the visual features to be classified and the preset video features, and obtains normal visual features and abnormal visual features. The visual feature detection model is trained by historical video data and corresponding historical visual feature classification results.

[0141] The visual feature fusion unit is used to fuse normal visual features and abnormal visual features to obtain fused visual features.

[0142] In some alternative implementations, the visual feature detection unit includes:

[0143] The normal visual feature extraction subunit is used to detect the visual features to be classified based on the first memory term through the first feature extraction network to obtain normal visual features;

[0144] The abnormal visual feature extraction subunit is used to detect the visual features to be classified based on the second memory term through the second feature extraction network to obtain abnormal visual features.

[0145] In some optional implementations, the video anomaly detection device further includes:

[0146] A similarity calculation unit is used to calculate the similarity between the first memory item and the second memory item of the visual features to be classified.

[0147] A target memory item determination unit is used to determine the most relevant target memory item based on the memory item similarity.

[0148] An update unit is used to update the target memory item based on the visual features to be classified.

[0149] In some optional implementations, the video anomaly detection device further includes:

[0150] The normalization module is used to normalize the similarity of memory items to obtain the attention values ​​corresponding to the first and second memory items.

[0151] A fusion visual feature determination module is obtained, which is used to fuse based on attention values ​​and corresponding memory items to obtain fusion visual features.

[0152] In some optional implementations, the first feature extraction module 502 includes:

[0153] The prompt information extraction unit is used to extract semantic features from the target video data to obtain general prompt information and status prompt information;

[0154] The prompt information fusion unit is used to fuse general prompt information and status prompt information to obtain semantic prior features.

[0155] In some alternative implementations, the feature fusion module 504 includes:

[0156] The semantic feature set acquisition module is used to acquire a preset semantic feature set;

[0157] The semantic feature extraction module is used to extract semantic features from a preset semantic feature set based on semantic prior features.

[0158] The final feature determination module is used to perform feature fusion based on the similarity between semantic features and fused visual features to obtain the final features.

[0159] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0160] In this embodiment, the video anomaly detection device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0161] This invention also provides a computer device having the above-described features. Figure 6 The video anomaly detection device shown.

[0162] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0163] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0164] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0165] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0167] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0168] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0169] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting video anomalies, characterized in that, The video anomaly feature detection method includes: Acquire the target video data to be detected; Based on the text feature dimension, the first feature is extracted from the target video data to obtain semantic prior features; A second feature is extracted from the target video data based on the video feature dimension to obtain a fused visual feature; wherein the fused visual feature is obtained by fusing the normal visual features and abnormal visual features of the target video data; Based on the semantic prior features and the fused visual features, feature fusion is performed to obtain the final features of the target video data; Based on the final features, anomaly detection is performed to obtain anomaly detection results; The first feature extraction based on the text feature dimension of the target video data yields semantic prior features, including: Extract semantic features from the target video data to obtain general prompt information and status prompt information; The semantic prior features are obtained by fusing the general prompt information and the status prompt information; The second feature extraction based on the video feature dimension of the target video data to obtain fused visual features includes: Extract video features from the target video data to obtain the visual features of the target video data to be classified; The visual features to be classified are input into a pre-trained visual feature detection model. The visual feature detection model detects the visual features to be classified based on their similarity to preset video features, thereby obtaining the normal visual features and the abnormal visual features. The visual feature detection model is trained using historical video data and corresponding historical visual feature classification results. The normal visual features and the abnormal visual features are fused to obtain the fused visual features; The visual feature detection model includes a first feature extraction network composed of a first memory term and a second feature extraction network composed of a second memory term; The detection based on the similarity between the visual features to be classified and preset video features includes: The normal visual features are obtained by detecting the visual features to be classified using a first feature extraction network based on a first memory term; The abnormal visual features are obtained by detecting the visual features to be classified using a second feature extraction network based on a second memory term; The prompt template for a normal video state is no tiptoeing, rope present, full body in the frame, frontal view, and upright video; the prompt template for an abnormal video state is non-skipping rope video, tiptoeing, completely sideways view, no full body in the frame, back view, and horizontal video.

2. The video anomaly feature detection method according to claim 1, characterized in that, The video anomaly feature detection method further includes: Calculate the memory similarity between the first memory item and the second memory item of the visual features to be classified; The most relevant target memory item is determined based on the similarity of the memory items; The target memory item is updated based on the visual features to be classified.

3. The video anomaly feature detection method according to claim 2, characterized in that, The video anomaly feature detection method further includes: The similarity of the memory items is normalized to obtain the attention values ​​corresponding to the first and second memory items. The fused visual features are obtained by fusing the attention value and the corresponding memory item.

4. The video anomaly feature detection method according to claim 1, characterized in that, The feature fusion based on the semantic prior features and the fused visual features includes: Obtain a preset set of semantic features; Based on the aforementioned semantic prior features, semantic features are extracted from a preset semantic feature set; The final feature is obtained by fusing features based on the similarity between the semantic features and the fused visual features.

5. A video anomaly detection device, characterized in that, The video anomaly detection device includes: The acquisition module is used to acquire the target video data to be detected for video anomalies. The first feature extraction module is used to extract first features from the target video data based on the text feature dimension to obtain semantic prior features; The second feature extraction module is used to perform second feature extraction on the target video data based on the video feature dimension to obtain fused visual features; wherein, the fused visual features are obtained by fusing the normal visual features and abnormal visual features of the target video data; The feature fusion module is used to perform feature fusion based on the semantic prior features and the fused visual features to obtain the final features of the target video data; An anomaly scoring module is used to score anomalies based on the final features to obtain anomaly detection results. The first feature extraction module includes: The prompt information extraction unit is used to extract semantic features from the target video data to obtain general prompt information and status prompt information; The prompt information fusion unit is used to fuse general prompt information and status prompt information to obtain semantic prior features; The second feature extraction module includes: The visual feature extraction unit is used to extract video features from the target video data to obtain the visual features to be classified from the target video data. The visual feature detection unit is used to input the visual features to be classified into a pre-trained visual feature detection model. The visual feature detection model detects the visual features to be classified based on the similarity between the visual features to be classified and the preset video features, and obtains normal visual features and abnormal visual features. The visual feature detection model is trained by historical video data and corresponding historical visual feature classification results. The visual feature fusion unit is used to fuse normal visual features and abnormal visual features to obtain fused visual features; The visual feature detection unit includes: The normal visual feature extraction subunit is used to detect the visual features to be classified based on the first memory term through the first feature extraction network to obtain normal visual features; The abnormal visual feature extraction subunit is used to detect the visual features to be classified based on the second memory term through the second feature extraction network to obtain abnormal visual features; The prompt template for a normal video state is no tiptoeing, rope present, full body in the frame, frontal view, and upright video; the prompt template for an abnormal video state is non-skipping rope video, tiptoeing, completely sideways view, no full body in the frame, back view, and horizontal video.

6. A computer device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the video anomaly feature detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the video anomaly feature detection method according to any one of claims 1 to 4.

Images