A method, apparatus, and device for model training based on visual reconstruction supervision

Abstract

This application discloses a model training method based on visual reconstruction supervision, comprising: constructing a video anomaly detection model to be trained, the model including a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules; obtaining an anomaly video training set, and performing weakly supervised training on the video anomaly detection model to be trained using the anomaly video training set to obtain a trained target detection model; wherein, the global-local temporal modeling module is used to obtain target visual features corresponding to each target input video in the anomaly video training set, and the feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on each target visual feature during the weakly supervised training process. Implementing this application embodiment can fully utilize the original visual signals of the video training set during the model training process, which is beneficial to improving the weakly supervised model's ability to understand fine-grained visual features and its temporal modeling ability.

Description

Technical Field

[0001] This application relates to the field of machine learning, specifically to a model training method, apparatus, and device based on visual reconstruction supervision. Background Technology

[0002] Currently, various types of computer vision solutions, especially automated video anomaly detection solutions, are increasingly relied upon in numerous application scenarios such as video surveillance, industrial quality inspection, and internet television. However, existing video anomaly detection solutions still have many shortcomings in their vision-language pre-training (VLP) models and their training. These shortcomings include the fact that, due to reliance on coarse-grained video-level labels for supervised learning, model performance is often limited by the quality of visual representations, thereby reducing the model's understanding ability to some extent. Summary of the Invention

[0003] This application discloses a model training method, apparatus, and device based on visual reconstruction supervision, which can make full use of the original visual signals of the video training set during the model training process, thereby improving the weakly supervised model's ability to understand fine-grained visual features and its temporal modeling ability.

[0004] The first aspect of this application discloses a model training method based on visual reconstruction supervision, comprising: Construct a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules; An abnormal video training set is obtained, and the video anomaly detection model to be trained is weakly supervised and trained using the abnormal video training set to obtain a trained target detection model; wherein, the global-local temporal modeling module is used to obtain the target visual features corresponding to each target input video in the abnormal video training set, and the feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on each of the target visual features during the weakly supervised training process.

[0005] The second aspect of this application discloses a video anomaly detection method, including: The target video to be detected is input into the trained target detection model for detection, and the anomaly detection result corresponding to the target video to be detected is output by the target detection model; wherein, the target detection model is trained by any visual reconstruction-supervised model training method disclosed in the first aspect of the embodiments of this application.

[0006] A third aspect of this application discloses a model training apparatus based on visual reconstruction supervision, comprising: The model building unit is used to build a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules; The model training unit is used to acquire an abnormal video training set and perform weakly supervised training on the video anomaly detection model to be trained using the abnormal video training set to obtain the trained target detection model; wherein, the global-local temporal modeling module is used to acquire the target visual features corresponding to each target input video in the abnormal video training set, and the feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on each of the target visual features during the weakly supervised training process.

[0007] The fourth aspect of this application discloses a video anomaly detection device, comprising: The detection unit is used to input the target video to be detected into a trained target detection model for detection, and obtain the anomaly detection result corresponding to the target video output by the target detection model; wherein, the target detection model is trained by any visual reconstruction-supervised model training method disclosed in the first aspect of the embodiments of this application.

[0008] The fifth aspect of this application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor enables the processor to implement any of the visual reconstruction-supervised model training methods disclosed in the first aspect of this application, or any of the video anomaly detection methods disclosed in the second aspect of this application.

[0009] The sixth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements any of the visual reconstruction-supervised model training methods disclosed in the first aspect of this application, or any of the video anomaly detection methods disclosed in the second aspect of this application.

[0010] Compared with related technologies, the embodiments of this application have the following beneficial effects: In this embodiment, a trained target detection model is obtained by constructing a video anomaly detection model to be trained and acquiring an abnormal video training set. The trained model is then weakly supervised through this training set. The trained video anomaly detection model may include at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules. The global-local temporal modeling module is used to acquire the target visual features corresponding to each target input video in the abnormal video training set. The feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on the visual features of each target during weakly supervised training. Therefore, implementing this embodiment allows full utilization of the original visual signals from the video training set during the training of the VLP model. This enables the model to not only utilize coarse-grained video-level labels for weakly supervised learning but also directly utilize input features—i.e., visual features obtained from the video training set—as additional visual supervision to further learn the rich details of the original visual signals. By introducing additional visual supervision, not only are the performance limitations that may arise from the model's over-reliance on video-level label supervision avoided, but the reconstruction of the aforementioned input features can also guide the model's optimization process, encouraging the model to maintain attention to low-level image details. This is beneficial for improving the weakly supervised model's ability to understand fine-grained visual features. Building upon this, a model containing corresponding global-local temporal modeling (GLT modeling) modules is constructed to perform temporal modeling on the aforementioned input features. This enhances the temporal modeling capability of the weakly supervised model, helps to dynamically modulate the temporal receptive field in the feature extraction backbone network, and thus more efficiently adapts to and processes diverse and extensive video temporal contexts, further enhancing the model's sensitivity to subtle differences in features. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a model training method based on visual reconstruction supervision disclosed in an embodiment of this application; Figure 2 This is a schematic diagram of the overall framework of the video anomaly detection model disclosed in the embodiments of this application; Figure 3 This is a flowchart illustrating another model training method based on visual reconstruction supervision disclosed in an embodiment of this application; Figure 4 This is a flowchart illustrating another model training method based on visual reconstruction supervision disclosed in the embodiments of this application; Figure 5This is another schematic diagram of the overall framework of the video anomaly detection model disclosed in the embodiments of this application; Figure 6 This is a flowchart illustrating a video anomaly detection method disclosed in an embodiment of this application; Figure 7 This is a modular schematic diagram of a model training device based on visual reconstruction supervision disclosed in an embodiment of this application; Figure 8 This is a modular schematic diagram of a video anomaly detection device disclosed in an embodiment of this application; Figure 9 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0014] This application discloses a model training method, apparatus, and device based on visual reconstruction supervision, which can make full use of the original visual signals of the video training set during the model training process, thereby improving the weakly supervised model's ability to understand fine-grained visual features and its temporal modeling ability.

[0015] First, some of the technical terms disclosed in the embodiments of this application will be explained.

[0016] Weakly supervised learning refers to a machine learning approach where the training sample data used to train the model contains only partial or inaccurate annotation information. Compared to strongly supervised learning, which relies on a large amount of precisely labeled data, weakly supervised learning requires less labeled training sample data with coarser accuracy, which helps reduce data annotation costs and improve the overall efficiency of model training. For example, in video anomaly detection applications, training sample videos with video-level labels can be used for weakly supervised training of the video anomaly detection model. Here, "video-level labels" means that each training sample video is simply labeled as "normal" or "abnormal" as a whole, lacking annotation information down to the segment or frame level.

[0017] Visual supervision refers to the process of guiding a model to learn based on the annotation information of visual data (such as images, videos, etc.). The annotation information helps the model better understand the visual data and guides the model to learn the relationship between its visual features and the target task, so as to achieve further specific functions such as image classification, object detection, and semantic segmentation.

[0018] Temporal modeling refers to the process of modeling data arranged sequentially over time. Since some data exhibit a temporal order and corresponding dependencies, building appropriate models to capture and analyze these dependencies helps the models better understand the dynamic trends of these data. For example, temporal modeling of visual features helps the model understand the temporal relationships between visual features, facilitating further functions such as anomaly detection and trend prediction.

[0019] The following will be described in detail with reference to the accompanying drawings.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating a visual reconstruction-supervised model training method disclosed in an embodiment of this application. Figure 1 As shown, the model training method may include the following steps: S102. Construct a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules.

[0021] In this embodiment, the video anomaly detection model to be trained can be a VLP model trained using weakly supervised learning. That is, it is a type of model that uses large-scale image (video)-text pairs as training sample data to learn a joint representation between vision and language. Preferably, the video anomaly detection model can be built based on a CLIP (Contrastive Language-Image Pre-training) model. The CLIP model can include a pre-trained image encoder and a text encoder, respectively used to process the input image (video) and text data, converting them into feature vectors with the same dimension.

[0022] In some embodiments, the video anomaly detection model to be trained may include a global-local temporal modeling module, which can be used to acquire visual features corresponding to the input image (video) data in subsequent steps. For example, the input image (video) data may include various target input videos in an anomaly video training set, and the global-local temporal modeling module can be used to acquire target visual features corresponding to each target input video.

[0023] Specifically, the CLIP image encoder described above can be used to extract the corresponding video frame features from each target input video. (i represents the i-th video frame of the target input video); the features of the above video frames are processed by the global-local temporal modeling module. After processing, it can output features from each video frame. Corresponding target visual features The visual features of the target It can be further used in subsequent visual reconstruction supervision and other processes.

[0024] In some embodiments, the video anomaly detection model to be trained may include a feature reconstruction module, which can be used to process the input visual features to obtain corresponding reconstructed features, and then apply them to visual supervision in subsequent steps.

[0025] For example, based on the above-mentioned target visual features The feature reconstruction module can reconstruct the feature and output the corresponding reconstructed feature. As an optional implementation, the CLIP text encoder described above can also be used to extract target text features corresponding to the target input video. ; target visual features Corresponding target text features Using these features as inputs, we can obtain the reconstructed features reconstructed and output by the feature reconstruction module. This reconstruction feature It can be used to calculate video frame features. The reconstruction loss between them provides visual supervision and guidance for model training.

[0026] In some embodiments, the video anomaly detection model to be trained may include one or more visual classification branch modules, wherein each visual classification branch module may include necessary layers or components, primarily a classifier. For example, each visual classification branch module may calculate target visual features based on its respective classification method and using a corresponding loss function. The corresponding training loss provides further supervision and guidance for model training.

[0027] S104. Obtain an abnormal video training set, and use the abnormal video training set to perform weakly supervised training on the above-mentioned video anomaly detection model to be trained, so as to obtain the trained target detection model; wherein, the above-mentioned global-local temporal modeling module is used to obtain the target visual features corresponding to each target input video in the abnormal video training set, and the above-mentioned feature reconstruction module and visual classification branch module are used to perform visual reconstruction supervision based on each target visual feature during the weakly supervised training process.

[0028] In this embodiment, the training set of abnormal videos used to train the video anomaly detection model may include multiple target input videos with video-level labels. For example, the video-level label of the target input video may be Y, where Y∈ 0 indicates that the content of the target input video is normal, and 1 indicates that the content of the target input video is abnormal. In some embodiments, the target input video may also have more text tags indicating abnormal content, such as "fighting", "abuse", "riot", etc., which are not specifically limited in this application embodiment.

[0029] It is understandable that weakly supervised training of the video anomaly detection model to be trained using an abnormal video training set can include inputting each target video from the abnormal video training set into the video anomaly detection model in sequence. The model's global-local temporal modeling module, feature reconstruction module, and one or more visual classification branch modules will then perform continuous and parallel processing. The corresponding loss will be calculated based on a pre-defined loss function and backpropagated to the video anomaly detection model via gradients, ultimately resulting in the trained target detection model.

[0030] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the overall framework of a video anomaly detection model disclosed in an embodiment of this application. Figure 2 As shown, this video anomaly detection model includes the aforementioned global-local temporal modeling module, feature reconstruction module, and one or more visual classification branch modules. Figure 2 Taking one example, the model can also include a CLIP image encoder and a CLIP text encoder. Specifically, the target input video and its corresponding text labels in the anomaly video training set can be used as inputs to the CLIP image encoder and CLIP text encoder, respectively. Their output feature vectors can then be further input to the global-local temporal modeling module and the feature reconstruction module. Based on this, the reconstructed features output by the feature reconstruction module can be used to calculate the corresponding reconstruction loss. Based on the feature vectors further processed by the global-local temporal modeling module and / or the feature reconstruction module, different types of visual classification branch modules can perform corresponding classifications and calculate the corresponding training losses. By backpropagating the above training losses and reconstruction losses to the video anomaly detection model for parameter tuning, the video anomaly detection model can be iterated step by step until training is complete, resulting in the trained target detection model.

[0031] It should be noted that, Figure 2 The overall framework shown only includes the simplest necessary modules required to implement the model training method disclosed in the embodiments of this application. Other necessary or non-necessary modules (including layers or components) not specifically shown can be added or deleted according to the model training requirements, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0032] As can be seen, implementing the model training method described in the above embodiments can fully utilize the original visual signals of the video training set during the training process of the VLP model. This allows the model to directly use the input visual features as additional visual supervision to learn the rich details of the original visual signals. By introducing additional visual supervision, not only are the performance limitations that may result from the model's over-reliance on video-level label supervision avoided, but the reconstruction of the aforementioned visual features can also guide the model's own optimization process, encouraging the model to maintain attention to low-level image details. This is beneficial for improving the weakly supervised model's ability to understand fine-grained visual features. Based on this, by constructing a model that includes a global-local temporal modeling module to perform temporal modeling for the aforementioned visual features, the temporal modeling capability of the weakly supervised model can be improved. This helps to dynamically modulate the temporal receptive field in the feature extraction backbone network, thereby more efficiently adapting to and processing diverse and extensive video temporal contexts, and further enhancing the model's sensitivity to subtle differences in features.

[0033] Please see Figure 3 , Figure 3 This is a flowchart illustrating another model training method based on visual reconstruction supervision disclosed in an embodiment of this application. Figure 3As shown, the model training method may include the following steps: S302. Construct a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules.

[0034] Step S302 is similar to step S102 above, and will not be described again here.

[0035] S304. Obtain the first input video in the abnormal video training set, and the first video text label corresponding to the first input video.

[0036] In this embodiment of the application, the abnormal video training set may include multiple target input videos as training samples, and each target input video may have a corresponding video text tag.

[0037] For example, in an abnormal video training set primarily focused on security monitoring, the video text tags corresponding to the target input video may include terms such as "fighting," "abuse," and "riot." In an abnormal video training set primarily focused on product quality inspection, the video text tags may include terms such as "product damage," "product loss," and "product deformation." In an abnormal video training set primarily focused on industrial production scenarios, the video text tags may also include terms such as "fire," "leakage," and "collapse." Furthermore, the abnormal video training set may contain target input videos with one or more themes as their main content, thus allowing the target input videos to carry more diverse video text tags. This application embodiment does not impose specific limitations on this.

[0038] Optionally, the target input video in the abnormal video training set can also be a video obtained using non-visible light shooting methods such as infrared photography or ultraviolet photography. For example, in an abnormal video training set obtained by infrared photography, the video text tags can also include "temperature too high" or "temperature too low", etc., and this application embodiment does not make specific limitations.

[0039] To train a video anomaly detection model using an abnormal video training set, each target input video can be sequentially input into the model. In this embodiment, the current target input video can be used as the first input video in the abnormal video training set, and its corresponding video text tag can be used as the first video text tag. By obtaining the first input video and its corresponding first video text tag, and executing subsequent steps S306 to S312, the training process based on the first input video and the first video text tag can be completed. Based on this, the next target input video is used as the new first input video, and its corresponding video text tag is used as the new first video text tag. By obtaining the new first input video and its corresponding first video text tag, and repeatedly executing subsequent steps S306 to S312, the training of the video anomaly detection model can be gradually completed based on each target input video in the abnormal video training set.

[0040] S306. Obtain the first visual feature corresponding to the first input video, and obtain the first text feature corresponding to the first video text label.

[0041] In this embodiment, the global-local temporal modeling module in the video anomaly detection model can be used to obtain the target visual features corresponding to the first input video, which are denoted as the first visual features (also using...). Let 'i' represent the i-th video frame of the first input video. For example, using a pre-trained CLIP image encoder, features from multiple corresponding video frames can be extracted from the first input video. (i represents the i-th video frame of the first input video); for each video frame feature By inputting it into the global-local temporal modeling module for processing, the output can be related to the features of that video frame. Corresponding first visual features .

[0042] In this embodiment of the application, the target text feature corresponding to the first video text tag can be denoted as the first text feature (also using...). (Representation). For example, to obtain the first text feature corresponding to the first video text tag. This can be extracted using a pre-trained CLIP text encoder. Specifically, using the CLIP text splitting component (Tokenizer), the original word embedding vector corresponding to the first video text tag can be extracted and encoded. Learnable cues with the same dimension as this original word embedding vector are then added (e.g., concatenated) to this original word embedding vector, and the corresponding first text features can be extracted using the frozen CLIP text encoder. .

[0043] S308. Through the feature reconstruction module, determine the first reconstructed feature corresponding to the first visual feature and the first text feature.

[0044] In this embodiment of the application, the feature reconstruction module in the video anomaly detection model can be used to reconstruct the aforementioned first visual features. and first text features Visual feature reconstruction is performed to obtain the corresponding first reconstructed feature. In some embodiments, the feature reconstruction module can apply the global-local temporal modeling module described above to perform reconstruction. For example, the reconstruction is based on the high-order first visual features output by the global-local temporal modeling module. After mapping, the data is re-input into the global-local temporal modeling module, combined with the aforementioned first text feature. It can reconstruct the features of the original input video frames. The corresponding first reconstruction feature .

[0045] S310. Through one or more visual classification branch modules, determine the first abnormal parameter corresponding to the first visual feature.

[0046] In this embodiment, the visual classification branch module can classify the output results of the global-local temporal modeling module and the feature reconstruction module based on the classification logic of its classifier. In some embodiments, the visual classification branch module can use a binary classification method to score the corresponding video frame or video segment (containing multiple video frames) based on the above output results, and determine whether the video frame or video segment is normal or abnormal based on the scoring results (e.g., the i-th video frame of the first input video has normal or abnormal content). In other embodiments, the visual classification branch module can also use a fine-grained classification method to further classify abnormal video frames or video segments (e.g., when the i-th video frame of the first input video has abnormal content, its specific abnormality is "fighting", "abuse", "riot", etc.).

[0047] For example, the aforementioned visual classification branch module may include a visual classification branch module based on Multiple Instance Learning (MIL), a vision-language alignment (VAL) classification branch module, etc.

[0048] For example, the visual classification branch module based on multi-instance learning can be used to obtain each of the aforementioned first visual features. The corresponding multiple frame-level anomaly score prediction results can then be used as the first visual feature, either all or part of the frame-level anomaly score prediction results. The corresponding first anomaly parameter is used to calculate the corresponding training loss in subsequent steps.

[0049] For example, the visual-language alignment classification branch module can be used to obtain each of the aforementioned first visual features. With the corresponding first text features The cosine similarity between the matching features can be used to calculate (e.g., calculating the mean, variance, standard deviation, etc.) based on the full or partial cosine similarity between the matching features, and the result can be used as the first visual feature. The corresponding first anomaly parameter is then used to calculate the corresponding training loss in subsequent steps.

[0050] It should be noted that the multiple visual classification branch modules in the video anomaly detection model can be used in parallel, or any one of them can be used according to the model training requirements. No specific limitations are made in the embodiments of this application.

[0051] S312. Calculate the reconstruction loss corresponding to the first reconstructed feature and the first visual feature, and the training loss corresponding to the first anomaly parameter, respectively. Based on the reconstruction loss and the training loss, backpropagate to the video anomaly detection model through gradient.

[0052] In this embodiment of the application, regarding the aforementioned first reconstruction feature With first visual features The corresponding reconstruction loss calculation, as well as the training loss calculation corresponding to the first abnormal parameter obtained for different visual classification branch modules, can be performed using different loss functions.

[0053] For example, to calculate the first reconstruction feature With first visual features The corresponding reconstruction loss can be achieved using the KL divergence loss function. Specifically, as shown in Equation 1 below, the first reconstructed feature can be calculated using the KL divergence loss function. With corresponding video frame features Reconstruction losses between , to serve as the first reconstruction feature With first visual features The corresponding reconstruction loss.

[0054] Formula 1:

[0055] Among them, reconstruction losses It can be used for gradient backpropagation to update the parameters of a video anomaly detection model. By minimizing this KL divergence loss, the distribution of latent variables is made closer to the prior distribution, which helps the video anomaly detection model generate more reasonable samples.

[0056] For example, to calculate the training loss of a video anomaly detection model, a Top-K mechanism can be used. This involves selecting the K most suitable data points based on the type of the visual classification branch module to calculate the first anomaly parameter (K ​​is a positive integer) corresponding to that visual classification branch module. Based on this, the corresponding training loss can be calculated separately for each visual classification branch module according to its first anomaly parameter.

[0057] In some embodiments, the visual classification branch module still includes a visual classification branch module based on multi-instance learning, a visual-language alignment classification branch module, etc., as examples.

[0058] For the visual classification branch module that uses binary classification based on multi-instance learning, the above-mentioned first visual features can be used. Frame-level anomaly score prediction is performed, and based on multiple frame-level anomaly score prediction results, the first anomaly score output by this visual classification branch module is determined. Based on this, video-level anomaly score prediction results can be further determined, and then the binary cross-entropy between the video-level anomaly score prediction results and the ground truth values ​​in the classifier can be calculated as the training loss for this multi-instance learning-based visual classification branch module. .

[0059] For the vision-language alignment classification branch module, the above-mentioned first visual features can be calculated. With the corresponding first text features The cosine similarity between the input video and the current classification result is calculated, and the average of multiple cosine similarities is used to measure the alignment between the first input video and the current classification result. Based on this, the training loss for this visual-language alignment classification branch module can be determined by cross-entropy calculation. .

[0060] Furthermore, the total loss of the deep neural network of the video anomaly detection model disclosed in the embodiments of this application can be calculated using the following formula 2.

[0061] Formula 2:

[0062] in, This represents the total loss of the deep neural network. The contrast loss is used to further encourage and guide video anomaly detection models to separate the category embeddings of different classification categories (i.e., the classification results of "normal" and "abnormal").

[0063] In this embodiment, by repeatedly executing steps S304 to S312, each target input video in the abnormal video training set can be gradually used as the first input video to train the video anomaly detection model until the entire weakly supervised training process of the video anomaly detection model is completed. For example, the order in which the target input videos are input into the video anomaly detection model can be randomly determined or preset according to model training requirements; no specific limitation is made in this embodiment.

[0064] As can be seen, implementing the model training method described in the above embodiments can fully utilize the original visual signals of the video training set during the training process of the VLP model. This allows the model to not only utilize coarse-grained video-level labels for weakly supervised learning, but also directly utilize the input visual features as additional visual supervision to further learn the rich details of the original visual signals. By introducing additional visual supervision, not only are the performance limitations that may result from the model's over-reliance on video-level label supervision avoided, but the reconstruction of the aforementioned visual features can also guide the model's own optimization process, encouraging the model to maintain attention to low-level image details, thereby improving the weakly supervised model's ability to understand fine-grained visual features. Based on this, by constructing a model that includes a global-local temporal modeling module to perform temporal modeling for the aforementioned visual features, the temporal modeling capability of the weakly supervised model can be improved. This helps to dynamically modulate the temporal receptive field in the feature extraction backbone network, thereby more efficiently adapting to and processing diverse and extensive video temporal contexts, and further enhancing the model's sensitivity to subtle differences in features. Furthermore, classifying through one or more visual classification branch modules and calculating the corresponding classification loss can measure the training loss of the model from different dimensions, which helps to improve the model's classification ability and further enhance the detection performance of the video anomaly detection model.

[0065] Please see Figure 4 , Figure 4 This is a flowchart illustrating another model training method based on visual reconstruction supervision disclosed in an embodiment of this application. Figure 4 As shown, the model training method may include the following steps: S402. Construct a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules.

[0066] Step S402 is similar to step S102 above, and will not be described again here.

[0067] S404. Obtain the first input video in the abnormal video training set, and the first video text label corresponding to the first input video.

[0068] Step S404 is similar to step S304 above, and will not be described again here.

[0069] S406. Based on the CLIP image encoder, obtain the video frame features corresponding to the first input video.

[0070] For example, a pre-trained CLIP image encoder can extract features from multiple video frames corresponding to the first input video. In subsequent steps, the features of this video frame are analyzed using a global-local temporal modeling module. After processing, the output can be related to the features of this video frame. Corresponding first visual features .

[0071] In this embodiment, the global-local temporal modeling module may include a one-dimensional temporal convolution module and a graph convolution module. By applying the one-dimensional temporal convolution module with temporal context awareness, convolution kernels with different receptive fields can be combined to more accurately extract local features corresponding to abnormal content in the first input video; by applying the graph convolution module that performs graph convolution calculation based on the adjacency matrix, the features of the aforementioned video frames can be used to extract local features. The model combines the original feature similarity and inter-frame difference features with a time decay mechanism (a mechanism that adjusts the importance of different temporal information by introducing time weights, typically focusing more on recent information) to enhance its ability to capture subtle changes and anomalous events in the first input video. Based on this, the global-local temporal modeling module can output features from the aforementioned video frames. The corresponding global-local temporal dependent visual features are used as the features of this video frame. Corresponding first visual features .

[0072] S408. Using a one-dimensional temporal convolution module, perform a one-dimensional convolution operation on the above video frame features to obtain the target local features corresponding to the video frame features.

[0073] In this embodiment, the one-dimensional temporal convolution module can use multiple (at least two) one-dimensional temporal convolution sub-modules to process the aforementioned video frame features. To perform one-dimensional convolution operations, the aforementioned multiple one-dimensional temporal convolution submodules can have gradually increasing kernel sizes and dilation rates to achieve different receptive field ranges.

[0074] In some embodiments, the aforementioned one-dimensional temporal convolution module may include at least a first one-dimensional convolution sub-module and a second one-dimensional convolution sub-module, wherein the kernel size and dilation rate of the first one-dimensional convolution sub-module are both smaller than those of the second one-dimensional convolution sub-module.

[0075] For example, the first one-dimensional convolutional submodule corresponds to the first one-dimensional convolutional layer, with a kernel size of 3, a stride of 1, and a dilation rate of 1. Correspondingly, the second one-dimensional convolutional submodule corresponds to the second one-dimensional convolutional layer, with a kernel size of 5, a stride of 1, and a dilation rate of 2. It should be noted that the number of input and output channels of the one-dimensional convolutional layers corresponding to the first and second one-dimensional convolutional submodules should be the same, for example, both using 512 channels, 256 channels, or 128 channels, etc. This embodiment does not impose specific limitations. Optionally, when processing video frame features... Before inputting the first one-dimensional convolutional submodule and the second one-dimensional convolutional submodule in sequence, you can pad the input sequence with one zero element on each side.

[0076] For example, the global-local temporal modeling module can sequentially process video frame features through the first one-dimensional convolutional sub-module and the second one-dimensional convolutional sub-module described above. Perform one-dimensional convolution operations to obtain the first local features output by the first one-dimensional convolution submodule. And the second local features output by the second one-dimensional convolutional submodule. One-dimensional convolution operation can be represented by the following formula 3.

[0077] Formula 3:

[0078] Where s represents the kernel size. , respectively corresponding to the first local features mentioned above Second local features .

[0079] Based on this, by analyzing the aforementioned first local features and second local features Temporal concatenation yields concatenated local features, which can then be combined with different receptive field ranges. By applying temporal-dimensional pooling operations to these concatenated local features, their temporal relationships can be efficiently extracted, facilitating the subsequent acquisition of the corresponding temporal attention map. This implementation method helps to dynamically modulate the temporal receptive field when applying a trained video anomaly detection model for actual detection, thereby more efficiently adapting to and processing diverse and extensive video temporal contexts, and further enhancing the model's sensitivity to subtle differences in features.

[0080] For example, by performing time-based average pooling and max pooling operations on the aforementioned spliced ​​local features, corresponding pooled local features can be obtained. These pooled local features can then be converted into temporal attention maps corresponding to each convolutional kernel in the aforementioned spliced ​​local features using convolutional layers. Specifically, for example, the aforementioned spliced ​​local features consist of a first local feature... Second local features In the case of splicing, the corresponding pooled local features can be converted into two corresponding temporal attention maps.

[0081] Furthermore, based on the aforementioned temporal attention maps, corresponding attention features S can be obtained through fusion. Based on this, the aforementioned video frame features... By performing element-wise multiplication with the attention feature S, we can obtain the feature of that video frame. Corresponding target local features .

[0082] For example, to fuse the various temporal attention maps mentioned above, a sigmoid activation function can be used to generate temporal selection masks for each temporal attention map. For the local features output by the one-dimensional convolutional submodule with a large kernel size, a spatial selection mask can be further obtained, which is used for weighting before fusion. Specifically, for example, in the above-mentioned concatenated local features, the first local feature... Second local features In the case of splicing, the second local features can be obtained. The spatial selection mask (the kernel size of the first one-dimensional convolutional submodule is smaller than that of the second one-dimensional convolutional submodule) corresponds to this. Based on this, by performing a weighted fusion operation on the above temporal selection mask and spatial selection mask through a convolutional layer, attention features S corresponding to the above-mentioned spliced ​​local features can be obtained.

[0083] S410. Through the graph convolution module, perform graph convolution operation on the local features of the target to obtain the global-local temporal dependent visual features corresponding to the local features of the target.

[0084] In this embodiment of the application, in order to further enhance the video anomaly detection model's ability to capture fine-grained visual features in the first input video, the aforementioned target local features can be further refined. Further global modeling is performed to obtain global-local temporal dependent visual features corresponding to the local features of the target, which are then used as the video frame features mentioned above. Corresponding first visual features For example, the graph convolution module in the global-local temporal modeling module can obtain the target adjacency matrix corresponding to the target local features, and then perform graph convolution operation on the target local features based on the target adjacency matrix to obtain the global-local temporal dependent visual features corresponding to the target local features.

[0085] In some embodiments, the target adjacency matrix may include a similarity matrix. and the distance adjacency matrix .

[0086] As an optional implementation, a similarity matrix corresponding to the target local feature can be obtained by fusing the original similarity and differential feature similarity between the video frame corresponding to the target local feature and its adjacent frames. For example, by calculating the cosine similarity between the video frame corresponding to the target's local features and its neighboring frames, an original similarity adjacency matrix can be generated. To highlight inter-frame variations and make the adjacency matrix more sensitive to subtle changes, the absolute difference between the video frame corresponding to the target's local features and its neighboring frames can be further calculated as its differential feature similarity. Then, the original similarity adjacency matrix can be weighted and fused with this differential feature similarity to obtain a similarity matrix. .

[0087] As an optional implementation, a distance adjacency matrix corresponding to the target local feature can also be generated based on the video frame corresponding to the target local feature and the distance between its adjacent frames. For example, by employing a time decay mechanism, a preset decay index is assigned to other video frames that are gradually moving away from the video frame corresponding to the target local feature in time sequence. Then, the distance between the video frame corresponding to the target local feature and its neighboring frames is calculated, thereby generating a distance adjacency matrix. This ensures that the model maintains its focus on local temporal relationships.

[0088] Based on this, through the aforementioned similarity matrix And distance adjacency matrix Local features of the target Performing the graph convolution operation as shown in Formula 4 below, the resulting graph convolution operation can be used as the local feature of the target. Corresponding global-local temporal dependent visual features .

[0089] Formula 4:

[0090] Here, the Softmax function is the normalization exponential function, and W represents the parameters of the graph convolutional layer corresponding to the graph convolutional module.

[0091] S412. Based on the CLIP text splitting component, obtain the original word embedding vector corresponding to the first video text tag.

[0092] S414. Add the target learnable cue vector to the original word embedding vector and obtain the first text feature corresponding to the original word embedding vector based on the CLIP text encoder.

[0093] In this embodiment, the CLIP text splitting component (Tokenizer) is used to extract and encode the original word embedding vector corresponding to the first video text tag. Based on this, a learnable cue with the same dimension as the original word embedding vector is added (e.g., concatenated) to the original word embedding vector, and then the corresponding first text feature is extracted using the frozen CLIP text encoder. .

[0094] For example, the aforementioned learnable hints can be concatenated before or after the original word embedding vector. After inputting them into the frozen CLIP text encoder, the instructions most suitable for the task of the video anomaly detection model (i.e., video anomaly detection) can be learned and saved during subsequent training, so that they can be reused when applying the trained video anomaly detection model for actual detection.

[0095] S416. Obtain the first implicit feature corresponding to the first visual feature mentioned above through linear layer mapping.

[0096] S418. Input the first implicit feature and the first text feature into the global-local temporal modeling module to obtain the first reconstructed feature output by the global-local temporal modeling module.

[0097] In this embodiment, the feature reconstruction module can apply the aforementioned global-local temporal modeling module for reconstruction. For example, the aforementioned first visual features... (i.e., global-local temporal dependent visual features) The input is fed into a linear layer, which can determine the relationship with the first visual feature. The corresponding first implicit feature. Based on this, through the global-local temporal modeling module, the first implicit feature and the aforementioned first text feature can be compared. Reconstruct the features of the original input video frames. The corresponding first reconstruction feature .

[0098] As an optional implementation, the video anomaly detection model in this application may include one or more visual classification branch modules, specifically a visual classification branch module based on multi-instance learning and / or a visual-language alignment classification branch module. Thus, steps S4021 and S4202 below may be executed in parallel or either one may be executed at will.

[0099] S4201. The visual classification branch module based on multi-instance learning performs frame-level anomaly score prediction on the first visual feature to obtain the first anomaly score, which is used as the first anomaly parameter corresponding to the first visual feature.

[0100] Optionally, the first visual features are processed through a visual classification branch module based on multi-instance learning. By performing frame-level anomaly score prediction, this first visual feature can be obtained. The corresponding multiple frame-level anomaly score prediction results are used. Based on this, the Top-K mechanism is adopted to select the top K1 frame-level anomaly score prediction results from the above multiple frame-level anomaly score prediction results from high to low (i.e., the top K1 prediction results with high anomaly confidence, where K1 is a positive integer). These results can be used as the first anomaly score and applied to the subsequent training loss calculation.

[0101] S4202. Through the visual-language alignment classification branch module, calculate the matching cosine similarity between the first visual feature and the first text feature, and use it as the first anomaly parameter corresponding to the first visual feature.

[0102] Optionally, the visual-language alignment classification branch module can be used to calculate the first visual features corresponding to each video frame in the first input video. With the corresponding first text features The cosine similarity between the matches is used. Based on this, a Top-K mechanism is adopted to select the top K2 cosine similarities from the above multiple matching cosine similarities from high to low (i.e., the matching cosine similarities of the top K2 high abnormal matches, where K2 is a positive integer), and calculate the average of the top K2 matching cosine similarities, which can be used as the first abnormal score and applied to the subsequent training loss calculation.

[0103] By adopting the Top-K mechanism, it is beneficial to more accurately identify video frames or video segments with a high probability of anomalies during the training process, thereby improving the actual detection capability of the video anomaly detection model for video anomalies.

[0104] S422. Calculate the reconstruction loss corresponding to the first reconstructed feature and the first visual feature, and the training loss corresponding to the first anomaly parameter, respectively. Based on the reconstruction loss and the training loss, backpropagate to the video anomaly detection model through gradient.

[0105] Step S422 is similar to step S312 described above. It should be noted that, in this embodiment, by repeatedly executing steps S404 to S422, each target input video in the abnormal video training set can be gradually used as the first input video to train the video anomaly detection model until the entire weakly supervised training process of the video anomaly detection model is completed.

[0106] For example, please refer to Figure 5 , Figure 5 This is another schematic diagram of the overall framework of the video anomaly detection model disclosed in the embodiments of this application. For example... Figure 5 As shown, the training loss determined by the visual classification branch module based on multi-instance learning. The training loss determined by the visual-language alignment classification branch module. This can be compared with the reconstruction loss determined by the aforementioned feature reconstruction module. Together, they are used for gradient backpropagation to update the parameters of the video anomaly detection model, providing supervision and guidance for the model training process. (Not specifically shown).

[0107] As can be seen, implementing the model training method described in the above embodiments can fully utilize the original visual signals of the video training set during the training process of the VLP model. This allows the model to not only utilize coarse-grained video-level labels for weakly supervised learning, but also directly utilize the input visual features as additional visual supervision to further learn the rich details of the original visual signals. By introducing additional visual supervision, not only are the performance limitations that may result from the model's over-reliance on video-level label supervision avoided, but the reconstruction of the aforementioned visual features can also guide the model's own optimization process, encouraging the model to maintain attention to low-level image details, thereby improving the weakly supervised model's ability to understand fine-grained visual features. Based on this, by constructing a model that includes a global-local temporal modeling module to perform temporal modeling for the aforementioned visual features, the temporal modeling capability of the weakly supervised model can be improved. This helps to dynamically modulate the temporal receptive field in the feature extraction backbone network, thereby more efficiently adapting to and processing diverse and extensive video temporal contexts, and further enhancing the model's sensitivity to subtle differences in features. Furthermore, classifying through one or more visual classification branch modules and calculating the corresponding classification loss allows for the measurement of the model's training loss from different dimensions, which helps improve the model's classification ability and further enhances the detection performance of the video anomaly detection model. In addition, employing the Top-K mechanism helps to more accurately identify video frames or segments with a high probability of anomalies during training, thereby improving the video anomaly detection model's actual ability to detect video anomalies.

[0108] Please see Figure 6, Figure 6 This is a flowchart illustrating a video anomaly detection method disclosed in an embodiment of this application. Figure 6 As shown, the video anomaly detection method may include the following steps: S602. Input the target video to be detected into the trained target detection model for detection, and obtain the anomaly detection result corresponding to the target video output by the target detection model; wherein, the target detection model is trained by any of the visual reconstruction-supervised model training methods disclosed in the above embodiments.

[0109] In this embodiment of the application, by inputting the target video to be detected into the trained video anomaly detection model (i.e., the target detection model), it is possible to determine whether the target video to be detected contains corresponding abnormal content based on the classification results of one or more visual classification branch modules included in the model.

[0110] As can be seen, the video anomaly detection method described in the above embodiments can train a video anomaly detection model based on any of the visual reconstruction-supervised model training methods disclosed in the above embodiments. It can make full use of the original visual signal of the target video to be detected, enhance attention to its low-level image details, thereby improving the understanding of fine-grained visual features and more efficiently adapting to and processing diverse and extensive video temporal contexts, maintaining sensitivity to its subtle differences, and helping to improve the accuracy and efficiency of video anomaly detection by electronic devices.

[0111] Please see Figure 7 , Figure 7 This is a modular schematic diagram of a model training device based on visual reconstruction supervision disclosed in an embodiment of this application. For example... Figure 7 As shown, the model training device may include a model building unit 701 and a model training unit 702, wherein: The model building unit 701 is used to build a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules. The model training unit 702 is used to acquire an abnormal video training set and to perform weakly supervised training on the video anomaly detection model to be trained using the abnormal video training set to obtain the trained target detection model. The global-local temporal modeling module is used to acquire the target visual features corresponding to each target input video in the abnormal video training set. The feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on the visual features of each target during the weakly supervised training process.

[0112] As can be seen, the model training apparatus described in the above embodiments can fully utilize the original visual signals of the video training set during the training process of the VLP model. This allows the model to directly use the input visual features as additional visual supervision to learn the rich details of the original visual signals. By introducing additional visual supervision, not only are the performance limitations that may result from the model's over-reliance on video-level label supervision avoided, but the model's own optimization process can also be guided by reconstructing the aforementioned visual features. This encourages the model to maintain attention to low-level image details, thereby improving the weakly supervised model's ability to understand fine-grained visual features. Based on this, by constructing a model that includes a global-local temporal modeling module to perform temporal modeling for the aforementioned visual features, the temporal modeling capability of the weakly supervised model can be improved. This helps to dynamically modulate the temporal receptive field in the feature extraction backbone network, thereby more efficiently adapting to and processing diverse and extensive video temporal contexts, and further enhancing the model's sensitivity to subtle differences in features.

[0113] In some embodiments, the model training unit 702 described above can be specifically used for: Obtain the first input video in the abnormal video training set, and the first video text label corresponding to the first input video; Obtain the first visual feature corresponding to the first input video, and obtain the first text feature corresponding to the first video text label; The feature reconstruction module determines the first reconstructed feature corresponding to the first visual feature and the first text feature; The first abnormal parameter corresponding to the first visual feature is determined by one or more visual classification branch modules; Calculate the reconstruction loss corresponding to the first reconstructed feature and the first visual feature, and the training loss corresponding to the first anomaly parameter, respectively. Based on the reconstruction loss and the training loss, backpropagate to the video anomaly detection model through gradient. Repeat the steps of obtaining the first input video in the abnormal video training set and the first video text label corresponding to the first input video until the weakly supervised training of the video anomaly detection model to be trained is completed through all target input videos in the abnormal video training set.

[0114] In some embodiments, the model training unit 702, when used to acquire the first visual features corresponding to the first input video, may specifically include: Based on the CLIP image encoder, the video frame features corresponding to the first input video are obtained; The video frame features are processed by the global-local temporal modeling module, and the first visual features corresponding to the video frame features are output.

[0115] In some embodiments, the global-local temporal modeling module may include a one-dimensional temporal convolution module and a graph convolution module, and the first visual feature may include a global-local temporal dependent visual feature. When the model training unit 702 processes video frame features through the global-local temporal modeling module and outputs the first visual feature corresponding to the video frame features, it may specifically include: By using a one-dimensional temporal convolution module, one-dimensional convolution operations are performed on the video frame features to obtain the target local features corresponding to the video frame features. The graph convolution module performs graph convolution operations on the local features of the target to obtain global-local temporal dependent visual features corresponding to the local features of the target.

[0116] In some embodiments, the one-dimensional temporal convolution module may include at least a first one-dimensional convolution sub-module and a second one-dimensional convolution sub-module, wherein the kernel size and dilation rate of the first one-dimensional convolution sub-module are both smaller than those of the second one-dimensional convolution sub-module. When the model training unit 702 is used to perform a one-dimensional convolution operation on video frame features through the one-dimensional temporal convolution module to obtain target local features corresponding to the video frame features, it may specifically include: The video frame features are sequentially processed by the first one-dimensional convolution submodule and the second one-dimensional convolution submodule to perform one-dimensional convolution operations, thereby obtaining the first local feature output by the first one-dimensional convolution submodule and the second local feature output by the second one-dimensional convolution submodule. The first local feature and the second local feature are sequentially concatenated to obtain the concatenated local feature. Obtain the temporal attention map corresponding to the spliced ​​local features, and fuse the corresponding attention features based on the temporal attention map; Element-wise multiplication of the video frame features and attention features yields the target local features corresponding to the video frame features.

[0117] In some embodiments, when the model training unit 702 is used to obtain the temporal attention map corresponding to the spliced ​​local features and to fuse the corresponding attention features based on the temporal attention map, it may specifically include: The spliced ​​local features are subjected to time-based average pooling and max pooling operations to obtain the corresponding pooled local features. Convolutional layers are used to convert pooled local features into temporal attention maps corresponding to each convolutional kernel in the concatenated local features; Based on the Sigmoid activation function, time selection masks are generated for each time attention map. The spatial selection mask corresponding to the second local feature in the spliced ​​local features is obtained, and the temporal selection mask and the spatial selection mask are fused by a convolutional layer to obtain the attention feature corresponding to the spliced ​​local features.

[0118] In some embodiments, when the model training unit 702 is used to perform graph convolution operations on the local features of the target through the graph convolution module to obtain the global-local temporal dependent visual features corresponding to the local features of the target, it may specifically include: Obtain the target adjacency matrix corresponding to the local features of the target; Graph convolution is performed on the local features of the target based on the target adjacency matrix to obtain global-local temporal dependent visual features corresponding to the local features of the target.

[0119] In some embodiments, the target adjacency matrix may include a similarity matrix and a distance adjacency matrix. When the model training unit 702 is used to obtain the target adjacency matrix corresponding to the local features of the target, it may specifically include: Based on the original similarity and differential feature similarity between the video frames corresponding to the target local features and their adjacent frames, a similarity matrix corresponding to the target local features is obtained by fusing them. Based on the video frames corresponding to the target local features and the distances between them and their adjacent frames, a distance adjacency matrix corresponding to the target local features is generated.

[0120] In some embodiments, the model training unit 702, when used to acquire the first text features corresponding to the first video text label, may specifically include: Based on the CLIP text splitting component, obtain the original word embedding vector corresponding to the first video text tag; The target learnable cue vector is added to the original word embedding vector, and the first text feature corresponding to the original word embedding vector is obtained based on the CLIP text encoder.

[0121] In some embodiments, when the model training unit 702 is used to determine the first reconstructed feature corresponding to the first visual feature and the first text feature through the feature reconstruction module, it may specifically include: By mapping through linear layers, the first implicit feature corresponding to the first visual feature is obtained; The first implicit feature and the first text feature are input into the global-local temporal modeling module to obtain the first reconstructed feature output by the global-local temporal modeling module.

[0122] In some embodiments, the above-mentioned one or more visual classification branch modules may include a visual classification branch module based on multi-instance learning and / or a visual-language alignment classification branch module. When the model training unit 702 determines the first abnormal parameter corresponding to the first visual feature through one or more visual classification branch modules, it may specifically include: The visual classification branch module, based on multi-instance learning, performs frame-level anomaly score prediction on the first visual feature to obtain the first anomaly score, which serves as the first anomaly parameter corresponding to the first visual feature; and / or, The visual-language alignment classification branch module calculates the matching cosine similarity between the first visual feature and the first text feature, which is used as the first anomaly parameter corresponding to the first visual feature.

[0123] In some embodiments, when the model training unit 702 is used to predict the first anomaly score by performing frame-level anomaly score prediction on the first visual features through the visual classification branch module based on multi-instance learning, it may specifically include: The visual classification branch module based on multi-instance learning obtains multiple frame-level anomaly score prediction results corresponding to the first visual feature. From multiple frame-level anomaly score prediction results, the top K1 frame-level anomaly score prediction results are selected from high to low and used as the first anomaly score; where K1 is a positive integer.

[0124] In some embodiments, when the model training unit 702 calculates the matching cosine similarity between the first visual feature and the first text feature through the visual-language alignment classification branch module, as the first anomaly parameter corresponding to the first visual feature, it may specifically include: The visual-language alignment classification branch module is used to obtain the matching cosine similarity between the first visual features and the first text features corresponding to each video frame in the first input video. From multiple matching cosine similarities, the top K2 matching cosine similarities are selected from high to low, and the average of the top K2 matching cosine similarities is calculated as the first outlier score; where K2 is a positive integer.

[0125] Please see Figure 8 , Figure 8 This is a modular schematic diagram of a video anomaly detection device disclosed in an embodiment of this application. For example... Figure 8 As shown, the video anomaly detection device may include a detection unit 801, which may be used for: The target video to be detected is input into the trained target detection model for detection, and the anomaly detection result corresponding to the target video is obtained from the output of the target detection model; wherein, the target detection model is trained by any of the visual reconstruction-supervised model training methods disclosed in the above embodiments.

[0126] As can be seen, the video anomaly detection device described in the above embodiments can train the video anomaly detection model based on any of the visual reconstruction-supervised model training methods disclosed in the above embodiments. It can make full use of the original visual signal of the target video to be detected, enhance attention to its low-level image details, thereby improving the understanding of fine-grained visual features and more efficiently adapting to and processing diverse and extensive video temporal contexts, maintaining sensitivity to its subtle differences, and helping to improve the accuracy and efficiency of video anomaly detection by electronic devices.

[0127] Please see Figure 9 , Figure 9 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. For example... Figure 9 As shown, the electronic device may include: Memory 901 storing executable program code; Processor 902 coupled to memory 901; The processor 902 can call the executable program code stored in the memory 901 to execute all or part of the steps in any visual reconstruction-supervised model training method described in the above embodiments, or to execute all or part of the steps in any video anomaly detection method described in the above embodiments.

[0128] Furthermore, embodiments of this application disclose a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program enables a computer to execute all or part of the steps in any of the visual reconstruction-supervised model training methods described in the above embodiments, or to execute all or part of the steps in any of the video anomaly detection methods described in the above embodiments.

[0129] Furthermore, this application further discloses a computer program product that, when run on a computer, enables the computer to execute all or part of the steps in any of the visual reconstruction-supervised model training methods described in the above embodiments, or to execute all or part of the steps in any of the video anomaly detection methods described in the above embodiments.

[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0131] The foregoing has provided a detailed description of a model training method, apparatus, and device based on visual reconstruction supervision disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for model training based on visual reconstruction supervision, characterized in that, The method comprises: constructing a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained comprises at least a global-local time sequence modeling module, a feature reconstruction module, and one or more visual classification branch modules; obtaining an abnormal video training set, and performing weak supervision training on the video anomaly detection model to be trained through the abnormal video training set to obtain a trained target detection model; wherein the global-local time sequence modeling module is configured to obtain target visual features corresponding to each target input video in the abnormal video training set, and the feature reconstruction module and the visual classification branch module are configured to perform visual reconstruction supervision based on each target visual feature during the weak supervision training.

2. The method of claim 1, wherein, The weak supervision training on the video anomaly detection model to be trained through the abnormal video training set comprises: obtaining a first input video in the abnormal video training set and a first video text label corresponding to the first input video; obtaining a first visual feature corresponding to the first input video and a first text feature corresponding to the first video text label; determining a first reconstruction feature corresponding to the first visual feature and the first text feature through the feature reconstruction module; determining a first abnormal parameter corresponding to the first visual feature through the one or more visual classification branch modules; calculating a reconstruction loss corresponding to the first reconstruction feature and the first visual feature and a training loss corresponding to the first abnormal parameter, and performing gradient backpropagation to the video anomaly detection model based on the reconstruction loss and the training loss; repeating the steps of obtaining the first input video in the abnormal video training set and the first video text label corresponding to the first input video until the weak supervision training on the video anomaly detection model to be trained is completed through all target input videos in the abnormal video training set.

3. The method of claim 2, wherein, The first visual feature corresponding to the first input video is obtained by: obtaining a video frame feature corresponding to the first input video based on a CLIP image encoder; processing the video frame feature through the global-local time sequence modeling module to output the first visual feature corresponding to the video frame feature.

4. The method of claim 3, wherein, The global-local time sequence modeling module comprises a one-dimensional time sequence convolution module and a graph convolution module, and the first visual feature comprises a global-local time sequence dependent visual feature. The processing of the video frame feature through the global-local time sequence modeling module to output the first visual feature corresponding to the video frame feature comprises: performing one-dimensional convolution operation on the video frame feature through the one-dimensional time sequence convolution module to obtain a target local feature corresponding to the video frame feature; performing graph convolution operation on the target local feature through the graph convolution module to obtain a global-local time sequence dependent visual feature corresponding to the target local feature.

5. The method of claim 4, wherein, The one-dimensional time sequence convolution module at least includes a first one-dimensional convolution submodule and a second one-dimensional convolution submodule, wherein the convolution kernel size and the expansion rate of the first one-dimensional convolution submodule are smaller than those of the second one-dimensional convolution submodule, and the one-dimensional time sequence convolution operation is performed on the video frame feature through the one-dimensional time sequence convolution module to obtain a target local feature corresponding to the video frame feature, including: The first one-dimensional convolution submodule and the second one-dimensional convolution submodule are sequentially used to perform one-dimensional convolution operation on the video frame feature to obtain a first local feature output by the first one-dimensional convolution submodule and a second local feature output by the second one-dimensional convolution submodule, respectively; The first local feature and the second local feature are time sequence spliced to obtain a spliced local feature; A time attention map corresponding to the spliced local feature is obtained, and a corresponding attention feature is fused based on the time attention map; The video frame feature and the attention feature are element-wise multiplied to obtain a target local feature corresponding to the video frame feature.

6. The method of claim 5, wherein, The time attention map corresponding to the spliced local feature is obtained, and a corresponding attention feature is fused based on the time attention map, including: The spliced local feature is subjected to time sequence-based average pooling and maximum pooling operations to obtain a corresponding pooled local feature; The pooled local feature is converted into a time attention map corresponding to each convolution kernel in the spliced local feature through a convolution layer; Based on a Sigmoid activation function, a time selection mask corresponding to each time attention map is generated; A spatial selection mask corresponding to the second local feature in the spliced local feature is obtained, and the time selection mask and the spatial selection mask are fused through a convolution layer to obtain an attention feature corresponding to the spliced local feature.

7. The method of claim 4, wherein, The graph convolution module is used to perform graph convolution operation on the target local feature to obtain a global-local time sequence dependent visual feature corresponding to the target local feature, including: A target adjacency matrix corresponding to the target local feature is obtained; The target local feature is subjected to graph convolution operation based on the target adjacency matrix to obtain a global-local time sequence dependent visual feature corresponding to the target local feature.

8. The method of claim 7, wherein, The target adjacency matrix includes a similarity matrix and a distance adjacency matrix, and the target adjacency matrix corresponding to the target local feature is obtained, including: The similarity matrix corresponding to the target local feature is obtained by fusing the original similarity and the differential feature similarity between the video frame corresponding to the target local feature and its adjacent frames; The distance adjacency matrix corresponding to the target local feature is generated according to the distance between the video frame corresponding to the target local feature and its adjacent frames.

9. The method of claim 2, wherein, The first text feature corresponding to the first video text label is obtained, including: An original word embedding vector corresponding to the first video text label is obtained based on a CLIP text splitting component; The target learnable prompt vector is added to the original word embedding vector, and a first text feature corresponding to the original word embedding vector is obtained based on a CLIP text encoder.

10. The method of claim 2, wherein, The first reconstruction feature corresponding to the first visual feature and the first text feature is determined by the feature reconstruction module, including: A first implicit feature corresponding to the first visual feature is obtained through linear layer mapping; The first implicit feature and the first text feature are input into the global-local temporal modeling module to obtain the first reconstruction feature output by the global-local temporal modeling module.

11. The method of claim 2, wherein, The one or more visual classification branch modules include a multi-instance learning-based visual classification branch module and / or a visual-linguistic alignment classification branch module, and the first abnormal parameter corresponding to the first visual feature is determined by the one or more visual classification branch modules, respectively, including: The first visual feature is subjected to frame-level abnormal score prediction by the multi-instance learning-based visual classification branch module to obtain a first abnormal score as the first abnormal parameter corresponding to the first visual feature; and / or, The matching cosine similarity between the first visual feature and the first text feature is calculated by the visual-linguistic alignment classification branch module as the first abnormal parameter corresponding to the first visual feature.

12. The method of claim 11, wherein, The first visual feature is subjected to frame-level abnormal score prediction by the multi-instance learning-based visual classification branch module to obtain a first abnormal score, including: A plurality of frame-level abnormal score prediction results corresponding to the first visual feature are obtained by the multi-instance learning-based visual classification branch module; From the plurality of frame-level abnormal score prediction results, the top K1 frame-level abnormal score prediction results are filtered from high to low as the first abnormal score; wherein K1 is a positive integer.

13. The method of claim 11, wherein, The matching cosine similarity between the first visual feature and the first text feature is calculated by the visual-linguistic alignment classification branch module as the first abnormal parameter corresponding to the first visual feature, including: The matching cosine similarity between the first visual feature and the first text feature corresponding to each video frame in the first input video is obtained by the visual-linguistic alignment classification branch module, respectively; From the plurality of matching cosine similarities, the top K2 matching cosine similarities are filtered from high to low, and the average value of the top K2 matching cosine similarities is calculated as the first abnormal score; wherein K2 is a positive integer.

14. A method of video anomaly detection, the method comprising: including: The target detection model is trained by the model training method based on visual reconstruction supervision of any one of claims 1 to 13.

15. A model training device based on visual reconstruction supervision, characterized in that, including: A model construction unit is configured to construct a video anomaly detection model to be trained, wherein the video anomaly detection model to be trained includes at least a global-local temporal modeling module, a feature reconstruction module, and one or more visual classification branch modules; The model training unit is used to acquire an abnormal video training set and perform weakly supervised training on the video anomaly detection model to be trained using the abnormal video training set to obtain the trained target detection model; wherein, the global-local temporal modeling module is used to acquire the target visual features corresponding to each target input video in the abnormal video training set, and the feature reconstruction module and the visual classification branch module are used to perform visual reconstruction supervision based on each of the target visual features during the weakly supervised training process.

16. A video anomaly detection apparatus, characterized by comprising: include: The detection unit is used to input the target video to be detected into a trained target detection model for detection, and obtain the anomaly detection result corresponding to the target video output by the target detection model; wherein, the target detection model is trained by the model training method based on visual reconstruction supervision as described in any one of claims 1 to 13.

17. An electronic device, comprising: The device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 13, or the method as described in claim 14.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 13, or the method as described in claim 14.

Images