Video anomaly detection method and device, equipment, medium and program product

Abstract

The invention provides a video anomaly detection method which can be applied to the technical field of audio and video. The method comprises the steps that image frames in a video are collected, visual features of the image frames are extracted through a pre-trained lightweight encoder, and the pre-trained lightweight encoder is of a compressed model structure; performing feature fusion on the visual features to generate enhanced visual features; the enhanced visual features are input into a pre-trained lightweight decoder, text description describing the image frame is generated, and the pre-trained lightweight decoder is provided with a compressed model structure; and performing anomaly judgment based on the text description. The invention further provides a video anomaly detection device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the field of audio and video technology, and more specifically, to a video anomaly detection method, apparatus, device, medium, and program product. Background Technology

[0002] Currently, intelligent video surveillance uses computer vision, pattern recognition, and machine learning to analyze surveillance videos in real-time or offline to detect abnormal events. Existing intelligent video anomaly detection mainly falls into two categories: object detection-based schemes and video action recognition-based schemes. Object detection-based schemes typically use deep learning models (e.g., real-time object detection algorithms based on convolutional neural networks) to identify targets in the surveillance video, outputting the target's category and bounding box location, and determining whether an anomaly exists based on the detected target category. Video action recognition-based schemes, on the other hand, use 3D convolutional networks or temporal models to extract spatiotemporal features from the video, predict action categories, and determine whether an abnormal event has occurred.

[0003] Existing intelligent video anomaly detection systems have high computational resource requirements, making them difficult to deploy on local terminal devices; they also lack intuitive and natural language descriptions of the detection results, which hinders users from quickly understanding and judging abnormal events. Summary of the Invention

[0004] In view of the above problems, this application provides video anomaly detection methods, apparatus, equipment, media and program products that reduce computing power requirements and improve anomaly detection efficiency.

[0005] According to a first aspect of this application, a video anomaly detection method is provided, comprising: acquiring image frames from a video; extracting visual features of the image frames using a pre-trained lightweight encoder, wherein the pre-trained lightweight encoder has a compressed model structure; performing feature fusion on the visual features to generate enhanced visual features; inputting the enhanced visual features into a pre-trained lightweight decoder to generate a text description of the image frames, wherein the pre-trained lightweight decoder has a compressed model structure; and determining anomalies based on the text description.

[0006] According to an embodiment of this application, the compressed model structure includes reducing at least one of the following: the number of attention heads, the embedding dimension, and the number of model layers.

[0007] According to an embodiment of this application, the embedding dimension of the pre-trained lightweight encoder is the same as the embedding dimension of the pre-trained lightweight decoder.

[0008] According to an embodiment of this application, the pre-trained lightweight encoder is a visual neural network model generated through knowledge distillation. The training of the pre-trained lightweight encoder includes: acquiring a visual neural network model and a lightweight visual neural network model, wherein the lightweight visual neural network model has a compressed model structure; acquiring a first training dataset; inputting a first training image from the first training dataset into the visual neural network model and the lightweight visual neural network model; the lightweight visual neural network model mimicking the output of the visual neural network model to generate a pre-trained lightweight visual neural network model, wherein the pre-trained lightweight visual neural network model serves as the pre-trained lightweight encoder.

[0009] According to embodiments of this application, both the lightweight visual neural network model and the visual neural network model perform image classification operations on the first training image.

[0010] According to an embodiment of this application, the step of fusing the visual features to generate enhanced visual features includes: extracting multi-layer visual features output from different network layers in the pre-trained lightweight encoder; and fusing the multi-layer visual features to obtain the enhanced visual features.

[0011] According to an embodiment of this application, the pre-trained lightweight decoder is a sequence generation model based on a neural network architecture, and the pre-trained lightweight decoder decodes the enhanced visual features into the text description through a cross-attention mechanism.

[0012] According to an embodiment of this application, training the pre-trained lightweight decoder includes: acquiring a second training dataset, the second training dataset including a second training image and a corresponding text description; inputting the second training image into the pre-trained lightweight encoder to acquire second visual features; acquiring a lightweight decoder, the lightweight decoder being a sequence generation model based on a neural network architecture with a compressed model structure; and training the lightweight decoder using the second visual features and the corresponding text description to generate the pre-trained lightweight decoder.

[0013] A second aspect of this application provides a video anomaly detection device, comprising: a feature extraction module for acquiring image frames from the video and extracting visual features of the image frames using a pre-trained lightweight encoder, wherein the pre-trained lightweight encoder has a compressed model structure; a feature fusion module for performing feature fusion on the visual features to generate enhanced visual features; a text generation module for inputting the enhanced visual features into a pre-trained lightweight decoder to generate a text description of the image frames, wherein the pre-trained lightweight decoder has a compressed model structure; and an anomaly determination module for determining anomalies based on the text description.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of video anomaly detection methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating a video anomaly detection method according to an embodiment of this application is shown schematically.

[0020] Figure 3 This illustration schematically shows a flowchart of converting an image frame into a text description according to an embodiment of this application;

[0021] Figure 4 A schematic diagram illustrating the structure of a pre-trained lightweight encoder according to an embodiment of this application is shown.

[0022] Figure 5 This illustration schematically shows a flowchart of feature fusion of visual features to generate enhanced visual features according to an embodiment of the present application;

[0023] Figure 6 This schematically illustrates a structural diagram of the training of a pre-trained lightweight decoder according to an embodiment of this application;

[0024] Figure 7 This illustration schematically shows a flowchart of the terminal processing flow of the video anomaly detection method according to an embodiment of this application;

[0025] Figure 8 This schematic diagram illustrates a structural block diagram of a video anomaly detection device according to an embodiment of this application;

[0026] Figure 9A block diagram schematically illustrates an electronic device suitable for implementing a video anomaly detection method according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0032] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0033] Traditional bank surveillance systems primarily rely on manual review of video footage, which suffers from drawbacks such as difficulty maintaining focus for extended periods and reactive responses to anomalies. Typically, recordings need to be retrieved for review after an incident. While intelligent video surveillance systems can achieve a degree of automated detection, most are based on complex deep learning models, requiring significant hardware computing power. Furthermore, some bank surveillance hosts have low-configuration settings, making replacement or upgrades costly and limiting the widespread adoption of existing technologies.

[0034] The embodiments of this application provide a video anomaly detection method, which involves acquiring image frames from a video, extracting visual features from the image frames using a pre-trained lightweight encoder (the pre-trained lightweight encoder has a compressed model structure), fusing the visual features to generate enhanced visual features, inputting the enhanced visual features into a pre-trained lightweight decoder to generate a text description of the image frames (the pre-trained lightweight decoder has a compressed model structure), and determining anomalies based on the text description.

[0035] The video anomaly detection method with low computing power requirements provided in this application identifies potential anomaly elements based on image description technology and is suitable for deployment on local terminals.

[0036] It should be noted that the video anomaly detection method provided in this application can be used in the field of audio and video technology, or in any field other than audio and video technology, such as the field of financial technology. Here, the application field of this application is not limited.

[0037] Figure 1 The illustration shows an application scenario diagram of the video anomaly detection method according to an embodiment of this application.

[0038] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0039] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0041] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0042] It should be noted that the video anomaly detection method provided in this application embodiment can generally be executed by server 105. Correspondingly, the video anomaly detection device provided in this application embodiment can generally be located in server 105. The video anomaly detection method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the video anomaly detection device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0043] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] The following will be based on Figure 1 The described scene, through Figures 2-7 A video anomaly detection method according to embodiments of this application will be described in detail.

[0045] Figure 2 A flowchart illustrating a video anomaly detection method according to an embodiment of this application is shown schematically.

[0046] like Figure 2 As shown, the video anomaly detection method 200 of this embodiment includes operations S210 to S240.

[0047] It should be noted that, although the steps of the method are described in a specific order in the embodiments of this application, it should be understood that the execution order of these steps is not fixed and can be adjusted according to actual needs. Specifically, some steps can be executed in reverse order, or they can be executed in parallel to improve efficiency without affecting the overall method effect. In addition, some steps may be optional and can be selectively executed according to specific application scenarios or requirements.

[0048] During operation S210, image frames from the video are acquired, and visual features of the image frames are extracted through a pre-trained lightweight encoder with a compressed model structure.

[0049] In embodiments of this application, user consent or authorization can be obtained before acquiring user information. For example, a request to acquire user information can be sent to the user before operation S210. If the user consents or authorizes the acquisition of user information, operation S210 is performed.

[0050] In operation S220, feature fusion is performed on visual features to generate enhanced visual features.

[0051] In operation S230, enhanced visual features are input into a pre-trained lightweight decoder to generate text descriptions of image frames. The pre-trained lightweight decoder has a compressed model structure.

[0052] In operation S240, anomaly detection is performed based on text descriptions.

[0053] In embodiments of this application, a corresponding operation entry point can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before performing anomaly determination based on text description, the user can provide an instruction to agree to or reject the anomaly determination through the corresponding operation entry point. If the user agrees to perform the anomaly determination, the anomaly determination is performed based on the text description, i.e., step S240 is executed. If the user rejects the anomaly determination, the expert decision-making process is initiated.

[0054] For example, the method of the embodiment includes: capturing real-time video from a surveillance camera; extracting frames from the video stream, adjusting each frame to a fixed size, for example, image frames of [224, 224, 3]; extracting features from the images using a pre-trained lightweight encoder to output multi-layer feature maps; fusing shallow features (including detail features), mid-level features (including structural features), and deep features (including semantic features) to improve the saliency of small targets, for example, using a data augmentation module for feature fusion; converting the enhanced visual features into a visual feature sequence, for example, a visual feature sequence of [196, 384]; converting the visual feature sequence into a natural language description using a pre-trained lightweight decoder; and determining whether the video has anomalies based on the natural language description, for example, determining whether the video has anomalies through keyword matching.

[0055] According to the embodiments of this application, the video anomaly detection method of the embodiments reduces the computing power requirement and can be deployed on a local terminal; it can enhance small targets and improve the ability to recognize distant objects; it provides text descriptions of image frames, increases the interpretability of anomalies, and facilitates post-event review.

[0056] Figure 3 The flowchart illustrating the conversion of an image frame into a text description according to an embodiment of this application is shown in the illustration.

[0057] like Figure 3 As shown, the image input is converted into a feature map by a pre-trained lightweight encoder, which includes visual features extracted from the image; enhanced visual features are generated after feature enhancement processing; and the enhanced visual features are input into a pre-trained lightweight decoder to generate a text description corresponding to the image. Figure 3 The steps in the dashed box, "inputting the training text into the word segmenter (e.g., a word segmentation scheme combining a Chinese word segmentation tool with a byte-pair encoding word segmentation algorithm), forming a text sequence, and inputting the text sequence," describe the operations during the training of the lightweight decoder. When applying a pre-trained lightweight decoder, no text sequence is required. The following will refer to the training of the pre-trained lightweight decoder and... Figure 6 This step will be explained in detail and will not be elaborated upon here.

[0058] According to embodiments of this application, the compressed model structure includes reducing at least one of the following: the number of attention heads, the embedding dimension, and the number of model layers.

[0059] For example, compressing the encoder and decoder is a key step in achieving low computing power requirements in the embodiments of this application. For instance, knowledge distillation is performed on the encoder to achieve performance close to that of the original model when the number of parameters is significantly less than that of the original model (e.g., the number of parameters is one-quarter of the number of parameters of the original model).

[0060] In some embodiments, model compression of the encoder can be achieved by reducing at least one of the number of attention heads, embedding dimension, and number of model layers. For example, reducing the number of attention heads can create a pre-trained lightweight encoder; reducing the embedding dimension can create a pre-trained lightweight encoder; reducing the number of model layers can create a pre-trained lightweight encoder; reducing both the number of attention heads and embedding dimension can create a pre-trained lightweight encoder; reducing both the number of attention heads and model layers can create a pre-trained lightweight encoder; reducing both the number of model layers and embedding dimension can create a pre-trained lightweight encoder; reducing both the number of attention heads, embedding dimension, and number of model layers can create a pre-trained lightweight encoder. The operation of model compression of the decoder is similar to that of the encoder and will not be described in detail here.

[0061] According to embodiments of this application, by compressing the encoder and decoder, the model computing power can be reduced to meet the device configuration requirements of relevant scenarios, thereby deploying low-computing-power encoders and decoders to local terminals and reducing hardware replacement costs; by compressing the encoder and decoder, the model processing efficiency can be improved and the processing time can be shortened.

[0062] For the compressed encoder, the embodiment obtains a pre-trained lightweight encoder for extracting visual features from images through model training.

[0063] According to an embodiment of this application, the pre-trained lightweight encoder is a visual neural network model generated through knowledge distillation. The training of the pre-trained lightweight encoder includes: acquiring a visual neural network model and a lightweight visual neural network model, wherein the lightweight visual neural network model has a compressed model structure; acquiring a first training dataset, and inputting a first training image from the first training dataset into the visual neural network model and the lightweight visual neural network model; the lightweight visual neural network model mimics the output of the visual neural network model to generate a pre-trained lightweight visual neural network model, which serves as the pre-trained lightweight encoder.

[0064] For example, a pre-trained lightweight encoder can be obtained by distilling a visual neural network model (VisionTransformer, ViT). The student model, i.e., the lightweight visual neural network model, can employ a 12-layer neural network structure, reducing the number of parameters (e.g., reducing the number of parameters by 22M). Key improvements include halving the embedding dimension through linear mapping layers; for example, mapping a patch [196, 768] of [224, 224, 3] to [196, 384]. The number of attention heads is reduced from 12 to 6. The teacher model, i.e., the visual neural network model, uses ViT-B / 16 (with 80M parameters). The following will refer to... Figure 4 This document describes in detail the processing flow for extracting visual features from images using a pre-trained lightweight encoder.

[0065] According to an embodiment of this application, training a pre-trained lightweight encoder may include: inputting training images from the training dataset into a visual neural network model and a lightweight visual neural network model, respectively; the lightweight visual neural network model mimics the output of the visual neural network model to generate a pre-trained lightweight visual neural network model, that is, generating the pre-trained lightweight encoder in the embodiment.

[0066] According to embodiments of this application, the student model can learn powerful feature representation capabilities from a large teacher model. Even with a reduction in the number of parameters, it can still maintain performance close to that of the original model, thereby accurately extracting visual features of images with low computing power and improving the efficiency of visual feature extraction.

[0067] Unlike traditional convolutional neural networks, pre-trained lightweight encoders represent image features as a sequence of image patches, making them more suitable for visual scene understanding combined with language tasks.

[0068] According to embodiments of this application, both the lightweight visual neural network model and the visual neural network model perform image classification operations on the first training image.

[0069] Considering that the purpose of distillation is to learn the feature representation capabilities of the original model, the distillation process is based on image classification tasks (using image classification datasets) rather than image description tasks. Classification tasks are more mature and stable, and can efficiently learn general and powerful feature representation capabilities, laying a solid foundation for image description tasks.

[0070] According to embodiments of this application, performing an image classification operation on the first training image ensures that the lightweight encoder (e.g., a student model) learns general visual features, avoiding instability caused by distillation on the description task.

[0071] Figure 4A schematic diagram of a pre-trained lightweight encoder according to an embodiment of this application is shown.

[0072] For example, such as Figure 4 As shown, the input image has dimensions [224, 224, 3]. The image is divided into 14×14 blocks with a 16×16 kernel and a stride of 16, each block having a dimension of 768. After flattening, it is transformed into a two-dimensional sequence [196, 768] (where 14*14=196), and then linearly mapped to [196, 384] using a reshape operation, resulting in a visual sequence of 196 dimensions of 384. Positional encoding is added to the reshaped sequence while preserving its shape [196, 384]. The pre-trained lightweight encoder (Transformer encoding module) has 12 identical encoding layers, with the output of each layer serving as the input to the next. Each layer mainly includes: layer normalization, multi-head self-attention (6 heads), residual connections, layer normalization, multi-layer perceptron blocks (feedforward network), and residual connections. The pre-trained lightweight encoder converts the input sequence into visual features. Visual features can be extracted from layers 3, 6, 9, and 12 of a pre-trained lightweight encoder for feature fusion, forming enhanced visual features and constituting a feature map [14, 14, 384]. This feature map is then flattened to convert it into a sequence [196, 384]. After multi-layer perceptron block and layer normalization, a visual sequence, i.e., the enhanced visual features, is formed. It should be noted that extracting visual features from layers 3, 6, 9, and 12 of the pre-trained lightweight encoder for feature fusion is merely an example and not a limitation of the embodiments of this application. In other embodiments, visual features from other layers can be extracted.

[0073] For example, in the pre-trained lightweight encoder, the position-encoded sequence [196, 384] is normalized and then enters a multi-head self-attention layer. A linear transformation generates a query vector (Q), a key vector (K), and a value vector (V), all with dimensions [196, 384]. The query vector Q, key vector K, and value vector V are reshaped according to the number of attention heads (e.g., 6), meaning each attention head has a vector dimension of 64. Attention weights are calculated for each attention head, yielding their respective outputs [196, 64], which are then concatenated to obtain [196, 384]. The output of the multi-head attention layer is residually connected to the original input and normalized. The normalized result is then input into a feedforward network. In the feedforward network, a linear transformation is first used to increase the dimensionality, transforming [196, 384] into [196, 1536] to mitigate some information loss from the activation function. After a non-linear activation function, the network's fitting ability is enhanced, and then dimensionality reduction is performed to maintain the consistency of the input and output shapes. Then, a residual connection is performed again, and the result, after normalization, serves as the output of one layer. It should be noted that the number of attention heads (6) is merely an example and not a limitation on the embodiments of this application. In other embodiments, the number of attention heads can be other values, such as 5 or 8.

[0074] For example, a multilayer perceptron block may mainly include fully connected layers, activation functions (e.g., GELU) and fully connected layers. For example, the fully connected layer converts the sequence [196, 384] into the sequence [196, 1536], and after passing through the activation function and another fully connected layer, it is converted back into the sequence [196, 384].

[0075] According to embodiments of this application, feature fusion of visual features to generate enhanced visual features includes: extracting multi-layer visual features output from different network layers in a pre-trained lightweight encoder; and fusing the multi-layer visual features to obtain enhanced visual features.

[0076] In deep neural networks, low-level semantic features are typically extracted from the first few layers of the network, containing more detailed information (e.g., small objects), while high-level semantic features are typically extracted from the later layers, reflecting the overall semantic information. The method in this embodiment achieves feature enhancement by fusing features from different levels.

[0077] For example, the shallow attention maps of a visual neural network model (e.g., layer 3) focus on local details, such as edge information; the middle layers (e.g., layer 6) focus on local structures and simple objects, achieving a smooth transition in information fusion; and the deep layers (e.g., layers 9 and 12) focus on global semantics, such as overall contours and object relationships. By fusing feature sequences from different levels, a visual sequence representation that takes into account both local details and global semantics can be obtained.

[0078] According to embodiments of this application, by fusing shallow local detail features and deep global semantic features, the expressive power of the features is enhanced, which can better identify distant and small-sized abnormal targets and improve the accuracy and robustness of anomaly detection.

[0079] Figure 5 The flowchart illustrating feature fusion of visual features to generate enhanced visual features according to an embodiment of this application is shown schematically.

[0080] like Figure 5 As shown, the input is a multi-layer visual feature sequence, for example, the outputs of layers 3, 6, 9, and 12 from a pre-trained lightweight encoder, all in the range [196, 384]. Each sequence is reshaped into a feature map of [14, 14, 384]. A linear transformation with shared parameters [384, 384] is applied to each feature map to achieve feature alignment, for example, using a 2D convolutional layer with a kernel size of 1×1 and 384 input and output channels. x1, x2, x3, and x4 are the outputs of the feature alignment operation, where x1 is the aligned feature from layer 12, x2 is the aligned feature from layer 9, x3 is the aligned feature from layer 6, and x4 is the aligned feature from layer 3. The features from different layers are then weighted and averaged to output a feature map of uniform size [14, 14, 384]. The feature map is then flattened back into a visual sequence of [196, 384] and input into the pre-trained lightweight decoder.

[0081] For example, multi-layer visual features can be fused using the following formula.

[0082]

[0083] Among them, F out The feature maps are for enhancing visual features, where P1=x1, P2=x1+x2, P3=x1+x2+x3, and P4=x1+x2+x3+x4. The values ​​of P1, P2, P3, and P4 remain [14, 14, 384]. w1, w2, w3, and w4 are learnable weights. It should be noted that the above formula for fusing multi-layer visual features is merely an example and not a limitation of the embodiments of this application. In other embodiments, other methods for fusing multi-layer visual features can be used, which will not be elaborated here.

[0084] The video anomaly detection method in this embodiment inputs enhanced visual features into a pre-trained lightweight decoder to generate text descriptions of image frames. To ensure that the pre-trained lightweight decoder can adapt to and quickly process enhanced visual features, the embedding dimensions of the pre-trained lightweight encoder and the pre-trained lightweight decoder need to be designed.

[0085] According to an embodiment of this application, the embedding dimension of the pre-trained lightweight encoder is the same as the embedding dimension of the pre-trained lightweight decoder.

[0086] For example, designing the embedding dimension of the pre-trained lightweight encoder to be the same as that of the pre-trained lightweight decoder can avoid the additional linear transformation layer required for dimension alignment, further reducing the overall number of model parameters and computational complexity, and helping to deploy the model on local terminals with limited computing power.

[0087] According to embodiments of this application, the embedding dimension of the pre-trained lightweight encoder is the same as that of the pre-trained lightweight decoder, which is used to ensure compatibility between the pre-trained lightweight encoder and the pre-trained lightweight decoder, reduce unnecessary dimension transformation operations, and reduce computational overhead.

[0088] It should be noted that the video anomaly detection method in this application embodiment inputs the image into a pre-trained lightweight encoder after it has been processed into blocks. The visual features output by the pre-trained lightweight encoder are then subjected to feature enhancement processing to obtain enhanced visual features. These enhanced visual features are then converted into an enhanced visual sequence and input into a pre-trained lightweight decoder.

[0089] According to embodiments of this application, the pre-trained lightweight decoder is a sequence generation model based on a neural network architecture. The pre-trained lightweight decoder decodes enhanced visual features into text descriptions through a cross-attention mechanism.

[0090] In some embodiments, the pre-trained lightweight decoder employs a 6-layer neural network structure (e.g., a 6-layer transformer structure), half the number of layers in a conventional decoder. Since the embedding dimension of the pre-trained lightweight encoder is halved, the embedding dimension of the pre-trained lightweight decoder is also adjusted accordingly, for example, to be the same as the embedding dimension of the pre-trained lightweight encoder, thereby further reducing computational requirements. The multi-head attention mechanism of the pre-trained lightweight decoder still retains 12 attention heads because text generation tasks require accurate modeling of language structure, referential relations, and language alignment. Retaining a sufficient number of attention heads helps maintain the model's expressive power in the low-dimensional embedding space, ensuring the fluency of the generated text description.

[0091] The pre-trained lightweight decoder is a sequence generation model based on a neural network architecture that can decode the enhanced visual features of the input into a sequence output, for example, the output: <bos>(Start identifier), image, description, anomaly, detection, <eos>(End marker). During the decoding process, a cross-attention mechanism is introduced to calculate the correlation between the generated text and the enhanced visual features. Based on the correlation, information is extracted from the enhanced visual features to generate the decoded text information.

[0092] According to embodiments of this application, enhanced visual features are decoded into text descriptions through a cross-attention mechanism, which enables attention to the most relevant regions in the image, ensuring that the generated text descriptions are highly consistent with the image content and improving the accuracy of the text descriptions.

[0093] In some embodiments, the complete flow of the anomaly detection method is as follows: The local processing terminal acquires video streams from a surveillance camera and extracts video frames. It then uses a distilled encoder to extract semantic features at different levels and enhances the saliency of small targets in the feature map (targets in surveillance scenes are typically small and inconspicuous) through a feature enhancement module (for feature fusion). The enhanced features are divided into visual sequence representations (e.g., obtained through image patch tokenization) and mapped to a unified dimension, serving as input to a lightweight sequence-to-sequence decoding module (e.g., a pre-trained lightweight decoder). The decoding module, based on a cross-attention mechanism, progressively generates natural language descriptions corresponding to the surveillance footage, enabling anomaly determination based on the generated natural language descriptions.

[0094] In some embodiments, the pre-trained lightweight encoder and pre-trained lightweight decoder models of the embodiments can also be deployed at the camera end to reduce the computing power requirements of the local terminal by utilizing edge computing. However, such deployment requires the camera to have corresponding computing capabilities. For example, the pre-trained lightweight encoder and pre-trained lightweight decoder models of the embodiments can be deployed in some new cameras.

[0095] According to an embodiment of this application, training a pre-trained lightweight decoder includes: acquiring a second training dataset, the second training dataset including a second training image and a corresponding text description; inputting the second training image into a pre-trained lightweight encoder to acquire second visual features; acquiring a lightweight decoder, the lightweight decoder being a sequence generation model based on a neural network architecture with a compressed model structure; and training the lightweight decoder using the second visual features and the corresponding text description to generate a pre-trained lightweight decoder.

[0096] For example, the method of this embodiment trains a lightweight decoder with compressed model results using training images and corresponding text descriptions. The lightweight decoder model can be fine-tuned based on the loss values ​​of the generated text descriptions and their corresponding text descriptions to generate a pre-trained lightweight decoder. The output and training loss of the lightweight decoder can be jointly optimized through end-to-end backpropagation, for example, by fine-tuning the pre-trained lightweight encoder and feature enhancement process and updating the model parameters. After joint optimization, the pre-trained lightweight encoder is fine-tuned to extract visual features more suitable for the text description task; the feature enhancement process is optimized to learn the optimal fusion weights of different levels of visual features, providing the lightweight decoder with more informative enhanced visual features. For example, in the feature enhancement process, weights w1, w2, w3, and w4 are trainable parameters, and the values ​​of these trainable parameters can be adjusted based on backpropagation information to dynamically change the contribution ratio of different levels of features during fusion.

[0097] Figure 6 The diagram illustrates the structure of training a pre-trained lightweight decoder according to an embodiment of this application.

[0098] According to embodiments of this application, the lightweight concept and basic module design of the decoder continue the characteristics of the aforementioned lightweight encoder. The difference lies in the adoption of a masked self-attention mechanism and a cross-attention mechanism. The input includes two sources: a text sequence and a visual sequence (e.g., the visual sequence as the key vector K, the value vector V, and the text sequence as the query vector Q). The visual sequence is the sequence formed after the output of the feature enhancement process passes through a feedforward network, transforming the feature map into a feature sequence. That is, the visual sequence. The text sequence comes from, for example, a database.

[0099] like Figure 6 As shown, input a visual sequence, for example, the sequence [196, 384]. Input a pre-generated text token (e.g., [...]). <bos>(Start Identifier), Image, Description, Anomaly, Detection]), which is represented as a string of text token identifiers, each text token identifier being a one-hot encoding of the vocabulary size. An embedding layer maps, for example, [5, vocabulary size] to the sequence [5, 384] (aligned with the visual sequence) and adds positional encoding. The positionally encoded sequence [5, 384] is input into a 6-layer neural network structure, each layer including: a masked multi-head attention layer (12 attention heads), layer normalization, cross-attention (12 attention heads), residual connections, layer normalization, feedforward network, and layer normalization, etc. The masked multi-head attention layer also differs from the pre-trained lightweight encoder; the query vector Q, key vector K, and value vector V all come from [ <bos>The system uses five tokens: image, description, anomaly, and detection. However, it only calculates the attention weights between tokens like "anomaly" and "description," excluding the attention weights between "anomaly" and the subsequent "detection" (where "anomaly" is the position of the query vector Q). The output of the neural network is mapped to [5, vocabulary size] through a fully connected layer. An activation function is then used to obtain the probability distribution of the next token. The token identifier with the highest probability (the next token identifier of the vocabulary size) is selected and superimposed onto the generated text sequence. This process is repeated until an end marker appears (e.g., ...). <eos>The output is generated by a tokenizer, for example, [start identifier, image, description, anomaly, detection, end identifier].

[0100] For example, the feedforward network in a lightweight decoder may mainly include fully connected layers, activation functions (e.g., GELU) and fully connected layers. For example, the fully connected layer converts the sequence [5, 384] into the sequence [5, 1536], and then converts it back into the sequence [5, 384] after passing through the activation function and another fully connected layer.

[0101] For example, after passing through a 6-layer lightweight decoder, the 384-dimensional hidden vector is linearly mapped to [5, vocabulary size]. This is equivalent to scoring each word in the vocabulary, then obtaining the probability of each word through an activation function, and taking the token identifier with the highest probability and converting it into a one-hot encoded form, which is then superimposed on the previously generated text token identifier. It should be noted that the 6-layer lightweight decoder is only an example and is not a limitation on the embodiments of this application. In other embodiments, for example, an 8-layer lightweight decoder can also be used.

[0102] According to embodiments of this application, a lightweight encoder is trained first, and then a lightweight decoder is trained while the lightweight encoder is fixed. This reduces the complexity of the training process and the system coupling, allowing for independent replacement, updating, or optimization of different modules (lightweight encoder / lightweight decoder), thus increasing deployment flexibility. The training of the lightweight decoder involves learning how to map enhanced visual features to text descriptions, simplifying the training process and accelerating training convergence. The training of the lightweight decoder utilizes the output of the pre-trained lightweight encoder, increasing the accuracy of the generated text descriptions.

[0103] According to embodiments of this application, anomaly determination based on text description may include: matching the text description with keywords in an anomaly keyword database; and recording and reporting anomaly information in response to a successful match between the text description and keywords in the anomaly keyword database.

[0104] For example, when keywords related to dangerous behavior (such as "knife", "flame", etc.) appear in the text description, the text description, the corresponding video frame and the timestamp can be stored together, and the relevant video frame and text description can be sent to the client. If the staff has any questions about the abnormal situation, they can retrieve the records through the client for verification.

[0105] According to embodiments of this application, the text description generated by the pre-trained lightweight decoder is bound to and stored together with the image frame, which facilitates later review and tracing, and improves the interpretability of anomalies.

[0106] Figure 7 A flowchart illustrating the terminal processing flow of the video anomaly detection method according to an embodiment of this application is shown.

[0107] like Figure 7 As shown, the monitoring device provides video through a camera and transmits the video to a local processing terminal. The local processing terminal performs the following operations: extracting image frames from the video and preprocessing the extracted image frames, such as removing noise and normalizing; extracting image features from the preprocessed image frames (e.g., extracting visual features using a pre-trained lightweight encoder), and enhancing the image features using, for example, a feature enhancement module; a decoding module (e.g., a pre-trained lightweight decoder) generates a corresponding text description based on the enhanced image features; if an anomaly is detected based on the text description, the text description, image frame, and timestamp are saved, and the local processing terminal reports the anomaly to the local client, which displays the text description and image frame; the user of the local client (e.g., a security user) can review the record based on the saved text description, image frame, and timestamp. For example, if security personnel have doubts about the anomaly, they can retrieve the stored record on the client for review.

[0108] Based on the above-described video anomaly detection method, this application also provides a video anomaly detection device. The following will be combined with... Figure 8 The device is described in detail.

[0109] Figure 8 A schematic block diagram of a video anomaly detection device according to an embodiment of this application is shown.

[0110] like Figure 8 As shown, the video anomaly detection device 800 of this embodiment includes a feature extraction module 810, a feature fusion module 820, a text generation module 830, and an anomaly determination module 840.

[0111] The feature extraction module 810 is used to acquire image frames from the video and extract visual features of the image frames through a pre-trained lightweight encoder. The pre-trained lightweight encoder has a compressed model structure. In one embodiment, the feature extraction module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0112] The feature fusion module 820 is used to perform feature fusion on visual features to generate enhanced visual features. In one embodiment, the feature fusion module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0113] The text generation module 830 is used to input enhanced visual features into a pre-trained lightweight decoder to generate text descriptions of image frames. The pre-trained lightweight decoder has a compressed model structure. In one embodiment, the text generation module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0114] The anomaly detection module 840 is used to perform anomaly detection based on text description. In one embodiment, the anomaly detection module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0115] According to embodiments of this application, the compressed model structure includes reducing at least one of the following: the number of attention heads, the embedding dimension, and the number of model layers.

[0116] According to an embodiment of this application, the embedding dimension of the pre-trained lightweight encoder is the same as the embedding dimension of the pre-trained lightweight decoder.

[0117] According to an embodiment of this application, the pre-trained lightweight encoder is a visual neural network model generated through knowledge distillation. The training of the pre-trained lightweight encoder includes: acquiring a visual neural network model and a lightweight visual neural network model, wherein the lightweight visual neural network model has a compressed model structure; acquiring a first training dataset, and inputting a first training image from the first training dataset into the visual neural network model and the lightweight visual neural network model; the lightweight visual neural network model mimics the output of the visual neural network model to generate a pre-trained lightweight visual neural network model, which serves as the pre-trained lightweight encoder.

[0118] According to embodiments of this application, both the lightweight visual neural network model and the visual neural network model perform image classification operations on the first training image.

[0119] According to an embodiment of this application, the feature fusion module 820 is further configured to: extract multi-layer visual features output from different network layers in a pre-trained lightweight encoder; and fuse the multi-layer visual features to obtain enhanced visual features.

[0120] According to embodiments of this application, the pre-trained lightweight decoder is a sequence generation model based on a neural network architecture. The pre-trained lightweight decoder decodes enhanced visual features into text descriptions through a cross-attention mechanism.

[0121] According to an embodiment of this application, training a pre-trained lightweight decoder includes: acquiring a second training dataset, the second training dataset including a second training image and a corresponding text description; inputting the second training image into a pre-trained lightweight encoder to acquire second visual features; acquiring a lightweight decoder, the lightweight decoder being a sequence generation model based on a neural network architecture with a compressed model structure; and training the lightweight decoder using the second visual features and the corresponding text description to generate a pre-trained lightweight decoder.

[0122] According to embodiments of this application, any multiple modules among the feature extraction module 810, feature fusion module 820, text generation module 830, and anomaly detection module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the feature extraction module 810, feature fusion module 820, text generation module 830, and anomaly detection module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the feature extraction module 810, feature fusion module 820, text generation module 830, and anomaly detection module 840 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0123] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a video anomaly detection method according to an embodiment of this application.

[0124] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0125] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0126] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0127] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0128] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0129] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the video anomaly detection method provided in the embodiments of this application.

[0130] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0132] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0133] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.< / eos> < / bos> < / bos> < / eos> < / bos>

Claims

1. A video anomaly detection method, characterized in that, include: Image frames are captured from a video, and visual features of the image frames are extracted using a pre-trained lightweight encoder, which has a compressed model structure. The visual features are fused to generate enhanced visual features; The enhanced visual features are input into a pre-trained lightweight decoder to generate a text description of the image frame. The pre-trained lightweight decoder has a compressed model structure. Anomaly detection is performed based on the text description.

2. The method according to claim 1, characterized in that, The compressed model structure includes reducing at least one of the following: the number of attention heads, embedding dimension, and number of model layers.

3. The method according to claim 2, characterized in that, The embedding dimension of the pre-trained lightweight encoder is the same as the embedding dimension of the pre-trained lightweight decoder.

4. The method according to claim 1, characterized in that, The pre-trained lightweight encoder is a visual neural network model generated through knowledge distillation. The training of the pre-trained lightweight encoder includes: Obtain a visual neural network model and a lightweight visual neural network model, wherein the lightweight visual neural network model has a compressed model structure; Obtain a first training dataset, and input the first training image of the first training dataset into the visual neural network model and the lightweight visual neural network model; The lightweight visual neural network model mimics the output of the visual neural network model to generate a pre-trained lightweight visual neural network model, which serves as the pre-trained lightweight encoder.

5. The method according to claim 4, characterized in that, Both the lightweight visual neural network model and the visual neural network model perform image classification operations on the first training image.

6. The method according to claim 1, characterized in that, The visual features are then fused. Generating enhanced visual features includes: Extract multi-layer visual features from the outputs of different network layers in the pre-trained lightweight encoder; The enhanced visual features are obtained by fusing the multi-layer visual features.

7. The method according to claim 1, characterized in that, The pre-trained lightweight decoder is a sequence generation model based on a neural network architecture. The pre-trained lightweight decoder decodes the enhanced visual features into the text description through a cross-attention mechanism.

8. The method according to claim 1, characterized in that, The training of the pre-trained lightweight decoder includes: Obtain a second training dataset, which includes second training images and corresponding text descriptions; The second training image is input into the pre-trained lightweight encoder to obtain the second visual features; Obtain a lightweight decoder, which is a sequence generation model based on a neural network architecture and has a compressed model structure; The lightweight decoder is trained using the second visual feature and the corresponding text description to generate the pre-trained lightweight decoder.

9. A video anomaly detection device, characterized in that, The device includes: The feature extraction module is used to acquire image frames from the video and extract the visual features of the image frames through a pre-trained lightweight encoder, wherein the pre-trained lightweight encoder has a compressed model structure. The feature fusion module is used to perform feature fusion on the visual features to generate enhanced visual features; A text generation module is used to input the enhanced visual features into a pre-trained lightweight decoder to generate a text description of the image frame, wherein the pre-trained lightweight decoder has a compressed model structure. An anomaly detection module is used to detect anomalies based on the text description.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images