Open vocabulary video anomaly detection method and device based on multi-modal information

Abstract

The invention particularly relates to an open vocabulary video anomaly detection method and device based on multi-modal information, and the method comprises the steps: obtaining a tag text and a noun concept slot text based on a preset large language model, and inputting the tag text and the noun concept slot text to a pre-trained multi-modal model, thereby obtaining a tag text code and a noun concept slot; obtaining a visual code of the video frame based on a pre-trained multi-modal model, and carrying out time sequence modeling to obtain a visual code after time sequence modeling; obtaining a first enhancement result based on the visual code after time sequence modeling and the noun concept slot, obtaining a second enhancement result based on the visual code after time sequence modeling and the label text code, and obtaining a final abnormal score according to the enhancement result; and fusing the visual code of the video frame and the visual code after time sequence modeling, performing multi-modal matching with the label text code, and selecting a label text meeting a preset distance condition as a target prediction result. Therefore, the problems of detection fuzziness, classification confusion and the like in related technologies are solved, and the detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and computer vision technology, and in particular to a method and device for detecting anomalies in open vocabulary videos based on multimodal information. Background Art

[0002] Video anomaly detection is a current research hotspot in the field of video understanding. Its core objective is to determine whether anomalies exist in a video. This task closely aligns with real-world applications and holds broad application prospects in fields such as surveillance and security. Compared to traditional video anomaly detection, open vocabulary video anomaly detection takes this a step further by introducing open-world requirements. This requires the model to be able to detect and identify new categories of anomalies, even when trained only on basic categories. The goal of this task is to first determine whether an anomaly exists in a video and, once detected, further determine its specific category. This open-world setting makes it more suitable for real-world deployments, such as security systems, and offers greater practical value.

[0003] Traditional video anomaly detection methods are mainly divided into semi-supervised and weakly supervised methods. Semi-supervised methods use only normal video data for training, learning normal patterns to identify videos that deviate from these patterns and classify them as anomalies. Weakly supervised methods use both normal and abnormal videos for training, but lack the ability to accurately label the time when anomalies occur.

[0004] However, semi-supervised and weakly supervised methods are usually only applicable to the detection of closed sets, that is, they perform well on specific anomaly categories, but in real applications, when faced with new anomaly categories, their performance often degrades severely; the open set method proposed in related technologies can only determine whether there are anomalies in the video, and cannot provide more detailed anomaly category information, which limits its interpretability and intelligence level in practical applications; the open vocabulary video anomaly detection proposed in related technologies is prone to detection ambiguity and classification confusion, which needs to be solved urgently. Summary of the Invention

[0005] The present application provides an open vocabulary video anomaly detection method and device based on multimodal information to solve problems such as detection ambiguity and classification confusion in related technologies, thereby improving detection accuracy.

[0006] The first embodiment of the present application provides an open vocabulary video anomaly detection method based on multimodal information, comprising the following steps:

[0007] Based on a preset large language model, label text and noun concept slot text are obtained, and the label text and noun concept slot text are input into a pre-trained multimodal model to obtain label text encoding and noun concept slot; based on the pre-trained multimodal model, visual encoding of the video frame is obtained, and temporal modeling is performed on the visual encoding of the video frame to obtain visual encoding after temporal modeling;

[0008] A first enhancement result is obtained by enhancing scene information based on the visual encoding of the video frame and the noun concept slot, and a second enhancement result is obtained by enhancing video dynamic change information based on the visual encoding after temporal modeling and the label text encoding, and a final anomaly score is obtained according to the first enhancement result and the second enhancement result;

[0009] The visual coding of the video frame and the visual coding after time series modeling are fused to obtain a fused visual coding, the fused visual coding is multimodally matched with the label text coding, and the label text corresponding to the label text coding that meets the preset distance condition is selected as the target prediction result.

[0010] Optionally, the step of obtaining label text and noun concept slot text based on a preset large language model, and inputting the label text and noun concept slot text into a pre-trained multimodal model to obtain label text encoding and noun concept slots includes:

[0011] Input a tag list, group the tag list based on the preset large language model to obtain the tag text, and extract keywords from the tag list based on the preset large language model to obtain the noun concept slot text;

[0012] The label text is encoded by the pre-trained multimodal model to obtain the label text encoding, and the noun concept slot text is encoded by the pre-trained multimodal model to obtain the noun concept slot.

[0013] Optionally, obtaining a final anomaly score according to the first enhancement result and the second enhancement result includes:

[0014] Calculating a first anomaly score according to the first enhancement result;

[0015] Calculating a second anomaly score according to the second enhancement result;

[0016] Based on a preset weighted formula, the first anomaly score and the second anomaly score are fused to obtain the final anomaly score.

[0017] Optionally, the preset weighting formula is:

[0018] S=β·S dyn +(1-β)·Ssta ;

[0019] Among them, S is the final anomaly score, β is the weight, S dyn is the anomaly score calculated for the dynamic flow, S sta Anomaly score calculated for static flows.

[0020] Optionally, the fusing of the visual coding of the video frame and the visual coding after temporal modeling to obtain a fused visual coding, performing multimodal matching on the fused visual coding and the label text coding, and selecting the label text corresponding to the label text coding that meets a preset distance condition as the target prediction result includes:

[0021] Adding the visual code of the video frame and the visual code after the temporal modeling to obtain the fused visual code;

[0022] Based on the fused visual encoding, a preset cosine similarity is used to calculate the distance between the fused visual encoding of each frame and the label text encoding, and an average calculation is performed based on the frame-level results to obtain a video-level multimodal matching result;

[0023] Based on the multimodal matching result, the label text corresponding to the label text encoding that meets the preset distance condition is selected as the target prediction result.

[0024] A second embodiment of the present application provides an open vocabulary video anomaly detection device based on multimodal information, comprising:

[0025] An acquisition module is configured to acquire label text and noun concept slot text based on a preset large language model, and input the label text and noun concept slot text into a pre-trained multimodal model to obtain label text encoding and noun concept slot; acquire visual encoding of a video frame based on the pre-trained multimodal model, and perform time series modeling on the visual encoding of the video frame to obtain a visual encoding after time series modeling;

[0026] an anomaly score calculation module, which uses the visual encoding based on the video frame and the noun concept slot to enhance the scene information to obtain a first enhancement result, and enhances the video dynamic change information based on the visual encoding after the time series modeling and the label text encoding to obtain a second enhancement result, and obtains a final anomaly score based on the first enhancement result and the second enhancement result;

[0027] The abnormal label determination module is used to fuse the visual coding of the video frame and the visual coding after time series modeling to obtain a fused visual coding, perform multimodal matching on the fused visual coding and the label text coding, and select the label text corresponding to the label text coding that meets the preset distance condition as the target prediction result.

[0028] Optionally, the acquisition module is specifically configured to:

[0029] Input a tag list, group the tag list based on the preset large language model to obtain the tag text, and extract keywords from the tag list based on the preset large language model to obtain the noun concept slot text;

[0030] The label text is encoded by the pre-trained multimodal model to obtain the label text encoding, and the noun concept slot text is encoded by the pre-trained multimodal model to obtain the noun concept slot.

[0031] Optionally, the anomaly score calculation module is specifically configured to:

[0032] Calculating a first anomaly score according to the first enhancement result;

[0033] Calculating a second anomaly score according to the second enhancement result;

[0034] Based on a preset weighted formula, the first anomaly score and the second anomaly score are fused to obtain the final anomaly score.

[0035] Optionally, the preset weighting formula is:

[0036] S=β·S dyn +(1-β)·S sta ;

[0037] Among them, S is the final anomaly score, β is the weight, S dyn is the anomaly score calculated for the dynamic flow, S sta Anomaly score calculated for static flows.

[0038] Optionally, the abnormal label determination module is specifically configured to:

[0039] Adding the visual code of the video frame and the visual code after the temporal modeling to obtain the fused visual code;

[0040] Based on the fused visual encoding, a preset cosine similarity is used to calculate the distance between the fused visual encoding of each frame and the label text encoding, and an average calculation is performed based on the frame-level results to obtain a video-level multimodal matching result;

[0041] Based on the multimodal matching result, the label text corresponding to the label text encoding that meets the preset distance condition is selected as the target prediction result.

[0042] A third aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the open vocabulary video anomaly detection method based on multimodal information as described in the above embodiment.

[0043] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the open vocabulary video anomaly detection method based on multimodal information as described in the above embodiment.

[0044] A fifth aspect of the present application provides a computer program product, which stores a computer program. When the program is executed by a processor, it implements the open vocabulary video anomaly detection method based on multimodal information as described in the above embodiment.

[0045] Therefore, the embodiment of the present application obtains the label text and the noun concept slot text based on the preset large language model, and inputs them into the pre-trained multimodal model to obtain the label text encoding and the noun concept slot; obtains the visual encoding of the video frame based on the pre-trained multimodal model and performs time series modeling to obtain the visual encoding after time series modeling; obtains the first enhancement result based on the visual encoding after time series modeling and the noun concept slot, and obtains the second enhancement result based on the visual encoding after time series modeling and the label text encoding, and obtains the final anomaly score according to the first enhancement result and the second enhancement result; fuses the visual encoding of the video frame and the visual encoding after time series modeling to obtain the fused visual encoding, performs multimodal matching with the label text encoding, and selects the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result. Thus, the problems of detection ambiguity and classification confusion in the related technology are solved, and the detection accuracy is improved.

[0046] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 Flowchart of a method for detecting anomalies in open-vocabulary videos based on multimodal information according to an embodiment of the present application;

[0049] Figure 2 A flowchart of an open vocabulary video anomaly detection method based on multimodal information according to an embodiment of the present application is provided;

[0050] Figure 3 Schematic diagram of the overall framework of an open vocabulary video anomaly detection method based on multimodal information according to one embodiment of the present application;

[0051] Figure 4 A schematic diagram of functional modules of an open vocabulary video anomaly detection method based on multimodal information provided according to one embodiment of the present application;

[0052] Figure 5 Schematic diagram of the results of an open vocabulary video anomaly detection method based on multimodal information on the XD-Violence and UCF-Crime datasets according to one embodiment of the present application;

[0053] Figure 6 Schematic diagram of an open vocabulary video anomaly detection method and apparatus based on multimodal information according to an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0056] Before introducing the open vocabulary video anomaly detection method based on multimodal information according to an embodiment of the present application, a brief introduction to the open vocabulary video anomaly detection method in related art is first given.

[0057] To address the limitations of closed-set detection in video anomaly detection, open-set methods have been proposed in related art. This method uses both normal and partially abnormal data during training, enabling the model to detect anomalies beyond the training categories, thereby improving generalization. Under this setup, anomalies observed during training are referred to as baseline anomalies, while unseen anomalies are considered novel anomalies. However, open-set methods can only determine the presence of anomalies in a video but cannot provide more detailed information about the anomaly category, limiting their interpretability and intelligence in practical applications.

[0058] To further enhance the practicality of anomaly detection, the open vocabulary video anomaly detection task has been proposed in related technologies. This task inherits the principles of the open set method, but while detecting anomalies, it also outputs specific anomaly category labels, making the detection results more informative and readable. This improvement not only enhances the practical value of the method, enabling it to provide more comprehensive video anomaly analysis in real-world scenarios, but also lays a more solid foundation for applications such as intelligent security systems.

[0059] Most methods in the open vocabulary field typically leverage the generalization capabilities of pre-trained multimodal models to mitigate performance degradation on new data, primarily in textual and visual modalities. Therefore, addressing these two challenges, and focusing on the core focus of this approach, is to deeply explore the potential of these two modalities in addressing the challenges, systematically analyze the root causes of the problems, and design diverse multimodal information application strategies based on the characteristics of the video anomaly detection task.

[0060] The open vocabulary video anomaly detection task faces two core challenges: detection ambiguity and classification confusion. Detection ambiguity refers to the difficulty a model faces in accurately assessing anomaly scores when faced with new anomalies due to its unfamiliarity with their features, resulting in decreased detection performance. Classification confusion stems from the fact that open vocabulary tasks typically rely on multimodal matching to determine classification labels. When processing new anomaly data, the model tends to extract visual features similar to those of the underlying anomaly, making it more likely to misclassify it as an underlying anomaly during the multimodal matching process, leading to classification errors. These two challenges directly impact the performance of open vocabulary video anomaly detection, especially when dealing with unseen anomaly data, which may lead to inaccurate detection or incorrect classification. In practical deployments, overcoming these issues is crucial to ensure that the system maintains stable and reliable performance in complex and changing real-world scenarios.

[0061] This application is based on the above-mentioned problems and proposes an open vocabulary video anomaly detection method based on multimodal information. In this method, the embodiment of the present application obtains label text and noun concept slot text based on a preset large language model, and inputs them into a pre-trained multimodal model to obtain label text encoding and noun concept slot; obtains the visual encoding of the video frame based on the pre-trained multimodal model and performs time series modeling to obtain the visual encoding after time series modeling; obtains a first enhancement result based on the visual encoding after time series modeling and the noun concept slot, and obtains a second enhancement result based on the visual encoding after time series modeling and the label text encoding, and obtains a final anomaly score based on the first enhancement result and the second enhancement result; fuses the visual encoding of the video frame and the visual encoding after time series modeling to obtain a fused visual encoding, performs multimodal matching with the label text encoding, and selects the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result. In this way, the problems of detection ambiguity and classification confusion in the related technology are solved, and the detection accuracy is improved.

[0062] Specifically, Figure 1 This is a flowchart of an open vocabulary video anomaly detection method based on multimodal information provided in an embodiment of the present application.

[0063] like Figure 1 As shown, the open vocabulary video anomaly detection method based on multimodal information includes the following steps:

[0064] In step S101, based on the preset large language model, label text and noun concept slot text are obtained, and the label text and noun concept slot text are input into the pre-trained multimodal model to obtain label text encoding and noun concept slot; based on the pre-trained multimodal model, the visual encoding of the video frame is obtained, and the visual encoding of the video frame is temporally modeled to obtain the visual encoding after temporal modeling.

[0065] Optionally, in some embodiments, based on a preset large language model, label text and noun concept slot text are obtained, and the label text and noun concept slot text are input into a pre-trained multimodal model to obtain label text encoding and noun concept slot, including: inputting a label list, grouping the label list based on the preset large language model to obtain label text, and extracting keywords from the label list based on the preset large language model to obtain noun concept slot text; encoding the label text through the pre-trained multimodal model to obtain label text encoding, and encoding the noun concept slot text through the pre-trained multimodal model to obtain noun concept slot.

[0066] Among them, the preset large language model is an artificial intelligence model that is pre-integrated during the system initialization phase and is used to generate semantic text descriptions.

[0067] Specifically, input a list of labels and use a preset large language model to group the labels, placing visually similar abnormal labels into the same group; based on the grouping results, let the preset large language model generate label descriptions; for labels in the same group, the description should contain one or two sentences summarizing their common features, and one or two sentences highlighting their differences, and keep the sentence format consistent to ensure that the encoded data is similar within the same group but still distinguishable. Input a list of labels and let the preset large language model extract key nouns that describe abnormal events. These nouns should cover core elements such as significant scenes and key actions of abnormal events to enhance the fine-grained information of the description and obtain noun concept slot text. Input the label text into the text encoder in the pre-trained multimodal model to generate the corresponding label text encoding; at the same time, input the noun concept slot text into the text encoder, and summarize the encoding output to obtain the noun concept slot. The video frames are input into the visual encoder of the pre-trained multimodal model to extract the visual encoding before temporal encoding; the powerful generalization ability of the pre-trained multimodal model is used to ensure that the visual encoding can effectively capture the core information of the video frame; the extracted visual encoding is input into the temporal modeler for temporal modeling, and the long short-term memory network is used as the temporal modeler to capture the dynamic information in the video and obtain the visual encoding after temporal modeling, laying a solid foundation for subsequent video tasks.

[0068] In step S102, the scene information is enhanced based on the visual coding and noun concept slots of the video frame to obtain a first enhancement result, and the video dynamic change information is enhanced based on the visual coding and label text coding after time series modeling to obtain a second enhancement result, and the final anomaly score is obtained based on the first enhancement result and the second enhancement result.

[0069] Among them, the final anomaly score is a comprehensive score value obtained by fusing static scene information and dynamic timing information to quantify the degree of anomaly of the video frame.

[0070] Specifically, the static stream is based on the visual encoding before temporal modeling and uses noun concept slots to enhance scene information; the dynamic stream is based on the visual encoding after temporal modeling and uses label text encoding to enhance the dynamic change information of the video; the static stream and the dynamic stream predict anomaly scores respectively, and their outputs are fused to determine the final result.

[0071] Optionally, in some embodiments, obtaining a final anomaly score based on the first enhancement result and the second enhancement result includes calculating a first anomaly score based on the first enhancement result; calculating a second anomaly score based on the second enhancement result; and obtaining a final anomaly score by fusing the first anomaly score and the second anomaly score based on a preset weighted formula.

[0072] Optionally, in some embodiments, the preset weighting formula is:

[0073] S=β·S dyn+(1-β)·S sta ;

[0074] Among them, S is the final anomaly score, β is the weight, S dyn is the second anomaly score calculated by dynamic flow, S sta The first anomaly score calculated for the static flow.

[0075] It is understandable that the embodiment of the present application includes an enhancer for improving the expressiveness of visual encoding based on text encoding. The enhancer takes text encoding and visual encoding as input and adopts a multi-head attention mechanism, where the visual encoding is used as a query and the text encoding is used as a key and value to extract text information related to the visual encoding. At the same time, the visual encoding is also processed through a fully connected layer. Subsequently, the outputs of the multi-head attention layer and the fully connected layer are connected and input into a multi-layer perceptron for dimensionality reduction, thereby obtaining the final enhancement result, which is calculated as follows:

[0076] E refine =MHA(E visual , E textual , E textual )

[0077] E aug =MLP([E refine ,FC(E visual )]);

[0078] Among them, E refine is the relevant text information extracted by multi-head attention, E visual is the input visual code, E textual is the input label text encoding, MHA(·) is the multi-head attention layer, MLP(·) is the multi-layer perceptron layer, FC(·) is the fully connected layer, E aug The final enhanced result.

[0079] In the static stream, the visual encoding before temporal modeling and the noun concept slot are fed into the enhancer to obtain the enhanced encoding. In the dynamic stream, the visual encoding after temporal modeling and the label encoding are fed into the enhancer to obtain an enhanced representation that includes temporal information. The static and dynamic streams share the same detector to calculate the anomaly score. The detector adds the enhanced encoding to the result of its processing through the multi-layer perceptron and inputs it into the fully connected layer. Finally, the sigmoid function is used to map the encoding to a probability space between 0 and 1. The calculation formula is:

[0080] S dyn =Sigmoid(FC(E dyn_aug +MLP(E dyn_aug )))

[0081] Ssta =Sigmoid(FC(E sta_aug +MLP(E sta_aug )));

[0082] Among them, S dyn is the second anomaly score output by the dynamic flow, MLP(·) is the multi-layer perceptron layer, FC(·) is the fully connected layer, and E dyn_aug is the visual encoding after dynamic flow enhancement, S sta is the first anomaly score output by the static flow, E sta_aug Enhanced visual encoding for static streams.

[0083] Finally, the results of the static flow and the dynamic flow are weighted and summed to determine the final anomaly score, where the weight is 0.5.

[0084] In step S103, the visual coding of the video frame and the visual coding after time series modeling are fused to obtain a fused visual coding, the fused visual coding is multimodally matched with the label text coding, and the label text corresponding to the label text coding that meets the preset distance condition is selected as the target prediction result.

[0085] Optionally, in some embodiments, the visual encoding of the video frame and the visual encoding after time series modeling are fused to obtain a fused visual encoding, the fused visual encoding is multimodally matched with the label text encoding, and the label text corresponding to the label text encoding that meets the preset distance condition is selected as the target prediction result, including: adding the visual encoding of the video frame and the visual encoding after time series modeling to obtain the fused visual encoding; based on the fused visual encoding, using the preset cosine similarity to calculate the distance between the fused visual encoding and the label text encoding of each frame, and performing an average calculation based on the frame-level results to obtain the multimodal matching result at the video level; based on the multimodal matching result, selecting the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result.

[0086] Among them, the preset distance condition refers to the screening criteria used to determine the similarity between the visual features of the video frame and the text features of the abnormal label in the multimodal matching process.

[0087] It can be understood that the visual encodings before and after temporal modeling are added together to fuse scene information and temporal information, so that the visual encoding has richer feature expression capabilities during classification; the fused visual encoding is multimodally matched with the label text encoding, and the cosine similarity is used to calculate the distance between the visual encoding and the label text of each frame, and the result is mapped between 0 and 1 to measure the correlation between the visual features and the text label. The calculation formula is:

[0088]

[0089] Among them, P frame is the result of multimodal correspondence for each frame, E fused is the fused visual code, L desc Text encoding for exception labels.

[0090] For each abnormal label text, select M frames with a close distance (i.e., high similarity) and calculate the average of their similarities as the probability that the video belongs to the abnormal label; finally, the label with the highest probability is used as the prediction result; where M is the total number of video frames divided by 16, and the calculation formula is:

[0091]

[0092] Among them, P avg To find the probability that the video belongs to each label after averaging, σ(·) is the softmax function, M is the number of frames that meet the preset distance condition, and P frame is the result of multimodal correspondence for each frame, c is the number of labels, j is the index of the label (from 1 to c inclusive), P video is the classification result, and argmax(·) is the label of the position with the maximum probability.

[0093] To facilitate those skilled in the art to further understand the open vocabulary video anomaly detection method based on multimodal information in the embodiment of the present application, the following is a description of the method. Figures 2 to 5 The illustrated embodiment will be described in detail.

[0094] Specifically, if Figure 2 As shown, Figure 2 This is a flow chart of an open vocabulary video anomaly detection method based on multimodal information according to an embodiment of the present application, comprising the following steps:

[0095] S201: Generate group-based label description text and noun concept slots describing anomalies using a large language model, and obtain corresponding text encoding through a text encoder of a multimodal model.

[0096] S202: First, obtain the visual encoding of the video frame through the visual encoder of the multimodal model, and then use the temporal information modeler to perform temporal modeling on the visual encoding.

[0097] S203: The static stream is based on the visual encoding before temporal modeling and uses noun concept slots to enhance scene information. The dynamic stream is based on the visual encoding after temporal modeling and uses label encoding to enhance information about dynamic changes in the video. Both streams predict anomaly scores separately, and their outputs are fused to determine the final result.

[0098] S204: By fusing the visual encodings before and after temporal modeling (retaining scene and temporal information), and performing multimodal matching with the label text encoding, the label corresponding to the text encoding closest to the label is selected as the final prediction result.

[0099] Therefore, the present embodiment proposes a dynamic and static dual-stream architecture that fuses multimodal information to provide richer feature support, thereby more accurately determining the anomaly score of new anomalous frames and effectively alleviating detection ambiguity. In addition, the present embodiment guides the spatial distribution of text label encodings by grouping descriptions to optimize multimodal alignment, significantly reducing classification confusion.

[0100] Further, if Figure 3 and Figure 4 As shown, Figure 3 A schematic diagram of the overall framework of an open vocabulary video anomaly detection method based on multimodal information provided by one embodiment of the present application; a preset large language model is used to group and describe anomaly labels to generate label description text, and a pre-trained multimodal large model is used to convert the label description text into a label text encoding; at the same time, the preset large language model is used to obtain key nouns describing the anomaly based on the anomaly label, and the pre-trained multimodal large model converts these nouns into noun concept slots. The video frame generates a visual encoding through a visual encoder, and these visual encodings are processed by a time series modeler to generate a visual encoding after time series modeling. The dynamic stream combines the visual encoding after time series modeling and the label text encoding to enhance the dynamic change information in the video; the static stream combines the visual encoding before time series modeling and the noun concept slot to enhance the static scene information in the video. The first enhancement results and the second enhancement results of the dynamic stream and the static stream are fused to obtain the final anomaly score; in addition, by fusing the visual encoding before and after time series modeling and performing multimodal alignment on the fused visual encoding and the label text encoding, the classification result of the anomaly label is output.

[0101] Figure 4A functional module diagram of an open vocabulary video anomaly detection method based on multimodal information provided for one embodiment of the present application; the functional modules of the embodiment of the present application include a large language model module, a pre-trained multimodal model module, a temporal information modeling module, a visual information enhancement module, an anomaly detection module, and an anomaly label classification module. The large language model module is used to utilize the text generation capability of the preset large language model to generate label grouping results, label text, and anomaly noun concept slots; the pre-trained multimodal model module is used to utilize the generalization capability of the pre-trained model to extract the visual encoding of the video frame, the label text encoding, and the noun concept slot; the temporal information modeling module is used to perform temporal modeling on the visual encoding, laying the foundation for video-level anomaly detection; the visual information enhancement module is used to utilize text encoding to enhance visual information to provide richer information support for detection, including utilizing noun concept slots to enhance static frame visual features and utilizing label text encoding to enhance the visual encoding after temporal modeling; the anomaly detection module is used to calculate anomaly scores using the enhanced visual encoding to evaluate the degree of anomaly in the video; the anomaly label classification module is used to match the fused visual encoding with the label text encoding using a multimodal alignment strategy to determine the anomaly category.

[0102] Further, if Figure 5 As shown, Figure 5 A schematic diagram of the results of an open-vocabulary video anomaly detection method based on multimodal information on the XD-Violence and UCF-Crime datasets, provided in one embodiment of the present application; the first and second rows correspond to the detection results on the XD-Violence and UCF-Crime datasets, respectively; the red boxes and rectangles represent real anomaly frames, and the blue curve represents the anomaly score predicted by the model. All example anomalies are accurately classified, indicating that the method can effectively address the two key challenges of detection ambiguity and classification confusion.

[0103] Therefore, the embodiment of the present application focuses on the dynamic flow and static flow of temporal information and scene information respectively, and the corresponding text encoding enhances the visual encoding, fully utilizes multimodal information, and provides more comprehensive information support for the model, thereby effectively solving the detection ambiguity problem; obtains text encoding based on grouped label descriptions for classification, guides the position of label text in space, thereby achieving more accurate multimodal correspondence, and effectively solving the classification confusion problem; the embodiment of the present application has achieved significant performance improvement on the two video anomaly detection datasets XD-Violence and UCF-Crime, and significantly enhanced the effect of open vocabulary video anomaly detection.

[0104] According to the open vocabulary video anomaly detection method based on multimodal information proposed in the embodiment of the present application, the embodiment of the present application obtains the label text and noun concept slot text based on the preset large language model, and inputs them into the pre-trained multimodal model to obtain the label text encoding and noun concept slot; obtains the visual encoding of the video frame based on the pre-trained multimodal model and performs time series modeling to obtain the visual encoding after time series modeling; obtains the first enhancement result based on the visual encoding after time series modeling and the noun concept slot, and obtains the second enhancement result based on the visual encoding after time series modeling and the label text encoding, and obtains the final anomaly score based on the first enhancement result and the second enhancement result; fuses the visual encoding of the video frame and the visual encoding after time series modeling to obtain the fused visual encoding, performs multimodal matching with the label text encoding, and selects the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result. In this way, the problems of detection ambiguity and classification confusion in the related technology are solved, and the detection accuracy is improved.

[0105] Next, the open vocabulary video anomaly detection device for multimodal information proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0106] Figure 6 4 is a block diagram of an open vocabulary video anomaly detection device for multimodal information according to an embodiment of the present application.

[0107] like Figure 6 As shown, the multimodal information open vocabulary video anomaly detection device 10 includes: an acquisition module 100, an anomaly score calculation module 200 and an anomaly label determination module 300.

[0108] The acquisition module 100 is used to acquire label text and noun concept slot text based on a preset large language model, and input the label text and noun concept slot text into a pre-trained multimodal model to obtain label text encoding and noun concept slot; obtain visual encoding of video frames based on the pre-trained multimodal model, and perform time series modeling on the visual encoding of the video frames to obtain visual encoding after time series modeling;

[0109] Anomaly score calculation module 200 uses visual encoding based on video frames and noun concept slots to enhance scene information to obtain a first enhancement result, and enhances video dynamic change information based on visual encoding and label text encoding after time series modeling to obtain a second enhancement result, and obtains a final anomaly score based on the first enhancement result and the second enhancement result;

[0110] The abnormal label determination module 300 is used to fuse the visual coding of the video frame and the visual coding after time series modeling to obtain a fused visual coding, perform multimodal matching on the fused visual coding and the label text coding, and select the label text corresponding to the label text coding that meets the preset distance condition as the target prediction result.

[0111] Optionally, the acquisition module 100 is specifically used to: input a tag list, group the tag list based on a preset large language model to obtain tag text, and extract keywords from the tag list based on the preset large language model to obtain noun concept slot text; encode the tag text through a pre-trained multimodal model to obtain tag text encoding, and encode the noun concept slot text through a pre-trained multimodal model to obtain noun concept slot.

[0112] Optionally, the anomaly score calculation module 200 is specifically configured to: calculate a first anomaly score according to the first enhancement result; calculate a second anomaly score according to the second enhancement result; and obtain a final anomaly score by fusing the first anomaly score and the second anomaly score based on a preset weighted formula.

[0113] Optionally, the preset weighting formula is:

[0114] S=β·S dyn +(1-β)·S sta ;

[0115] Among them, S is the final anomaly score, β is the weight, S dyn is the anomaly score calculated for the dynamic flow, S sta Anomaly score calculated for static flows.

[0116] Optionally, the abnormal label determination module 300 is specifically used to: add the visual encoding of the video frame and the visual encoding after time series modeling to obtain a fused visual encoding; based on the fused visual encoding, use a preset cosine similarity to calculate the distance between the fused visual encoding and the label text encoding of each frame, and perform an average calculation based on the frame-level results to obtain a multimodal matching result at the video level; based on the multimodal matching result, select the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result.

[0117] It should be noted that the aforementioned explanation of the embodiment of the open vocabulary video anomaly detection method based on multimodal information is also applicable to the open vocabulary video anomaly detection method device based on multimodal information of this embodiment, and will not be repeated here.

[0118] According to the open vocabulary video anomaly detection device based on multimodal information proposed in the embodiment of the present application, the embodiment of the present application obtains label text and noun concept slot text based on a preset large language model, and inputs them into a pre-trained multimodal model to obtain label text encoding and noun concept slot; obtains the visual encoding of the video frame based on the pre-trained multimodal model and performs time series modeling to obtain the visual encoding after time series modeling; obtains a first enhancement result based on the visual encoding after time series modeling and the noun concept slot, and obtains a second enhancement result based on the visual encoding after time series modeling and the label text encoding, and obtains a final anomaly score based on the first enhancement result and the second enhancement result; fuses the visual encoding of the video frame and the visual encoding after time series modeling to obtain a fused visual encoding, performs multimodal matching with the label text encoding, and selects the label text corresponding to the label text encoding that meets the preset distance condition as the target prediction result. In this way, problems such as detection ambiguity and classification confusion in related technologies are solved, and detection accuracy is improved.

[0119] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0120] Memory 701 , processor 702 , and computer programs stored in the memory 701 and executable on the processor 702 .

[0121] When the processor 702 executes the program, the open vocabulary video anomaly detection method based on multimodal information provided in the above embodiment is implemented.

[0122] Furthermore, the electronic device further includes:

[0123] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0124] The memory 701 is used to store computer programs that can be run on the processor 702 .

[0125] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0126] If the memory 701, processor 702, and communication interface 703 are implemented independently, the communication interface 703, memory 701, and processor 702 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0128] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0129] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned open vocabulary video anomaly detection method based on multimodal information.

[0130] An embodiment of the present application further provides a computer program product, which stores a computer program. When the program is executed by a processor, the above-mentioned open vocabulary video anomaly detection method based on multimodal information is implemented.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0132] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0133] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0134] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

Claims

1. An open vocabulary video anomaly detection method based on multimodal information, characterized in that The method comprises the following steps: Based on a preset large language model, label text and noun concept slot text are obtained, and the label text and noun concept slot text are input into a pre-trained multimodal model to obtain label text encoding and noun concept slot; based on the pre-trained multimodal model, visual encoding of the video frame is obtained, and temporal modeling is performed on the visual encoding of the video frame to obtain visual encoding after temporal modeling; A first enhancement result is obtained by enhancing scene information based on the visual encoding of the video frame and the noun concept slot, and a second enhancement result is obtained by enhancing video dynamic change information based on the visual encoding after temporal modeling and the label text encoding, and a final anomaly score is obtained according to the first enhancement result and the second enhancement result; The visual coding of the video frame and the visual coding after time series modeling are fused to obtain a fused visual coding, the fused visual coding is multimodally matched with the label text coding, and the label text corresponding to the label text coding that meets the preset distance condition is selected as the target prediction result.

2. The method according to claim 1, characterized in that The method of obtaining label text and noun concept slot text based on a preset large language model, and inputting the label text and noun concept slot text into a pre-trained multimodal model to obtain label text encoding and noun concept slots includes: Input a tag list, group the tag list based on the preset large language model to obtain the tag text, and extract keywords from the tag list based on the preset large language model to obtain the noun concept slot text; The label text is encoded by the pre-trained multimodal model to obtain the label text encoding, and the noun concept slot text is encoded by the pre-trained multimodal model to obtain the noun concept slot.

3. The method according to claim 1, characterized in that Obtaining a final anomaly score according to the first enhancement result and the second enhancement result includes: Calculating a first anomaly score according to the first enhancement result; Calculating a second anomaly score according to the second enhancement result; Based on a preset weighted formula, the first anomaly score and the second anomaly score are fused to obtain the final anomaly score.

4. The method according to claim 3, characterized in that The preset weighted formula is: S=β·S dyn +(1-β)·S sta ; Among them, S is the final anomaly score, β is the weight, S dyn is the anomaly score calculated for the dynamic flow, S sta Anomaly score calculated for static flows.

5. The method according to claim 1, wherein The method further comprises: fusing the visual coding of the video frame and the visual coding after temporal modeling to obtain a fused visual coding, performing multimodal matching on the fused visual coding and the label text coding, and selecting the label text corresponding to the label text coding that meets a preset distance condition as a target prediction result, including: Adding the visual code of the video frame and the visual code after the temporal modeling to obtain the fused visual code; Based on the fused visual encoding, a preset cosine similarity is used to calculate the distance between the fused visual encoding of each frame and the label text encoding, and an average calculation is performed based on the frame-level results to obtain a video-level multimodal matching result; Based on the multimodal matching result, the label text corresponding to the label text encoding that meets the preset distance condition is selected as the target prediction result.

6. An open vocabulary video anomaly detection device based on multimodal information, characterized in that include: An acquisition module is configured to acquire label text and noun concept slot text based on a preset large language model, and input the label text and noun concept slot text into a pre-trained multimodal model to obtain label text encoding and noun concept slot; acquire visual encoding of a video frame based on the pre-trained multimodal model, and perform time series modeling on the visual encoding of the video frame to obtain a visual encoding after time series modeling; an anomaly score calculation module, which uses the visual encoding based on the video frame and the noun concept slot to enhance the scene information to obtain a first enhancement result, and enhances the video dynamic change information based on the visual encoding after the time series modeling and the label text encoding to obtain a second enhancement result, and obtains a final anomaly score based on the first enhancement result and the second enhancement result; The abnormal label determination module is used to fuse the visual coding of the video frame and the visual coding after time series modeling to obtain a fused visual coding, perform multimodal matching on the fused visual coding and the label text coding, and select the label text corresponding to the label text coding that meets the preset distance condition as the target prediction result.

7. The device according to claim 6, characterized in that The acquisition module is specifically used to: Input a tag list, group the tag list based on the preset large language model to obtain the tag text, and extract keywords from the tag list based on the preset large language model to obtain the noun concept slot text; The label text is encoded by the pre-trained multimodal model to obtain the label text encoding, and the noun concept slot text is encoded by the pre-trained multimodal model to obtain the noun concept slot.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the open vocabulary video anomaly detection method based on multimodal information according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the open vocabulary video anomaly detection method based on multimodal information as described in any one of claims 1 to 5.

10. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the program is executed by a processor, the open vocabulary video anomaly detection method based on multimodal information according to any one of claims 1 to 5 is implemented.

Images