Scene-dependent video anomaly detection method, device, equipment, and storage medium

Abstract

The present invention provides a scene-dependent video anomaly detection method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method comprises: using a large language model to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of a target scene to obtain an abnormal activity list of the target scene; uniformly sampling multiple frames within a time window centered on each frame of the video to be detected to construct a video segment corresponding to each frame; inputting the subtitles of all frames in each video segment and a summary generation prompt for describing time information into the large language model to generate a time summary corresponding to each video segment; and inputting the time summary corresponding to each video segment, the abnormal activity list of the target scene, an anomaly scoring prompt, and an output format requirement prompt into the large language model to obtain an anomaly score for each frame of the video to be detected. The present invention implements scene-dependent anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scene-dependent video anomaly detection method, apparatus, device, and storage medium. Background Art

[0002] Video Anomaly Detection (VAD) aims to accurately identify abnormal events from large-scale video streams and is widely used in the field of public security.

[0003] In recent years, video anomaly detection based on large language models (LLMs) has made significant progress. However, while the pre-trained LLMs used in previous methods contain rich general domain knowledge, they lack context-specific knowledge when determining anomalies, such as in campus environments, limiting their anomaly detection performance. Summary of the Invention

[0004] The present invention provides a scene-dependent video anomaly detection method, apparatus, device and storage medium, which are used to solve the defect of the prior art that the anomaly detection performance is limited when the judgment of abnormal events needs to rely on the scene, and realize anomaly detection in a scene-dependent manner.

[0005] The present invention provides a scene-dependent video anomaly detection method, comprising:

[0006] Using a large language model to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of a target scenario, obtaining an abnormal activity list of the target scenario; the target scenario is the scenario that anomaly detection relies on;

[0007] Uniformly sampling multiple frames of images within a time window with each frame of the video to be detected as the time center, and constructing a video clip corresponding to each frame of the image;

[0008] Inputting the subtitles of all frame images in each of the video clips and summary generation prompts for describing time information into a large language model to generate a time summary corresponding to each of the video clips;

[0009] The time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt and the output format requirement prompt are input into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0010] In some embodiments, the use of a large language model to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario to obtain an abnormal activity list of the target scenario includes:

[0011] inputting a first generation prompt for prompting generation of a universal exception management rule into a large language model to obtain the universal abnormal activity rule;

[0012] Inputting the security management specification of the target scenario and a second generation prompt for prompting generation of an abnormality management rule for the target scenario into a large language model to obtain an abnormal activity rule for the target scenario;

[0013] The general abnormal activity rules, the abnormal activity rules of the target scenario, and a third generation prompt for prompting generation of an abnormal activity list of the target scenario are input into a large language model to obtain the abnormal activity list of the target scenario.

[0014] In some embodiments, the method further comprises:

[0015] Taking each video segment as the center, the multiple video segments adjacent to it are grouped as a group of video segments;

[0016] The video encoder of the ImageBind model is used to encode the central video segment in each group of video segments to obtain the video features of the central video segment in each group of video segments;

[0017] Using the text encoder of the ImageBind model, respectively encode the time summary corresponding to each video segment in each group of video segments to obtain the time summary feature corresponding to each video segment in each group of video segments;

[0018] Based on the anomaly score of the frame image corresponding to each video clip in each group of video clips, the time summary feature corresponding to each video clip in each group of video clips, and the video feature of the central video clip in each group of video clips, the anomaly score of the frame image corresponding to the central video clip in each group of video clips is refined.

[0019] In some embodiments, the method further comprises:

[0020] An adaptive motion smoothing strategy is adopted to smooth the refined anomaly score of each frame image in the video to be detected, so as to obtain a smoothed anomaly score of each frame image in the video to be detected.

[0021] In some embodiments, the adaptive motion smoothing strategy is used to smooth the refined anomaly score of each frame image in the video to be detected to obtain the smoothed anomaly score of each frame image in the video to be detected, including:

[0022] Multiplying the number of frame images in the video to be detected by the proportional coefficient to obtain a product value;

[0023] The larger value between the product value and the minimum value of the sliding window is used as the size of the sliding window;

[0024] According to the size of the sliding window, the refined anomaly scores of all frame images in each sliding window centered on each frame image in the video to be detected are averaged to obtain a smoothed anomaly score for each frame image in the video to be detected.

[0025] In some embodiments, the method further comprises:

[0026] Using multiple BLIP2 models with different parameter sizes to generate candidate subtitles for each frame image in the video to be detected, to obtain a candidate subtitle set;

[0027] Encode each frame of the video to be detected using the image encoder in the ImageBind model to obtain image features of each frame of the image;

[0028] Encoding each candidate subtitle in the candidate subtitle set using a text encoder in the ImageBind model to obtain a subtitle feature of each candidate subtitle;

[0029] Calculate the cosine similarity between the image features of each frame in the video to be detected and the subtitle features of each candidate subtitle in the candidate subtitle set, and use the candidate subtitle corresponding to the maximum cosine similarity as the subtitle of each frame in the video to be detected.

[0030] The present invention also provides a scene-dependent video anomaly detection device, comprising:

[0031] A first acquisition module is configured to use a large language model to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario to obtain an abnormal activity list of the target scenario; the target scenario is the scenario that anomaly detection relies on;

[0032] A construction module is used to uniformly sample multiple frames of images within a time window with each frame of the video to be detected as the time center, and construct a video segment corresponding to each frame of the image;

[0033] A generation module, configured to input the subtitles of all frame images in each of the video segments and summary generation prompts for describing time information into a large language model to generate a time summary corresponding to each of the video segments;

[0034] The second acquisition module is used to input the time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the scene-dependent video anomaly detection method as described above is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described scene-dependent video anomaly detection methods.

[0037] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described scene-dependent video anomaly detection methods.

[0038] The scene-dependent video anomaly detection method, apparatus, device and storage medium provided by the present invention analyze general abnormal activity rules and abnormal activity rules of the target scene through a large language model, generate an abnormal activity list of the target scene, and improve the adaptability to the scene-dependent abnormal detection; uniformly sample multiple frames with each frame image as the center to construct the video segment corresponding to each frame image, which not only retains the local temporal context but also avoids the high overhead of frame-by-frame calculation, and balances computational efficiency and information integrity; generate prompts for the summary of subtitles and description time information in the video segment through the large language model, explicitly model the event logic in the time dimension, and make up for the implicit learning defect of traditional visual models for temporal logic; the large language model outputs anomaly scores based on the abnormal activity list and time summary of the target scene, providing a traceable decision basis, which is better than the unexplainable output of the black box model, and realizes anomaly detection in scene-dependent situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is a flow chart of the scene-dependent video anomaly detection method provided by the present invention.

[0041] Figure 2 This is an abnormal score result diagram provided by the present invention without using the moving smoothing strategy.

[0042] Figure 3 This is an abnormal score result diagram using the moving smoothing strategy provided by the present invention.

[0043] Figure 4 It is a structural diagram of the scene-dependent video anomaly detection device provided by the present invention.

[0044] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] Video anomaly detection is a key area in artificial intelligence and computer vision, aiming to accurately identify abnormal events from large-scale video streams. Abnormal events in videos are defined contextually based on real-world scenarios and settings and can generally be divided into two categories. The first category involves normal behaviors or events that occur in a limited number of locations within a specific scene, such as riding a bicycle or scooter on a sidewalk. The second category includes abnormal behaviors or events that may occur anywhere in the scene, such as fires or explosions. The first category is also called scene-dependent anomaly detection, because the judgment of abnormal events depends not only on the event itself, but also on whether the rules of the environment in which the event occurs restrict the occurrence of the event. In addition, the differences in rules in different scenarios increase the difficulty and uncertainty of judging scene-dependent anomalies. For example, Table 1 shows abnormal events in different scenarios, one of which is scene-dependent anomalies in a campus setting.

[0046] Table 1 Abnormal events in different scenarios

[0047]

[0048] In recent years, significant progress has been made in video anomaly detection using large language models (LLMs). However, while pre-trained LLMs in previous approaches contain rich general domain knowledge, their anomaly detection performance is limited in scenarios where abnormal event detection depends on context, such as campus environments.

[0049] Therefore, the present invention provides a scene-dependent video anomaly detection method, which automatically generates anomaly detection rules according to management specifications of different scenes through a rule enhancement strategy, and then performs anomaly detection based on the anomaly detection rules.

[0050] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] Figure 1 : is a flow chart of the scene-dependent video anomaly detection method provided by the present invention, such as Figure 1As shown, the present invention provides a scene-dependent video anomaly detection method, comprising:

[0052] Step 110 , using a large language model to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario, to obtain an abnormal activity list of the target scenario; the target scenario is the scenario that anomaly detection relies on.

[0053] Specifically, the general scenario is a scenario that anomaly detection does not rely on. The target scenario is a scenario that anomaly detection relies on, such as a campus scenario, a hospital scenario, etc. The general abnormal activity rules in the general scenario and the abnormal activity rules in the target scenario are obtained.

[0054] The general abnormal activity rules and the abnormal activity rules of the target scenario are input into a large language model (such as GPT-4). The large language model combines the general abnormal activity rules and the abnormal activity rules of the target scenario to perform abnormal activity analysis. That is, the large language model determines the abnormal activity list in the general scenario based on the general abnormal activity rules, determines the preliminary abnormal activity list in the target scenario based on the abnormal activity rules of the target scenario, and combines the abnormal activity list in the general scenario with the preliminary abnormal activity list in the target scenario to obtain the abnormal activity list in the target scenario.

[0055] Step 120 : uniformly sampling multiple frames of images within a time window with each frame of the video to be detected as the time center, and constructing a video segment corresponding to each frame of the image.

[0056] Specifically, the following is the Frame Image Take the following as an example to explain how to construct a video clip:

[0057] To be tested in the video Frame Image As the time center, extend forward and backward by T / 2 seconds to construct a time window with a time length of T. In this time window, uniformly sample N frames of images to construct the first Frame Image Corresponding video clips Therefore, the video clip Contains N+1 frames of images, among which the Frame Image For video clips The center image.

[0058] In the case that the time window exceeds the time boundary of the video to be detected, processing such as truncation, padding or looping can be performed to ensure that each frame image in the video to be detected corresponds to a video segment.

[0059] Step 130 : Input the subtitles of all the frame images in each video segment and the summary generation prompt for describing the time information into the large language model to generate a time summary corresponding to each video segment.

[0060] Specifically, since the description at the frame level lacks time information, the description at the frame level may not reflect the actual action or behavior and lacks the continuity of the action. Therefore, a large language model is used to generate a temporal summary. Take the following as an example to generate a time summary:

[0061] Video clips The captions of all the frames and the summary of the time information are input into the large language model, which generates prompts based on the summary to generate video clips. Corresponding time summary . Time Summary The expression is as follows:

[0062]

[0063] Where, Indicates a video clip The corresponding time summary, Represents a large language model, Indicates summary generation hint, Indicates a video clip Captions for all frames in .

[0064] For example, summary generation tips The question is “Please summarize what happened in the following scenario in a few sentences, focusing on the time information and not including any unnecessary details or descriptions.”

[0065] Step 140 , input the time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt, and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0066] Specifically, the video clip The corresponding time summary, abnormal activity list of the target scene, abnormality scoring prompts and output format requirements are input into the large language model, and the large language model is based on the video clips. The corresponding time summary is used to determine whether the abnormal activity in the abnormal activity list of the target scene is consistent. If it is consistent, the abnormal activity is scored according to the abnormal scoring prompt, and then the abnormal score is output according to the output format requirements. Frame Image The expression for the anomaly score is as follows:

[0067]

[0068] Where, Indicates the first Frame Image The anomaly score, Represents a large language model, Indicates abnormal score prompt, A list of abnormal activities representing the target scenario, Indicates output format requirement prompt, Indicates a video clip The corresponding time summary.

[0069] For example, anomaly scoring prompts For: "If you are a law enforcement agency, please score the anomaly of the described scene semantics based on the list of abnormal activities in the target scene. The score ranges from 0 to 1, where 0 indicates normal and 1 indicates abnormal."

[0070] The scene-dependent video anomaly detection method provided by the present invention analyzes general abnormal activity rules and abnormal activity rules of the target scene through a large language model, generates an abnormal activity list of the target scene, and improves adaptability to scenes dependent on anomaly detection; uniformly samples multiple frames with each frame image as the center to construct the video segment corresponding to each frame image, which not only retains the local temporal context but also avoids the high overhead of frame-by-frame calculation, and balances computational efficiency and information integrity; generates a temporal summary of the subtitles and description time information in the video segment through the large language model, explicitly models the event logic in the time dimension, and makes up for the implicit learning defect of the traditional visual model for temporal logic; the large language model outputs anomaly scores based on the abnormal activity list and time summary of the target scene, providing a traceable decision basis, which is superior to the unexplainable output of the black box model and realizes anomaly detection in scene-dependent situations.

[0071] In practical vision applications, each specific scenario, such as a hospital, supermarket, or different classrooms, laboratories, and playgrounds within a school, follows unique safety management protocols. These protocols implicitly define abnormal activities that are prohibited within these environments. Due to the diversity of scenarios, the types of abnormal events are almost infinite. Furthermore, the implicit knowledge acquired by LLMs during pre-training focuses primarily on general context and lacks a deep understanding of specific domain knowledge. Therefore, it is necessary to design a framework that can automatically generate anomaly rules based on safety management specifications, which can effectively guide LLMs in anomaly detection and judgment.

[0072] In some embodiments, a large language model is used to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of a target scenario to obtain an abnormal activity list of the target scenario, including:

[0073] inputting a first generation prompt for prompting generation of a general exception management rule into a large language model to obtain a general exception activity rule;

[0074] Inputting the security management specification of the target scenario and the second generation prompt for prompting the generation of the abnormal management rules of the target scenario into the large language model to obtain the abnormal activity rules of the target scenario;

[0075] The general abnormal activity rules, the abnormal activity rules of the target scenario, and a third generation prompt for prompting generation of an abnormal activity list of the target scenario are input into the large language model to obtain the abnormal activity list of the target scenario.

[0076] Specifically, first, using the implicit knowledge in the large language model, the first generation prompt for generating general exception management rules is input into the large language model. Based on the first generation prompt, the large language model outputs the general exception activity rule. The expression of the general exception activity rule is as follows:

[0077]

[0078] Where, Indicates a general abnormal activity rule, Represents a large language model, Indicates a first generation prompt for prompting generation of a general exception management rule.

[0079] For example, the first generation prompt The question is "Based on your general knowledge, list the rules for unusual human activities that are prohibited on the sidewalk. No explanation is required after listing the activities; just provide the rules for the prohibited activities."

[0080] The security management specifications for the target scenario and the second generation prompt for generating the exception management rules for the target scenario are input into the large language model. The large language model, guided by the second generation prompt, combines the security management specifications for the target scenario to generate the exception management rules for the target scenario. The expression for the exception management rules for the target scenario is as follows:

[0081]

[0082] Where, Indicates the abnormal activity rules of the target scenario, Represents a large language model, represents a second generation prompt for prompting the generation of anomaly management rules for a target scenario, Indicates the security management specifications for the target scenario.

[0083] For example, when the target scene is a sidewalk scene on campus, the second generation prompt To "list the rules for unusual human activities prohibited on campus sidewalks in accordance with the above campus safety management regulations. No explanation is required after listing the activities, just provide the prohibited activities directly."

[0084] After obtaining the general abnormal activity rules and the abnormal activity rules for the target scenario, the general abnormal activity rules, the abnormal activity rules for the target scenario, and a third generation prompt for generating an abnormal activity list for the target scenario are input into the large language model. Under the guidance of the third generation prompt, the large language model combines the general abnormal activity rules and the abnormal activity rules for the target scenario to generate an abnormal activity list for the target scenario. The expression for the abnormal activity list for the target scenario is as follows:

[0085]

[0086] Where, A list of abnormal activities representing the target scenario, Represents a large language model, Indicates a general abnormal activity rule, Indicates the abnormal activity rules of the target scene, Indicates a third generation prompt for prompting generation of an abnormal activity list for a target scenario.

[0087] Exemplary, third generation prompt "Provide a simplified list of unusual human activities prohibited on campus sidewalks based on the general unusual activity rules you listed earlier using general knowledge and the unusual activity rules on campus sidewalks listed based on the 'Campus Safety Management Code.' Avoid adding any explanations or numbers; simply separate the activity names with commas. Please provide your answer in this format."

[0088] The scenario-dependent video anomaly detection method provided by the present invention utilizes the powerful understanding and summarizing capabilities of a large language model to design multiple different generation prompts and automatically generate a list of abnormal activities according to the security management specifications of different scenarios.

[0089] In some embodiments, the scene-dependent video anomaly detection method provided by the present invention further includes:

[0090] Multiple BLIP2 models with different parameter sizes are used to generate candidate subtitles for each frame of the video to be detected, and a set of candidate subtitles is obtained;

[0091] The image encoder in the ImageBind model is used to encode each frame of the video to be detected to obtain the image features of each frame;

[0092] Use the text encoder in the ImageBind model to encode each candidate subtitle in the candidate subtitle set to obtain the subtitle features of each candidate subtitle;

[0093] Calculate the cosine similarity between the image features of each frame in the video to be detected and the subtitle features of each candidate subtitle in the candidate subtitle set, and use the candidate subtitle corresponding to the maximum cosine similarity as the subtitle of each frame in the video to be detected.

[0094] Specifically, multiple BLIP2 models with different parameter sizes are used to generate candidate subtitles for each frame of the video to be detected. Each frame can correspond to multiple candidate subtitles. The candidate subtitles corresponding to all frames of the video to be detected are combined into a candidate subtitle set.

[0095] The ImageBind model is a multimodal learning model that can process data from modalities such as images, text, and audio. The image encoder in the ImageBind model encodes each frame in the video to be detected, obtaining the image features of each frame. The text encoder in the ImageBind model encodes each candidate subtitle in the candidate subtitle set, obtaining the subtitle features of each candidate subtitle.

[0096] For any frame image in the video to be detected, calculate the cosine similarity between the image features of the frame image and the subtitle features of each candidate subtitle in the candidate subtitle set, and take the candidate subtitle corresponding to the maximum cosine similarity as the subtitle of the frame image. Frame Image The subtitle expression is as follows:

[0097]

[0098] Where, Indicates the first subtitles for frame images, An image encoder representing the ImageBind model, Indicates the Frame image, represents the text encoder of the ImageBind model, represents the mth candidate subtitle in the candidate subtitle set, represents a set of candidate subtitles, Represents cosine similarity.

[0099] The scene-dependent video anomaly detection method proposed in this paper generates candidate subtitles for each frame of the video being tested using multiple BLIP2 models with varying parameter sizes. The resulting set of candidate subtitles covers a more comprehensive semantic scene, reducing the likelihood of missed detections or incorrect descriptions. By aligning the image encoder and text encoder in a shared embedding space, cosine similarity calculation directly reflects the semantic match between the image and text, assigning the most semantically relevant subtitle to each frame.

[0100] In some embodiments, the scene-dependent video anomaly detection method provided by the present invention further includes:

[0101] Taking each video clip as the center, multiple video clips adjacent to it are regarded as a group of video clips;

[0102] The video encoder of the ImageBind model is used to encode the central video segment in each group of video segments to obtain the video features of the central video segment in each group of video segments;

[0103] The text encoder of the ImageBind model is used to encode the time summary corresponding to each video clip in each group of video clips, and the time summary features corresponding to each video clip in each group of video clips are obtained;

[0104] Based on the anomaly score of the frame image corresponding to each video clip in each group of video clips, the time summary features corresponding to each video clip in each group of video clips, and the video features of the central video clip in each group of video clips, the anomaly score of the frame image corresponding to the central video clip in each group of video clips is refined.

[0105] Specifically, the anomaly score of each frame in the video to be detected is refined based on a group of video clips. Take the center as an example to illustrate:

[0106] Video clips As the center, multiple video clips adjacent to it are grouped together to obtain a group of video clips. , a set of video clips Includes K video clips.

[0107] First, the video encoder of the ImageBind model is used to encode the central video segment (i.e., the video segment) in a group of video segments. ) to encode and obtain the video clip The text encoder of the ImageBind model is used to encode a set of video clips. Encode the time summary corresponding to each video clip in to obtain a set of video clips The temporal summary features corresponding to each video clip in .

[0108] Secondly, the video features of the central video clip are calculated and compared with a set of video clips The cosine similarity of the temporal summary features corresponding to each video clip in is calculated, and the cumulative value of all cosine similarities is calculated.

[0109] Then, calculate a set of video segments The proportion of the cosine similarity corresponding to each video clip in the cosine similarity accumulation value. The abnormality score of the frame image corresponding to each video clip in is multiplied by the proportion corresponding to the video clip.

[0110] Finally, a set of video clips The products corresponding to all video clips in are accumulated to obtain the video clips The refined anomaly score of the corresponding frame image is obtained, that is, the first Refined anomaly score of the frame image.

[0111] The video to be tested The expression of the refined anomaly score of a frame image is as follows:

[0112]

[0113] Where, Indicates the first Refined anomaly score of the frame image, Indicates video clip A set of video clips centered on Middle The anomaly score corresponding to each video clip, Indicates a video clip A corresponding set of video clips, Indicates the A video clip centered on a frame image. A video encoder representing the ImageBind model, represents the text encoder of the ImageBind model, Indicates video clip A set of video clips centered on Middle The temporal summary of each video segment.

[0114] The scene-dependent video anomaly detection method proposed in this paper captures local temporal context (previous and next video segments) by constructing segment groups based on a central video segment, overcoming the field of view limitations of traditional single-segment detection. A refined anomaly score is generated by integrating the three elements of the original anomaly score, the semantic encoding of the summary, and the visual encoding of the central video segment.

[0115] In some embodiments, the scene-dependent video anomaly detection method provided by the present invention further includes:

[0116] An adaptive motion smoothing strategy is used to smooth the refined anomaly score of each frame in the video to be detected, and a smoothed anomaly score of each frame in the video to be detected is obtained.

[0117] Specifically, when evaluating anomaly scores using large language models, the scores are primarily based on the current description information, often ignoring the temporal consistency and action continuity of the anomaly video frames. This may, to a certain extent, limit the comprehensiveness and accuracy of the evaluation.

[0118] In order to further improve the robustness of anomaly score judgment, an adaptive motion smoothing strategy is adopted to smooth the refined anomaly score of each frame image in the video to be detected, and obtain the smoothed anomaly score of each frame image in the video to be detected.

[0119] It should be noted that the adaptability is mainly reflected in the different window size settings. The window size is not fixed, but the size of the smoothing time window is automatically selected according to the length of the video.

[0120] In some embodiments, an adaptive motion smoothing strategy is used to smooth the refined anomaly score of each frame of the video to be detected, thereby obtaining a smoothed anomaly score for each frame of the video to be detected, including:

[0121] Multiply the number of frame images in the video to be detected by the proportional coefficient to obtain a product value;

[0122] The larger value of the product value and the minimum value of the sliding window is used as the size of the sliding window;

[0123] According to the size of the sliding window, the refined anomaly scores of all frame images in each sliding window centered on each frame image in the video to be detected are averaged to obtain the smoothed anomaly score of each frame image in the video to be detected.

[0124] Specifically, the number of frame images in the video to be detected is multiplied by the scale factor, the product is compared with the minimum value of the sliding window (for example, the minimum value is 3), and the larger value is taken as the size of the sliding window. The expression of the sliding window size is as follows:

[0125]

[0126] Where, represents the size of the sliding window, Indicates rounding down. Indicates the video to be detected. Indicates the number of frame images in the video to be detected, Represents the scale factor.

[0127] It should be noted that the proportionality coefficient It is obtained based on a large number of experimental verifications and is an empirical value. The minimum value is 3, and the maximum value is determined by the number of frame images in the video to be detected.

[0128] Each sliding window is centered on each frame image in the video to be detected, and the average value of the refined anomaly scores of all frame images in each sliding window is calculated to obtain the smoothed anomaly score of each frame image in the video to be detected. Frame Image The expression for the smoothed anomaly score is as follows:

[0129]

[0130] Where, Indicates the first Frame Image The smoothed anomaly score of Indicates the Frame Image The size of the sliding window centered on Indicates the Frame Image The sliding window centered at Refined anomaly score of the frame image.

[0131] The scene-dependent video anomaly detection method provided by the present invention further improves the robustness of anomaly scoring judgment through an adaptive motion smoothing strategy.

[0132] An experiment was conducted using the scene-dependent video anomaly detection method provided by the present invention. The specific experimental process is as follows.

[0133] (1) Dataset

[0134] In this study, three widely used video anomaly detection (VAD) datasets were used: the ShanghaiTech dataset (ShanghaiTech), the UCF-Crime dataset, and the XD-Violence dataset. The ShanghaiTech dataset contains 437 videos collected from multiple surveillance cameras on a university campus, recording 130 abnormal events across 13 different scenarios and embodying 17 anomaly categories. The dataset is divided into 238 training videos and 199 test videos. In this experiment, only the 199 test videos were used. The UCF-Crime dataset is a large-scale, real-world surveillance video dataset containing 1,900 uncropped videos, showcasing 13 real-world anomaly scenarios of high relevance to public safety. The training set includes 800 normal videos and 810 abnormal videos, while the test set includes 150 normal videos and 140 abnormal videos. The XD-Violence dataset is a large-scale dataset specifically designed for violence detection, containing 4,754 uncropped videos with audio signals and weak labels, of which 3,954 videos are used for training and 800 for testing. The dataset has a total length of 217 hours and covers a variety of scenarios and six anomaly categories.

[0135] (2) Experimental details

[0136] To improve computational efficiency, the video is sampled every 16 frames, following the configuration of the LAVAD method. BLIP-2 is used as the subtitle generation module, and Llama-2-13b-chat is used as the LLMs module. For multimodal encoding, the pre-trained multimodal encoder provided by ImageBind is used. Specifically, the time window is set to T = 10 seconds, which is consistent with the pre-training of the ImageBind video encoder. Ki is set to 10, that is, 10 video clips are grouped together. In addition, GPT-4 is used as LLMs for rule summarization. The adaptive window scale coefficient r is set to 0.15 for the motion smoothing strategy.

[0137] (3) Experimental results

[0138] Experiments were conducted using the UCF-Crime, XD-Violence, and ShanghaiTech datasets, and the proposed algorithm was compared with state-of-the-art untrained video anomaly detection methods. This is because the events in these datasets are highly identifiable, and the large language model has rich implicit knowledge about dangerous events in general scenarios.

[0139] The method provided by the present invention is compared with the latest weakly-supervised, unsupervised, and training-free video anomaly detection methods on the XD-Violence dataset. The comparison results are shown in Table 2. Compared with the latest untrained LAVAD method, the average precision (AP) is improved by +2.05%, and the area under the curve (AUC) increases by nearly one percentage point.

[0140] Table 2. Comparison of the proposed method with the latest weakly supervised, unsupervised, and untrained methods on the XD-Violence dataset

[0141]

[0142] The method provided by the present invention is compared with the latest weakly supervised, unsupervised, and untrained video anomaly detection methods on the UCF-Crime dataset. The comparison results are shown in Table 3. It can be observed that the method provided by the present invention performs better than the untrained methods, with an improvement of +2.18%.

[0143] Table 3 Comparison of the proposed method with the latest weakly supervised, unsupervised and untrained methods on the UCF-Crime dataset

[0144]

[0145] The method proposed in this paper was compared with the latest unsupervised and untrained video anomaly detection methods on the ShanghaiTech dataset. The results are shown in Table 4. Without the incorporation of rules and an adaptive motion smoothing strategy, the AUC was only 53.51%. This significant accuracy drop is attributed to the large model's lack of in-depth understanding of specific domain knowledge in certain scenarios. To address this, the present invention further introduced rules and a motion smoothing strategy, achieving significant improvements, increasing accuracy by nearly 13 percentage points.

[0146] Table 4. Comparison of the proposed method with the latest unsupervised and untrained methods on the ShanghaiTech dataset

[0147]

[0148] (4) Ablation experiment

[0149] Ablation experiments were conducted on the ShanghaiTech dataset to evaluate the impact of different strategies on the accuracy of video anomaly detection. The experiments explored the effects of introducing rules, applying the adaptive motion smoothing strategy, and applying both simultaneously. The ablation experiment results on the ShanghaiTech dataset are shown in Table 5, where "×" indicates no introduction and "√" indicates introduction. As shown in Table 5, adding rules alone improves the AUC by 6 percentage points, while applying the adaptive motion smoothing strategy alone improves the AUC by 4 percentage points. Notably, when both rules and the adaptive motion smoothing strategy are introduced, the AUC significantly improves to 66.43%. These results strongly verify the effectiveness of the proposed strategy in improving the accuracy of video anomaly detection.

[0150] Table 5 Ablation experiment results on ShanghaiTech dataset

[0151]

[0152] Figure 2 This is the anomaly score result graph provided by the present invention on the UCF-Crime dataset without using the motion smoothing strategy. Figure 3 This is the anomaly score result graph using the motion smoothing strategy provided by the present invention on the UCF-Crime dataset. The video name is Arson007_x264. The horizontal axis is the frame number and the vertical axis is the anomaly score. The closer the anomaly score is to 1, the more abnormal the frame is. The gray background highlights the time period of the abnormal event in the video, and the black lines represent the anomaly score of each frame. Figure 2 As shown in , in the gray background area, some abnormal frames have lower scores, while some non-abnormal frames have higher scores. Figure 3 As shown, by introducing a moving average smoothing strategy, this problem is significantly alleviated and the final score is better aligned with the temporal consistency and action continuity between video frames.

[0153] In this experiment, attention was paid to the limitations of large language models, especially their lack of domain-specific knowledge and the neglect of temporal consistency and action continuity in the scoring evaluation process. This paper proposes a video anomaly detection method that combines rule enhancement and perceptual smoothing. The method uses adaptive templates to automatically generate anomaly detection rules and introduces an adaptive temporal smoothing strategy to enhance the robustness of scoring judgment based on LLMs. Experimental results show that the proposed method not only surpasses the state-of-the-art untrained methods on general datasets such as UCF-Crime and XD-Violence, but also achieves significant improvements on the specific scene dataset ShanghaiTech. In future research, we will focus on optimizing visual language models to obtain more accurate video descriptions. This improvement aims to provide a more reliable description basis for large language models in the scoring evaluation process.

[0154] The scene-dependent video anomaly detection device provided by the present invention is described below. The scene-dependent video anomaly detection device described below and the scene-dependent video anomaly detection method described above can be referenced to each other.

[0155] Figure 4 Schematic diagram of the structure of the scene-dependent video anomaly detection device provided by the present invention, such as Figure 4 As shown, the present invention provides a scene-dependent video anomaly detection device, comprising:

[0156] A first acquisition module 410 is configured to use a large language model to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario to obtain an abnormal activity list of the target scenario; the target scenario is the scenario that anomaly detection relies on;

[0157] A construction module 420 is configured to uniformly sample multiple frames of images within a time window centered at each frame of the video to be detected, and construct a video segment corresponding to each frame of the image;

[0158] A generation module 430 is configured to input the subtitles of all the frame images in each of the video segments and a summary generation prompt for describing the time information into a large language model to generate a time summary corresponding to each of the video segments;

[0159] The second acquisition module 440 is used to input the time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0160] In some embodiments, the first acquisition module 410 is specifically configured to:

[0161] inputting a first generation prompt for prompting generation of a universal exception management rule into a large language model to obtain the universal abnormal activity rule;

[0162] Inputting the security management specification of the target scenario and a second generation prompt for prompting generation of an abnormality management rule for the target scenario into a large language model to obtain an abnormal activity rule for the target scenario;

[0163] The general abnormal activity rules, the abnormal activity rules of the target scenario, and a third generation prompt for prompting generation of an abnormal activity list of the target scenario are input into a large language model to obtain the abnormal activity list of the target scenario.

[0164] In some embodiments, the apparatus further comprises:

[0165] a combining module, configured to take each video segment as a center and combine a plurality of adjacent video segments before and after the video segment into a group of video segments;

[0166] A first encoding module is configured to encode a central video segment in each group of video segments using a video encoder of an ImageBind model to obtain video features of the central video segment in each group of video segments;

[0167] A second encoding module is configured to use a text encoder of the ImageBind model to encode the time summary corresponding to each video segment in each group of video segments, respectively, to obtain a time summary feature corresponding to each video segment in each group of video segments;

[0168] A refinement module is used to refine the anomaly score of the frame image corresponding to each video clip in each group of video clips, based on the anomaly score of the frame image corresponding to each video clip in each group of video clips, the time summary feature corresponding to each video clip in each group of video clips, and the video feature of the central video clip in each group of video clips.

[0169] In some embodiments, the apparatus further comprises:

[0170] The smoothing module is used to adopt an adaptive motion smoothing strategy to smooth the refined anomaly score of each frame image in the video to be detected, so as to obtain a smoothed anomaly score of each frame image in the video to be detected.

[0171] In some embodiments, the smoothing model is specifically used to:

[0172] Multiplying the number of frame images in the video to be detected by the proportional coefficient to obtain a product value;

[0173] The larger value between the product value and the minimum value of the sliding window is used as the size of the sliding window;

[0174] According to the size of the sliding window, the refined anomaly scores of all frame images in each sliding window centered on each frame image in the video to be detected are averaged to obtain a smoothed anomaly score for each frame image in the video to be detected.

[0175] In some embodiments, the apparatus further comprises:

[0176] A third acquisition module is configured to generate candidate subtitles for each frame of the video to be detected using a plurality of BLIP2 models with different parameter sizes to obtain a set of candidate subtitles;

[0177] A third encoding module is used to encode each frame of the video to be detected using an image encoder in the ImageBind model to obtain image features of each frame of the image;

[0178] a fourth encoding module, configured to encode each candidate subtitle in the candidate subtitle set using a text encoder in the ImageBind model to obtain a subtitle feature of each candidate subtitle;

[0179] The calculation module is used to calculate the cosine similarity between the image features of each frame image in the video to be detected and the subtitle features of each candidate subtitle in the candidate subtitle set, and use the candidate subtitle corresponding to the maximum cosine similarity as the subtitle of each frame image in the video to be detected.

[0180] It should be noted here that the above-mentioned scenario-dependent video anomaly detection device provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0181] Figure 5 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communications bus 540. The processor 510 may call logic instructions in the memory 530 to execute a scene-dependent video anomaly detection method, which includes: using a large language model to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of a target scene to obtain an abnormal activity list for the target scene; the target scene is a scene dependent on anomaly detection; uniformly sampling multiple frames within a time window centered on each frame of the video to be detected to construct a video segment corresponding to each frame; inputting the subtitles of all frames in each video segment and a summary generation prompt for describing time information into the large language model to generate a time summary corresponding to each video segment; and inputting the time summary corresponding to each video segment, the abnormal activity list of the target scene, an anomaly scoring prompt, and an output format requirement prompt into the large language model to obtain an anomaly score for each frame of the video to be detected.

[0182] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scene-dependent video anomaly detection method provided by the above-mentioned methods, the method including: using a large language model to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of the target scene to obtain an abnormal activity list of the target scene; the target scene is a scene that the anomaly detection depends on; uniformly sampling multiple frames of images in a time window with each frame image in the video to be detected as the time center to construct a video segment corresponding to each frame image; inputting the subtitles of all frame images in each of the video segments and the summary generation prompt for prompting the description of time information into the large language model to generate a time summary corresponding to each of the video segments; inputting the time summary corresponding to each of the video segments, the abnormal activity list of the target scene, the abnormality scoring prompt and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0184] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the scene-dependent video anomaly detection method provided by the above-mentioned methods, the method comprising: using a large language model to perform abnormal activity analysis on general abnormal activity rules and abnormal activity rules of the target scene to obtain an abnormal activity list of the target scene; the target scene is a scene that the anomaly detection depends on; uniformly sampling multiple frames of images within a time window with each frame image in the video to be detected as the time center to construct a video segment corresponding to each frame image; inputting the subtitles of all frame images in each of the video segments and a summary generation prompt for prompting the description of time information into the large language model to generate a time summary corresponding to each of the video segments; inputting the time summary corresponding to each of the video segments, the abnormal activity list of the target scene, the abnormality scoring prompt and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0186] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A scene-dependent video anomaly detection method, characterized in that: include: Use a large language model to analyze common abnormal activity rules and abnormal activity rules of the target scenario to obtain a list of abnormal activities in the target scenario; The target scenario is the scenario that anomaly detection relies on; Uniformly sampling multiple frames of images within a time window with each frame of the video to be detected as the time center, and constructing a video clip corresponding to each frame of the image; Inputting the subtitles of all frame images in each of the video clips and summary generation prompts for describing time information into a large language model to generate a time summary corresponding to each of the video clips; Inputting the time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt, and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected; The method further comprises: Adopting an adaptive motion smoothing strategy to smooth the refined anomaly score of each frame image in the video to be detected, thereby obtaining a smoothed anomaly score of each frame image in the video to be detected; The method adopts an adaptive motion smoothing strategy to smooth the refined anomaly score of each frame image in the video to be detected to obtain a smoothed anomaly score of each frame image in the video to be detected, including: Multiplying the number of frame images in the video to be detected by the proportional coefficient to obtain a product value; The larger value between the product value and the minimum value of the sliding window is used as the size of the sliding window; According to the size of the sliding window, the refined anomaly scores of all frame images in each sliding window centered on each frame image in the video to be detected are averaged to obtain a smoothed anomaly score for each frame image in the video to be detected.

2. The scene-dependent video anomaly detection method according to claim 1, characterized in that: The large language model is used to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario to obtain an abnormal activity list of the target scenario, including: inputting a first generation prompt for prompting generation of a universal exception management rule into a large language model to obtain the universal abnormal activity rule; Inputting the security management specification of the target scenario and a second generation prompt for prompting generation of an abnormality management rule for the target scenario into a large language model to obtain an abnormal activity rule for the target scenario; The general abnormal activity rules, the abnormal activity rules of the target scenario, and a third generation prompt for prompting generation of an abnormal activity list of the target scenario are input into a large language model to obtain the abnormal activity list of the target scenario.

3. The scene-dependent video anomaly detection method according to claim 1, characterized in that: The method further comprises: Taking each video segment as the center, the multiple video segments adjacent to it are grouped as a group of video segments; The video encoder of the ImageBind model is used to encode the central video segment in each group of video segments to obtain the video features of the central video segment in each group of video segments; Using the text encoder of the ImageBind model, respectively encode the time summary corresponding to each video segment in each group of video segments to obtain the time summary feature corresponding to each video segment in each group of video segments; Based on the anomaly score of the frame image corresponding to each video clip in each group of video clips, the time summary feature corresponding to each video clip in each group of video clips, and the video feature of the central video clip in each group of video clips, the anomaly score of the frame image corresponding to the central video clip in each group of video clips is refined.

4. The scene-dependent video anomaly detection method according to claim 1, characterized in that: The method further comprises: Using multiple BLIP2 models with different parameter sizes to generate candidate subtitles for each frame image in the video to be detected, to obtain a candidate subtitle set; Encode each frame of the video to be detected using the image encoder in the ImageBind model to obtain image features of each frame of the image; Encoding each candidate subtitle in the candidate subtitle set using a text encoder in the ImageBind model to obtain a subtitle feature of each candidate subtitle; Calculate the cosine similarity between the image features of each frame in the video to be detected and the subtitle features of each candidate subtitle in the candidate subtitle set, and use the candidate subtitle corresponding to the maximum cosine similarity as the subtitle of each frame in the video to be detected.

5. A scene-dependent video anomaly detection device, characterized in that: include: A first acquisition module is configured to use a large language model to perform abnormal activity analysis on the general abnormal activity rules and the abnormal activity rules of the target scenario to obtain an abnormal activity list of the target scenario; The target scenario is the scenario that anomaly detection relies on; A construction module is used to uniformly sample multiple frames of images within a time window with each frame of the video to be detected as the time center, and construct a video segment corresponding to each frame of the image; A generation module, configured to input the subtitles of all frame images in each of the video segments and summary generation prompts for describing time information into a large language model to generate a time summary corresponding to each of the video segments; The second acquisition module is configured to input the time summary corresponding to each video clip, the abnormal activity list of the target scene, the abnormality score prompt, and the output format requirement prompt into the large language model to obtain the abnormality score of each frame image in the video to be detected; The device further comprises: a smoothing module, configured to smooth the refined anomaly score of each frame of the video to be detected by using an adaptive motion smoothing strategy, so as to obtain a smoothed anomaly score of each frame of the video to be detected; The smoothing module is specifically used for: Multiplying the number of frame images in the video to be detected by the proportional coefficient to obtain a product value; The larger value between the product value and the minimum value of the sliding window is used as the size of the sliding window; According to the size of the sliding window, the refined anomaly scores of all frame images in each sliding window centered on each frame image in the video to be detected are averaged to obtain a smoothed anomaly score for each frame image in the video to be detected.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the scene-dependent video anomaly detection method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the scene-dependent video anomaly detection method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the scene-dependent video anomaly detection method according to any one of claims 1 to 4 is implemented.

Images