A Cross-Modal Anomaly Detection and Localization Method and System Based on Context and Dual Text

Abstract

This invention provides a cross-modal anomaly detection and localization method and system based on context and dual text. The method includes: extracting global semantic visual features, spatial fine-grained visual features, and fragment-level visual context information from the video frame to be detected; generating instance-level conditional cue vectors using the fragment-level visual context information as conditions; mapping and generating a first text prototype feature representing a normal state and a second text prototype feature representing an abnormal state; performing cross-modal comparison between the generated dual text prototype features and the extracted global semantic visual features and spatial fine-grained visual features, generating frame-level anomaly scores in parallel to complete anomaly detection, and generating pixel-level anomaly localization heatmaps to complete anomaly localization. This invention enables the generated normal and abnormal text prototype features to adaptively adjust with the context of the current video scene, overcoming the shortcomings of static text cue's lack of scene adaptability and susceptibility to environmental interference.

Description

Technical Field

[0001] This invention belongs to the field of anomaly detection technology, and specifically relates to a cross-modal anomaly detection and localization method and system based on context and dual text. Background Technology

[0002] Video anomaly detection is a fundamental task in intelligent video analytics, aiming to identify events that deviate from expected behavioral patterns in continuous video streams. Existing semi-supervised methods typically rely on training only on normal samples, identifying anomalies through reconstruction or prediction errors; weakly supervised methods, on the other hand, utilize video-level labels to achieve anomaly detection within a multi-instance learning framework. Despite these advancements, detection performance often degrades significantly when anomaly types not observed during training appear in the test data.

[0003] Visual-language models offer a novel technical approach for open-set anomaly detection by aligning visual features with natural language descriptions in a shared embedding space. Existing methods have attempted to incorporate visual-language models into anomaly detection tasks to improve the model's zero-shot transfer capability; however, current solutions still primarily rely on frame-level global features for discrimination, making it difficult to directly achieve pixel-level fine-grained localization.

[0004] Furthermore, existing prompt learning methods typically use static text templates, lacking the ability to adapt to different visual contexts and are easily affected by changes in background, viewpoint, and lighting conditions. Additionally, the method of constructing multi-classification boundaries around specific known anomaly categories weakens the model's ability to generalize to unknown anomaly categories. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-modal anomaly detection and localization method and system based on context and dual text, so as to solve the problems in the existing technology of open set anomaly detection where the decision boundary is easily bound to known anomaly categories, the text prompts lack scene adaptation ability, and it is difficult to achieve pixel-level localization.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a cross-modal anomaly detection and localization method based on context and dual text, comprising: Extract global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and aggregate them based on the global semantic visual features of each frame in the current video segment to generate segment-level visual context information; Instance-level conditional cue vectors are generated based on fragment-level visual context information. The conditional prompt vector is combined with the preset normal state text template and abnormal state text template respectively, and then input into the text encoder with frozen parameters to generate the first text prototype feature representing the normal state and the second text prototype feature representing the abnormal state. Based on the generated dual-text prototype features, cross-modal comparisons are performed with the extracted global semantic visual features and spatial fine-grained visual features. Frame-level anomaly scores are generated in parallel to complete anomaly detection, and pixel-level anomaly localization heatmaps are generated to complete anomaly localization.

[0007] Furthermore, the step of extracting global semantic visual features and spatial fine-grained visual features of the video frame to be detected, and aggregating and generating segment-level visual context information based on the global semantic visual features of each frame within the current video segment, includes: The system acquires the spatial fine-grained visual features, global semantic visual features, and corresponding segment-level visual context information of the video frame to be detected. The spatial fine-grained visual features are dense visual features that preserve spatial structure at the feature map resolution, used to support subsequent pixel-level localization. The global semantic visual features are compact feature vectors that characterize the overall state attributes of the image, used to support frame-level anomaly detection. The segment-level visual context information is the segment-level mean feature obtained by aggregating the global semantic visual features of each frame in the video segment along the time dimension, used to reflect the overall visual semantic state of the current video segment.

[0008] Furthermore, the step of generating instance-level conditional cue vectors based on fragment-level visual context information includes: Using fragment-level visual context information as a condition, the fragment-level visual context information is input into the meta-network to generate a context condition offset of the same dimension as the basic shared learnable cue. The context condition offset is then added element-wise to the basic shared learnable cue to generate an instance-level condition cue vector.

[0009] Furthermore, the specific calculation process of the conditional prompt vector is as follows:

[0010] in, Represents fragment-level visual context information. This indicates that basic shared learning prompts are available. Represents the meta-network. This represents an instance-level conditional hint vector.

[0011] Furthermore, the step of combining the conditional prompt vector with preset normal state text templates and abnormal state text templates, and inputting them into a text encoder with frozen parameters, to map and generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state, includes: The same conditional cue vector is combined with normal state text templates and abnormal state text templates respectively to obtain the corresponding complete cue sequence; then, the complete cue sequence is input into the text encoder with frozen parameters to generate text prototype features. ,in .

[0012] Furthermore, the generated dual-text prototype features are compared cross-modally with the extracted global semantic visual features and spatial fine-grained visual features to generate frame-level anomaly scores in parallel to complete anomaly detection, including: Before calculating the similarity difference, the global semantic visual features, spatial fine-grained visual features, first text prototype features, and second text prototype features are normalized, and the corresponding similarity is calculated using dot product operation; the global semantic visual features, spatial fine-grained visual features, first text prototype features, and second text prototype features are in the same embedding space. The calculation process for frame-level logic margin and frame-level anomaly score is as follows:

[0013]

[0014] in, and These represent the global semantic visual features and textual prototype features after L2 regularization, respectively. This represents the learnable global logical scaling factor. This represents the Sigmoid activation function.

[0015] Furthermore, the step of generating a pixel-level anomaly localization heatmap to complete anomaly localization includes: The calculation process for pixel-level logic margin and pixel-level anomaly location heatmap is as follows:

[0016]

[0017] pixel-level anomaly response map After upsampling to the target spatial resolution using bilinear interpolation, and then normalizing using the Sigmoid function, a pixel-level anomaly localization heatmap is obtained. ; in, This indicates the spatial fine-grained visual features after L2 regularization at the location level. The feature vector at that location, This represents the learnable local logical scaling factor. This represents the pixel logical temperature parameter.

[0018] Secondly, the present invention provides a cross-modal anomaly detection and localization system based on context and dual text, comprising: The feature extraction module is used to extract the global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and to aggregate the global semantic visual features of each frame in the current video segment along the time dimension to generate segment-level visual context information. The vector generation module is used to generate instance-level conditional cue vectors based on fragment-level visual context information. The mapping module is used to combine the conditional prompt vector with the preset normal state text template and abnormal state text template respectively, and input the text encoder with frozen parameters to generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state. The output module is used to perform cross-modal comparisons based on the generated dual-text prototype features with the extracted global semantic visual features and spatial fine-grained visual features, generate frame-level anomaly scores in parallel to complete anomaly detection, and generate pixel-level anomaly localization heatmaps to complete anomaly localization.

[0019] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the cross-modal anomaly detection and localization method based on context and dual text.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the cross-modal anomaly detection and localization method based on context and dual text.

[0021] Compared with the prior art, the present invention has the following technical effects: This invention uses fragment-level visual context information as a conditional input meta-network to dynamically generate instance-level conditional cue vectors. This allows the generated normal and abnormal text prototype features to adaptively adjust to the context of the current video scene, overcoming the shortcomings of static text prompts, such as lack of scene adaptation and susceptibility to environmental interference. By employing the pair of normal and abnormal text prototypes as a unified reference for cross-modal discrimination, the entire discrimination boundary is established on the generalized semantic difference between normal and abnormal, rather than being bound to any specific, known abnormal category description. This enhances the generalization ability to unknown abnormal types and solves the performance degradation problem caused by binding the decision boundary to known categories in open-set scenarios. Attached Figure Description

[0022] Figure 1This is a schematic diagram of the overall process of the cross-modal anomaly detection and localization method and system based on context and dual text provided in the embodiments of the present invention.

[0023] Figure 2 This is a partial schematic diagram illustrating the principle of condition prompt generation and text prototype mapping in an embodiment of the present invention.

[0024] Figure 3 This is a partial schematic diagram illustrating the principle of pixel-level logic margin calculation and abnormal heatmap generation in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings: Example 1, please refer to Figure 1 This invention provides a cross-modal anomaly detection and localization method based on context and dual text, including: S1, extract the global semantic visual features and spatial fine-grained visual features of the video frame to be detected, and aggregate the global semantic visual features of each frame in the current video segment to generate segment-level visual context information. Global semantic visual features are an abstraction and summary of the entire video frame's content. They encode the overall semantic information of the scene (such as environment type, main objects, and general behavior), serving as a macroscopic basis for judging whether the frame as a whole is abnormal. Spatial fine-grained visual features preserve the spatial structure of the feature map, essentially performing grid-based encoding on the image. The feature vector at each location corresponds to the local details of the original image. This feature allows the model to clearly see the visual patterns at each specific location in the image, which is key to achieving pixel-level anomaly localization. Segment-level visual context information is generated by aggregating global features from multiple consecutive frames within the current video segment. It is no longer an isolated snapshot of a single frame but carries dynamic and contextual information within a short time sequence.

[0026] S2 generates instance-level conditional cue vectors based on fragment-level visual context information. Fragment-level visual context information is input into a lightweight meta-network that learns the mapping from visual context to text cue space. The network outputs a contextual conditional offset with the same dimension as the base shared learnable cue. This offset essentially contains the direction and magnitude of fine-tuning the general text cue for the current specific video fragment. This offset is element-wise added to the base shared learnable cue to generate an instance-level conditional cue vector. The final text cue is no longer fixed but varies according to the specific visual content of each video fragment to be detected, allowing the subsequently generated text prototypes to flexibly adapt to different scenes, viewpoints, or lighting conditions, thus enhancing the model's robustness.

[0027] S3, combine the conditional prompt vector with the preset normal state text template and abnormal state text template respectively, and input them into the text encoder with frozen parameters to map and generate the first text prototype feature representing the normal state and the second text prototype feature representing the abnormal state. The same conditional cue vector generated by S2 is combined with pre-defined normal state text templates and abnormal state text templates, respectively. These two templates typically have the same sentence structure, differing only in their core state descriptors. Then, the two combined complete text sequences are input into a pre-trained text encoder (such as the CLIP text encoder) with its parameters frozen. The encoder maps the text to a high-dimensional semantic space, outputting a first text prototype feature (representing normal) and a second text prototype feature (representing abnormal). Freezing the encoder ensures that it utilizes the fundamental linguistic semantic knowledge obtained through large-scale pre-training; while the shared conditional cue guarantees that the pair of prototype features are generated under the same visual context.

[0028] S4, based on the generated dual-text prototype features, performs cross-modal comparison with the extracted global semantic visual features and spatial fine-grained visual features, generates frame-level anomaly scores in parallel to complete anomaly detection, and generates pixel-level anomaly localization heatmaps to complete anomaly localization.

[0029] The dual text prototype features generated by S3 are used as a unified, cross-modal discrimination standard, and their similarity is compared with the two visual features extracted by S1. In the frame-level anomaly detection pathway, cosine similarity is calculated between the global semantic visual features and the normal and abnormal text prototype features respectively. Then, the anomaly similarity is subtracted from the normal similarity to obtain a logical margin, which is then mapped to a frame-level anomaly score through an activation function. This score reflects the degree to which the overall content of the current frame deviates from normal and tends towards anomaly. In the pixel-level anomaly localization pathway, the feature vector of each spatial location (corresponding to a local region of the image) of the spatial fine-grained visual features is similarly compared with the two text prototypes, and the difference is calculated to obtain the initial anomaly response for each pixel location. This response map is adjusted by temperature parameters, upsampled to the original image resolution, and normalized to generate a pixel-level anomaly localization heatmap, visually marking the specific areas of suspected anomalies in the image. Both branches share the same pair of text prototypes as a reference, ensuring consistency in semantic standards between global judgment and local localization.

[0030] Example 2: This invention provides a cross-modal anomaly detection and localization method based on context and dual text, including: Step 1: Obtain the spatial fine-grained visual features, global semantic visual features, and corresponding segment-level visual context information of the video frame to be detected. The spatial fine-grained visual features are dense visual features that preserve spatial structure at the feature map resolution, used to support subsequent pixel-level localization; the global semantic visual features are compact feature vectors that characterize the overall state attributes of the image, used to support frame-level anomaly detection; the segment-level visual context information is used to reflect the overall visual semantic state of the current video segment.

[0031] Step 2: Predefine normal state text templates and abnormal state text templates, and set the normal state text templates and abnormal state text templates to use the same basic shared learnable prompts.

[0032] Step 3: Conditional cue generation. Using the fragment-level visual context information as a condition, the fragment-level visual context information is input into the meta-network to generate a contextual conditional offset of the same dimension as the basic shared learnable cue. The contextual conditional offset is then added element-wise to the basic shared learnable cue to generate an instance-level conditional cue vector.

[0033] Step 4: Text Prototype Mapping. The conditional cue vector is combined with the normal state text template and the abnormal state text template respectively, and input into the text encoder with frozen parameters to generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state.

[0034] Furthermore, neither the frame-level detection branch nor the pixel-level localization branch has a separate classification head for learning independent classification boundaries. Instead, they both use the first text prototype feature and the second text prototype feature as cross-modal discrimination references and complete anomaly discrimination based on their respective relative logical margins.

[0035] Step 5: Frame-level anomaly detection. In the frame-level detection branch, the frame-level logical margin is obtained based on the similarity difference between the global semantic visual features and the first text prototype features and the second text prototype features, and then the frame-level anomaly score is generated by mapping through an activation function.

[0036] Step 6: Pixel-level logical margin calculation. In the pixel-level positioning branch, the pixel-level logical margin corresponding to each spatial location is obtained based on the similarity difference between the spatial fine-grained visual features and the first text prototype features and the second text prototype features at each spatial location.

[0037] Step 7: Generating a pixel-level anomaly localization heatmap. Based on the pixel-level logical margin, a pixel-level anomaly localization heatmap for the video frame to be detected is generated through temperature scaling, bilinear interpolation upsampling, and activation function mapping.

[0038] Furthermore, the segment-level visual context information is the segment-level mean feature obtained by aggregating the global semantic visual features of each frame within the video segment along the time dimension.

[0039] Furthermore, the specific calculation process of the conditional prompt vector is as follows:

[0040] in, Represents fragment-level visual context information. This indicates that basic shared learning prompts are available. Represents the meta-network. This represents an instance-level conditional hint vector.

[0041] Furthermore, the generation process of the first text prototype feature and the second text prototype feature is as follows: The same conditional cue vector is combined with normal state text templates and abnormal state text templates respectively to obtain the corresponding complete cue sequence; then, the complete cue sequence is input into the text encoder with frozen parameters to generate text prototype features. ,in .

[0042] Furthermore, before calculating the similarity difference, the global semantic visual features, the spatial fine-grained visual features, and the first and second text prototype features are normalized, and the corresponding similarity is calculated using dot product operation. The global semantic visual features, the spatial fine-grained visual features, the first and second text prototype features are in the same embedding space; in some embodiments, the global semantic visual features and the spatial fine-grained visual features are linearly projected and then similarity calculated with the text prototype features on the same embedding dimension.

[0043] Furthermore, the calculation process for the frame-level logic margin and frame-level anomaly score is as follows:

[0044]

[0045] in, and These represent the global semantic visual features and textual prototype features after L2 regularization, respectively. This represents the learnable global logical scaling factor. This represents the Sigmoid activation function.

[0046] Furthermore, the calculation process for the pixel-level logic margin and the pixel-level anomaly location heatmap is as follows:

[0047]

[0048] pixel-level anomaly response map After upsampling to the target spatial resolution using bilinear interpolation, and then normalizing using the Sigmoid function, a pixel-level anomaly localization heatmap is obtained. .

[0049] in, This indicates the spatial fine-grained visual features after L2 regularization at the location level. The feature vector at that location, This represents the learnable local logical scaling factor. This represents the pixel logical temperature parameter.

[0050] Furthermore, the method also includes a model training phase; during the model training phase, the text encoder parameters are kept frozen, and the meta-network, basic shared learnable hints, global logical scaling factor, and local logical scaling factor are optimized. The loss function used for training, in addition to frame-level global detection loss and pixel-level local localization loss, also includes a text separation regularization constraint and a cross-granularity consistency constraint; wherein, the text separation regularization constraint calculates the dot product similarity between the first text prototype feature and the second text prototype feature after L2 regularization, and applies an interval penalty when the dot product similarity is greater than a preset interval threshold, thereby expanding the discrimination interval between the first text prototype feature and the second text prototype feature; the cross-granularity consistency constraint is constructed based on the difference between the frame-level anomaly score and the mean of the first k maximum values ​​of the pixel-level anomaly localization heatmap, and is used to constrain the output consistency of the frame-level detection branch and the pixel-level localization branch under the shared first and second text prototype feature discrimination reference.

[0051] in, A preset positive integer is used to select the element with the largest response in the pixel-level anomaly localization heatmap. Each position participates in the consistency constraint calculation.

[0052] By employing anomaly category binary merging, shared conditional cues, and normal / abnormal dual text prototypes during the training phase, the model establishes a discrimination boundary around the binary semantic relationship between normal and abnormal. A cue generation method based on fragment-level visual context information is adopted, enabling text prototype features to adjust according to changes in the visual context of the current video fragment. Logical margins based on normal / abnormal dual text prototypes are used for discrimination in both frame-level and pixel-level branches, achieving frame-level anomaly detection and pixel-level anomaly localization.

[0053] Example 3: This invention provides a cross-modal anomaly detection and localization method based on context and dual text, including: Step 1: Obtain Feature Representation. Receive pre-extracted spatial fine-grained visual features of the current video frame to be detected. Global semantic visual features and the corresponding fragment-level visual context information .in, and This represents the spatial resolution of the feature map. For the channel dimension of local features, For global feature dimensions. Spatial fine-grained visual features retain the spatial structure information of the original frame, for each spatial location. eigenvectors at location Encodes the local visual semantics at that location. Fragment-level visual context information. This is the average of the global semantic visual features of each frame within a video segment along the time dimension, used to provide a stable scene-level semantic overview. The spatial fine-grained visual features, global semantic visual features, and segment-level visual context information can be obtained from the upstream visual feature extraction module.

[0054] Step 2: Predefine normal state text templates and abnormal state text templates, and set the normal state text templates and abnormal state text templates to use the same basic shared learnable prompts.

[0055] During the model training phase, all abnormal samples in the training set are uniformly mapped to abnormal classes, which together with the normal classes form a binary label space. For all videos in the training set, normal videos are labeled as... All abnormal videos, regardless of their specific abnormality category, are uniformly labeled as... This binary merging allows the model to learn around the concept of "whether it is abnormal" and establishes a normal / abnormal binary decision boundary accordingly.

[0056] Combination Figure 2 Step 3: Conditional prompt generation. Define basic shared learnable prompts. This is a set of continuous vectors with the same dimensions as the word embeddings in the text encoder, which are continuously optimized through backpropagation during training. This incorporates fragment-level visual context information. Input Metanetwork Output context condition offset Conditional hint vector The calculation formula is:

[0057] in, It consists of basic shared learnable cues and conditional offsets generated based on fragment-level visual context information.

[0058] Step 4: Text prototype mapping. Map the same conditional prompt vector... The text prompts are combined with normal state text templates and abnormal state text templates respectively to obtain corresponding complete prompt sequences; then, the complete prompt sequences are input into the text encoder with frozen parameters to generate text prototype features. ,in .

[0059] Step 5: Frame-level anomaly detection. This involves analyzing global semantic visual features. and text prototype features Perform L2 regularization separately to obtain And introduce a learnable global logical scaling factor. Calculate frame-level logic margin:

[0060]

[0061] in, This represents the dot product operation. This represents the Sigmoid activation function. The confidence level that the current frame is abnormal.

[0062] Step 6: Calculate pixel-level logical margin. For fine-grained spatial visual features... and text prototype features L2 regularization is performed separately, and a learnable local logical scaling factor is introduced. Calculate pixel-level logical margins at each spatial location:

[0063]

[0064] in, These are the preset pixel logic temperature parameters.

[0065] Step 7: Generate an anomaly location heatmap. (This involves using the logic margin matrix.) Upsampled to the original frame resolution using bilinear interpolation. The pixel-level anomaly localization heatmap is obtained by mapping with the Sigmoid activation function. .

[0066] Furthermore, the model training phase of this invention will be described. During training, the text encoder parameters are kept frozen, and the parameters to be optimized include: basic shared learnable hints. Meta-network Network parameters, global logical scaling factor and local logical scaling factor The loss function used for training includes the following components: frame-level global detection loss, which employs a binary cross-entropy loss function applied to frame-level anomaly scores. Pixel-level local localization loss includes pixel-level binary cross-entropy classification loss, Dice loss, and ranking constraint loss.

[0067] The sorting constraint loss is used to constrain the pixel scores in abnormal regions to be higher than those in normal regions. Specifically, it is constructed by introducing an interval penalty on the score difference between the abnormal pixel set and the normal pixel set.

[0068] The text separation regularization constraint can be expressed as:

[0069] in, and Let represent the first and second text prototype features after L2 regularization, respectively. This represents a preset interval threshold. When the dot product similarity between the two features exceeds the interval threshold, the text separation regularization constraint term generates a positive penalty to increase the discriminative interval between the first text prototype feature and the second text prototype feature.

[0070] The cross-granularity consistency constraint is constructed based on the difference between the frame-level anomaly score output by the frame-level detection branch and the mean of the first k maximum values ​​of the pixel-level anomaly localization heatmap output by the pixel-level localization branch. It is used to constrain the output consistency of the two branches when they share the first text prototype feature and the second text prototype feature as discrimination references.

[0071] In summary, the embodiments of the present invention achieve frame-level anomaly detection and pixel-level anomaly localization by constructing normal / abnormal dual-text prototypes driven by shared condition prompts and by uniformly employing logical margin discrimination in frame-level and pixel-level branches.

[0072] This invention addresses the problems of existing anomaly detection models' inability to generalize to unknown anomaly categories and their lack of pixel-level localization capabilities. It employs a binary soft cue strategy, mapping diverse anomaly categories to a normal / abnormal binary semantic space during the training phase. After acquiring the spatial fine-grained visual features, global semantic visual features, and fragment-level visual context information formed by aggregating global semantic visual features from consecutive frames along the time dimension, a contextual conditional offset of the same dimension as the basic shared learnable cue is generated through a meta-network using the fragment-level visual context information as a condition. This offset is then added element-wise to the basic shared learnable cue to generate an instance-level conditional cue vector. Subsequently, this conditional cue vector is combined with normal and abnormal state text templates, respectively, and mapped through a parameter-frozen text encoder to generate text prototype features representing normal and abnormal states. In the frame-level detection branch, a frame-level anomaly score is obtained based on the relative logical margin between the global semantic visual features and the two types of text prototypes. In the pixel-level localization branch, a pixel-level anomaly localization heatmap is generated based on the spatial fine-grained visual features and the relative logical margin between the two types of text prototypes at various spatial locations, after temperature scaling and spatial upsampling. This invention uses a shared conditional prompt-driven normal / abnormal dual text prototype as a unified cross-modal discrimination reference to achieve consistent discrimination of frame-level detection and pixel-level localization without setting an independent classification head, and improves the generalization ability of abnormal semantics.

[0073] In another embodiment of the present invention, a cross-modal anomaly detection and localization system based on context and dual text is provided, which can be used to implement the above-mentioned cross-modal anomaly detection and localization method based on context and dual text. Specifically, the system includes: The feature extraction module is used to extract the global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and to aggregate the global semantic visual features of each frame in the current video segment along the time dimension to generate segment-level visual context information. The vector generation module is used to generate instance-level conditional cue vectors based on fragment-level visual context information. The mapping module is used to combine the conditional prompt vector with the preset normal state text template and abnormal state text template respectively, and input the text encoder with frozen parameters to generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state. The output module is used to perform cross-modal comparisons based on the generated dual-text prototype features with the extracted global semantic visual features and spatial fine-grained visual features, generate frame-level anomaly scores in parallel to complete anomaly detection, and generate pixel-level anomaly localization heatmaps to complete anomaly localization.

[0074] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0075] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a cross-modal anomaly detection and localization method based on context and dual text.

[0076] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the context- and dual-text-based cross-modal anomaly detection and localization method in the above embodiments.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A cross-modal anomaly detection and localization method based on context and dual text, characterized in that, include: Extract global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and aggregate them based on the global semantic visual features of each frame in the current video segment to generate segment-level visual context information; Instance-level conditional cue vectors are generated based on fragment-level visual context information. The conditional prompt vector is combined with the preset normal state text template and abnormal state text template respectively, and then input into the text encoder with frozen parameters to generate the first text prototype feature representing the normal state and the second text prototype feature representing the abnormal state. Based on the generated dual-text prototype features, cross-modal comparisons are performed with the extracted global semantic visual features and spatial fine-grained visual features. Frame-level anomaly scores are generated in parallel to complete anomaly detection, and pixel-level anomaly localization heatmaps are generated to complete anomaly localization.

2. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, The step of extracting global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and aggregating and generating segment-level visual context information based on the global semantic visual features of each frame within the current video segment, includes: The system acquires the spatial fine-grained visual features and global semantic visual features of the video frames to be detected, and aggregates the global semantic visual features of each frame at the current video segment level along the time dimension to generate corresponding segment-level visual context information. The spatial fine-grained visual features are dense visual features that preserve spatial structure at the feature map resolution, used to support subsequent pixel-level localization. The global semantic visual features are compact feature vectors that characterize the overall state attributes of the image, used to support frame-level anomaly detection. The segment-level visual context information is the segment-level mean feature obtained by aggregating the global semantic visual features of each frame within the video segment along the time dimension, used to reflect the overall visual semantic state of the current video segment.

3. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, The step of generating instance-level conditional cue vectors based on fragment-level visual context information includes: Using fragment-level visual context information as a condition, the fragment-level visual context information is input into the meta-network to generate a context condition offset of the same dimension as the basic shared learnable cue. The context condition offset is then added element-wise to the basic shared learnable cue to generate an instance-level condition cue vector. The specific calculation process for the conditional prompt vector is as follows: in, Represents fragment-level visual context information. This indicates that basic shared learning prompts are available. Represents the meta-network. This represents an instance-level conditional hint vector.

4. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, The step of combining the conditional cue vector with preset normal state text templates and abnormal state text templates, and inputting them into a text encoder with frozen parameters, to map and generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state, includes: The same conditional cue vector is combined with normal state text templates and abnormal state text templates respectively to obtain the corresponding complete cue sequence; then, the complete cue sequence is input into the text encoder with frozen parameters to generate text prototype features. ,in .

5. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, Neither the frame-level detection branch nor the pixel-level localization branch has a separate classification head for learning independent classification boundaries. Instead, they both use the first text prototype feature and the second text prototype feature as cross-modal discrimination references and complete anomaly discrimination based on their respective logical margins.

6. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, The generated dual-text prototype features are compared across modalities with extracted global semantic visual features and spatial fine-grained visual features to generate frame-level anomaly scores in parallel to complete anomaly detection, including: Before calculating the similarity difference, the global semantic visual features, spatial fine-grained visual features, first text prototype features, and second text prototype features are normalized, and the corresponding similarity is calculated using dot product operation; the global semantic visual features, spatial fine-grained visual features, first text prototype features, and second text prototype features are in the same embedding space. The calculation process for frame-level logic margin and frame-level anomaly score is as follows: in, and These represent the global semantic visual features and textual prototype features after L2 regularization, respectively. This represents the learnable global logical scaling factor. This represents the Sigmoid activation function.

7. The cross-modal anomaly detection and localization method based on context and dual text as described in claim 1, characterized in that, The process of generating a pixel-level anomaly localization heatmap to complete anomaly localization includes: The calculation process for pixel-level logic margin and pixel-level anomaly location heatmap is as follows: pixel-level anomaly response map After upsampling to the target spatial resolution using bilinear interpolation, and then normalizing using the Sigmoid function, a pixel-level anomaly localization heatmap is obtained. ; in, This indicates the spatial fine-grained visual features after L2 regularization at the location level. The feature vector at that location, This represents the learnable local logical scaling factor. This represents the pixel logical temperature parameter.

8. A cross-modal anomaly detection and localization system based on context and dual text, characterized in that, include: The feature extraction module is used to extract the global semantic visual features and spatial fine-grained visual features of the video frames to be detected, and to aggregate the global semantic visual features of each frame in the current video segment along the time dimension to generate segment-level visual context information. The vector generation module is used to generate instance-level conditional cue vectors based on fragment-level visual context information. The mapping module is used to combine the conditional prompt vector with the preset normal state text template and abnormal state text template respectively, and input the text encoder with frozen parameters to generate a first text prototype feature representing the normal state and a second text prototype feature representing the abnormal state. The output module is used to perform cross-modal comparisons based on the generated dual-text prototype features with the extracted global semantic visual features and spatial fine-grained visual features, generate frame-level anomaly scores in parallel to complete anomaly detection, and generate pixel-level anomaly localization heatmaps to complete anomaly localization.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the cross-modal anomaly detection and localization method based on context and dual text as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the cross-modal anomaly detection and localization method based on context and dual text as described in any one of claims 1 to 7.

Images