Anomaly detection methods, electronic devices, storage media, and application products based on multimodal and multi-level features.

Abstract

This disclosure provides an anomaly detection method, electronic device, storage medium, and program product based on multimodal, multi-level features. The method includes: acquiring an image to be detected; scaling and cropping the image to obtain a globally scaled image patch and multiple local image patches; extracting features from the globally scaled image patch to obtain global visual features; extracting multi-level features from each of the multiple local image patches to obtain multi-level visual features for each local image patch; processing the global visual features using a cross-attention mechanism to obtain a global descriptive vector; determining local anomaly information for each local image patch based on its multi-level visual features and the global descriptive vector; and determining the anomaly detection result based on the local anomaly information from multiple local image patches. This method enables local anomaly judgment guided by global semantics, achieving precise localization of fine-grained anomalies.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer vision, and more specifically, to an anomaly detection method, electronic device, storage medium, and program product based on multimodal, multi-level features. Background Technology

[0002] Personnel safety monitoring technology plays a crucial role in industrial production, public safety, and other fields. Its core function is to identify abnormal personnel behavior through technological means to minimize the risk of safety accidents. With the rapid development of computer vision and deep learning technologies, this field has gradually evolved from traditional manual monitoring to intelligent and automated monitoring.

[0003] Currently, the mainstream personnel safety monitoring methods mainly include: (1) deep learning methods, which use convolutional neural networks or Transformer models to extract image features and train models such as autoencoders and generative adversarial networks to distinguish between normal and abnormal modes. However, their performance is highly dependent on large-scale labeled data and their generalization ability to unseen abnormal types is limited; (2) multimodal fusion methods, some studies have introduced audio, text and other modalities to assist visual detection, especially methods based on pre-trained visual language models. With the help of cross-modal semantic alignment capabilities, abnormal identification is guided by text prompts, which reduces the dependence on labeled data to a certain extent. However, relying only on a single level of visual features, there is a problem of insufficient localization accuracy for fine-grained abnormalities. Summary of the Invention

[0004] This disclosure provides an anomaly detection method based on multimodal and multi-level features to solve at least one of the above-mentioned problems.

[0005] According to a first aspect of the present disclosure, an anomaly detection method based on multimodal, multi-level features is provided, comprising: acquiring an image to be detected; scaling and cropping the image to be detected to obtain a globally scaled image patch and multiple local image patches; performing feature extraction on the globally scaled image patch to obtain global visual features; performing multi-level feature extraction on the multiple local image patches respectively to obtain multi-level visual features for each local image patch; processing the global visual features using a cross-attention mechanism to obtain a global description vector, wherein the global description vector is used to describe normal events and / or abnormal events; for each local image patch, determining local anomaly information of the local image patch based on the multi-level visual features of the local image patch and the global description vector; and determining an anomaly detection result based on the local anomaly information of the multiple local image patches.

[0006] Optionally, determining the local anomaly information of each local image patch based on its multi-level visual features and the global description vector includes: determining the anomaly value of the local image patch at each level based on the visual features at that level and the global description vector; and aggregating the anomaly values ​​of the local image patch at each level to obtain the local anomaly information of the local image patch.

[0007] Optionally, the step of processing the global visual features using the cross-attention mechanism to obtain a global description vector includes: processing the global visual features, normal mode learnable embedding vector, and abnormal mode learnable embedding vector using the cross-attention mechanism to obtain a global description vector including a normal global description vector and an abnormal global description vector, wherein the normal mode learnable embedding vector and the abnormal mode learnable embedding vector are obtained through training.

[0008] Optionally, determining the outlier value of the local image patch at the level based on the hierarchical visual features and the global descriptive vector includes: determining the similarity between the hierarchical visual features and the normal global descriptive vector as normal similarity, and determining the similarity between the hierarchical visual features and the abnormal global descriptive vector as abnormal similarity; and determining the outlier value of the local image patch at the level based on the normal similarity and the abnormal similarity.

[0009] Optionally, determining the anomaly detection result of the image to be detected based on the local anomaly information of each of the plurality of local image patches includes: acquiring normal mode text prompts and abnormal mode text prompts, and extracting normal prompt features and abnormal prompt features from them respectively, wherein the normal mode text prompts and abnormal mode text prompts are obtained through training; determining global anomaly information based on the normal prompt features, the abnormal prompt features and the global visual features; and determining the anomaly detection result based on the global anomaly information and the local anomaly information of the plurality of local image patches.

[0010] Optionally, determining global anomaly information based on the normal prompt features, the abnormal prompt features, and the global visual features includes: determining global normal weights and global anomaly weights based on the normal prompt features, the abnormal prompt features, and the global visual features; determining local anomaly weights based on the local anomaly information of the multiple local image patches; performing weighted fusion of the normal prompt features and the abnormal prompt features based on the global normal weights, the global anomaly weights, and the local anomaly weights to obtain fused prompt features; using the fused prompt features to modulate the global visual features to obtain modulated global visual features; and determining the global anomaly information based on the modulated global visual features, the normal prompt features, and the abnormal prompt features.

[0011] Optionally, determining the global anomaly information based on the modulation global visual features, the normal prompt features, and the anomaly prompt features includes: acquiring multiple historical anomaly attributes and extracting multiple historical anomaly attribute features from them; determining a preset number of historical anomaly attribute features as reference anomaly attribute features based on the similarity between each of the multiple historical anomaly attribute features and the modulation global visual features; enhancing the anomaly prompt features based on the reference anomaly attribute features to obtain enhanced anomaly prompt features; and determining the global anomaly information based on the modulation global visual features, the normal prompt features, and the enhanced anomaly prompt features.

[0012] Optionally, determining the anomaly detection result based on the global anomaly information and the local anomaly information of the plurality of local image blocks includes: performing weighted fusion processing on the global anomaly information and the local anomaly information of the plurality of local image blocks to obtain fused anomaly information; performing bilinear interpolation processing on the fused anomaly information to obtain a full-resolution anomaly heatmap, wherein the full-resolution anomaly heatmap includes the anomaly heatmap values ​​of multiple pixels in the image to be detected; and determining the regions in the full-resolution anomaly heatmap where the anomaly heatmap values ​​are greater than a heatmap threshold as anomaly regions to obtain the anomaly detection result.

[0013] According to a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform an anomaly detection method based on multimodal, multi-level features according to exemplary embodiments of the present disclosure.

[0014] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by at least one processor, cause at least one processor to perform an anomaly detection method based on multimodal, multi-level features according to exemplary embodiments of the present disclosure.

[0015] According to a fourth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, cause at least one processor to perform an anomaly detection method based on multimodal, multi-level features according to exemplary embodiments of the present disclosure.

[0016] The technical solutions provided by the embodiments of this disclosure offer at least the following beneficial effects: According to the anomaly detection method, electronic device, storage medium, and program product based on multimodal, multi-level features disclosed herein, by processing the image to be detected into a globally scaled image patch and multiple local image patches, and extracting global visual features from the former and simultaneously extracting multiple visual features at different levels for each local image patch, both global semantics and local details can be considered, improving fine-grained anomaly detection capabilities. Furthermore, by employing two-stage feature matching—first generating a global description vector based on global visual features, and then matching the multi-level visual features of each local image patch with the global description vector to obtain the local anomaly information of the corresponding local image patch—local anomaly judgment guided by global semantics can be achieved, reducing the possibility of misjudgment due to the separation of global and local information, and enabling accurate localization of fine-grained anomalies.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 This is a flowchart of an anomaly detection method based on multimodal, multi-level features according to exemplary embodiments of the present disclosure.

[0020] Figure 2 This is a logical schematic diagram of extracting multi-level visual features according to exemplary embodiments of the present disclosure.

[0021] Figure 3 This is a logical schematic diagram illustrating the determination of local anomaly information according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 This is a logical diagram illustrating the determination of global anomaly information according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0025] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0027] Hereinafter, with reference to the accompanying drawings, an anomaly detection method, electronic device, storage medium, and program product based on multimodal and multi-level features according to exemplary embodiments of the present disclosure will be described in detail.

[0028] Figure 1 This is a flowchart of an anomaly detection method based on multimodal, multi-level features according to exemplary embodiments of the present disclosure. The method can be executed on an electronic device with sufficient computing power.

[0029] Reference Figure 1 In step S101, the image to be detected is acquired.

[0030] The image to be detected can be a high-resolution image, specifically an image with a pixel count of a certain threshold or higher. This specific threshold can be selected according to actual needs. High-resolution images include, but are not limited to, images of 1080P or higher. The image to be detected can be a single image or a video frame from a video. This disclosure does not impose any restrictions on this.

[0031] In step S102, the image to be detected is scaled and locally cropped to obtain a globally scaled image block and multiple local image blocks.

[0032] As an example, the LLaVA-NeXT strategy can be used to segment the image to be detected into a globally scaled image patch and multiple local image patches. Specifically, the AnyRes strategy introduced by the LLaVA-NeXT strategy can be used to segment the image to be detected into multiple sub-regions, i.e., multiple local image patches.

[0033] .

[0034] In step S103, feature extraction processing is performed on the globally scaled image patch to obtain global visual features.

[0035] In step S104, multi-level feature extraction is performed on multiple local image blocks to obtain multi-level visual features for each local image block.

[0036] Step S104 extracts multiple visual features at different levels from each local image patch. This extraction can be achieved, for example, through cascaded feature extraction layers in a visual encoder. Earlier feature extraction layers can extract low-level local features from the image patch and input them into the next layer to extract more complex, higher-level features. These visual features at various levels collectively constitute multi-level visual features. The specific number of extracted levels can be adjusted as needed, and this disclosure does not impose any limitations on this. Furthermore, these visual features at various levels are intermediate features extracted by the visual encoder. These intermediate features undergo further processing, such as post-processing including but not limited to normalization, pooling, and projection, to obtain a more compact feature representation. The global visual features extracted in step S103 are this more compact feature representation. The exemplary embodiments of this disclosure, by simultaneously extracting global visual features from a globally scaled image patch and multi-level visual features from each local image patch, can balance global semantics with local details, improving fine-grained anomaly detection capabilities.

[0037] As an example, a CLIP-based pre-trained ViT model can be used as the visual encoder, and step S104 can be executed. As an example, such as... Figure 2As shown, the visual encoder has four different Transformer layers (represented by layer1, layer2, layer3, and layer4 respectively), which can be obtained from each local image patch ( Figure 2 The example shown (using one globally scaled image patch and three local image patches) extracts four levels of visual features, which can be represented as:

[0038]

[0039] In the above formula, Indicates the hierarchical sequence number. Indicates hierarchy Visual features.

[0040] As an example, the multi-level visual features of the extracted local image patches can be further concatenated to form a feature vector (i.e., Figure 2 The token features in the image (the multi-level visual features of each local image patch after stitching together) can be represented as:

[0041]

[0042] In step S105, a cross-attention mechanism is used to process global visual features to obtain a global description vector.

[0043] Global description vectors are used to describe normal events and / or abnormal events. That is, they can include only vectors describing normal events, only vectors describing abnormal events, or both. These vectors can all guide the distinction between normal and abnormal events from a global perspective.

[0044] In step S106, for each local image block, local anomaly information of the local image block is determined based on the multi-level visual features of the local image block and the global description vector.

[0045] By employing a two-stage feature matching approach—that is, first generating a global description vector and then obtaining local anomaly information through local feature matching (i.e., multi-level visual features of local image patches)—it is possible to achieve local anomaly judgment guided by global semantics, reduce the possibility of misjudgment caused by the separation of global and local information, and achieve accurate localization of fine-grained anomalies.

[0046] In step S107, the anomaly detection result is determined based on the local anomaly information of multiple local image blocks.

[0047] This step summarizes and processes the local anomaly information of each local image block to obtain the overall anomaly detection result of the video.

[0048] The following section provides a further description of an anomaly detection method based on multimodal and multi-level features according to exemplary embodiments of the present disclosure.

[0049] Regarding step S105, in some embodiments, optionally, this step can determine the local anomaly information of the local image block based on the multi-level visual features of each local image block after stitching together and the global description vector.

[0050] In some other embodiments, step S106 may optionally include: for each level of visual features of each local image patch, determining the outlier value of the local image patch at that level based on the level of visual features and the global descriptive vector; and aggregating the outlier values ​​of the local image patch at each level to obtain the local anomaly information of the local image patch. By determining the outlier value for each level of visual features separately, anomaly detection at different levels can be performed in a targeted manner, improving the interpretability of the solution. As an example, the similarity between each level of visual features and the global descriptive vector can be calculated, and then the outlier value can be obtained accordingly. It should be understood that since the global descriptive vector is used to describe normal events and / or abnormal events, the similarity between the hierarchical visual features and the global descriptive vector also includes normal similarity components related to normal events and / or abnormal similarity components related to abnormal events. Furthermore, since outliers are estimates of the probability of anomalies at that level, they should be negatively correlated with normal similarity components and positively correlated with abnormal similarity components. Based on this, a calculation method for obtaining outliers from similarity can be constructed. This calculation method can be constructed as needed, as long as it meets the detection requirements. This disclosure does not impose any specific restrictions on it.

[0051] As an example, the above aggregation process is a hierarchical weighted aggregation, which can be represented as:

[0052]

[0053] In the above formula, This indicates local anomaly information for the current local image patch. This indicates the weight of the corresponding level; the specific rules for setting the weights are not specified here. This indicates an outlier at the corresponding level of the current local image patch.

[0054] Regarding step S105, optionally, this step includes: processing global visual features and learnable embedding vectors for normal and abnormal modes using a cross-attention mechanism to obtain a global description vector including normal and abnormal global description vectors, wherein the learnable embedding vectors for normal and abnormal modes are obtained through training. By introducing the pre-trained learnable embedding vectors for normal and abnormal modes, these can be used as references to generate normal and abnormal global description vectors for global visual features using the cross-attention mechanism, serving as the global description vector. In other words, the global description vector includes both vectors describing normal events and vectors describing abnormal events. As an example, the dimension of the learnable embedding vector is consistent with the CLIP text embedding space.

[0055] As an example, the use of the cross-attention mechanism can be represented as:

[0056]

[0057] In the above formula, Q(query) = [ ], and Let K (key) and V (value) represent the learnable embedding vectors for abnormal and normal modes, respectively. K and V are both global visual features, and T is the matrix transpose. The dimension of the key.

[0058] As an example, the anomaly detection method of the exemplary embodiments of this disclosure can be implemented based on an anomaly detection model. This model may include, for example, an image patch segmentation network performing step S102, a visual encoder performing steps S103 and S104, and a cross-attention network performing step S105. The anomaly detection phase using this model belongs to the inference phase of the model. Prior to this, there is a training phase for adjusting the model's parameters. For example, this includes, but is not limited to, using supervised training methods to obtain sample videos with text labels. These text labels describe the normal or abnormal state of the sample video. During training, the model can process the sample videos, output anomaly detection results, and then update the model parameters through backpropagation based on the similarity loss between the output results and the text labels of the sample videos. Here, the learnable embedding vectors for normal and anomaly modes can be used as parameters of the model and participate in the model training.

[0059] Based on the embodiments where the global descriptor vector includes the aforementioned normal global descriptor vector and abnormal global descriptor vector, in some further optional embodiments of step S105, the operation of determining the outlier value of the local image patch at that level based on the hierarchical visual features and the global descriptor vector includes: determining the similarity between the hierarchical visual features and the normal global descriptor vector as normal similarity, and determining the similarity between the hierarchical visual features and the abnormal global descriptor vector as abnormal similarity; determining the outlier value of the local image patch at that level based on the normal similarity and the abnormal similarity. That is, the normal similarity and the abnormal similarity are determined separately, and then the two are combined to obtain the outlier value. It should be understood that, as mentioned above, outlier values ​​are negatively correlated with normal similarity and positively correlated with abnormal similarity.

[0060] As an example, the global description vector includes global description vectors at multiple levels, and the global description vector at each level can be represented as:

[0061]

[0062] In the above formula, and Representing levels The abnormal global description vector and the normal global description vector, and This refers to MLP (Multilayer Perceptron), used to fuse global visual features. Learnable embedding vectors with abnormal patterns Normal mode can learn embedding vectors . This indicates a token selection linear layer. In computer vision, each local image patch is mapped to a visual token. The token selection linear layer filters the input tokens according to certain rules or algorithms; here, it represents filtering visual features at different levels to obtain the desired level. The global description vector.

[0063] As an example, when calculating similarity, cosine similarity can be specifically calculated, i.e.:

[0064]

[0065] In the above formula, and Representing levels Abnormal similarity and normal similarity, Indicates the current local image patch at the layer level. Visual features.

[0066] As an example, for hierarchy When identifying outliers, Softmax can be used to measure the similarity of outliers. Similarity to normal Normalization to outliers For example, it can be represented as:

[0067]

[0068] like Figure 3 The diagram shown is a flowchart of a specific embodiment of steps S103 to S106.

[0069] Regarding step S107, in some embodiments, optionally, the local anomaly information of each local image block is the aggregated value of the anomaly values ​​described in the above embodiments, referred to here as the aggregated anomaly value. This step can obtain an anomaly threshold, compare the aggregated anomaly value of each local image block with the anomaly threshold, and mark the local image block corresponding to the aggregated anomaly value that is greater than the anomaly threshold as an anomaly image block to obtain the anomaly detection result.

[0070] In some other embodiments, optionally, this step includes: acquiring normal mode text prompts and abnormal mode text prompts, and extracting normal prompt features and abnormal prompt features from them respectively, wherein the normal mode text prompts and abnormal mode text prompts are obtained through training; determining global anomaly information based on the normal prompt features, abnormal prompt features, and global visual features; and determining the anomaly detection result based on the global anomaly information and the local anomaly information of multiple local image patches. By further introducing normal mode text prompts and abnormal mode text prompts, and combining them with global visual features to obtain global anomaly information, and using it as a reference for determining the anomaly detection result, it is possible to utilize the rich semantics of text descriptions to guide anomaly recognition, enhance the model's sensitivity to security-related anomalies, and enhance cross-scene generalization ability, while also further considering both global semantics and local details, thereby improving anomaly detection capabilities.

[0071] As an example, both normal mode text hints and abnormal mode text hints can be constructed by concatenating a learnable prefix vector with a fixed text template, represented as follows: and The corresponding text embedding vectors are generated by the CLIP text encoder and used as normal prompt features. and abnormal prompt features The entire dataset, comprising normal mode text hints and abnormal mode text hints, can be called the learnable hint pool. and middle, and These represent the fixed text templates for normal and abnormal modes, respectively. The preceding vectors represent the learnable prefix vectors for each mode. The learnable prefix vectors, similar to the learnable embedding vectors for normal and abnormal modes described earlier, can also be used as parameters for the anomaly detection model. They participate in the model's training during the training phase, with the goal of better aligning the text features generated from the concatenated text prompts with the visual features.

[0072] Optionally, in some of the embodiments described above, the operation of determining global anomaly information based on normal prompt features, abnormal prompt features, and global visual features includes: determining global normal weights and global anomaly weights based on normal prompt features, abnormal prompt features, and global visual features; determining local anomaly weights based on local anomaly information of multiple local image patches; weightedly fusing the normal prompt features and abnormal prompt features based on the global normal weights, global anomaly weights, and local anomaly weights to obtain fused prompt features; using the fused prompt features to modulate the global visual features to obtain modulated global visual features; and determining global anomaly information based on the modulated global visual features, normal prompt features, and abnormal prompt features. By using both global normal weights and global anomaly weights, and local anomaly weights determined by local anomaly information, the weighted fusion of normal prompt features and abnormal prompt features can preserve both the matching between global semantics and prompt features, as well as local anomalies. Based on this, using the obtained fused prompt features to modulate the global visual features can yield a more accurate global semantic feature representation, which, combined with the prompt features, provides more accurate global anomaly information.

[0073] As an example, when determining the global normal weight and global abnormal weight based on normal prompt features, abnormal prompt features, and global visual features, the similarity between the global visual features and the normal prompt features can be used as the normal weight. The similarity between global visual features and anomaly cue features is calculated and used as the anomaly weight. .

[0074] As an example, the local anomaly information of each local image patch is the anomaly value described in the above embodiment. When determining the local anomaly weight based on the local anomaly information of multiple local image patches, the global average value of the anomaly value can be calculated as the local anomaly weight. .

[0075] As an example, the operation of weighted fusing of normal and abnormal prompt features based on global normal weight, global abnormal weight, and local abnormal weight to obtain the fused prompt features can be performed according to the following formula:

[0076]

[0077] In the above formula, This indicates a fusion prompt feature; the meanings of other symbols are the same as above.

[0078] As an example, using fused cue features to modulate global visual features, the operation of modulating global visual features can be expressed as the following formula:

[0079]

[0080] In the above formula, Indicates modulation of global visual features. Represents global visual features. This indicates element-wise multiplication.

[0081] Further optional, such as Figure 4 As shown, the above-mentioned operation of determining global anomaly information based on modulated global visual features, normal prompt features, and anomaly prompt features includes: acquiring multiple historical anomaly attributes and extracting multiple historical anomaly attribute features from them; determining a predetermined number of historical anomaly attribute features as reference anomaly attribute features based on the similarity between each of the multiple historical anomaly attribute features and the modulated global visual features; enhancing the anomaly prompt features based on the reference anomaly attribute features to obtain enhanced anomaly prompt features; and determining global anomaly information based on the modulated global visual features, normal prompt features, and enhanced anomaly prompt features. By using an attention mechanism to retrieve reference anomaly attribute features similar to the modulated global visual features from multiple historical anomaly attribute features, and enhancing the anomaly prompt features accordingly, the discriminative ability of the anomaly prompt features can be dynamically enhanced, further improving the model's sensitivity to security-related anomalies. It should be understood that the preset quantity represents the number of reference abnormal attribute features. In specific implementations, the value of the preset quantity can be directly configured, and the historical abnormal attribute features with the highest similarity can be used as reference abnormal attribute features; alternatively, a similarity threshold can be configured, and historical abnormal attribute features with similarity greater than the similarity threshold can be used as reference abnormal attribute features. The number of reference abnormal attribute features obtained in this case is the preset quantity. These are all implementation methods disclosed in this publication.

[0082] As an example, historical anomaly attributes include semantic attributes of historical anomaly scenarios. These can be embedded into vectors using a corresponding encoder, serving as historical anomaly attribute features. A feature set composed of multiple historical anomaly attribute features can be used... This indicates that each of the historical anomaly attribute features can be used It means that among them This represents the sequence number of the corresponding historical anomaly attribute. As an example, multiple historical anomaly attributes can be stored in a database, which can be called an attribute memory.

[0083] As an example, multiple historical anomaly attribute features Each with modulated global visual features The similarity is cosine similarity, which can be expressed as:

[0084]

[0085] As an example, enhancing anomaly indication features based on reference anomaly attribute features can be achieved by fusing the retrieved reference anomaly attribute features with the anomaly indication features, as shown below:

[0086]

[0087] In the above formula, This indicates enhanced anomaly alert features. Indicates the preset quantity.

[0088] As an example, the operation of determining global anomaly information based on modulated global visual features, normal cue features, and enhanced anomaly cue features is similar to the method of calculating the anomaly values ​​of each local image patch at each level, as described earlier. The similarity scores between the normal cue features and the enhanced anomaly cue features and the modulated global visual features can be calculated separately. , :

[0089]

[0090] Global outliers (i.e., global outlier information) .

[0091] As an example, during the model training phase, a lightweight projector can also be used to map modulated global visual features onto the CLIP text embedding space. And calculate the mapped visual features Features of normal prompts and enhanced anomaly alert features Similarity loss This optimizes cross-modal feature alignment. The similarity loss L can be expressed as:

[0092]

[0093] In the above formula, The temperature parameter is a hyperparameter used to control the sharpness of the similarity score distribution.

[0094] Optionally, in some of the above embodiments, the operation of determining the anomaly detection result based on global anomaly information and local anomaly information of multiple local image blocks includes: performing weighted fusion processing on the global anomaly information and the local anomaly information of multiple local image blocks to obtain fused anomaly information; performing bilinear interpolation processing on the fused anomaly information to obtain a full-resolution anomaly heatmap, wherein the full-resolution anomaly heatmap includes the anomaly heatmap values ​​of multiple pixels in the image to be detected; and determining the regions in the full-resolution anomaly heatmap where the anomaly heatmap values ​​are greater than the heatmap threshold as anomaly regions to obtain the anomaly detection result. By performing weighted fusion processing, global and local anomaly information can be easily fused. For example, in embodiments where the anomaly information is anomaly value, for each local image block, the weighted sum of the anomaly value of that local image block and the global anomaly value can be calculated as the fused anomaly value of that local image block. Building upon this foundation, since the granularity of fused anomaly information only reaches the level of local image patches, bilinear interpolation is applied to the fused anomaly information. This involves finding the four nearest known points on the low-resolution feature map corresponding to the pixel to be interpolated, and performing two linear interpolations based on the distances to these four points to estimate the anomaly value of that pixel. This allows for the supplementation of the anomaly heatmap values ​​for each pixel in the image to be detected, resulting in a full-resolution anomaly heatmap and achieving a finer-grained representation of anomaly information. Combined with a pre-set heatmap threshold, it is possible to quickly determine whether the anomaly heatmap value of each pixel is too high (i.e., whether it exceeds the heatmap threshold). The regions formed by pixels with excessively high anomaly heatmap values ​​are then identified as anomaly regions, thus achieving anomaly detection in the image to be detected.

[0095] As an example, weighted fusion processing can be represented as:

[0096]

[0097] In the above formula, It's weight. This provides a bias to the entire heatmap; if the global assessment indicates a high degree of anomaly, A high score will improve Outliers.

[0098] In summary, this disclosure provides an anomaly detection method based on multimodal cue learning and hierarchical feature alignment, which can be used, for example, for personnel safety monitoring. Addressing the shortcomings of existing video anomaly detection methods, such as insufficient fine-grained anomaly description, cross-domain generalization, and semantic understanding in complex scenarios, as well as poor robustness, this method designs a complete framework of multi-resolution visual encoding, two-stage feature matching, and dynamic cue weighting. Multi-resolution visual encoding segments high-resolution images and extracts multi-level visual features using a pre-trained visual language model, taking into account both global semantics and local details. The two-stage feature matching method first generates a description vector of the global anomaly pattern, and then calculates pixel-level anomaly values ​​through local feature matching, achieving accurate anomaly localization. Furthermore, the dynamic cue weighting mechanism utilizes a learnable cue pool and attribute memory to dynamically modulate global visual features, thereby obtaining global anomaly information and enhancing the model's sensitivity to safety-related anomalies. This disclosure achieves high-precision, low-false-positive personnel safety monitoring, significantly improving detection capabilities in complex environments while reducing annotation dependence, and is effectively suited for intelligent security, industrial monitoring, and other scenarios.

[0099] Figure 5 A structural block diagram of an electronic device 500 according to an exemplary embodiment of the present disclosure is shown.

[0100] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502. The at least one memory 501 stores computer-executable instructions. When the computer-executable instructions are executed by the at least one processor 502, they cause the at least one processor to execute the anomaly detection method based on multimodal multi-level features as described in the exemplary embodiments above.

[0101] As an example, electronic device 500 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 500 is not necessarily a single electronic device 500, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 500 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device 500 locally or remotely (e.g., via wireless transmission) through an interface.

[0102] In electronic device 500, processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 502 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0103] The processor 502 can execute instructions or code stored in the memory 501, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0104] The memory 501 may be integrated with the processor 502, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 501 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 502 to read files stored in the memory.

[0105] In addition, electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 500 can be interconnected via a bus and / or network.

[0106] According to exemplary embodiments of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause at least one processor to perform the anomaly detection method based on multimodal, multi-level features as described in the exemplary embodiments above. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0107] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including computer instructions that, when executed by at least one processor, perform the anomaly detection method based on multimodal, multi-level features as described in the exemplary embodiments above.

[0108] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0109] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An anomaly detection method based on multimodal and multi-level features, characterized in that, include: Acquire the image to be detected; The image to be detected is scaled and locally cropped to obtain a globally scaled image block and multiple local image blocks; The global scaled image patch is subjected to feature extraction processing to obtain global visual features; Multi-level feature extraction processing is performed on the multiple local image blocks to obtain the multi-level visual features of each local image block; The global visual features are processed using a cross-attention mechanism to obtain a global description vector, wherein the global description vector is used to describe normal events and / or abnormal events; For each local image patch, local anomaly information of the local image patch is determined based on the multi-level visual features of the local image patch and the global description vector; Obtain normal mode text prompts and abnormal mode text prompts, and extract normal prompt features and abnormal prompt features from them respectively, wherein the normal mode text prompts and abnormal mode text prompts are obtained through training; Based on the normal prompt features, the abnormal prompt features, and the global visual features, determine the global normal weight and the global abnormal weight; Based on the local anomaly information of the multiple local image blocks, the local anomaly weights are determined; Based on the global normal weight, the global abnormal weight, and the local abnormal weight, the normal prompt features and the abnormal prompt features are weighted and fused to obtain the fused prompt features; Using the fused cue features, the global visual features are modulated to obtain modulated global visual features; Global anomaly information is determined based on the modulated global visual features, the normal prompt features, and the abnormal prompt features; The anomaly detection result is determined based on the global anomaly information and the local anomaly information of the multiple local image patches.

2. The anomaly detection method as described in claim 1, characterized in that, For each local image patch, the local anomaly information of that local image patch is determined based on its multi-level visual features and the global description vector, including: For each level of visual features of each local image patch, the outlier value of the local image patch at that level is determined based on the level of visual features and the global description vector; The outliers of the local image patch at each level are aggregated to obtain the local anomaly information of the local image patch.

3. The anomaly detection method as described in claim 2, characterized in that, The process of using a cross-attention mechanism to process the global visual features to obtain a global description vector includes: The cross-attention mechanism is used to process the global visual features and the learnable embedding vectors for normal and abnormal modes to obtain a global description vector that includes a normal global description vector and an abnormal global description vector. The learnable embedding vectors for normal and abnormal modes are obtained through training. The step of determining the outlier value of the local image patch at that level based on the visual features of that level and the global description vector includes: The similarity between the hierarchical visual feature and the normal global description vector is determined as normal similarity, and the similarity between the hierarchical visual feature and the abnormal global description vector is determined as abnormal similarity. Based on the normal similarity and the abnormal similarity, the abnormal value of the local image patch at this level is determined.

4. The anomaly detection method according to any one of claims 1 to 3, characterized in that, The step of determining global anomaly information based on the modulated global visual features, the normal prompt features, and the abnormal prompt features includes: Obtain multiple historical anomaly attributes and extract multiple historical anomaly attribute features from them; Based on the similarity between each of the multiple historical anomaly attribute features and the modulated global visual features, a predetermined number of historical anomaly attribute features among the multiple historical anomaly attribute features are determined as reference anomaly attribute features. Based on the reference anomaly attribute features, the anomaly indication features are enhanced to obtain enhanced anomaly indication features; The global anomaly information is determined based on the modulated global visual features, the normal prompt features, and the enhanced anomaly prompt features.

5. The anomaly detection method according to any one of claims 1 to 3, characterized in that, Determining the anomaly detection result based on the global anomaly information and the local anomaly information of the multiple local image patches includes: The global anomaly information and the local anomaly information of the multiple local image blocks are weighted and fused to obtain fused anomaly information; The fused anomaly information is subjected to bilinear interpolation to obtain a full-resolution anomaly heatmap, wherein the full-resolution anomaly heatmap includes the anomaly heatmap values ​​of multiple pixels in the image to be detected; The regions in the full-resolution anomaly heatmap where the anomalous thermal values ​​are greater than the thermal threshold are identified as anomalous regions, and the anomaly detection results are obtained.

6. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to execute the anomaly detection method based on multimodal and multi-level features as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the anomaly detection method based on multimodal, multi-level features as described in any one of claims 1 to 5.

8. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the at least one processor causes the processor to perform the anomaly detection method based on multimodal, multi-level features as described in any one of claims 1 to 5.

Images