Abnormal scene detection method based on visual and semantic feature fusion

Abstract

The invention discloses an abnormal scene detection method based on visual and semantic feature fusion, and relates to the technical field of safety monitoring and intelligent identification, and the method comprises the steps: carrying out the preprocessing of a collected original image, and obtaining a preprocessed image; forming a multi-modal input pair by the preprocessed image and a predefined structured prompt statement; inputting the multi-modal input pair into the visual language large model, and outputting semantic features including visual feature vectors and text vectors; inputting the preprocessed image into a target detection model, and outputting visual features; fusing the semantic features and the visual features through a cross-modal attention mechanism to obtain multi-scale fusion features; and inputting the multi-scale fusion features into detection heads of all scales, executing abnormal scene detection, and outputting an abnormal detection result. When the unconventional object is identified in the abnormal scene, the visual features and the semantic features are fused to perform abnormal scene detection, so that the strong perception capability of a complex scene is realized, and false alarm or missing alarm is effectively avoided.

Description

Technical Field

[0001] This application relates to the fields of security monitoring and intelligent recognition technology, and in particular to an abnormal scene detection method based on the fusion of visual and semantic features. Background Technology

[0002] For public or semi-public environments (such as elevator cars, subway cars, and smart warehouses), operational safety and behavioral norms are of paramount importance. In these scenarios, due to the high concentration of people or goods, the probability of abnormal behaviors or states such as handling transparent items, carrying dangerous items, and leashing pets increases significantly. Failure to detect these abnormalities in a timely manner could lead to safety accidents. Existing anomaly detection methods primarily rely on sensor measurements or traditional image detection algorithms. However, relying solely on sensor data often fails to accurately identify complex anomalies. While purely vision-based detection methods (such as the YOLO series of object detection algorithms) can identify regular objects, they are prone to false positives or false negatives when dealing with transparent objects (such as glass and transparent plastic bags), small items (such as pet leashes and slender objects), and objects with incomplete structures due to occlusion, due to a lack of deep understanding of the object's semantics and context. Current technology lacks an efficient anomaly detection method that can integrate visual features and semantic information and possess strong perception capabilities for the aforementioned complex scenarios. Summary of the Invention

[0003] The purpose of this application is to provide an abnormal scene detection method based on the fusion of visual and semantic features. When identifying unconventional objects in abnormal scenes, it not only extracts visual features but also extracts semantic features by combining a large visual language model. By fusing visual and semantic features, it can detect abnormal scenes, achieve strong perception capabilities for complex scenes, and effectively avoid false alarms or missed alarms.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides an abnormal scene detection method based on the fusion of visual and semantic features, including:

[0006] The original acquired images are preprocessed to obtain preprocessed images.

[0007] The multimodal input pairs consist of a preprocessed image and predefined structured prompts.

[0008] The multimodal input is fed into a large visual language model, and the output contains semantic features including visual feature vectors and text vectors.

[0009] The preprocessed image is input into the target detection model, which outputs visual features; the target detection model is trained on a preset deep learning network model based on the image to be trained and its visual features.

[0010] Semantic features and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fused features.

[0011] The multi-scale fusion features are input into the detection heads at each scale to perform abnormal scene detection and output the abnormal detection results.

[0012] Secondly, this application provides an abnormal scene detection device based on the fusion of visual and semantic features, including: a data acquisition and processing module, a feature extraction module, a feature fusion module, and a detection module.

[0013] The data acquisition and processing module is used to preprocess the acquired raw images to obtain preprocessed images. The preprocessed images and predefined structured prompts constitute a multimodal input pair.

[0014] The feature extraction module is used to input multimodal input pairs into a large visual language model and output semantic features containing visual feature vectors and text vectors; it also inputs preprocessed images into an object detection model and outputs visual features; the object detection model is trained on a pre-defined deep learning network model based on the image to be trained and its visual features. Specifically, the preprocessed image and predefined structured prompts constitute a multimodal input pair, which is input into the semantic feature extraction module, while the preprocessed image is input separately into the visual feature extraction module.

[0015] The feature fusion module is used to fuse semantic features and visual features through a cross-modal attention mechanism to obtain multi-scale fused features.

[0016] The detection module is used to input multi-scale fused features into the detection head at each scale, perform abnormal scene detection, and output abnormal detection results.

[0017] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the abnormal scene detection method based on visual and semantic feature fusion as described above.

[0018] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the abnormal scene detection method based on the fusion of visual and semantic features described above.

[0019] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the abnormal scene detection method based on the fusion of visual and semantic features described above.

[0020] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0021] This application provides an anomaly scene detection method based on the fusion of visual and semantic features. The method preprocesses the acquired raw images to standardize and normalize them, meeting the input requirements of subsequent visual language models and object detection models. A multimodal input pair is formed by the preprocessed image and predefined structured prompts, fully utilizing the visual information of the image and the semantic information of the text to provide richer contextual information, helping the model better understand the image content. Semantic features are extracted based on the visual language model, with visual feature vectors and text vectors forming dual supervision, providing strong semantic understanding capabilities and capturing the complex relationships between images and text. Semantic and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fused features. Semantic features guide the weighted reconstruction of visual features, avoiding the false positives and false negatives caused by traditional detection models relying solely on local textures or edge cues. The multi-scale fused features are input to detection heads at different scales, outputting anomaly detection results. This enables the detection of anomaly scenes at different scales, improving the comprehensiveness and accuracy of anomaly detection and avoiding false positives or false negatives due to scale differences. It also achieves parallel detection of multiple potential anomaly targets in high-density areas. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an application environment diagram of an abnormal scene detection method based on the fusion of visual and semantic features according to an embodiment of this application;

[0024] Figure 2 A flowchart illustrating an abnormal scene detection method based on the fusion of visual and semantic features, provided as an embodiment of this application;

[0025] Figure 3 A flowchart of an abnormal scene detection method based on the fusion of visual and semantic features, provided as an embodiment of this application;

[0026] Figure 4 A partial structural diagram of a large visual language model provided in an embodiment of this application;

[0027] Figure 5 This is a partial structural diagram of a target detection model provided in an embodiment of this application;

[0028] Figure 6 A cross-modal fusion structure diagram provided in an embodiment of this application;

[0029] Figure 7 This is a detection effect diagram provided in one embodiment of this application;

[0030] Figure 8 This is a framework diagram of a detection model provided in an embodiment of this application;

[0031] Figure 9 A schematic diagram of the functional modules of an abnormal scene detection device based on the fusion of visual and semantic features, provided in an embodiment of this application;

[0032] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The abnormal scene detection method based on the fusion of visual and semantic features provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can send the acquired raw image to server 102. Server 102 receives the acquired raw image, preprocesses it to obtain a preprocessed image, and uses the preprocessed image and predefined structured prompts to form a multimodal input pair. The multimodal input pair is input into a large visual language model, outputting semantic features containing visual feature vectors and text vectors. The preprocessed image is input into a target detection model, outputting visual features. The target detection model is trained on a preset deep learning network model based on the image to be trained and its visual features. The semantic features and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fusion features. The multi-scale fusion features are input to the detection heads at each scale to perform abnormal scene detection and output abnormal detection results. Server 102 can feed back the obtained abnormal detection results to terminal 101. In addition, in some embodiments, the abnormal scene detection method based on the fusion of visual and semantic features can also be implemented by the server 102 or the terminal 101 alone. For example, the terminal 101 can directly perform abnormal scene detection based on the fusion of visual and semantic features on the original image to be processed, or the server 102 can obtain the original image from the data storage system and perform abnormal scene detection based on the fusion of visual and semantic features on the original image.

[0036] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 102 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0037] In one exemplary embodiment, such as Figure 2 As shown, an abnormal scene detection method based on the fusion of visual and semantic features is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps 201 to 206. Wherein:

[0038] Step 201: Preprocess the acquired raw image to obtain a preprocessed image.

[0039] Step 202: A multimodal input pair is formed by the preprocessed image and the predefined structured prompts.

[0040] Step 203: Input the multimodal input to the visual language large model and output semantic features containing visual feature vectors and text vectors.

[0041] Step 204: Input the preprocessed image into the target detection model and output visual features; the target detection model is obtained by training a preset deep learning network model based on the image to be trained and its visual features.

[0042] Step 205: The semantic features and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fused features.

[0043] Step 206: Input the multi-scale fusion features into the detection heads of each scale respectively, perform abnormal scene detection, and output the abnormal detection results.

[0044] Steps 201 to 206 are implemented as follows: The acquired raw images are preprocessed to standardize and normalize them, meeting the input requirements of the subsequent visual language large-scale model and object detection model. The preprocessed images and predefined structured prompts form a multimodal input pair, fully utilizing the visual information of the images and the semantic information of the text, providing richer contextual information to help the model better understand the image content. Semantic features are extracted based on the visual language large-scale model, with visual feature vectors and text vectors forming dual supervision, possessing powerful semantic understanding capabilities and able to capture the complex relationships between images and text. Semantic features and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fused features. Semantic features guide the weighted reconstruction of visual features, avoiding the false positives and false negatives caused by traditional detection models relying solely on local textures or edge cues. The multi-scale fused features are input to the detection heads at each scale, outputting anomaly detection results. This enables the detection of abnormal scenes at different scales, improving the comprehensiveness and accuracy of anomaly detection, avoiding false positives or false negatives due to scale differences, and achieving parallel detection of multiple potential abnormal targets in high-density areas.

[0045] This application acquires real-time images from complex scenarios (such as elevator cars, subway cars, and smart warehouses) and performs preprocessing such as normalization and enhancement. The preprocessed images are then paired with structured prompts to form a multimodal input pair, which is input into a large-scale visual language model. Multidimensional semantic features and multi-scale visual features are extracted from the images. A cross-modal feature fusion module fuses the semantic and visual features to generate a multi-scale fused feature representation. Finally, the multi-scale fused features are input into detection heads of different scales, where multi-scale feature fusion is performed to enhance feature representation capabilities. Anomaly detection results containing target location, category, and confidence are output. Simultaneously, incremental fine-tuning is used to adapt and update the visual encoder, extracting its intermediate-layer semantic features to guide the training of the target detection model. This invention integrates high-level semantic information and multi-scale visual features, significantly enhancing the perception of abnormal behavior in complex scenarios. It effectively improves the accuracy and robustness of recognition, especially when dealing with transparent objects, small items, and targets with incomplete structures due to occlusion. By introducing a semantic guidance mechanism, the shortcomings of traditional pure visual detection methods in semantic understanding and context modeling are compensated for, thus solving the problems of frequent false positives and false negatives. At the same time, the proposed method possesses good flexibility and versatility, supporting the replacement or extension of specific model structures according to application requirements, and adapting to anomaly detection tasks in complex scenarios.

[0046] Optionally, the preprocessing operation in step 201 includes: performing data enhancement operations such as scale normalization, rotation, exposure and contrast enhancement on the acquired real-time raw image, and mapping the color values ​​to a unified color space range.

[0047] Specifically, the original images (acquired in real time within complex scene areas) are uniformly adjusted to a fixed resolution of 448×448 to complete scale standardization. Then, random rotation within a range of ±15° is performed. The image brightness is adjusted by gamma correction, and the local contrast of the image is improved by histogram equalization. Finally, the color values ​​are mapped to a unified color space range, and the pixel values ​​of the RGB channels are scaled to the [0, 1] interval.

[0048] The predefined structured prompts in step 202 refer to natural language descriptions generated according to a fixed template based on preset target categories or abnormal behaviors. These descriptions guide the language model to focus on regions in the image that are semantically related to the target. Optionally, the target category can be an abnormal object (e.g., glass, transparent plastic bag, oversized item, dragged object (e.g., long plank)), a small object (e.g., pet leash, mop cable), or a difficult-to-identify object (e.g., fallen object); the abnormal behavior can be carrying dangerous items (e.g., carrying flammable or explosive items (e.g., gas canisters), sharp objects, etc.). The fixed template is manually labeled, and its content can be: You are {role}, and your task is {specific item or state, target or behavior}. For example, you are an elevator safety inspector, please identify whether there is a gas canister in the current scene; or you are an elevator safety monitoring system, please determine whether there is transparent glass in the image; or you are an elevator safety patrolman, your task is to determine whether there is an electric vehicle entering the elevator in the image, etc.

[0049] In another exemplary embodiment of this application, in order to combine text information and image information to improve semantic understanding, step 203 is replaced by the following steps 301 to 302:

[0050] Step 301: Based on the large visual language model, extract multi-scale high-dimensional visual feature vectors from the preprocessed images of multimodal input pairs.

[0051] Step 302: Encode the structured prompts of the multimodal input pairs into text vectors, and replace the special placeholders in the text vectors with high-dimensional visual feature vectors to obtain semantic features.

[0052] This application extracts multi-scale high-dimensional visual features based on a large visual language model. These multi-scale high-dimensional visual features can simultaneously contain global semantics (object category, scene) and local details (edges, texture). After these complementary information are injected into the text stream at special placeholders, the performance degradation caused by scale changes, occlusion, or noise can be significantly reduced, and the robustness of the model to understanding complex scenes can be improved. The replacement of special placeholders only involves tensor slicing operations, which hardly increases the number of parameters and latency, and can be easily "plug and play" in the framework of large language models.

[0053] In addition, the visual encoder of the visual language big model extracts multi-scale high-dimensional visual feature vectors from the preprocessed image and maps them to the dimensions required by the language model of the visual language big model through the projection module.

[0054] Preferably, the large visual language model is a multimodal pre-trained model with image understanding and language processing capabilities, enabling image-text alignment and semantic feature extraction. The large visual language model used in this application employs a visual encoder based on a multi-layer Transformer network with a ViT architecture, capable of extracting multi-level, high-dimensional semantic features from images for identifying target categories, semantic relationships, and scene information.

[0055] Preferably, the projection module normalizes the visual features in the local area, rearranges and splices the visual features of multiple adjacent positions into a unified high-dimensional representation, and then performs a nonlinear transformation through a two-layer feedforward network to map it to the target dimension.

[0056] In another exemplary embodiment of this application, in order to effectively extract high-dimensional visual feature vectors of the image, facilitate the provision of more accurate and richer feature representations, and improve the generalization ability and computational efficiency of the model, step 301 is replaced by the following steps 311 to 313:

[0057] Step 311: Divide the preprocessed image into multiple non-overlapping image blocks according to a preset step size.

[0058] Step 312: Input the sequence of non-overlapping image patches into the 3D convolutional embedding module for embedding processing, and output the initial token sequence.

[0059] Step 313: Input the initial token sequence into the visual language big model and output a high-dimensional visual feature vector with unified dimensions and multiple scales. The visual language big model is based on the visual Transformer and includes multiple stacked visual blocks. Each visual block includes the following components connected in sequence: a first normalization layer, a multi-head self-attention module, a second normalization layer, and a feedforward network.

[0060] This application divides an image into multiple non-overlapping image patches, enabling analysis from a local perspective while avoiding redundant computation caused by overlapping partitioning, thus facilitating the capture of local detail features. Three-dimensional convolution captures the spatial and channel-dimensional features of image patches, better preserving spatial structural information compared to traditional two-dimensional convolution, providing more accurate input for subsequent feature learning. The large visual language model, based on the Visual Transformer, uses a normalization layer to normalize features, making feature distribution more stable and beneficial for subsequent feature learning. A multi-head self-attention module extracts features from multiple subspaces simultaneously, capturing long-distance dependencies between image features, better handling complex relationships in images compared to traditional convolutional neural networks. A feedforward network performs further non-linear transformations on features, enhancing their expressive power. Residual connections alleviate the vanishing and exploding gradient problems in deep network training, improving training stability and enabling the network to learn deeper features. Through non-overlapping image patch merging and downsampling operations, a high-dimensional visual feature vector with unified dimensions and multiple scales is output, providing a more efficient and unified feature representation for subsequent tasks.

[0061] In another exemplary embodiment of this application, in order to facilitate the accurate fusion of high-dimensional visual feature vectors and text vectors, step 302 is replaced by steps 321 to 323:

[0062] Step 321: The structured prompt statement is encoded by a word segmenter to obtain a discrete Token ID sequence, which contains reserved special placeholders.

[0063] Step 322: Construct a mask matrix and mark the expanded portion of the special placeholder positions in the Token ID sequence.

[0064] Step 323: Inject the high-dimensional visual feature vector into the text embedding based on the mask matrix, and replace the special placeholder positions as semantic features.

[0065] This application encodes structured prompts using a word segmenter, converting textual information into a numerical form that the model can process, while preserving the semantic structure of the text. Reserved special placeholders provide clear location markers for subsequent visual feature injection. A mask matrix ensures that visual features are accurately injected into the text embedding, avoiding errors or confusion in feature injection. By injecting high-dimensional visual feature vectors into the text embedding, visual information and textual semantics are precisely combined to form unified semantic features. This facilitates subsequent model inference using both visual information and textual semantics, enhancing the model's understanding of complex scenes, improving the performance of multimodal tasks, and strengthening the model's generalization and adaptability.

[0066] Optionally, the multimodal input is fed into the large visual-language model, and the output includes semantic features containing visual feature vectors and text vectors, specifically including:

[0067] 1) Divide the preprocessed image (input image) into H / 14 and W / 14 non-overlapping image patches in the height and width directions with a stride of 14, generating a total of N = (H / 14) × (W / 14) patches. Each patch is 14 × 14 in size and has 3 channels.

[0068] The Patch sequence is treated as a three-dimensional data block with a time length of 2, and input into a three-dimensional convolutional embedding module for embedding processing, outputting an initial Token sequence T. vit ∈R N×1280 , which serves as the initial input for the visual Transformer.

[0069] To introduce spatial structure perception capabilities, a two-dimensional rotational position encoding (2D RoPE) is introduced. Let the dimension of each attention head be d. head Split it into d head / 2 is used for the height direction, and d is used for the height direction. head / 2 is used in the width direction.

[0070] For each position (h, w) in the image, construct its rotation encoding vector:

[0071] R h (h)=[cos(h·θ),sin(h·θ)].

[0072] R w (w)=[cos(w·θ),sin(w·θ)].

[0073] Where, θ∈R dhead / 2 For a shared frequency base group, h represents the row direction, w represents the column direction, and R... h (h) represents the position encoding for the row direction, R w (w) is the position encoding for the column direction.

[0074] The final RoPE encoding result is:

[0075]

[0076] Among them, T vit This is a mapping function, typically a linear transformation / projection, that transforms the concatenated vector to the embedding dimension required by ViT. This is a compound operator for functions, meaning "concatenate first, then input into T". vit ".

[0077] This encoding format directly affects the query vector (Query) and key vector (Key) in the Attention module.

[0078] The initial token sequence is fed into a 32-layer stacked visual Transformer, with each layer being a visual block B. i The unified structure is as follows:

[0079] First normalization layer: The root mean square normalization (RMSNorm) algorithm is used to perform root mean square normalization on the input x.

[0080]

[0081] Where, w∈R 1280 For trainable vectors, ε = 10 -6 The term is the smoothing term, and mean is the function for calculating the mean, mean(x) 2 () is the average of the squares of x.

[0082] Multi-head self-attention module: Linear mapping yields the query vector (Q), key vector (K), and value vector (V), which are respectively:

[0083] Q = Linear 3584→3584 (x).

[0084]

[0085] K = Linear 3584→512 (x).

[0086]

[0087] V = Linear 3584→512 (x).

[0088] Where Linear represents a linear transformation operation. The location-aware query vector encoded for RoPE. The position-aware key vector encoded for RoPE.

[0089] After adding RoPE encoding to the query vector Q and key vector K, the structure is divided into a multi-head structure, and scaled dot product attention is performed:

[0090]

[0091] The output is restored to 1280 dimensions through a linear transformation.

[0092] Second normalization layer: RMSNorm is executed again to process the Attention output.

[0093] Multilayer Perceptron (MLP): Consists of two linearly mapped layers and the nonlinear activation function SiLU.

[0094] MLP(x) = Linear 3420→1280 (SiLU(Linear 1280→3420 (x))).

[0095] Each submodule (attention, MLP) uses residual linking to directly add the submodule output to the input to build a deep information flow path.

[0096] Some layers employ a global attention mechanism for their multi-head self-attention modules, while the remaining layers use a local window attention mechanism. The local window size is 112×112, corresponding to 8×8 patches. After windowing, self-attention is only performed within each window, reducing computational complexity.

[0097] After a specified layer (e.g., layer 16, layer 24, layer 32), patch merging and downsampling operations are performed. This operation uses 2×2 patches as units, concatenating four adjacent tokens to form a feature vector of length 5120, which is then input into the merging module Qwen2_5_VLPatchMerger, using an MLP with the following structure:

[0098] MLP(x) = Linear 5120→3584 (GELU(Linear 5120→5120 (x))).

[0099] The output is a token sequence T of uniform dimensions. merged ∈R (N / 4)×3584 The spatial dimension is reduced to H / 28×W / 28, the temporal dimension is 1, and the channel dimension is unified to 3584, thus forming three progressive visual scales.

[0100] 2) The input structured prompt statement (text sequence) is first processed by a tokenizer, which converts the original text into a discrete sequence of TokenIDs:

[0101] I = [i1, i2, ..., i L ].

[0102] Where L is the length of the TokenID sequence, i is the ID of a single Token, and I is the Token ID sequence corresponding to the entire text. The sequence contains reserved special placeholders (i.e., visual placeholders) Token<|image_pad|>, which are used for subsequent image embedding.

[0103] Each special placeholder token <|image_pad|> corresponds to one input image. To achieve mapping and fusion of multiple visual tokens, each special placeholder is expanded according to the number of tokens for the visual features. The specific expansion length is determined by the spatial downsampling of the visual features and is defined as follows:

[0104]

[0105] Among them, grid t The time dimension of the image is 1 (for still images); grid h grid w These represent the height and width of the merged space, respectively, grid h =H / 28, grid w =W / 28; merge_size is the patch merging size, which defaults to 2 (i.e., 2×2 patches merged into one token); N img This represents the number of visual tokens obtained from the final mapping of a single image.

[0106] The number of special placeholders Token<|image_pad|> is denoted as n. pad The length of the expanded token sequence (or the expanded text token) is:

[0107] L'=Ln pad +n pad ×N img =L+n pad (N img -1).

[0108] The expanded token sequence is mapped to a continuous vector representation through an embedding layer:

[0109] E text ∈R L'×D .

[0110] Where D is the unified embedding dimension of the model, which is 3584 dimensions, and E text This is the embedding representation matrix corresponding to the extended token sequence.

[0111] The expanded token sequence (i.e., the visual token embedding sequence) replaces the corresponding special placeholder positions in the text sequence in sequence. The specific steps are as follows:

[0112] Construct a mask matrix M∈{0,1} L'×1 , marks the expanded portion of a text sequence at the position of a special placeholder.

[0113] Using the mask matrix M, the token sequence T (i.e., multi-scale high-dimensional visual feature vectors) of the same dimension is... merged Inject text embeddings respectively text Replace special placeholder positions as semantic features:

[0114]

[0115] in, For semantic features, ⊙ represents element-wise selection and replacement, where i∈{1,2,3}.

[0116] Optionally, the semantic features of this application are represented by a set of high-dimensional tokens, in the form of:

[0117]

[0118] Where R is a multidimensional tensor of a specific dimension, and T i Let i be the number of tokens, and D be the dimension of the hidden layer, where i ∈ {1, 2, 3}.

[0119] Preferably, the semantic features are token-level high-dimensional representations extracted by a large visual language model, used to describe the semantic information in the image. This application selects high, medium, and low-level semantic features from the large visual language model, each corresponding to a different level of semantic abstraction, possessing good context awareness and expressive capabilities.

[0120] In step 204, the preprocessed image is input into the object detection model (or object detection backbone network), and the output is (multi-scale) visual features. The (multi-scale) visual features are in the following form:

[0121]

[0122] Where R is a multidimensional tensor of a specific dimension, and C i H represents the number of channels. i W i represents the spatial dimension of the feature map.

[0123] Preferably, the visual feature is a multi-scale spatial feature map extracted by the target detection model from the input image, typically in the form of R. C×W×H The feature map is a three-dimensional tensor, where R is a multidimensional tensor with a specific dimension, C is the number of channels, and H and W are the height and width of the feature map. This feature can effectively characterize the salient information of local regions in an image, and has advantages such as clear spatial structure and high target localization accuracy, making it suitable for efficient target detection tasks in complex scenes.

[0124] In another exemplary embodiment of this application, in order to achieve accurate fusion of semantic features and visual features to obtain multi-scale fused features, step 205 is replaced by steps 401 to 404:

[0125] Step 401: Perform dimensional mapping and compression on the semantic features to obtain semantically aligned features that are aligned with the dimensions of the visual features.

[0126] Step 402: Use independent linear transformations to map visual features to a query matrix and semantically aligned features to a key matrix and a value matrix.

[0127] Step 403: Calculate the fused features using a scaling dot product attention mechanism based on the query matrix, key matrix, and value matrix.

[0128] Step 404: Enhance the fused features using a residual convolution module to obtain multi-scale fused features.

[0129] This application performs dimensional mapping and compression on semantic features to avoid inefficient attention computation or information loss due to dimensionality mismatch, reduces redundant information, and improves computational efficiency. Through independent linear transformations, the representations of visual and semantic features are optimized separately to avoid mutual interference, generating query, key, and value matrices, which provide input for the scaled dot product attention mechanism. The scaled dot product attention mechanism dynamically allocates weights based on the similarity between the query and key matrices, highlighting important feature information, capturing long-distance dependencies between features, and improving the model's semantic understanding ability. The residual convolution module further extracts local and global information from the features, enhancing their expressive power.

[0130] Specifically, to align the spatial dimensions of semantic features with those of visual features, the semantic features are dimensionally mapped and compressed to obtain semantically aligned features that are aligned with the dimensions of the visual features, including:

[0131] The semantic features output by the large visual language model are subjected to layer normalization.

[0132] Use a linear transformation to map the semantic features of dimension D to H. i ×W i The spatial representation of the dimension yields the mapped features, where H i ×W i Dimensions of visual features.

[0133] The mapped features are rearranged in dimensions and compressed using adaptive one-dimensional average pooling to generate a form like The semantic features (i.e., semantic alignment features), where R is a multidimensional tensor of a specific dimension, and L = H i ×W i .

[0134] Preferably, the layer normalization process involves calculating the mean and variance of each feature dimension of each sample, and normalizing the outputs of all neurons in that layer to ensure they conform to a standard normal distribution.

[0135] After obtaining the semantic alignment features, the semantic alignment features and visual features are fused together through a cross-modal attention mechanism, specifically including:

[0136] Align semantic alignment features with visual features, and map the visual features to a query matrix Q = W. Q F Y Simultaneously, the semantic alignment feature tensor is mapped to a key matrix. Sum matrix Among them, W Q W K W V These are learnable projection matrices used to map input features to query (Q), key (K), and value (V) spaces, respectively.

[0137] The fused features are obtained using the following calculation formula:

[0138]

[0139] Among them, F attn For the fused feature representation, d k Let K be the dimension of the key matrix. The softmax function is used to calculate the attention weights.

[0140] Building upon this, a residual convolution module is introduced to enhance the fused features:

[0141] F i =conv(F attn )+F vis_{i} .

[0142] Among them, F i For the feature enhanced at the i-th scale, F vis_{i} Let i represent the visual features before the initial mapping, where i∈{1,2,3}.

[0143] In another exemplary embodiment of this application, in order to detect anomalous scenes at different scales and improve the comprehensiveness and accuracy of anomaly detection, step 206 is replaced by steps 501 to 503:

[0144] Step 501: Connect the multi-scale fused features to the detection heads of each scale respectively, and each detection head processes the feature information of the corresponding resolution.

[0145] Step 502: Within each detection head, large-scale features are passed down to small-scale branches level by level, and feature information of different scales is fused at each layer.

[0146] Step 503: The detection heads at each scale output structured anomaly detection results, which include the target's location, category, and corresponding confidence level.

[0147] This application enables the detection of anomalous scenes at multiple scales by inputting multi-scale fused features into detection heads of different scales. This helps capture targets and anomalous regions of different sizes. Each detection head independently processes features at its corresponding scale, improving computational efficiency and reducing the time overhead of serial processing. Large-scale features contain more contextual information, and through hierarchical transmission and fusion, they help small-scale features better understand local information, enabling better capture of both global and local features of the target and improving detection accuracy. The fused features, output by the detection head, can be directly applied to scenarios such as anomaly recognition and alarms, possessing advantages such as high efficiency, robustness, and deployability.

[0148] In one specific implementation, the multi-scale fusion feature F i Each detection head is connected to a different scale of target detection, and each detection head processes feature information at different resolutions. Multi-scale features are fused within the detection head to improve the detection capability for targets of different sizes. Each detection head outputs structured detection results, including the target's location, category, and corresponding confidence level, which are used to identify abnormal behaviors or states in high-density areas.

[0149] Optionally, the detection head includes a multi-scale convolutional prediction structure for generating multiple detection outputs at different scales, thereby enabling parallel detection of multiple potential anomalous targets within a high-density region. The fusion of multi-scale features refers to progressively passing large-scale features down to smaller-scale branches and integrating feature information at different scales at each layer.

[0150] Preferably, the detection head is the output structure of the target detection network, responsible for predicting the location, category, and confidence level of potential targets in the image based on input features. The detection head in this application contains prediction branches at three scales, adapted for small, medium, and large target detection respectively, and its output structure is as follows:

[0151]

[0152] Where R is a multidimensional tensor of a specific dimension, B is the batch size, A is the number of anchor frames, and H is the number of anchor frames. k ×W k Let C be the spatial dimension of the k-th layer, C be the number of target categories, 5 represent the location and confidence fields (x,y,w,h,s), and O be the number of target categories. k Let be the output tensor of the k-th layer detector head.

[0153] Finally, the model outputs the location, category, and confidence information of all potential anomalous targets in the image, and generates the final detection result by setting thresholds and non-maximum suppression (NMS) and other mechanisms.

[0154] Furthermore, the training process of the visual language large model and object detection model in this application includes: incrementally fine-tuning the visual language large model using multimodal input pairs, extracting intermediate layer features from the visual language large model, and using these features as guiding information to participate in feature fusion during the training process of the object detection model, thereby achieving feature-guided fine-tuning.

[0155] In one specific implementation, the above training process can be as follows:

[0156] The preprocessed multimodal input is used to input a large visual language model, and the visual encoder is adaptively trained using incremental fine-tuning. Incremental fine-tuning refers to training only the low-rank adaptation modules inserted into a subset of layers while freezing the parameters of the large-scale pre-trained model. This incremental optimization on new data enables the model to adapt to new tasks.

[0157] The preprocessed image is input into the object detection model, and the multi-scale semantic feature representation of the intermediate layer is extracted from the fine-tuned visual language model. The extracted semantic features are used as guiding information to fuse with the visual features in the object detection model, guiding its feature representation learning and realizing feature-guided fine-tuning.

[0158] This application introduces a large visual language model to extract semantic features and fuses them with the visual features of a target detection model, which can effectively enhance the representation ability of abnormal targets in monitoring images of complex scenes. The cross-modal fusion module is built based on an attention-like mechanism, which realizes deep alignment and complementarity of visual and semantic information, improving the model's recognition accuracy and robustness for weakly salient targets (such as glass, traction ropes, etc.) in complex scenes. At the same time, this invention does not rely on additional hardware and can improve the intelligence level of existing monitoring systems simply by optimizing the model structure, which has high practicality, scalability and engineering deployment value.

[0159] The following uses an elevator scenario as an application example to illustrate this application. The method described is not limited to this type of scenario and can be widely applied to anomaly detection tasks in various complex environments such as subway cars and warehouse spaces.

[0160] like Figure 3 As shown, the specific steps include:

[0161] S1: Data Acquisition and Preprocessing

[0162] Raw image frames are acquired from cameras installed on the top or inside the elevator car and used as input images for subsequent anomaly detection. The raw images are uniformly adjusted to a fixed resolution of 448×448 for scale standardization, then randomly rotated within ±15°. Image brightness is adjusted using linear transformation or gamma correction, and histogram equalization is used to enhance local contrast. Finally, color values ​​are mapped to a unified color space, and pixel values ​​in the RGB channels are scaled to the [0, 1] interval to obtain the preprocessed image.

[0163] Furthermore, the preprocessed image and predefined structured prompts are combined to form a multimodal input pair, which is then input into the semantic feature extraction module. Simultaneously, the preprocessed image is input separately into the visual feature extraction module.

[0164] S2: Feature Extraction

[0165] The multimodal input obtained in step S1 is fed into the Qwen2.5-VL large-scale visual language model to extract deep semantic features from the image. Qwen2.5-VL is a multimodal model based on the Transformer architecture, such as... Figure 4 As shown, its visual encoder part uses the Vision Transformer (ViT) module.

[0166] Preferably, the ViT module contains two normalization layers (RMSNorm), one window attention mechanism (WindowAttention), and one feedforward network with SwiGLU activation function (FFN with SwiGLU), and each layer is connected by residual connections, with the output being input to the next module.

[0167] Preferably, the output features of layers 16, 24, and 32 of the ViT encoder are extracted to obtain multi-scale semantic information. The multi-scale features extracted from each layer are input into the VLPM module. The VLPM module internally uses normalization (RMSNorm), linear transformation, and the GELU activation function to perform semantic mapping and cross-modal alignment of the image features, ultimately outputting a unified feature representation (i.e., semantic features), in the form of:

[0168]

[0169] Where R is a multidimensional tensor of a specific dimension, and T i Where is the number of tokens, and D is the dimension of the hidden layer.

[0170] Input the preprocessed image into the YOLO series object detection model, such as Figure 5 As shown, visual feature information of the image is extracted.

[0171] Preferably, this embodiment uses YOLOv5x, which has better performance in the YOLOv5 series, as the target detection model.

[0172] The YOLO model includes a convolutional backbone, a feature fusion network (Neck), and a multi-scale detection head (Head), which can generate multi-scale visual features.

[0173] The extracted multi-scale visual features are in the following form:

[0174]

[0175] Where R is a multidimensional tensor of a specific dimension, and C i Number of channels H i W i represents the spatial dimension of the feature map.

[0176] The multi-scale visual features contain saliency information of each spatial location in the scene, which is used for target location perception and structural modeling.

[0177] S3: Cross-modal feature fusion:

[0178] To align semantic features with multi-scale visual feature structures, further mapping and compression processes are performed, including:

[0179] Layer normalization (LNorm) is performed on semantic features.

[0180] Use a linear transformation to map it from the original dimension to H. i ×W i A spatial representation of dimension H, where H i ×W i Dimensions of visual features.

[0181] Transpose the mapped sequence tensor.

[0182] Its channel dimension is compressed using adaptive one-dimensional average pooling (AvgPool).

[0183] Finally, it becomes like The semantic alignment feature aligns with the visual feature in the spatial dimension, where R is a multidimensional tensor of a specific dimension, and L = H. i ×W i .

[0184] The semantically aligned features, aligned with the visual features, and the multi-scale visual features obtained in step S2 are input into the cross-modal feature fusion module (CMF Block), such as... Figure 6As shown, this module achieves semantic fusion and feature enhancement between two modalities. It is built upon a self-attention-like mechanism, and the fusion process is as follows:

[0185] Map visual features to a query matrix Q = W Q F Y .

[0186] Map the semantic feature tensor to the key matrix AND-value matrix

[0187] The fused representation is computed using a scaled dot product attention mechanism:

[0188]

[0189] Among them, F attn For the fused feature representation, d k Let K be the dimension of the key. The softmax function is used to calculate the attention weights.

[0190] Building upon this, a residual convolution module is introduced to enhance the fused features:

[0191] F i =conv(F attn +F vis_{i} ).

[0192] Among them, F i For the feature enhanced at the i-th scale, F vis_{i} Let i represent the visual features before the initial mapping, where i∈{1,2,3}.

[0193] The multi-scale fusion feature F i It preserves the spatial structure of the image while introducing linguistic semantic information, enabling the model to have stronger semantic perception capabilities when detecting targets, especially suitable for detection scenarios with complex backgrounds or low-salience targets (such as glass, pet leashes, etc.) inside elevators.

[0194] S4: Anomaly Detection Output:

[0195] The multi-scale fused features output from step S3 are input into the multi-scale detection head of the YOLO model to perform the elevator anomaly target detection task. The detection head adopts a top-down feature fusion strategy, which progressively passes and fuses large-scale features into small-scale features to enhance the perception capability of targets of different sizes.

[0196] Each detection head outputs structured detection results, including the target's location, category, and corresponding confidence level, used to identify abnormal behaviors or states within the elevator car. Each detection head contains three prediction branches at three scales, with each branch outputting the following structure:

[0197]

[0198] Where R is a multidimensional tensor of a specific dimension, B is the batch size, A is the number of predicted anchor boxes per location, 5 represents the location information of the bounding box (x,y,w,h,s), and C is the number of target categories.

[0199] Finally, the model outputs the location, category, and confidence information of all potential anomalous targets in the elevator image. The final detection results are generated through mechanisms such as thresholding and non-maximum suppression (NMS). Figure 7 As shown.

[0200] The test results can be used for automatic alarms, behavior recording, or follow-up actions in elevator safety systems, thereby improving the intelligence level and response efficiency of the monitoring system.

[0201] S5: Model fine-tuning:

[0202] The preprocessed image obtained in step S1 is input into the visual language large-scale model Qwen2.5-VL. The visual encoder is adaptively trained using the LoRa incremental fine-tuning method, and its intermediate layer multi-scale semantic features are extracted. The extracted intermediate layer visual features are then introduced into the YOLO object detection model through the CMF_block module as guiding information in the feature fusion process. The fused features are further input into the multi-scale detection head for object detection and recognition training, thereby achieving feature enhancement and fine-tuning optimization based on visual language guidance, such as... Figure 8 As shown.

[0203] This invention proposes an anomaly scene detection method based on the fusion of visual and semantic features. By fusing semantic perception features with spatial visual features, it achieves high-precision detection of potential anomaly targets in complex scenes. To overcome the problems of existing target detection methods being susceptible to occlusion, background interference, and low-salience targets in complex environments, this invention employs a large visual language model to extract high-level semantic features and combines this with multi-scale visual features from the YOLO target detection model. Through a cross-modal feature fusion module with a self-attention-like structure, the model's ability to discriminate anomaly targets is effectively enhanced.

[0204] In feature fusion, semantic features are used to guide the weighted reconstruction of visual features, avoiding the false positives and false negatives caused by traditional detection models that rely solely on local textures or edge cues. Compared to ordinary fusion strategies that require manual design or simple splicing of multimodal features, the fusion method proposed in this invention has a clear semantic alignment mechanism and good structural adaptability.

[0205] Furthermore, this invention employs a fixed prompt word and image-based input method, eliminating the need for external semantic annotation or text generation processes, thus ensuring the stability and versatility of the method. The fused features, output by the detection head, can be directly applied to scenarios such as anomaly detection and alarms, offering advantages such as high efficiency, robustness, and deployability. This method also exhibits strong adaptability to scenarios involving rain, changing lighting, and occlusion, making it a highly practical and stable anomaly detection technology for complex scenes.

[0206] Based on the same inventive concept, this application also provides an anomaly scene detection device based on visual and semantic feature fusion for implementing the above-mentioned anomaly scene detection method based on visual and semantic feature fusion. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more anomaly scene detection device embodiments based on visual and semantic feature fusion provided below can be found in the limitations of the video tag processing method above, and will not be repeated here.

[0207] In one exemplary embodiment, such as Figure 9 As shown, an abnormal scene detection device based on the fusion of visual and semantic features is provided, including: a data acquisition and processing module, a feature extraction module, a feature fusion module, and a detection module.

[0208] The data acquisition and processing module is used to preprocess the acquired raw images to obtain preprocessed images. The preprocessed images and predefined structured prompts constitute a multimodal input pair.

[0209] The feature extraction module is used to input multimodal input pairs into a large visual language model and output semantic features containing visual feature vectors and text vectors; it also inputs preprocessed images into an object detection model and outputs visual features; the object detection model is trained on a pre-defined deep learning network model based on the image to be trained and its visual features. Specifically, the preprocessed image and predefined structured prompts constitute a multimodal input pair, which is input into the semantic feature extraction module, while the preprocessed image is input separately into the visual feature extraction module.

[0210] The feature fusion module is used to fuse semantic features and visual features through a cross-modal attention mechanism to obtain multi-scale fused features. Specifically, the semantic features are dimensionally mapped and compressed to obtain semantically aligned features that are aligned with the dimensions of the visual features; the semantically aligned features and visual features are then fused together through a cross-modal attention mechanism to obtain multi-scale fused features.

[0211] The detection module is used to input multi-scale fused features into detection heads of different scales, perform abnormal scene detection, and output abnormal detection results. Specifically, multi-scale fused features are input into detection heads of different scales, and the multi-scale features are fused step by step within the detection heads to improve the detection capability for targets of different sizes; each detection head outputs structured detection results.

[0212] Furthermore, the abnormal scene detection device based on the fusion of visual and semantic features also includes a model fine-tuning module. The model fine-tuning module uses multimodal input pairs to incrementally fine-tune the large visual language model, then extracts intermediate layer features from the large visual language model, and uses them as guiding information to participate in feature fusion during the training process of the target detection model, thereby realizing feature-guided fine-tuning.

[0213] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database is used for anomaly scene detection data based on the fusion of visual and semantic features. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an anomaly scene detection method based on the fusion of visual and semantic features.

[0214] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0215] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0216] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0217] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0219] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0220] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0221] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0222] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An abnormal scene detection method based on the fusion of visual and semantic features, characterized in that, include: The acquired raw images are preprocessed to obtain preprocessed images; A multimodal input pair is formed by a preprocessed image and predefined structured prompts; The multimodal input is fed into a large visual language model, and the output is semantic features containing visual feature vectors and text vectors. The preprocessed image is input into the object detection model, which outputs visual features. The target detection model is obtained by training a preset deep learning network model based on the image to be trained and its visual features; Semantic features and visual features are fused through a cross-modal attention mechanism to obtain multi-scale fused features; The multi-scale fusion features are input into the detection heads at each scale to perform abnormal scene detection and output the abnormal detection results.

2. The abnormal scene detection method based on visual and semantic feature fusion according to claim 1, characterized in that, The multimodal input is fed into a large visual language model, and the output consists of semantic features including visual feature vectors and text vectors, specifically: Based on a large visual language model, high-dimensional visual feature vectors of multiple scales are extracted from preprocessed images of multimodal input pairs. The structured prompts for the multimodal input pairs are encoded into text vectors, and the special placeholders in the text vectors are replaced with high-dimensional visual feature vectors to obtain semantic features.

3. The abnormal scene detection method based on visual and semantic feature fusion according to claim 2, characterized in that, Based on a large visual language model, high-dimensional visual feature vectors of multiple scales are extracted from preprocessed images of multimodal input pairs, specifically including: The preprocessed image is divided into multiple non-overlapping image blocks according to a preset step size; The sequence of non-overlapping image patches is input into the 3D convolutional embedding module for embedding processing, and the initial token sequence is output. The initial token sequence is input into the visual language big model, and the output is a high-dimensional visual feature vector with unified dimensions and multiple scales. The visual language big model is based on the visual Transformer and includes multiple stacked visual blocks. Each visual block includes the following components connected in sequence: a first normalization layer, a multi-head self-attention module, a second normalization layer, and a feedforward network.

4. The abnormal scene detection method based on visual and semantic feature fusion according to claim 2, characterized in that, The structured prompts for the multimodal input pairs are encoded into text vectors, and special placeholders in the text vectors are replaced with high-dimensional visual feature vectors to obtain semantic features, specifically including: The structured prompt statement is encoded by a word segmenter to obtain a discrete Token ID sequence, which contains reserved special placeholders; Construct a mask matrix to mark the expanded portions of special placeholder positions in the Token ID sequence; High-dimensional visual feature vectors are injected into the text embedding based on a mask matrix, replacing the positions of special placeholders, and used as semantic features.

5. The abnormal scene detection method based on visual and semantic feature fusion according to any one of claims 1-4, characterized in that, Semantic and visual features are fused using a cross-modal attention mechanism to obtain multi-scale fused features, specifically including: The semantic features are dimensionally mapped and compressed to obtain semantically aligned features that are aligned with the dimensions of the visual features; Independent linear transformations are used to map visual features to a query matrix and semantically aligned features to a key matrix and a value matrix. Based on the query matrix, key matrix, and value matrix, the fused features are calculated using a scaled dot product attention mechanism. Multi-scale fused features are obtained by enhancing the fused features using a residual convolution module.

6. The abnormal scene detection method based on visual and semantic feature fusion according to claim 1, characterized in that, The multi-scale fused features are input into the detection heads at each scale to perform anomaly scene detection and output the anomaly detection results, specifically including: The multi-scale fused features are connected to the detection heads at each scale, and each detection head processes the feature information at the corresponding resolution. Within each detection head, large-scale features are passed down to small-scale branches step by step, and feature information of different scales is fused at each layer; The detection heads at various scales output structured anomaly detection results, which include the target's location, category, and corresponding confidence level.

7. An abnormal scene detection device based on the fusion of visual and semantic features, characterized in that, include: The data acquisition and processing module is used to preprocess the acquired raw images to obtain preprocessed images. The preprocessed images and predefined structured prompts constitute a multimodal input pair. The feature extraction module is used to input multimodal inputs into a large visual language model and output semantic features containing visual feature vectors and text vectors. The preprocessed image is input into the object detection model, which outputs visual features. The target detection model is obtained by training a preset deep learning network model based on the image to be trained and its visual features; The feature fusion module is used to fuse semantic features and visual features through a cross-modal attention mechanism to obtain multi-scale fused features; The detection module is used to input multi-scale fused features into the detection heads of each scale, perform abnormal scene detection, and output abnormal detection results.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the abnormal scene detection method based on the fusion of visual and semantic features as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal scene detection method based on the fusion of visual and semantic features as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the abnormal scene detection method based on the fusion of visual and semantic features as described in any one of claims 1-6.

Images