A method and system for surveillance video retrieval based on a large visual model under fuzzy description.

Abstract

This invention discloses a method and system for retrieving surveillance videos based on a large visual model under fuzzy descriptions. The method includes: Step 1, creating a fuzzy behavior description and feature dataset, and training a large text-to-text model; Step 2, setting up a text encoder to encode the text and generate text feature vectors; Step 3, preprocessing the videos, aggregating the proposed features into a KAN network, setting up a video encoder to encode several surveillance videos to form video feature vectors; Step 4, training a text-video matching model based on video-text matching pairs using a hybrid parallel dual-network parallel mode; Step 5, using the trained large text interpretation model to parse and expand the user-input text, and finding matching videos based on the new text descriptions. This invention can find corresponding videos based on user descriptions, has a wide range of applications, and good practicality and feasibility.

Description

Technical Field

[0001] This invention relates to the field of security monitoring and visual large model, and in particular to a method and system for multi-retrieval of security monitoring videos based on a video-text large model in fuzzy description scenarios. Background Technology

[0002] In modern society, as the importance of security and prevention becomes increasingly prominent, security issues have gradually become a focus of public attention. Security video surveillance, as the mainstream monitoring method today, is widely used in various locations, including city streets, train stations, airports, and shopping malls. It is an important means of ensuring public safety and an indispensable part of modern social management, playing a vital role in maintaining public order, combating crime, and responding to emergencies.

[0003] Currently, video surveillance retrieval still relies primarily on manual labor. The varied perspectives and complex content of surveillance videos make retrieval time-consuming and labor-intensive, hindering efficient risk assessment and suspect apprehension. Manual video surveillance retrieval is insufficient to meet the security demands of today's big data era. In recent years, with the continuous development of computer vision and artificial intelligence large-scale modeling technologies, image processing and video-text large-scale models have been applied to security surveillance video retrieval. However, there is no precedent for video retrieval in fuzzy scenarios. Therefore, proposing a novel, efficient method for retrieving security surveillance videos in fuzzy scenarios is urgently needed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a security monitoring video retrieval method based on a video-text large model in fuzzy scenarios, which addresses the deficiencies of the existing technology. The method is simple and practical, has good adaptability to description retrieval based on different degrees of fuzziness, and has high accuracy.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] This invention provides a method for retrieving security surveillance videos based on a video-text large model in ambiguous scenarios. The method includes the following steps:

[0007] Step 1: Create a fuzzy behavior description and feature dataset, and train a large text-to-text model to convert fuzzy descriptions into more specific and structured text information;

[0008] Step 2: First, preprocess the text information obtained in Step 1, and then set up a Transformer-based text encoder to encode the preprocessed text and generate text feature vectors.

[0009] Step 3: First, perform video frame sampling and feature extraction. Then, perform temporal and semantic aggregation based on the KAN network to obtain feature vectors representing the semantics of the entire video and extract key frames.

[0010] Step 4: Train the text-video matching model using a hybrid parallel dual-network parallel mode. The text-video matching model includes a text branch and a video branch, wherein the text branch is the Transformer-based text encoder in Step 2, and the video branch is the KAN network.

[0011] Step 5: Use the trained text-video matching model to parse and expand the user input text, and find matching videos based on the new text description.

[0012] Furthermore, the large text-to-text model described in step 1 is either the T5 or BART model.

[0013] Furthermore, in step 2, the preprocessing of the text information includes:

[0014] Step 201: Perform word segmentation on the input text, separating it into several words by spaces.

[0015] Step 202: By finding the embedding matrix of clip, each word is converted into a vector of fixed dimensions: X=[e1,e2,…,en], where ei represents the embedding vector of the i-th word;

[0016] Step 203: Use a sine function to mark the position of each word, as shown in the following formula:

[0017]

[0018] Where: i represents the position of the current word in the sentence, j represents half of the embedding dimension, d is the word embedding dimension, and 10000 is the scaling factor;

[0019] Finally, a fixed vector P representing the position is generated for each position i. This vector is added to the word embedding X to obtain the final vector:

[0020] .

[0021] Furthermore, in step 2, the specific implementation method for generating text feature vectors is as follows:

[0022] First, a multi-head self-attention mechanism is used to calculate the relationships between words, specifically through the following sub-steps:

[0023] Step 20411: For each Transformer layer, the input vector T undergoes a linear transformation to obtain the query Q, key K, and value V:

[0024]

[0025]

[0026]

[0027] in , , All are learnable parameter matrices with the shape (feature vector dimension, feature dimension of each attention head).

[0028] Step 20412, calculate the attention score:

[0029]

[0030] in, Calculate the dot product of the query and the key to measure the relevance between each pair of words; Scaling is applied to avoid gradient explosion caused by excessively large values.

[0031] Step 20413: Normalize the attention score to obtain the attention weight, and multiply the value V by the weight to obtain the final output;

[0032] Step 2042, skip connection: The input vector T is added to the vector T after the self-attention mechanism in step 2041, and after layer normalization, the vector T1 is obtained;

[0033] Step 2043: The vector T1 obtained in the previous step is further processed using a feedforward neural network (FFN). The calculation formula is as follows:

[0034]

[0035] Where x is the vector T1 obtained in step 2042, w1 and w2 are the weight matrices learned during the training process, and b1 and b2 are updated together during the training process.

[0036] Step 2044, Stack multiple Transformer layers: The Transformer encoder contains several transformer layers. After the first layer is completed, the output of the first layer is used as the input for the second layer. Repeat this process multiple times until all layers are completed.

[0037] After L layers of Transformer operations, a processed text vector sequence is obtained, and the feature vector at the end position is used as the final global text feature vector.

[0038] Furthermore, in step 3, a fixed frame rate r is used for uniform sampling, and a fixed number of frames Ns = frame rate * sampling interval are extracted per second. For each frame, the embedded features and high-level semantic information are extracted by the image encoder, i.e., the VIT model.

[0039] Furthermore, in step 3, the processing procedure of the KAN network is as follows:

[0040] First, features extracted from the video... After preprocessing and mapping to the intermediate feature space, it is represented as: ,in and These are learnable parameters;

[0041] During temporal aggregation, the inherent temporal dynamics of the video frame sequence are addressed by processing the features at each moment layer by layer through several layers of KAN modules, forming an expression of local temporal information. Specifically, the output of each layer can be written as... , where the initial input This refers to the preprocessed features, where l represents the layer number. This represents a KAN module; each layer can embed a local attention mechanism or a kernel-based weighting strategy, utilizing B-spline functions to capture fine-grained changes between adjacent frames in the local temporal domain; after processing, the outputs of each frame in the temporal branch are weighted or averaged to obtain global temporal aggregated features. ;

[0042] In the semantic aggregation process, high-level semantic information is extracted from the features of each frame by the image encoder. After processing through several layers of KAN modules, another set of features is obtained. Then, a set of independent KAN modules performs hierarchical processing, with each layer outputting the following format: , Each KAN module is represented independently, then undergoes a nonlinear transformation using a B-spline activation function. Finally, the processing results from each frame are integrated into a global semantic aggregation feature using learnable semantic attention or a weighted summation mechanism. ;

[0043] Finally, by fusing global temporal aggregated features and global semantic aggregated features using a weighted summation method, that is... Where λ∈[0,1] is a hyperparameter that adjusts the contributions of the two parts, the global video features with temporal dynamics and semantic consistency are obtained, and after normalization, the feature vector representing the semantics of the entire video is obtained. .

[0044] Furthermore, the formula for extracting keyframes in step 3 is as follows:

[0045]

[0046] in, Let d(i) represent the feature vector of the i-th frame. The frame with the largest d(i) is the keyframe.

[0047] Furthermore, the specific implementation method of step 4 is as follows:

[0048] First, calculate the global similarity, which is the similarity between the video and text vectors obtained in steps 2 and 3, using the following formula:

[0049]

[0050] Where v and t represent the global feature vectors of the video and text, respectively;

[0051] Then, local matching degree is calculated based on keyframes: the similarity between the feature vector of each keyframe i and the text description is calculated, denoted as . And assign weights to each keyframe based on similarity, as shown in the following formula:

[0052]

[0053] Weight of each keyframe The weight of a frame is automatically adjusted based on its degree of matching with the text; frames with better matching have higher weights. The total local matching degree is calculated as follows:

[0054]

[0055] The final similarity S is obtained by weighted summation of global similarity and local matching degree.

[0056] Then, the model is trained based on the idea of ​​contrastive learning, and the contrastive loss function is:

[0057]

[0058] in, Indicates the index of a positive sample pair. The index for traversing all sample pairs in the batch, where, For temperature parameters;

[0059] During training, a hybrid parallel dual-network training approach is adopted, combining data parallelism and model parallelism. The text branch and video branch are deployed on different devices to achieve parallel computation of forward propagation and independent gradient calculation on each branch. Then, a synchronous gradient reduction mechanism is used to ensure consistent parameter updates across devices. Thus, during backpropagation, the gradient generated by the contrastive loss function is simultaneously passed to the Transformer-based text encoder and KAN network. Finally, training is completed when the value of the contrastive loss function converges to a certain value.

[0060] Furthermore, the specific implementation method of step 5 is as follows:

[0061] Input the text into the text-to-text model in step 1 to obtain a detailed description of the text after semantic expansion. Then, input the detailed text description into the video-text model obtained in step 4 to obtain the video with the highest similarity, that is, the surveillance video that best matches the description.

[0062] This invention also provides a surveillance video retrieval system based on a large visual model under fuzzy description, comprising:

[0063] The system includes a processor and a memory. The memory stores program instructions, and the processor calls the stored instructions in the memory to execute a surveillance video retrieval method based on a large visual model under fuzzy description, as described in the above technical solution.

[0064] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0065] 1. Based on a video-text big data model, it retrieves surveillance videos with strong learning capabilities, unique advantages in accuracy, and wide applicability. It can retrieve different types of surveillance videos in different scenarios based on the content description text.

[0066] 2. A large text-to-text model was introduced to process user input, which can parse vague and imprecise descriptions into detailed and comprehensive descriptions, significantly improving the ability to retrieve surveillance videos in ambiguous scenarios and further enhancing the accuracy and practicality of the retrieval.

[0067] 3. Using the KAN network for video semantic enhancement and the Transformer for text semantic enhancement, and training based on the dual network, the video-text large model has high accuracy and superiority that is difficult for other methods to match. It fills the technical gap in video retrieval in this area and provides new ideas and methods for my country's security industry.

[0068] In summary, the method of the present invention is simple and practical, and has good adaptability to the retrieval of surveillance videos under fuzzy descriptions. It can find surveillance videos that meet the requirements based on inaccurate or incomplete descriptions, and has good practicality and feasibility. Attached Figure Description

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0070] Figure 1 This is a flowchart of the present invention.

[0071] Figure 2 This is a schematic diagram of the multi-head attention mechanism.

[0072] Figure 3 This is a schematic diagram illustrating the core idea of ​​the KAN network.

[0073] Figure 4 This is a schematic diagram illustrating the final effect of the present invention. Specific implementation methods

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] Fuzzy scenarios refer to situations where descriptions are unclear due to user memory gaps. These mainly include situations where adjectives are too general, specific details are missing, or abstract or emotional terms are used. Video-text large-scale models are a class of large-scale deep learning models that jointly model textual and visual information using multimodal learning techniques. They can extract semantic features from natural language descriptions and align them with the feature space of visual content, thereby achieving cross-modal understanding, generation, and retrieval tasks. These models are typically pre-trained on large-scale multimodal data and employ dual-encoder or joint-encoder architectures, significantly improving performance in scenarios such as image-text alignment, inter-modal retrieval, content classification, and intelligent parsing.

[0076] Accordingly, this invention develops a technology for retrieving security surveillance videos in fuzzy scenarios based on a video-text large model. This technology can quickly and accurately find the required surveillance video through fuzzy content descriptions when users have missing memories or are unable to accurately describe the content of the video. This is of great significance to social security and provides strong support for maintaining public order and combating crime.

[0077] like Figure 1 As shown, the video retrieval method based on a video-text large model in fuzzy description scenarios according to an embodiment of the present invention includes the following steps:

[0078] Step 1: Create a fuzzy behavior description and feature dataset, and train a large-scale text-to-text model. To achieve intelligent parsing of fuzzy descriptions, we first need to build a behavior description dataset covering different scenarios and train it using video scene features. This dataset contains various possible user input texts, such as "someone is loitering" or "a suspicious person is approaching a restricted area," and we also establish corresponding standardized behavior descriptions to improve the accuracy of text parsing. We use data augmentation methods to expand the multiple expressions of the same behavior to adapt to the different expression habits of users. In addition, we build a large-scale text-to-text model based on natural language processing technology, which can understand and transform fuzzy descriptions into more specific and structured text information as input data for subsequent video retrieval.

[0079] Step 2: Set up a text encoder to encode the text and generate text feature vectors. After training, we use the pre-trained text encoder to vectorize the user-input descriptive text. The text encoder converts the input text into a high-dimensional feature vector for subsequent calculation of its similarity to the video content. In this process, we employ deep learning models, such as Transformer or BERT variants, to enhance the semantic understanding capabilities of the text encoding. Furthermore, we normalize the text encoding results to ensure consistent feature representations for text descriptions of different lengths during the matching process, thereby improving matching accuracy.

[0080] Step 3: Set up a video encoder to encode several surveillance videos. While completing the text encoding, we need to extract features from the surveillance video data for comparison with text features. We use a pre-trained video encoding model, such as one based on 3D CNN or spatiotemporal Transformer architecture, to perform frame-by-frame analysis of the input video and extract keyframe features. During feature extraction, we normalize the features for videos with different frame rates and resolutions to ensure feature stability. After encoding, each video segment is mapped to a high-dimensional feature vector, which is then compared with the text vector in the same feature space.

[0081] Step 4: Train a text-video matching model using video-text matching pairs and contrastive learning. To achieve efficient text-video matching, we construct a training dataset using text-video matching pairs and optimize the matching model using contrastive learning. During training, we find the corresponding video segment for each text description and introduce negative sample contrastive loss to help the model learn to distinguish between relevant and irrelevant videos. During training, we introduce a self-supervised learning strategy, utilizing unlabeled data to further improve the model's generalization ability, thus achieving accurate matching even in complex scenarios.

[0082] Step 5: Use the trained text interpretation model to parse and expand the user's input text, and find matching videos based on the new text description. After the model training is complete, we use the trained text interpretation model to parse the fuzzy description of the user's input and automatically expand its semantic range. For example, when the user inputs "someone is walking strangely," the system will parse it as "someone is loitering in the same place for a long time" or "someone is repeatedly entering and leaving a certain area," etc., for more specific descriptions. Then, based on the expanded text description, we use a video-text matching model to retrieve relevant segments from the surveillance video library. The search results are sorted by similarity and multiple candidate videos are provided for users or security personnel to verify.

[0083] The specific process of the above-mentioned video-text large model-based surveillance video retrieval method in fuzzy description scenarios is as follows:

[0084] Step 101, Behavior Description and Feature Collection: Collect possible target behaviors (such as "running"), appearance features (such as "wearing red clothes"), and scene features (such as "dark environment") in security monitoring to generate a description sample set {d1,d2,...,dn}.

[0085] Step 102, Semantic fuzzification: Perform semantic fuzzification on the description samples (e.g., expand "running" to "moving quickly" and "sneaking" to "moving repeatedly near a location") to form a preliminary fuzzy behavior description dataset D={t1,t2,…,tn}.

[0086] Step 103: Train the text generation model: Use the constructed dataset to train the generative text model (such as T5 or BART) so that it can generate more accurate and diverse descriptions from vague descriptions.

[0087] The specific method in step 2 is as follows:

[0088] Step 201: Perform word segmentation on the input text, separating it into several words by spaces.

[0089] Step 202: By finding the embedding matrix of clip, convert each word into a fixed-dimensional vector: X=[e1,e2,…,en], where ei represents the embedding vector of the i-th word.

[0090] Step 203: Use a sine function to mark the position of each word, as shown in the formula below:

[0091]

[0092] Where: i represents the position of the current word in the sentence, j represents half of the embedding dimension, d is the word embedding dimension, and 10000 is the scaling factor to ensure that the frequency changes of different dimensions are reasonable.

[0093] Finally, a fixed vector P representing the position is generated for each position i. This vector is added to the word embedding X to obtain the final vector:

[0094]

[0095] Step 204: Calculate global features using a multi-layer Transformer encoder:

[0096] Step 2041, as follows Figure 2 As shown, a multi-head self-attention mechanism is used to calculate the correlation between words:

[0097] Step 20411: For each Transformer layer, the input vector T undergoes a linear transformation to obtain the query (Q), key (K), and value (V):

[0098]

[0099]

[0100]

[0101] in , , All are learnable parameter matrices with the shape (feature vector dimension, feature dimension of each attention head).

[0102] Step 20412: Calculate the attention score:

[0103]

[0104] in, Calculate the dot product of the query and the key to measure the relevance between each pair of words; Scaling is applied to avoid gradient explosion caused by excessively large values.

[0105] Step 20413: Normalize the attention score to obtain the attention weight. Multiply the value V by the weight to obtain the final output.

[0106] Step 2042, Skip Connection: Add the input vector T to the vector T processed by the self-attention mechanism in step 2041, and then perform layer normalization to obtain vector T. 1 .

[0107] Step 2043: Use a feedforward neural network (FFN) to process the vector T obtained in the above steps. 1 Further processing is required; the core calculation formula is as follows:

[0108]

[0109] Here, x is the vector obtained in the previous step, and w1 and w2 are the weight matrices learned during training. b1 and b2 are usually updated together during training to adjust the network output.

[0110] Step 2044: Stack multiple Transformer layers: The Transformer encoder contains several Transformer layers. After the first layer is completed, the output of the first layer is used as the input for the second layer. Repeat the above steps until all layers are completed.

[0111] Step 205: Extract global text features: After processing by L layers of Transformer operations, a processed text vector sequence is obtained. The feature vector at the end position is used as the final global text feature vector.

[0112] The specific method in step 3 is as follows:

[0113] Step 301, Video Frame Sampling and Feature Extraction: Divide the surveillance video into several frame sequences, and use a fixed frame rate r for uniform sampling, extracting a fixed number of frames per second, Ns = frame rate * sampling interval. For each frame, extract its embedded features and high-level semantic information using an image encoder (VIT model).

[0114] Step 302: In order to generate feature vectors that can represent the semantics of the entire video Based on the embedding features of KAN for all frames Perform temporal modeling aggregation and semantic enhancement:

[0115] First, the frame embedding features extracted from the video... After preprocessing and mapping to the intermediate feature space, it can be formally represented as: ,in and These are learnable parameters. The KAN network utilizes the theoretical foundation of the Kolmogorov–Arnold representation theorem, representing a multivariable continuous function as a combination of a finite number of univariate functions and addition operations. Therefore, in KAN, each layer consists of univariate functions, which are typically parameterized using B-spline activation functions, allowing switching between coarse-grained and fine-grained grids to achieve flexible approximation of input features, such as... Figure 3 As shown, the left figure shows the activation symbols flowing through the network; the right figure shows the activation function parameterized as a B-spline, which can switch between coarse-grained and fine-grained grids.

[0116] Based on the above principles, during temporal aggregation, considering the inherent temporal dynamics of the video frame sequence, several layers of KAN modules (denoted as...) are used. The features at each time step are processed layer by layer to form a representation of local temporal information. Specifically, the output of each layer can be written as... , where the initial input This refers to the preprocessed features. Since video frames are sequentially continuous, each layer can embed a local attention mechanism or a kernel-based weighting strategy to capture fine-grained changes between adjacent frames in the local temporal domain using B-spline functions. After processing, the outputs of each frame in the temporal branch are appropriately weighted or averaged to obtain the global temporal aggregated features. .

[0117] Meanwhile, during the semantic aggregation process, the high-level semantic information of each frame's features extracted by the image encoder is processed in the same way as described above to obtain another set of features. Then, a set of independent KAN modules (denoted as...) The process is hierarchical, and the output of each level is as follows: This branch focuses on extracting semantic features within frames. It performs a nonlinear transformation on the input using a B-spline activation function and finally integrates the processing results of each frame into global semantic features using learnable semantic attention or a weighted summation mechanism. .

[0118] Finally, by fusing temporal aggregated features and semantic aggregated features using a weighted summation method, that is... Where λ∈[0,1] is a hyperparameter that adjusts the contributions of the two parts, a global video feature that comprehensively considers temporal dynamics and semantic consistency is obtained. To facilitate distance calculation or classification in subsequent tasks, the fused features need to be normalized, and the final output is the normalized video feature. .

[0119] This KAN-based bi-branch aggregation method not only theoretically guarantees the ability to approximate multivariable continuous functions based on the Kolmogorov–Arnold theorem, but also, in practice, utilizes the B-spline activation function to switch between coarse and fine grids, giving the model greater flexibility and expressive power, thereby effectively improving the discriminativeness and robustness of video feature representation.

[0120] Step 303: Extract keyframes:

[0121] Using a 5-frame sliding window, for each frame from frame 2 to frame 5 within the window, the cosine distance from each frame to the previous frame is calculated as a standard to measure the difference between frames, as shown in the following formula:

[0122]

[0123] in, Let represent the feature vector of the i-th frame, where the numerator represents the difference between the feature vectors of the two frames, and the denominator is the product of the magnitudes of the respective vectors.

[0124] Within the window, the frame with the largest d(i) is the keyframe that changes the most and best represents the window.

[0125] The specific method in step 4 is as follows:

[0126] First, calculate the global similarity, which is the similarity between the video and text vectors obtained in steps 2 and 3, using the following formula:

[0127]

[0128] Here, v and t represent the global feature vectors of the video and text, respectively.

[0129] Then, local matching degree is calculated based on keyframes. The similarity between the feature vector and the text vector corresponding to each keyframe i is calculated and denoted as . And assign weights to each keyframe based on similarity, as shown in the following formula:

[0130]

[0131] This gives the weight of each keyframe. It will automatically adjust based on how well it matches the text; the better the match, the greater the weight of the frame.

[0132] Calculate the local total matching degree:

[0133]

[0134] The final similarity is obtained by weighted summation of global similarity and local matching degree. . After multiple experiments, the value was set at 0.3, which can be flexibly adjusted according to actual task requirements.

[0135] Then, the model is trained based on the idea of ​​contrastive learning, and the contrastive loss function is:

[0136]

[0137] in, i This represents the index of the current positive sample pair (the matched video-text pair). In the loss function, each... S i The similarity corresponding to a positive sample pair. j This is the index for traversing all sample pairs in the batch (including positive and negative sample pairs).

[0138] Temperature is a parameter used to adjust the smoothness of the distribution, which helps to control the discriminative power of similarity between different samples during the comparison process, thereby making the similarity of positive sample pairs more significant than that of negative sample pairs.

[0139] During training, a hybrid parallel dual-network training approach is adopted, combining data parallelism and model parallelism. The text branch and video branch are deployed on different devices to achieve parallel computation during forward propagation, and gradients are calculated independently on each branch. Subsequently, a synchronous gradient reduction mechanism is used to ensure consistent parameter updates across devices. Thus, during backpropagation, the gradients generated by the contrastive loss function are simultaneously passed to the Transformer encoder and the KAN network. Specifically, the normalized feature vectors output by the two branches are used to construct positive and negative sample pairs through cosine similarity calculation. The contrastive loss function drives the similarity of matched video-text pairs to approach 1, and the similarity of unmatched pairs to approach 0. The value of the contrastive loss function converges to a certain minimum value. During the dynamic update process using optimizers such as Adam or SGD combined with learning rate scheduling strategies, the model encoder and weighting parameters are continuously adjusted to achieve accurate cross-modal semantic alignment and continuous optimization of the overall training effect.

[0140] The specific method in step 5 is as follows:

[0141] Input the text into the text model obtained in step 1 to obtain a detailed description of the text after semantic expansion. Then input the detailed text description into the video-text model obtained in step 4 to obtain the video with the highest similarity, that is, the surveillance video that best matches the description.

[0142] To verify the effectiveness of the security surveillance video retrieval method based on a large video-text model proposed in this invention, extensive experimental evaluations were conducted. The experiments used Recall@1, Recall@5, and Recall@10 (representing the number of queries that contain the correct answer in the first, fifth, and tenth returned results, respectively) and mean AP as the main performance indicators, while also measuring the system's average query response latency. The dataset used in the experiments included surveillance videos with various scenarios and corresponding fuzzy user descriptions.

[0143] In the comparative experiment, the following two schemes were established:

[0144] Baseline approach: Employs global video feature extraction and directly matches fuzzy text input by the user. This method relies solely on the overall semantic representation of the video and the original fuzzy text description, failing to further supplement or optimize the text semantics. Specific data is as follows:

[0145] Recall@1: 42.0%

[0146] Recall@5: 65.0%

[0147] Recall@10: 75.0%

[0148] Average accuracy (mAP): 38.0%

[0149] Average query response time: approximately 90 milliseconds

[0150] The present invention proposes the following approach: First, a large generative model is used to semantically supplement the ambiguous text input by the user, making the text description more detailed and richer. Then, global video features and local features extracted from keyframes are combined through weighted fusion to achieve matching. This method not only preserves the overall semantic information of the video but also utilizes local details to further improve the accuracy of matching. Specific data is as follows:

[0151] Recall@1: 60.0%

[0152] Recall@5: 80.0%

[0153] Recall@10: 88.0%

[0154] Average accuracy (mAP): 52.0%

[0155] Average query response latency: approximately 100 milliseconds

[0156] As can be seen from the above data, compared with the baseline solution, the present invention improves Recall@1 by 18 percentage points (approximately a 43% relative improvement) and mAP by 14 percentage points (approximately a 37% relative improvement), while the query response time only increases slightly by about 10 milliseconds, and the impact on the online real-time retrieval system is negligible.

[0157] These experimental data fully demonstrate that the present invention significantly enhances the video retrieval performance under fuzzy descriptions by using a large generative model to supplement text descriptions and combining it with a global and local weighted fusion matching method, thereby improving the system's recall and accuracy while maintaining a high response speed.

[0158] On the other hand, embodiments of the present invention also provide a surveillance video retrieval system based on a large visual model under fuzzy description, comprising:

[0159] The system includes a processor and a memory. The memory stores program instructions, and the processor calls the stored instructions in the memory to execute a surveillance video retrieval method based on a large visual model under fuzzy description, as described in the above technical solution.

[0160] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for retrieving surveillance videos based on a large visual model under fuzzy description, characterized in that, Includes the following steps: Step 1: Create a fuzzy behavior description and feature dataset, and train a large text-to-text model to convert fuzzy descriptions into more specific and structured text information; Step 2: First, preprocess the text information obtained in Step 1, and then set up a Transformer-based text encoder to encode the preprocessed text and generate text feature vectors. Step 3: First, perform video frame sampling and feature extraction. Then, perform temporal and semantic aggregation based on the KAN network to obtain feature vectors representing the semantics of the entire video and extract key frames. In step 3, the processing procedure of the KAN network is as follows: First, features extracted from the video... After preprocessing and mapping to the intermediate feature space, it is represented as: ,in and These are learnable parameters; During temporal aggregation, the inherent temporal dynamics of the video frame sequence are addressed by processing the features at each moment layer by layer through several layers of KAN modules, forming an expression of local temporal information. Specifically, the output of each layer can be written as... , where the initial input This refers to the preprocessed features, where l represents the layer number. This represents a KAN module; each layer can embed a local attention mechanism or a kernel-based weighting strategy, utilizing B-spline functions to capture fine-grained changes between adjacent frames in the local temporal domain; after processing, the outputs of each frame in the temporal branch are weighted or averaged to obtain global temporal aggregated features. ; In the semantic aggregation process, high-level semantic information is extracted from the features of each frame by the image encoder. After processing through several layers of KAN modules, another set of features is obtained. Then, a set of independent KAN modules performs hierarchical processing, with each layer outputting the following format: , Each KAN module is represented independently, then undergoes a nonlinear transformation using a B-spline activation function. Finally, the processing results from each frame are integrated into a global semantic aggregation feature using learnable semantic attention or a weighted summation mechanism. ; Finally, by fusing global temporal aggregated features and global semantic aggregated features using a weighted summation method, that is... Where λ∈[0,1] is a hyperparameter that adjusts the contributions of the two parts, the global video features with temporal dynamics and semantic consistency are obtained, and after normalization, the feature vector representing the semantics of the entire video is obtained. ; Step 4: Train the text-video matching model using a hybrid parallel dual-network parallel mode. The text-video matching model includes a text branch and a video branch, wherein the text branch is the Transformer-based text encoder in Step 2, and the video branch is the KAN network. Step 5: Use the trained text-video matching model to parse and expand the user input text, and find matching videos based on the new text description.

2. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: The large text-to-text model mentioned in step 1 is either the T5 or BART model.

3. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: Step 2, the preprocessing of the text information includes: Step 201: Perform word segmentation on the input text, separating it into several words by spaces. Step 202: By finding the embedding matrix of clip, each word is converted into a vector of fixed dimensions: X=[e1,e2,…,en], where ei represents the embedding vector of the i-th word; Step 203: Use a sine function to mark the position of each word, as shown in the following formula: Where: i represents the position of the current word in the sentence, j represents half of the embedding dimension, d is the word embedding dimension, and 10000 is the scaling factor; Finally, a fixed vector P representing the position is generated for each position i. This vector is added to the word embedding X to obtain the final vector: 。 4. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: In step 2, the specific implementation method for generating text feature vectors is as follows: First, a multi-head self-attention mechanism is used to calculate the relationships between words, specifically through the following sub-steps: Step 20411: For each Transformer layer, the input vector T undergoes a linear transformation to obtain the query Q, key K, and value V: in, All are learnable parameter matrices with the shape (feature vector dimension, feature dimension of each attention head). Step 20412, calculate the attention score: in, It calculates the dot product of the query and the key, used to measure the relevance between each pair of words; it employs... Scaling is applied to prevent gradient explosion caused by excessively large values; Step 20413: Normalize the attention score to obtain the attention weight, and multiply the value V by the weight to obtain the final output; Step 2042, Skip Connection: Add the input vector T to the vector T processed by the self-attention mechanism in step 2041, and then normalize the result to obtain vector T. 1 ; Step 2043: Use the feedforward neural network FFN to process the vector T obtained in the previous step. 1 Further processing is required; the calculation formula is as follows: Where x is the vector T obtained in step 2042 1 w1 and w2 are weight matrices learned during training, and b1 and b2 are updated together during training. Step 2044, Stack multiple Transformer layers: The Transformer encoder contains several transformer layers. After the first layer is completed, the output of the first layer is used as the input for the second layer. Repeat this process multiple times until all layers are completed. After L layers of Transformer operations, a processed text vector sequence is obtained, and the feature vector at the end position is used as the final global text feature vector.

5. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: In step 3, a fixed frame rate r is used for uniform sampling, and a fixed number of frames Ns = frame rate * sampling interval are extracted per second. For each frame, the embedded features and high-level semantic information are extracted by the image encoder, i.e., the VIT model.

6. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: The formula for extracting keyframes in step 3 is: in, Let d(i) represent the feature vector of the i-th frame. The frame with the largest d(i) is the keyframe.

7. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: The specific implementation method of step 4 is as follows: First, calculate the global similarity, which is the similarity between the video and text vectors obtained in steps 2 and 3, using the following formula: Where v and t represent the global feature vectors of the video and text, respectively; Then, local matching degree is calculated based on keyframes: the similarity between the feature vector of each keyframe i and the text description is calculated, denoted as . And assign weights to each keyframe based on similarity, as shown in the following formula: Weight of each keyframe The weight of a frame is automatically adjusted based on its degree of matching with the text; frames with better matching have higher weights. The total local matching degree is calculated as follows: The final similarity S is obtained by weighted summation of global similarity and local matching degree. Then, the model is trained based on the idea of ​​contrastive learning, and the contrastive loss function is: in, Indicates the index of a positive sample pair. The index for traversing all sample pairs in the batch, where, For temperature parameters; During training, a hybrid parallel dual-network training approach is adopted, combining data parallelism and model parallelism. The text branch and video branch are deployed on different devices to achieve parallel computation of forward propagation and independent gradient calculation on each branch. Then, a synchronous gradient reduction mechanism is used to ensure consistent parameter updates across devices. Thus, during backpropagation, the gradient generated by the contrastive loss function is simultaneously passed to the Transformer-based text encoder and the KAN network. Finally, training is completed when the value of the contrastive loss function converges to a certain value.

8. The surveillance video retrieval method based on a large visual model under fuzzy description as described in claim 1, characterized in that: The specific implementation method of step 5 is as follows: Input the text into the text-to-text model in step 1 to obtain a detailed description of the text after semantic expansion. Then, input the detailed text description into the video-text model obtained in step 4 to obtain the video with the highest similarity, that is, the surveillance video that best matches the description.

9. A surveillance video retrieval system based on a large visual model under fuzzy description, characterized in that, include: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the surveillance video retrieval method based on a visual large model under fuzzy description as described in any one of claims 1-8.

Images