A video analysis processing system and method based on a multi-modal large model

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a video analysis processing system and method based on a multi-modal large model, the method comprising: a multi-modal control large model receiving a task required to be completed by natural language description input by a user, performing semantic analysis on the input of the user, extracting task information, decomposing a task related to video understanding into subtasks and sending the subtasks to a video understanding module; the video understanding module receiving a video or a video segment required by the task, performing analysis on the video by executing the subtasks, generating embedding and description of video content and inputting the multi-modal control large model; when the multi-modal control large model receives a user query request, the query task of the user is parsed into executable subtasks and mapped to the task of video understanding, and a response result based on the query request is generated based on the embedding and description of the video content generated by the video understanding large model. Thus, effective fusion and mutual enhancement of cross-modal information are realized, and the precision of video understanding is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a video analysis processing system and method based on a multi-modal large model. BACKGROUND

[0002] With the vigorous development of large model technology, natural language processing (NLP) and computer vision (CV) fields have made great progress, which has promoted the ability of intelligent systems in semantic understanding and multi-modal fusion to be significantly improved. However, in the field of video understanding, due to the rich complex spatio-temporal information of video data, its processing difficulty is far beyond that of single modal data, which has become a very challenging task. Traditional video processing methods often separate video analysis and language processing, ignoring the close relationship and potential mutual enhancement effect between the two.

[0003] In the smart park scene, the traditional video monitoring system highly depends on human resources for monitoring and abnormal behavior recognition, which not only leads to high cost expenditure, but also easily causes misjudgment and omission problems, seriously affecting the monitoring efficiency and safety. Therefore, how to quickly and accurately identify abnormal behaviors in the intelligent security system and improve the intelligent security capability of the park has become a key problem to be solved for the current video monitoring system. SUMMARY

[0004] The existing video analysis processing usually separates video processing and language processing, lacks the close relationship and mutual enhancement capability of cross-modal, and the present application provides a video analysis processing system and method based on a multi-modal large model.

[0005] In the first aspect, the technical scheme of the present application provides a video analysis processing system based on a multi-modal large model, which comprises a multi-modal control large model and a video understanding module.

[0006] The multi-modal control large model receives the task required to be completed by the user input through natural language description, performs semantic analysis on the user's input, extracts task information, and decomposes the task involving video understanding into subtasks and sends them to the video understanding module.

[0007] The video understanding module receives the video or video segment required by the task, performs analysis on the video by executing the subtask, and generates the embedding and description of the video content and inputs the multi-modal control large model.

[0008] When the multi-modal control large model receives the user query request, it parses the user's query task into executable subtasks and maps them to the task of video understanding, and generates the response result based on the query request based on the embedding and description of the video content generated by the video understanding large model.

[0009] As a further limitation of the technical solution of the application, the video understanding module comprises a video analyzer and a video embedder;

[0010] The video analyzer receives video input and outputs the analysis result of the video in text form to the multi-modal control large model; the text includes video subtitles, detailed descriptions of all events in the video and timestamps, labels, IDs and bounding boxes;

[0011] The video embedder uses a trained CLIP model to extract video features from the input video, encodes the corresponding text description into text features through the text encoder in the CLIP model to generate an embedding vector input into the multi-modal control large model.

[0012] As a further limitation of the technical solution of the application, the video analyzer decodes the input video, converts it from a compressed format to a sequence of original image frames, extracts consecutive image frames from the decoded video, extracts image features of each image frame, and extracts video features of the video. Analyze the extracted features, detect key events related to the task in the video, and generate detailed descriptions of the events when the events are detected; based on the results of event detection, the video analyzer generates text containing video subtitles, event descriptions, timestamps, labels, IDs and bounding box information.

[0013] As a further limitation of the technical solution of the application, the CLIP model is provided with a visual encoder, and the CLIP model training includes random masking of input video frames, allowing the model to reconstruct the masked part of the mask reconstruction training; specifically including: decomposing the video into frames and further into patches, each patch representing a token; randomly selecting a set proportion of tokens for masking; that is, a mask; inputting the unmasked tokens into the visual encoder of the CLIP model to generate feature embeddings of the tokens;

[0014] The unmasked token feature embeddings are aggregated into aggregated features as reference information for the model to predict the masked tokens; the aggregated features and position encodings are input into a reconstruction network to predict the embeddings of the masked tokens; the reconstruction error of the masked tokens is measured using a mean square error loss ; calculate the spatio-temporal smoothing loss ; calculate the cosine similarity of the reconstructed tokens and the unmasked tokens to calculate a contrastive loss function for distinguishing positive sample pairs and negative sample pairs ; calculate a loss function for measuring the difference between the generated image and the real image Generate a joint loss by weighted combination , the training process minimizes the joint loss;

[0015]

[0016] wherein is a weight coefficient.

[0017] As a further limitation of the technical scheme of the application, the CLIP model is provided with a text encoder, and during model training, the video features and the text features are aligned in a common semantic space, and the video features and the text features are calculated, and contrast learning is performed to update the parameters of the video encoder and the text encoder in the CLIP model;

[0018] The cross-modal contrast loss function is:

[0019]

[0020] wherein represents the similarity between the th video feature and the th text feature, is a hyperparameter for adjusting the smoothness of the distribution.

[0021] As a further limitation of the technical scheme of the application, the video embedder obtains a long video sequence , and according to the time label, the original video is divided into several video segments; using the trained CLIP model, the video features of each video are extracted, and the corresponding text description is encoded into text features by the text encoder.

[0022] As a further limitation of the technical scheme of the application, when the multi-modal control large model responds to a user query, for each sub-task , according to the task type parsed from the query, the video embed and the text description are used to calculate the similarity between the user's query embed and the video embed, and if the similarity exceeds a set threshold, it is considered that the video segment is related to the query; combining the query parsing result, the key segment in the video is located according to the time stamp, i.e. the event time period that meets the user's demand, and after the relevant video segment is found, the key frame or time node with the required is extracted therefrom for intuitive display of the video content.

[0023] As a further limitation of the technical scheme of the application, the key frame extraction formula is as follows:

[0024]

[0025] wherein representing key frames correlation measure with the query .

[0026] As a further limitation of the technical solution of the application, during the multi-round interaction process between the user and the multi-modal control large model based on the same query, the multi-modal control large model will generate a new sub-task according to the user's additional query and pass it to the video understanding module; the multi-modal control large model will generate a refined response based on the response data of the video understanding module.

[0027] In a second aspect, the technical solution of the application provides a video analysis processing method based on a multi-modal large model, comprising:

[0028] The multi-modal control large model receives the task required to be completed by the user input through natural language description, performs semantic analysis on the user's input, extracts task information, and decomposes the task involving video understanding into sub-tasks and sends them to the video understanding module;

[0029] The video understanding module receives the video or video segment required by the task, performs analysis on the video according to the sub-task, and generates the embedding and description of the video content and inputs the multi-modal control large model;

[0030] When the multi-modal control large model receives the user query request, it parses the user's query task into executable sub-tasks and maps them to the task of video understanding, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding large model.

[0031] As can be seen from the above technical solution, the application has the following advantages: the application realizes the close integration and mutual enhancement between video data and language description through the multi-modal control large model and the video understanding module. The multi-modal control large model can accurately understand the task requirements described by the user through natural language and decompose them into specific sub-tasks and send them to the video understanding module for processing. The video understanding module can generate accurate embedding and description based on video data and further feedback to the multi-modal control large model, thereby realizing effective fusion and mutual enhancement of cross-modal information. With the powerful ability of the multi-modal large model, the application can more accurately understand the video content, identify abnormal behavior, and generate accurate description and response results. This not only significantly improves the accuracy of video understanding, but also greatly improves the processing efficiency, making the security monitoring of the smart park more intelligent and efficient.

[0032] Traditional video monitoring systems rely heavily on human resources for monitoring and identification, which not only has high costs, but also is prone to misjudgment and missed judgment. The present application introduces a multi-modal large model for automatic processing, effectively reducing labor costs and significantly reducing misjudgment and missed judgment, improving the overall performance and security of the monitoring system. The multi-modal control large model and video understanding module of the present application have high scalability and flexibility, and can be customized and optimized according to different scene and task requirements. This makes the system better adapt to the complex and variable security environment of the smart park, and provides users with more intelligent, convenient and efficient security services. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0034] Figure 1 is a schematic block diagram of a system of an embodiment of the present application.

[0035] Figure 2 is a schematic flow chart of a method of an embodiment of the present application. DETAILED DESCRIPTION

[0036] The system integrates a multi-modal control large model (ControlLLM) and a video understanding module (Video Model) to realize user understanding of video content, answer and execution of related actions. The core of the system is to identify user intent through ControlLLM, and distribute related tasks involving video understanding, analysis and processing to the video understanding large model to perform specific operations. The video understanding module realizes video content analysis and embedding based on the Video-Embedder of ClIP, providing more rich multi-modal input for the large model. This method combines the advantages of video embedder and video analyzer, can realize fine-grained semantic understanding and high-quality video representation, enhance the semantic alignment ability between video and text, so as to effectively deal with complex multi-modal queries and execute corresponding actions, such as video description, video question answering (VQA), video positioning and content understanding.

[0037] In order to enable a person skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative labor should fall within the scope of protection of the present application.

[0038] As Figure 1 shown, the embodiment of the present application provides a video analysis processing system based on a multi-modal large model, which includes a multi-modal control large model and a video understanding module.

[0039] The multi-modal control large model receives a task required to be completed by natural language description input by a user, performs semantic analysis on the input of the user, extracts task information, and sends a task involving video understanding to the video understanding module.

[0040] ControlLLM first performs the task decomposition work. The user request is decomposed into a series of parallel subtasks . This process can be implemented by a large language model .

[0041] (1)

[0042] wherein, represents the th subtask, represents the number of all subtasks. Through task division of ControlLLM, tasks involving video understanding, analysis and processing are identified , and this part of the task is extracted and assigned to a specific video understanding module.

[0043] The video understanding module receives a video or a video segment required by the task, performs a subtask to analyze the video, and generates an embedding and a description of the video content input to the multi-modal control large model.

[0044] When the multi-modal control large model receives a user query request, the query task of the user is parsed into executable subtasks mapped to the task of video understanding, and a response result based on the query request is generated based on the embedding and the description of the video content generated by the video understanding large model.

[0045] It should be noted that the video understanding module includes a video analyzer and a video embedder.

[0046] ​The video analyzer receives video input and outputs the analysis results of the video in text form to the multimodal control large model; the text includes video captions, detailed descriptions and timestamps of all events in the video, labels, IDs and bounding boxes;

[0047] Specifically, the video analyzer accepts video input and outputs the analysis of the video, usually in text form. The above-mentioned text includes video captions, dense video captions (detailed descriptions and timestamps of all events in the video), object tracking results (labels, IDs and bounding boxes), and transcriptions of other modalities in the video (speech recognition results or caption recognition results). The text generated by the video analyzer can be directly input into the subsequent ControlLLM, or inserted into a pre-prepared template and then input into the ControlLLM, or converted into a temporary database format for subsequent retrieval by the ControlLLM.

[0048] The ControlLLM is mainly responsible for the overall coordination of the system, and generates commands to call various video analyzers to obtain the required analysis results. The ControlLLM can also further process the obtained analysis results after calling the video analyzer, and feed back the final results to the user to ensure that the user gets the accurate information required. In addition, the ControlLLM has the ability of multi-round interaction, which can continuously communicate with the video analyzer to gradually obtain and process information in complex tasks, and meet higher-level analysis requirements.

[0049] The video embedder uses a trained CLIP model to extract video features from the input video, and encodes the corresponding text description into text features through the text encoder in the CLIP model to generate embedding vectors input into the multimodal control large model.

[0050] The video embedder, i.e. the visual backbone, uses the CLIP model in this patent to convert the input video into vector representation, i.e. to generate video embedding. In this application, ControlLLM acts as a text decoder, receiving embedding vectors from the video embedder as input, and decoding them into text output according to prompts or instructions. These tasks usually do not require fine-grained understanding or precise spatio-temporal localization. Using the text analysis results provided by the video analyzer and the visual feature representation generated by the video embedder, ControlLLM provides more rich input information. ControlLLM can perform various tasks such as summarizing video content, answering questions, generating descriptive text, etc. according to these inputs.

[0051] CLIP can provide strong visual semantic representation by aligning the embedding space of images and texts. To enhance the spatio-temporal perception ability of CLIP, the technology introduces a mask reconstruction strategy in VideoMAE. Specifically, random masks are applied to the input video frames, and then the model is asked to reconstruct the masked parts. Through the above way, the model is forced to learn the spatio-temporal structure of the video to accurately reconstruct the missing information.

[0052] 1. Video pre-processing and spatio-temporal masking

[0053] First, the video is decomposed into frames and further divided into patches, each patch representing a token. A masking strategy is used to select part of the tokens for covering.

[0054] Suppose the video has frames, each frame is divided into small patches (patches), each patch represents a token. Then the video can be represented as:

[0055] (2)

[0056] where represents the th patch of the th frame.

[0057] Randomly select 90-95% of the tokens for masking, i.e. cover them so that they are invisible. Define the mask matrix :

[0058] (3)

[0059] Uncovered tokens will serve as reference information, and covered tokens will serve as reconstruction targets.

[0060] 2. Generate embeddings of unmasked tokens using CLIP visual encoder

[0061] Input the unmasked tokens into the visual encoder of CLIP to generate the feature embeddings of these tokens. Denote the CLIP encoder as :

[0062] (4)

[0063] If , generate the feature embedding , otherwise is empty.

[0064] The unmasked tokens embeddings are aggregated into a contextual feature as a reference for the model to predict the masked tokens:

[0065] (5)

[0066] 3. Reconstructing network to predict the masked tokens

[0067] A reconstructing network (lightweight network) is used to predict the embeddings of the masked tokens. The input is the aggregated feature and the position encoding, and the output is the representation of the masked tokens.

[0068] (6)

[0069] where is the position information of the th position in the th frame. The lightweight prediction network is represented by

[0070] 4. Loss function design

[0071] Mean Squared Error (MSE) loss is used to measure the reconstruction error of the masked tokens.

[0072] (7)

[0073] where is the total number of masked tokens.

[0074] To ensure that the reconstructed tokens maintain spatial and temporal consistency, a temporal and spatial smoothing loss is added.

[0075] (8)

[0076] is a loss function used to measure the amount of change in features or states between adjacent time points in time series data. This loss function is commonly used in tasks such as processing time series data, video analysis, motion estimation, etc. to evaluate the ability of the model to capture the continuity or smoothness of data over time.

[0077] is a normalization factor to ensure that the loss value is not affected by the length of the time series. T represents the total number of time points in the time series, so T-1 represents the number of adjacent time point pairs.

[0078] denotes the sum over all pairs of adjacent time points. Here, t denotes the index of time points in the time series. denotes the sum over all feature or state indices. Here, p can denote a spatial position, a channel index, or other indices used to distinguish different features or states.

[0079] denotes the squared Euclidean distance of the feature or state change between adjacent time points t and t+1 at feature or state index p. This distance is used to measure the continuity or smoothness of the feature or state between adjacent time points.

[0080] In summary, The loss function measures the continuity or smoothness of the time series data by computing the Euclidean distance of the feature or state change between adjacent time points.

[0081] The cosine similarity between the reconstructed tokens and the unmasked tokens is computed using the unmasked tokens as positive samples and the masked tokens as negative samples, encouraging the model to distinguish between masked and unmasked tokens.

[0082] (9)

[0083] where denotes the cosine similarity, is a hyperparameter.

[0084] is a contrastive loss,

[0085] denotes the sum over all negative sample pairs . Here, is a mask or indicator that = 0 indicates is a negative sample pair (i.e., a dissimilar or mismatched pair).

[0086] The logarithmic function is used to measure the model's ability to distinguish negative sample pairs. If the model can well distinguish negative sample pairs, the fraction inside the logarithm will be close to 1 (because the numerator is close to the numerator and the denominator is small), resulting in the logarithmic term close to 0 (which is what we want because we want to minimize the loss).

[0087] denotes the similarity between the two feature vectors ( and ) in the negative sample pair, scaled by the exponential function and the parameter . denotes the first feature vector in the negative sample pair The sum of similarities between all K feature vectors (including positive and negative samples) is also passed through an exponential function and a parameter The denominator is used for normalization, ensuring that the score is between 0 and 1.

[0088] is a hyperparameter that controls the "sharpness" of the similarity score. Smaller values make the similarity score more sharp (i.e., closer to 0 or 1), while larger values make the score more smooth.

[0089] In summary, The loss function encourages the model to reduce the similarity between negative sample pairs (i.e., scores close to 0) by comparing the feature vectors in negative sample pairs with all feature vectors and adjusting the sensitivity of the loss using a logarithmic function and a parameter.

[0090] The semantic consistency of predicted tokens is measured using high-level visual features. Suppose a high-level feature representation provided by a visual model (which can be CLIP itself) is .

[0091] (10)

[0092] Perceptual Loss is a loss function used to measure the difference between generated images or features and real images or features. This loss function is commonly used in image generation, image style transfer, image super-resolution, etc. tasks to evaluate the visual perception of generated images close to real images.

[0093] represents the sum of image pairs that need to calculate perceptual loss. is a normalization factor to ensure that the loss value is not affected by the number of image pairs. represents the total number of image pairs that need to calculate perceptual loss.

[0094] represents the sum of all image pairs that need to calculate perceptual loss Here, is a mask or indicator, when is an image pair that needs to calculate perceptual loss. is an image pair that needs to calculate perceptual loss. phiis a feature extraction function, usually the output of a certain layer or several layers of a pre-trained convolutional neural network (such as VGG network). This function is used to map images or image blocks into a high-dimensional feature space in order to compare their similarities or differences.

[0095] represents the square of the Euclidean distance between the generated image or feature and the real image or feature. This distance is used to measure the degree of difference between them in the feature space.

[0096] In summary, The loss function measures the visual perceptual difference between the generated image or feature and the real image or feature by calculating the Euclidean distance between them in the feature space. This loss function is very useful in image generation and style conversion tasks, as it can capture the high-level features of the image and encourage the generated image to be visually closer to the real image.

[0097] The final training objective is to minimize the joint loss, which is a weighted combination of the above losses, to ensure that the model performs well in reconstruction, spatio-temporal consistency, and contrastive learning.

[0098] (11)

[0099] where is the weight coefficient.

[0100] Through this mode, the model can use the CLIP-generated features in self-supervised learning to reconstruct the masked tokens, and thus learn the spatio-temporal structure and temporal consistency of the video, enhancing the spatio-temporal perception ability.

[0101] In video understanding tasks, traditional methods usually rely on a large amount of data and multi-modal information (such as video, audio, etc.) to improve the performance of the model. In this application, a small amount of labeled timestamps and corresponding text descriptions are used to enhance the model's understanding of video content. This method not only reduces the dependence on data volume, but also to some extent improves the model's ability to capture and understand key events.

[0102] 1. Dataset

[0103] The dataset used contains fire detection, behavior recognition, and other tasks applied in intelligent security systems. Each dataset includes:

[0104] (1) Long video sequence : contains rich spatio-temporal information.

[0105] (2) Time annotation , representing the start and end times of the th event in the video, with a total of events.

[0106] (3) Text Description A written introduction corresponding to the content expressed by the video in each time period.

[0107] These annotations provide precise positioning and semantic information of key events in the video, helping the model learn the association between video and text.

[0108] 2. Video and text alignment

[0109] The invention aligns video segments with corresponding text descriptions to form cross-modal training samples, which can fully utilize timestamp and text information and other related content

[0110] According to the time annotation, the original video is divided into several video segments :

[0111] (12)

[0112] Using the trained CLIP model, the video features of each video are extracted . Then, the corresponding text description is encoded into text features by a text encoder.

[0113] 3. Cross-modal alignment learning

[0114] In order to align the video features and text features in the same semantic space, the invention uses a cross-modal alignment loss function for training.

[0115] The similarity between video features and text features is calculated using cosine similarity:

[0116] (13)

[0117] For a batch of samples, the cross-modal contrast loss function is defined as:

[0118] (14)

[0119] where represents the similarity between the th video feature and the th text feature, is a hyperparameter used to adjust the smoothness of the distribution.

[0120] The optimization goal of this loss function is: positive samples are close (so that the similarity of the corresponding feature pairs is maximized), and negative samples are separated (to reduce the similarity of non-corresponding feature pairs similarity).

[0121] The ControlLLM is also provided with a video query response module, which is mainly responsible for processing user query requests for video content and generating responses matching the user query content. The module receives user text queries, parses the user query task into executable subtasks through the ControlLLM, and generates video content embeddings and descriptions using the video understanding module, thereby generating response results that meet user needs.

[0122] (I) Text query analysis

[0123] Map user query requests to video analysis and understanding tasks Generate structured query commands. For example, the query "Is there a fire scene in the video?" can be mapped to a fire detection task for the video analyzer; the query "Is there anyone in the garage?" can be mapped to a person detection task; the query "Is there unauthorized personnel entering?" can be mapped to a task of identifying visitor identity or detecting area boundary intrusion; the query "Is there a vehicle parked in the unauthorized area at night?" can be mapped to a vehicle recognition and parking area detection task; the query "Is there vandalism?" can be mapped to an action recognition task to detect vandalism or suspicious behavior, etc.

[0124] (II) Video query and feature retrieval

[0125] 1. Feature retrieval: For each subtask , according to the task type parsed from the query, select the corresponding features for retrieval. The video understanding module has generated the video embedding representation and text description .

[0126] 2. Video feature matching: Using the generated video embeddings and text descriptions, embed the user's query into these video embeddings for similarity calculation. Use cosine similarity to measure the similarity between the user query and the video content:

[0127] (15)

[0128] If the similarity exceeds the set threshold, it is considered that the video segment is related to the query.

[0129] (III) Key frame extraction based on spatiotemporal information

[0130] 1. Spatiotemporal precise positioning: Combined with the query analysis result, locate the key segment in the video according to the timestamp, i.e., the event time period that meets the user's needs.

[0131] 2. Keyframe extraction: After finding relevant video segments, extract representative keyframes or important time nodes for intuitive presentation of video content. Keyframe extraction formula:

[0132] (16)

[0133] where represents the keyframe and the relevance measure of the query .

[0134] (Four) Response generation and multi-round interaction

[0135] 1. Response generation: ControlLLM generates natural language responses based on query results, including descriptions of relevant video segments and event explanations. Response example: "Fire scene found at [0:30 - 1:15] in the video, flames spread from the left edge."

[0136] 2. Multi-round interaction: When users have further questions about the initial query results (e.g., "How did the fire develop?"), the system supports multi-round interaction to further analyze video content. ControlLLM will generate new sub-tasks based on user's additional queries during multi-round interaction. For example:

[0137] ① Task update and sub-task decomposition: ControlLLM will analyze the user's additional query "How did the fire develop?" and generate sub-tasks such as "Identify the spread direction and speed of the flames" or "Detect the trend of the fire-affected area changes". These sub-tasks are passed to the video analysis module for further processing of specific fire dynamics.

[0138] ② Context information preservation: ControlLLM will preserve the context information of user queries, including the initial location and time period of the fire (e.g. [0:30 - 1:15]). In new analysis, these context data will be used as limiting conditions to reduce the analysis range of the system, avoiding the interference of irrelevant content, so as to improve the processing efficiency.

[0139] ③ Gradually refined response generation: After the video analysis module returns new results, ControlLLM will generate more refined responses based on these data. For example, "Flames spread gradually to the right side in the [0:45 - 1:15] time period, covering most of the room area." If the user further asks "Is there anyone trapped?" the system can also add sub-tasks to detect the presence of personnel and escape routes to form a complete event description.

[0140] (4) Dynamic interaction and feedback adjustment: If the user still has higher demands (such as "can you provide specific diffusion speed") in multiple rounds of interaction, ControlLLM will adjust the task in real time according to the feedback, further refine the calculation of fire spread, and provide dynamic updated and more practical analysis results.

[0141] This multi-round interaction not only improves the accuracy of the query, but also enhances the flexible interaction between the user and the system, meeting the high requirements of complex event monitoring in the intelligent security application scenario.

[0142] (5) Final loss function and optimization

[0143] To ensure the accuracy and efficiency of the video query response module, the following loss functions can be introduced for optimization:

[0144] 1. Query matching loss: used to measure the similarity between the query and the retrieved video segments to maximize the relevance of the query results.

[0145] (17)

[0146] 2. Multi-round interaction consistency loss: ensures the coherence and consistency of the context in multi-round queries, defined as the context similarity between queries, specifically as follows:

[0147] (18)

[0148] where and are consecutive queries, is the query round.

[0149] 3. Joint loss: combines query matching and multi-round interaction consistency to optimize the overall performance of the query response module.

[0150] (19)

[0151] where and are weights.

[0152] It should be noted that the model training steps in this application are as follows:

[0153] Step 1: Model initialization. The video encoder is initialized using the pre-trained CLIP model parameters, the text encoder also uses the pre-trained text encoder parameters of CLIP, and the ControlLLM uses the pre-trained model.

[0154] Step 2: Reconstructing mask training. Train the video encoder of CLIP with reconstructing mask to enhance its understanding of spatio-temporal structure. Randomly mask the input video frames, reconstruct the masked frames by the model, and calculate the reconstruction loss. Update the parameters of the video encoder.

[0155] Step 3: Cross-modal alignment training. Align the video features and text features in a common semantic space, calculate the cosine similarity between the video features and the text features , and use for contrastive learning, thereby updating the parameters of the video encoder and text encoder of CLIP.

[0156] Step 4: Joint training of video encoder and ControlLLM.

[0157] Step 5: Overall optimization. In each training step, the total loss is calculated, and all loss terms are minimized. Balance the individual loss terms by adjusting the loss weight λ, and update all trainable parameters using the optimizer.

[0158] As shown in Figure 2 , the embodiment of the present application provides a video analysis processing method based on a multi-modal large model, comprising:

[0159] Step 1: Task acceptance and analysis.

[0160] The user describes the task to be completed through natural language, such as "Please tell me the time period when the fire occurred in the video." The system performs semantic analysis on the user input and extracts key task information. Determine whether the task involves video understanding, if so, decompose the task into subtasks.

[0161] Step 2: Task decomposition and delivery.

[0162] ControlLLM decomposes the complex user request into executable subtasks and sends the video understanding related subtasks to the video large model for processing.

[0163] Step 3: Video feature extraction.

[0164] Input the specified video or video segment V. Use the video encoder to extract the video features . If the task specifies a specific time period, extract the corresponding frame sequence.

[0165] Step 4: Feature matching.

[0166] According to the task requirements, match between the video features and the text description to generate task-related text features. Calculate the similarity between the video features and the task text features to locate the most relevant video frames or time period for the task.

[0167] Step 5: Text description generation.

[0168] Input video features and task context, generate the corresponding text answer describing the video content using the ControlLLM. Combine the context information to ensure that the generated text meets the user's needs.

[0169] In another embodiment, the method is based on the system described in the above embodiment, and the specific process is as follows:

[0170] S1: The multi-modal control large model receives the user's input of the task to be completed through natural language description, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into sub-tasks, and sends the sub-tasks to the video understanding module;

[0171] S2: The video understanding module receives the video or video segment required by the task, performs sub-tasks to analyze the video, generates the embedding and description of the video content, and inputs the multi-modal control large model;

[0172] S3: When the multi-modal control large model receives the user's query request, the user's query task is parsed into executable sub-tasks mapped to the task of video understanding, and the response result based on the query request is generated based on the embedding and description of the video content generated by the video understanding large model.

[0173] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0174] The embodiment of the video analysis processing method based on the multi-modal large model provided by the embodiment of the present application, the method and the above-mentioned multi-modal large model-based video analysis processing system belong to the same inventive concept, and the details not described in the embodiment of the multi-modal large model-based video analysis processing method can be referred to the embodiment of the multi-modal large model-based video analysis processing system.

[0175] Although the present application has been described in detail through reference to preferred embodiments, it should be understood that the present application is not limited to the preferred embodiments. Any modification or replacement of the embodiments of the present application made by those skilled in the art without departing from the spirit and essence of the present application should be within the scope of the present application. Any modification or replacement made by those skilled in the art within the technical scope disclosed by the present application can be easily thought of, and should be covered within the protection scope of the present application.

Claims

1. A multi-modal large model based video analytics processing system, characterized in that, The multi-modal control large model and a video understanding module are included. The multi-modal control large model receives a user input of a task to be completed through a natural language description, performs semantic analysis on the user input, extracts task information, and decomposes a task involving video understanding into subtasks and sends the subtasks to the video understanding module. The video understanding module receives a video or a video segment required by the task, performs analysis on the video through the subtasks, and generates an embedding and a description of the video content and inputs the embedding and the description to the multi-modal control large model. When the multi-modal control large model receives a user query request, the multi-modal control large model parses the user query task into executable subtasks and maps the subtasks to the task of the video understanding, generates a response result based on the query request based on the embedding and the description of the video content generated by the video understanding large model. The video understanding module includes a video analyzer and a video embedder. The video analyzer receives a video input and outputs an analysis result of the video in a text form to the multi-modal control large model; the text includes a video caption, a detailed description of all events in the video and a timestamp, a label, an ID and a bounding box. The video embedder uses a trained CLIP model to extract video features of an input video, encodes corresponding text descriptions into text features through a text encoder in the CLIP model to generate an embedding vector and inputs the embedding vector to the multi-modal control large model. The video analyzer decodes an input video, converts the video from a compressed format into a sequence of original image frames, extracts continuous image frames from the decoded video, extracts image features of each image frame and video features of the video, analyzes the extracted features, detects key events related to a task in the video, generates a detailed description of the events when the events are detected, and generates a text including a video caption, event descriptions, a timestamp, a label, an ID and a bounding box information based on a result of the event detection. The CLIP model is provided with a visual encoder, and the CLIP model is trained by including mask reconstruction training of randomly masking input video frames, and specifically includes: The video is decomposed into frames and further into patches, and each patch represents a token. A set proportion of tokens are randomly selected for masking, i.e., masking. Unmasked tokens are input into the visual encoder of the CLIP model to generate feature embeddings of the tokens. The unmasked token feature embeddings are aggregated into aggregated features as reference information for the model to predict the masked tokens. The aggregated features and position encodings are input into a reconstruction network to predict the embeddings of the masked tokens. Using mean squared error loss measure the reconstruction error of the masked tokens; Computing spatio-temporal smoothness loss Cosine similarity of reconstructed tokens and unmasked tokens is computed to compute a contrastive loss function for distinguishing positive sample pairs from negative sample pairs computing a loss function for measuring a difference between a generated image and a real image The joint loss is generated by a weighted combination The training process minimizes the joint loss; Wherein α, β, γ, δ are weight coefficients. 2.The multi-modal large model based video analytics processing system of claim 1, wherein, The CLIP model is equipped with a text encoder. During model training, the video features and text features are aligned in a common semantic space and the video features v are calculated. i and text features t i and use Perform comparative learning to update the parameters of the video encoder and text encoder in the CLIP model; The cross-modal contrast loss function is: where s i,j represents the similarity between the ith video feature and the jth text feature, and τ is a hyperparameter used to adjust the smoothness of the distribution. 3.The multi-modal large model based video analytics processing system of claim 2, wherein, The video embedder obtains a long video sequence V, cuts the original video V into several video clips according to time labels; uses a trained CLIP model to obtain a video feature v i Extract the video feature v i Encode the corresponding text description D i into a text feature t i by using a text encoder.

4. The multi-modal large model based video analytics processing system of claim 3, wherein, For each subtask s vi , according to the task type parsed from the query, the user's query embedding and the video embedding are calculated for similarity using the generated video embedding and text description. If the similarity s(q, v i ) exceeds the set threshold, it is considered that the video segment v i is related to the query; combined with the query analysis result, the key segment V ki in the video is located according to the timestamp, that is, the event time period that meets the user's demand, and the relevant video segment is found to extract the key frame or time node with the required key frame or time node for intuitive display of the video content.

5. The multi-modal large model based video analytics processing system of claim 4, wherein, The key frame extraction formula is as follows: Wherein relevance(f, q) represents a relevance measure of the key frame f and the query q.

6. The multi-modal large model based video analytics processing system of claim 5, wherein, During a multi-round interaction process between the user and the multi-modal control large model based on the same query request, the multi-modal control large model generates new subtasks based on additional queries of the user and delivers the new subtasks to the video understanding module. The multi-modal control large model generates refined responses based on response data of the video understanding module.

7. A multi-modal large model-based video analysis processing method, characterized in that, The multi-modal control large model includes a multi-modal control large model and a video understanding module. The multi-modal control large model receives a user input of a task to be completed through a natural language description, performs semantic analysis on the user input, extracts task information, decomposes a task involving video understanding into sub-tasks, and sends the sub-tasks to the video understanding module; The video understanding module receives a video or a video segment required by the task, performs sub-tasks to analyze the video, and generates an embedding and a description of the video content to input the multi-modal control large model; When the multi-modal control large model receives a user query request, the user's query task is parsed into executable sub-tasks mapped to the task of video understanding, and a response result based on the query request is generated based on the embedding and the description of the video content generated by the video understanding large model; The video understanding module includes a video analyzer and a video embedder; The video analyzer receives a video input and outputs an analysis result of the video in text form to the multi-modal control large model; the text includes a video caption, a detailed description of all events in the video and a timestamp, a label, an ID, and a bounding box; The video embedder uses a trained CLIP model to extract video features from the input video, encodes the corresponding text description into text features through a text encoder in the CLIP model to generate an embedding vector input to the multi-modal control large model; The video analyzer decodes the input video, converts it from a compressed format to a sequence of original image frames, extracts continuous image frames from the decoded video, extracts image features of each image frame, and extracts video features of the video, analyzes the extracted features, detects key events related to the task in the video, and generates a detailed description of the events when the events are detected; based on the event detection result, the video analyzer generates text containing video caption, event description, timestamp, label, ID, and bounding box information; The CLIP model is provided with a visual encoder, and the CLIP model training includes random masking of input video frames and mask reconstruction training of the model to reconstruct the masked part; specifically including: The video is decomposed into frames and further into patches, and each patch represents a token; Randomly select a set proportion of tokens for masking; that is, masking; The unmasked tokens are input into the visual encoder of the CLIP model to generate feature embeddings of the tokens; The unmasked token feature embeddings are aggregated into aggregated features as reference information for the model to infer the masked tokens; The reconstruction network is used to input the aggregated features and position encoding to predict the embedding of the masked tokens; Using mean squared error loss measure the reconstruction error of the masked tokens; Computing spatio-temporal smoothness loss computing cosine similarity of reconstructed tokens and unmasked tokens to compute a contrastive loss function for distinguishing positive pairs from negative pairs computing a loss function for measuring a difference between a generated image and a real image The joint loss is generated by a weighted combination The training process minimizes the joint loss; Wherein α, β, γ, δ are weight coefficients.

Images