Video analysis processing system and method based on multi-modal large model

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a video analysis processing system and method based on a multi-modal large model, and the method comprises the steps: enabling the multi-modal control large model to receive a task which is input by a user and needs to be completed through natural language description, carrying out the semantic analysis of the input of the user, extracting task information, and carrying out the analysis of the task information; a task related to video understanding is decomposed into sub-tasks and the sub-tasks are sent to a video understanding module; the video understanding module receives a video or a video clip required by a task, executes a subtask to analyze the video, and generates embedding and description of video content to be input into a multi-modal control large model; when the multi-modal control large model receives a query request of a user, a query task of the user is analyzed into executable subtasks and mapped to a video understanding task, and a response result based on the query request is generated based on embedding and description of video content generated by the video understanding large model. Therefore, effective fusion and mutual enhancement of cross-modal information are realized, and the video understanding precision is improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a video analysis and processing system and method based on a multimodal large model. Background Art

[0002] With the vigorous development of large model technology, the fields of natural language processing (NLP) and computer vision (CV) have made great progress, which has significantly improved the capabilities of intelligent systems in semantic understanding and multimodal fusion. However, in the field of video understanding, since video data is rich in complex spatiotemporal information, its processing difficulty is far greater than that of single-modal data, making it an extremely challenging task. Traditional video processing methods often separate video analysis from language processing, ignoring the close connection between the two and the potential mutual reinforcement effect.

[0003] In the smart campus scenario, traditional video surveillance systems are highly dependent on human resources for monitoring and abnormal behavior identification, which not only leads to high costs, but also easily leads to misjudgment and missed judgments, seriously affecting monitoring efficiency and security. Therefore, how to quickly and accurately identify abnormal behaviors in smart security systems and improve the smart security capabilities of the campus has become a key issue that needs to be urgently addressed in the current video surveillance system. Summary of the invention

[0004] Existing video analysis processing usually separates video processing and language processing, lacking close cross-modal connection and mutual enhancement capabilities. The present invention provides a video analysis processing system and method based on a multi-modal large model.

[0005] In a first aspect, the technical solution of the present invention provides a video analysis and processing system based on a multimodal large model, comprising a multimodal control large model and a video understanding module; The multimodal control model receives the task to be completed described in natural language from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

[0006] As a further limitation of the technical solution of the present invention, the video understanding module includes a video analyzer and a video embedder; The video analyzer receives video input and outputs the analysis results of the video in text form to the multimodal control model; the text includes video captions, detailed descriptions of all events in the video and timestamps, tags, IDs and bounding boxes; The video embedder uses the trained CLIP model to extract video features from the input video, and encodes the corresponding text description into text features through the text encoder in the CLIP model to generate an embedding vector that is input into the multimodal control model.

[0007] As a further limitation of the technical solution of the present invention, the video analyzer decodes the input video, converts it from a compressed format into a sequence of original image frames, extracts continuous image frames from the decoded video, extracts image features of each frame of the image, and extracts video features of the video, analyzes the extracted features, detects key events related to the task in the video, and after detecting the event, the video analyzer generates a detailed description of the event; based on the result of the event detection, the video analyzer generates text containing video subtitles, event description, timestamp, label, ID and bounding box information.

[0008] As a further limitation of the technical solution of the present invention, the CLIP model is provided with a visual encoder, and the CLIP model training includes randomly masking the input video frames and allowing the model to reconstruct the masked part of the mask reconstruction training; specifically including: decomposing the video into frames and further dividing them into patches, each patch representing a token; randomly selecting a set proportion of tokens for masking; i.e., masking; inputting the unmasked tokens into the visual encoder of the CLIP model to generate feature embedding of the tokens; The unmasked tokens features are embedded into aggregated features as reference information for the model to infer the masked tokens; the reconstruction network is used to input the aggregated features and position encoding to predict the embedding of the masked tokens; the mean square error loss is used Measuring the reconstruction error of masked tokens; calculating spatiotemporal smoothing loss ; Calculate the cosine similarity between the reconstructed tokens and the unmasked tokens to calculate the contrast loss function used to distinguish between positive and negative sample pairs ; Calculate the loss function used to measure the difference between the generated image and the real image Generate joint loss through weighted combination , the training process minimizes the joint loss;

[0009] in is the weight coefficient.

[0010] As a further limitation of the technical solution of the present invention, the CLIP model is provided with a text encoder. When the model is trained, the video features and the text features are aligned in a common semantic space, and the video features are calculated. and text features and use Perform comparative learning and update the parameters of the video encoder and text encoder in the CLIP model; The cross-modal contrast loss function is:

[0011] in Indicates Video features and The similarity between text features, It is a hyperparameter used to adjust the smoothness of the distribution.

[0012] As a further limitation of the technical solution of the present invention, the video embedder obtains a long video sequence , according to the time tag, the original video Divide into several video clips; use the trained CLIP model to train each video Extracting video features , through the text encoder to describe the corresponding text Encoding as text features .

[0013] As a further limitation of the technical solution of the present invention, when the multi-modal control large model performs user query response, for each subtask , according to the task type parsed from the query, the generated video embedding and text description are used to calculate the similarity between the user's query embedding and the video embedding. If the similarity If the threshold is exceeded, the video clip is considered Related to the query; combined with the query parsing results, locate the key segments in the video based on the timestamp , that is, the event time period that meets the user's needs. After finding the relevant video clips, the required key frames or time nodes are extracted from them to intuitively display the video content.

[0014] As a further limitation of the technical solution of the present invention, the key frame extraction formula is as follows:

[0015] in Represents a key frame With query The correlation measure of .

[0016] As a further limitation of the technical solution of the present invention, during multiple rounds of interaction between the user and the multimodal control big model based on the same query request, the multimodal control big model will generate new subtasks according to the user's additional queries and pass them to the video understanding module; the multimodal control big model will generate detailed responses based on the response data of the video understanding module.

[0017] In a second aspect, the technical solution of the present invention provides a video analysis and processing method based on a multimodal large model, comprising: The multimodal control model receives the task to be completed described in natural language from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

[0018] It can be seen from the above technical solutions that the present invention has the following advantages: The present invention realizes the close integration and mutual enhancement between video data and language description through the multimodal control large model and the video understanding module. The multimodal control large model can accurately understand the task requirements described by the user in natural language, and decompose it into specific subtasks and send it to the video understanding module for processing. The video understanding module can generate accurate embeddings and descriptions based on the video data, and further feed back to the multimodal control large model, thereby realizing the effective integration and mutual enhancement of cross-modal information. With the powerful capabilities of the multimodal large model, the present invention can more accurately understand the video content, identify abnormal behavior, and generate accurate descriptions and response results. This not only significantly improves the accuracy of video understanding, but also greatly improves the processing efficiency, making the security monitoring of the smart park more intelligent and efficient.

[0019] Traditional video surveillance systems are highly dependent on human resources for monitoring and identification, which is not only costly but also prone to misjudgment and missed judgments. The present invention effectively reduces labor costs and significantly reduces the occurrence of misjudgments and missed judgments by introducing a multimodal large model for automated processing, thereby improving the overall performance and security of the monitoring system. The multimodal control large model and video understanding module of the present invention are both highly scalable and flexible, and can be customized and optimized according to different scenarios and task requirements. This enables the system to better adapt to the complex and changeable security environment of smart parks, and provide users with more intelligent, convenient and efficient security services. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 2 is a schematic flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The system integrates a multimodal control large model (ControlLLM) and a video understanding module (Video Model) to enable users to understand, answer, and perform related actions on video content. The core of the system is to identify user intent through ControlLLM and distribute related tasks involving video understanding, analysis, and processing to the video understanding large model to perform specific operations. The video understanding module analyzes and embeds video content based on ClIP's Video-Embedder, providing richer multimodal input for the large model. This method combines the advantages of video embedders and video analyzers, and can achieve fine-grained semantic understanding and high-quality video representation, enhance the semantic alignment between video and text, and effectively respond to complex multimodal queries and perform corresponding actions, such as video description, video question answering (VQA), video positioning, and content understanding.

[0024] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a video analysis and processing system based on a multimodal large model, including a multimodal control large model and a video understanding module; The multimodal control model receives the task to be completed described in natural language from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; ControlLLM first performs the task decomposition work. Decompose into a series of parallel subtasks This process can be done with a large language model accomplish.

[0026] (1) in, Indicates subtasks, Indicates the number of all subtasks. Perform task division and identify tasks involving video understanding, analysis, and processing , and extract this part of the task and assign it to a specific video understanding module.

[0027] The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

[0028] It should be noted that the video understanding module includes a video analyzer and a video embedder; The video analyzer receives video input and outputs the analysis results of the video in text form to the multimodal control model; the text includes video captions, detailed descriptions of all events in the video and timestamps, tags, IDs and bounding boxes; Specifically, the video analyzer accepts video input and outputs analysis of the video, usually in text form. The above text includes video captions, dense video captions (detailed descriptions of all events in the video and timestamps), object tracking results (labels, IDs and bounding boxes), and transcriptions of other modalities in the video (speech recognition results or caption recognition results). The text generated by the video analyzer can be directly input into the subsequent ControlLLM, or inserted into a pre-prepared template and then input into ControlLLM, or converted into a temporary database format for subsequent retrieval by ControlLLM.

[0029] ControlLLM is mainly responsible for the overall coordination of the system, and calls various video analyzers by generating commands to obtain the required analysis results. After calling the video analyzer, ControlLLM can also further process the obtained analysis results and feedback the final results to the user to ensure that the user gets the accurate information they need. In addition, ControlLLM has multi-round interaction capabilities and can communicate continuously with the video analyzer to gradually obtain and process information in complex tasks to meet higher-level analysis needs.

[0030] The video embedder uses the trained CLIP model to extract video features from the input video, and encodes the corresponding text description into text features through the text encoder in the CLIP model to generate an embedding vector that is input into the multimodal control model.

[0031] The video embedder is the visual backbone. In this patent, the CLIP model is selected to convert the input video into a vector representation, that is, to generate video embedding. In this application, ControlLLM acts as a text decoder, receiving the embedding vector from the video embedder as input, and decoding it into text output according to prompts or instructions. These tasks usually do not require fine-grained understanding or precise spatiotemporal positioning. The text analysis results provided by the video analyzer and the visual feature representation generated by the video embedder are used to provide ControlLLM with richer input information. ControlLLM can perform various tasks based on these inputs, such as summarizing video content, answering questions, generating descriptive text, etc.

[0032] CLIP can provide powerful visual semantic representation by aligning the embedding spaces of images and text. In order to enhance the spatiotemporal perception of CLIP, the technology introduces the mask reconstruction strategy in VideoMAE. Specifically, the input video frames are randomly masked, and then the model is asked to reconstruct the masked parts. In this way, the model is forced to learn the spatiotemporal structure of the video to accurately reconstruct the missing information.

[0033] 1. Video Preprocessing and Spatiotemporal Masking First, the video is decomposed into frames and further divided into patches. Each patch represents a token, and a masking strategy is used to select some tokens for masking.

[0034] Assuming the video has Frames, each frame is divided into Each patch represents a token. Then the video can be represented as: (2) in, Indicates Frame No. patches.

[0035] Randomly select 90-95% of the tokens to mask, that is, cover them and make them invisible. Define the mask matrix : (3) Unmasked tokens will be used as reference information, and masked tokens will be used as reconstruction targets.

[0036] 2. Generate embeddings of unmasked tokens using CLIP visual encoder The unmasked tokens are input into the visual encoder of CLIP to generate feature embeddings of these tokens. The CLIP encoder is denoted as : (4) if Then generate feature embedding ,otherwise Is empty.

[0037] The unmasked tokens are embedded and aggregated into a context feature as a reference for the model to infer the masked tokens: (5) 3. Reconstruct the network prediction masked tokens Use a reconstruction network (lightweight network) to predict the embedding of the masked tokens. Input aggregate features and positional encodings, predicting representations of masked tokens.

[0038] (6) in It is Frame No. Location information for a location. It stands for lightweight prediction network, and the Transformer-Encoder is used in the patent.

[0039] 4. Loss function design The mean squared error (MSE) loss is used to measure the reconstruction error of masked tokens.

[0040] (7) in is the total number of masked tokens.

[0041] In order to ensure that the reconstructed tokens maintain spatiotemporal consistency, a spatiotemporal smoothing loss is added.

[0042] (8) It is a loss function used to measure the amount of feature or state change between adjacent time points in time series data. This loss function is often used to process time series data, video analysis, motion estimation and other tasks to evaluate the model's ability to capture the continuity or smoothness of data changes over time.

[0043] is a normalization factor to ensure that the loss value is not affected by the length of the time series. T represents the total number of time points in the time series, so T-1 represents the number of adjacent time point pairs.

[0044] It represents the sum of all adjacent time point pairs. Here, t represents the time point index in the time series. Indicates the sum of all feature or state indices. Here p can represent a spatial position, a channel index, or other index used to distinguish different features or states.

[0045] It represents the square of the Euclidean distance of the feature or state change at feature or state index p between adjacent time points t and t+1. This distance is used to measure the continuity or smoothness of features or states between adjacent time points.

[0046] In summary, The loss function measures the continuity or smoothness of time series data by calculating the Euclidean distance between the feature or state changes between adjacent time points.

[0047] Using unmasked tokens as positive samples and masked tokens as negative samples, the cosine similarity between reconstructed tokens and unmasked tokens is calculated to encourage the model to distinguish between masked and unmasked tokens.

[0048] (9) in represents the cosine similarity, It is a super parameter.

[0049] It is a contrast loss (Contrastive Loss), Represents all negative sample pairs Here, is a mask or indicator, when =0, indicating is a negative pair (i.e., a pair that is dissimilar or mismatched).

[0050] The logarithmic function is used to measure how well the model distinguishes the negative pairs. If the model is able to distinguish the negative pairs well, then the score within the logarithm will be close to 1 (because the numerator is close and the denominator is small), resulting in the logarithmic term being close to 0 (which is what we want since we want to minimize the loss).

[0051] Represents the two feature vectors in the negative sample pair ( and ), after the exponential function and parameter of scaling. Represents the first eigenvector in the negative sample pair The sum of the similarities with all K feature vectors (including positive and negative samples) is also passed through the exponential function and parameter This denominator is used for normalization to ensure that the fraction is between 0 and 1.

[0052] is a hyperparameter that controls the “sharpness” of the similarity scores. Smaller A value of 0 will make the similarity scores more sharp (that is, closer to 0 or 1), while larger A higher value will make the scores smoother.

[0053] In summary, The loss function encourages the model to learn effective feature representations by reducing the similarity between negative sample pairs (i.e., the score is close to 0). This is achieved by comparing the similarity between the feature vectors in the negative sample pairs and all feature vectors, and uses a logarithmic function and parameters to adjust the sensitivity of the loss.

[0054] Use high-level visual features to measure the semantic consistency of predicted tokens. Assume that there is a visual model (which can be CLIP itself) that provides high-level features represented as .

[0055] (10) Perceptual Loss is a loss function used to measure the difference between a generated image or feature and a real image or feature. This loss function is often used in tasks such as image generation, image style transfer, and image super-resolution to evaluate how close the generated image is to the real image in visual perception.

[0056] Represents the summation of image pairs for which perceptual loss needs to be calculated. is a normalization factor to ensure that the loss value is not affected by the number of image pairs. Represents the total number of image pairs that need to calculate the perceptual loss.

[0057] Represents all image pairs that need to calculate perceptual loss Here, is a mask or indicator, when When is an image pair for which the perceptual loss needs to be calculated. ϕ It is a feature extraction function, usually the output of one or several layers of a pre-trained convolutional neural network (such as a VGG network). This function is used to map images or image patches into a high-dimensional feature space in order to compare the similarities or differences between them.

[0058] Represents the square of the Euclidean distance between the generated image or feature and the real image or feature. This distance is used to measure the degree of difference between them in the feature space.

[0059] In summary, The loss function measures the visual perceptual difference between the generated image or feature and the real image or feature by calculating the Euclidean distance in feature space. This loss function is very useful in tasks such as image generation and style transfer because it can capture the high-level features of the image and encourage the generated image to be visually closer to the real image.

[0060] The final training goal is to minimize the joint loss, which is a weighted combination of the above losses to ensure that the model performs well in reconstruction, spatiotemporal consistency, and contrastive learning.

[0061] (11) in is the weight coefficient.

[0062] Through this mode, the model can use the features generated by CLIP to reconstruct mask tokens in self-supervised learning, and then learn the spatiotemporal structure and temporal consistency of the video to enhance the spatiotemporal perception ability.

[0063] In video understanding tasks, traditional methods usually rely on a large amount of data and multimodal information (such as video, audio, etc.) to improve the performance of the model. In this application, a small amount of annotated timestamps and corresponding text descriptions are used to enhance the model's understanding of video content. This method not only reduces the dependence on the amount of data, but also can improve the model's ability to capture and understand key events to a certain extent.

[0064] 1. Dataset The datasets used include tasks such as fire detection and behavior recognition applied in smart security systems. Each dataset includes: (1) Long video sequences : Contains rich spatiotemporal information.

[0065] (2) Time stamp , indicating that The start and end time of events, an event.

[0066] (3) Text description , corresponding to the textual description of the content expressed in the video in each time period.

[0067] These annotations provide the model with precise positioning and semantic information of key events in the video, helping the model learn the association between video and text.

[0068] 2. Video and text alignment The present invention aligns video clips with corresponding text descriptions to form cross-modal training samples, which can make full use of relevant content such as timestamps and text information. According to the time tag, the original video Split into several video clips : (12) Use the trained CLIP model to train each video Extracting video features Then, the corresponding text description is transformed into Encoding as text features .

[0069] 3. Cross-modal alignment learning In order to align video features and text features in the same semantic space, the invention adopts a cross-modal alignment loss function for training.

[0070] Use cosine similarity to calculate the similarity between video features and text features: (13) For a batch of samples, the cross-modal contrast loss function is defined as: (14) in Indicates Video features and The similarity between text features, It is a hyperparameter used to adjust the smoothness of the distribution.

[0071] The optimization goal of this loss function is: the positive samples are close (so that the corresponding The similarity of feature pairs is the largest), negative sample separation (reducing non-corresponding feature pairs similarity).

[0072] ControlLLM also has a video query response module, which is mainly responsible for processing user query requests for video content and generating responses that match the user's query content. This module receives the user's text query, parses the user's query task into executable subtasks through ControlLLM, and uses the video understanding module to generate embedding and description of the video content, thereby generating a response result that meets the user's needs.

[0073] 1. Text query parsing Map user query requests to video analysis and understanding tasks. The task of video understanding Generate structured query commands. For example, the query "Is there a fire scene in the video" can be mapped to the fire detection task of the video analyzer; the query "Is there anyone in the garage" can be mapped to the person detection task; the query "Is there any unauthorized person entering" can be mapped to the task of identifying the visitor or detecting the intrusion of the area boundary; the query "Is there any vehicle parked in an unauthorized area at night" can be mapped to the vehicle identification and parking area detection task; the query "Is there any vandalism" can be mapped to the action recognition task to detect vandalism or suspicious behavior, etc.

[0074] 2. Video query and feature retrieval 1. Feature retrieval: For each subtask , according to the task type parsed from the query, select the corresponding features for retrieval. The video understanding module has generated an embedded representation of the video and text description .

[0075] 2. Video feature matching: Using the generated video embeddings and text descriptions, the user's query embedding is similar to these video embeddings. Cosine similarity is used to measure the similarity between the user query and the video content: (15) If the similarity If the threshold is exceeded, the video clip is considered Related to the query.

[0076] 3. Key frame extraction based on spatiotemporal information 1. Precise spatial and temporal positioning: Combine query parsing results to locate key segments in the video based on timestamps , that is, the event time period that meets user needs.

[0077] 2. Key frame extraction: After finding relevant video clips, extract representative key frames or important time nodes from them to intuitively display the video content. The key frame extraction formula is as follows: (16) in Represents a key frame With query The correlation measure of .

[0078] 4. Response Generation and Multi-round Interaction 1. Response generation: ControlLLM generates a natural language response based on the query results, including a description of the relevant video clips and an explanation of the events. Example response: "A fire scene was found at [0:30 - 1:15] of the video, with flames starting to spread from the left edge."

[0079] 2. Multi-round interaction: When the user has further questions about the initial query results (for example, "How did the fire develop"), the system supports multi-round interaction to analyze the video content at a deeper level. ControlLLM will generate new subtasks based on the user's additional queries during the multi-round interaction process. For example: ① Task update and subtask decomposition: ControlLLM analyzes the user's additional query "How does the fire develop" and generates subtasks, such as "Identify the spread direction and speed of the flame" or "Detect the changing trend of the flame-affected area." These subtasks are passed to the video analysis module to further process specific fire dynamics.

[0080] ② Contextual information retention: ControlLLM retains the contextual information of the user's query, including the initial location and time period of the fire (such as [0:30 - 1:15]). In the new analysis, this contextual data will be used as a qualifier to reduce the system's analysis scope and avoid interference from irrelevant content, thereby improving processing efficiency.

[0081] ③Generate a more detailed response: After the video analysis module returns new results, ControlLLM will generate a more detailed response based on this data. For example, "The fire gradually spread to the right during the period [0:45 - 1:15], covering most of the room." If the user further asks "Is there anyone trapped?", the system can also add subtasks to detect the presence of people and escape routes to form a complete event description.

[0082] ④ Dynamic interaction and feedback adjustment: If in multiple rounds of interaction, the user still has higher requirements for the generated response (such as "Can you provide a specific diffusion speed"), ControlLLM will adjust the task in real time based on the feedback, further refine the calculation of fire spread, and provide dynamically updated and more practical analysis results.

[0083] This multi-round interaction not only improves the accuracy of the query, but also enhances the flexible interaction between users and the system, meeting the high requirements for complex event monitoring in smart security application scenarios.

[0084] (V) Final loss function and optimization In order to ensure the accuracy and efficiency of the video query response module, the following loss function can be introduced for optimization: 1. Query matching loss: used to measure the similarity between the query and the retrieved video segments to maximize the relevance of the query results.

[0085] (17) 2. Multi-round interaction consistency loss: Ensure the coherence and consistency of the context in multiple rounds of queries, defined as the context similarity between queries, as follows: (18) in and It is a continuous query. is the query round.

[0086] 3. Joint loss: Combines query matching and multi-round interaction consistency to optimize the overall performance of the query response module.

[0087] (19) in and is the weight.

[0088] It should be noted that the model training steps in this application are as follows: Step 1: Model initialization. The video encoder is initialized using the pre-trained CLIP model parameters, the text encoder also uses the pre-trained text encoder parameters of CLIP, and ControlLLM uses the pre-trained model.

[0089] Step 2: Reconstruction mask training. Use the reconstruction mask to train CLIP's video encoder to enhance its understanding of spatiotemporal structure. Randomly mask the input video frames, reconstruct the masked frames through the model, and calculate the reconstruction loss. Update the parameters of the video encoder.

[0090] Step 3: Cross-modal alignment training. Align video features and text features in a common semantic space and calculate video features. and text features and use Perform contrastive learning to update the parameters of CLIP's video encoder and text encoder.

[0091] Step 4: Jointly train the video encoder and ControlLLM.

[0092] Step 5: Overall optimization. In each training step, the total loss is calculated comprehensively , minimize all loss terms. Balance the individual loss terms by adjusting the loss weights 𝜆 and update all trainable parameters using the optimizer.

[0093] like Figure 2 As shown, an embodiment of the present invention provides a video analysis and processing method based on a multimodal large model, comprising: Step 1: Task acceptance and analysis.

[0094] The user describes the task to be completed in natural language, such as "Please tell me the time period when the fire occurred in the video." The system performs semantic analysis on the user input and extracts key task information. It determines whether the task involves video understanding. If so, it decomposes the task into subtasks.

[0095] Step 2: Task decomposition and distribution.

[0096] ControlLLM breaks down complex user requests into executable subtasks and sends subtasks related to video understanding to the large video model for processing.

[0097] Step 3: Video feature extraction.

[0098] Input the specified video or video clip 𝑉. Use the video encoder to extract video features If the task specifies a specific time period, the corresponding frame sequence is extracted.

[0099] Step 4: Feature matching.

[0100] According to the task requirements, match the video features and text descriptions to generate task-related text features. Calculate the similarity between the video features and the task text features to locate the video frames or time periods most relevant to the task.

[0101] Step 5: Text description generation.

[0102] Input video features Based on the task context, ControlLLM is used to generate corresponding text answers to describe the video content. The generated text is ensured to meet user needs by combining context information.

[0103] In another embodiment, the method is based on the system described in the above embodiment, and the specific process is as follows: S1: The multimodal control model receives the task to be completed in natural language input from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; S2: The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; S3: When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

[0104] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0105] An embodiment of the video analysis and processing method based on a multimodal large model provided in an embodiment of the present invention belongs to the same inventive concept as the video analysis and processing system based on a multimodal large model in the above-mentioned embodiments. For details not described in detail in the embodiment of the video analysis and processing method based on a multimodal large model, reference can be made to the embodiment of the video analysis and processing system based on a multimodal large model.

[0106] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A video analysis and processing system based on a multimodal large model, characterized in that: Includes a large multimodal control model and a video understanding module; The multimodal control model receives the task to be completed described in natural language from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

2. The video analysis and processing system based on multimodal large model according to claim 1 is characterized in that: The video understanding module includes a video analyzer and a video embedder; The video analyzer receives video input and outputs the analysis results of the video in text form to the multimodal control model; the text includes video captions, detailed descriptions of all events in the video and timestamps, tags, IDs and bounding boxes; The video embedder uses the trained CLIP model to extract video features from the input video, and encodes the corresponding text description into text features through the text encoder in the CLIP model to generate an embedding vector that is input into the multimodal control model.

3. The video analysis and processing system based on multimodal large model according to claim 2 is characterized in that: The video analyzer decodes the input video, converts it from a compressed format into a sequence of raw image frames, extracts continuous image frames from the decoded video, extracts image features of each frame, and extracts video features of the video, analyzes the extracted features, detects key task-related events in the video, and when an event is detected, the video analyzer generates a detailed description of the event; based on the results of event detection, the video analyzer generates text containing video subtitles, event description, timestamp, label, ID, and bounding box information.

4. The video analysis and processing system based on multimodal large model according to claim 3 is characterized in that: The CLIP model is equipped with a visual encoder. The training of the CLIP model includes randomly masking the input video frames and letting the model reconstruct the masked parts. Specifically, it includes: Decompose the video into frames and further divide them into patches, each patch represents a token; Randomly select a set proportion of tokens to cover up; that is, mask; Input the unmasked tokens into the visual encoder of the CLIP model to generate feature embeddings of the tokens; The unmasked tokens features are embedded and aggregated into aggregated features, which serve as reference information for the model to infer the masked tokens. Use the reconstruction network input aggregated features and positional encoding to predict the embedding of the masked tokens; Using mean squared error loss Measures the reconstruction error of masked tokens; Compute spatiotemporal smoothing loss ; Calculate the cosine similarity between the reconstructed tokens and the unmasked tokens to calculate the contrast loss function used to distinguish between positive and negative sample pairs ; Compute the loss function that measures the difference between the generated image and the real image Generate joint loss through weighted combination , the training process minimizes the joint loss; in is the weight coefficient.

5. The video analysis and processing system based on multimodal large model according to claim 4 is characterized in that: The CLIP model is equipped with a text encoder. During model training, the video features and text features are aligned in a common semantic space and the video features are calculated. and text features and use Perform comparative learning and update the parameters of the video encoder and text encoder in the CLIP model; The cross-modal contrast loss function is: in Indicates Video features and The similarity between text features, It is a hyperparameter used to adjust the smoothness of the distribution.

6. The video analysis and processing system based on multimodal large model according to claim 5 is characterized in that: Video embedder to get long video sequences , according to the time tag, the original video Divide into several video clips; use the trained CLIP model to train each video Extracting video features , through the text encoder to describe the corresponding text Encoding as text features .

7. The video analysis and processing system based on multimodal large model according to claim 6 is characterized in that: When the multimodal control model responds to user queries, for each subtask , according to the task type parsed from the query, the generated video embedding and text description are used to calculate the similarity between the user's query embedding and the video embedding. If the similarity If the threshold is exceeded, the video clip is considered Related to the query; Combine the query parsing results to locate the key segments in the video based on the timestamp , that is, the event time period that meets the user's needs. After finding the relevant video clips, the required key frames or time nodes are extracted from them to intuitively display the video content.

8. The video analysis and processing system based on multimodal large model according to claim 7 is characterized in that: The key frame extraction formula is as follows: in Represents a key frame With query The correlation measure of .

9. The video analysis and processing system based on multimodal large model according to claim 8, characterized in that: Based on the same query request, the user conducts multiple rounds of interaction with the multimodal control model. The multimodal control model will generate new subtasks based on the user's additional query and pass them to the video understanding module. The multimodal control model will generate refined responses based on the response data of the video understanding module.

10. A video analysis and processing method based on a multimodal large model, characterized in that: include: The multimodal control model receives the task to be completed described in natural language from the user, performs semantic analysis on the user's input, extracts task information, decomposes the task involving video understanding into subtasks and sends them to the video understanding module; The video understanding module receives the video or video clips required for the task, performs subtasks to analyze the video, and generates embeddings and descriptions of the video content to input into the multimodal control model; When the multimodal control model receives a user query request, it parses the user's query task into executable subtasks and maps them to the video understanding task, and generates a response result based on the query request based on the embedding and description of the video content generated by the video understanding model.

Images