Video timestamp event identification and reasoning method based on multi-modal large model

Abstract

The embodiment of the invention relates to the technical field of video processing, and particularly discloses a video timestamp event identification and reasoning method based on a multi-modal large model. According to the embodiment of the invention, an input video is processed, and visual information and text information are extracted; performing interaction and fusion on the visual information and the text information to generate a video event semantic representation; and performing event identification on the semantic representation of the video event to obtain an event timestamp mark and event logical reasoning. According to the method, deep fusion and semantic association understanding of multi-modal information such as vision and texts can be realized, the problem that a traditional method is insufficient in semantic consistency and feature expression ability among modals is solved, and deep fusion of multi-modal features and accurate understanding of semantic association are realized; and the frame-level time information can be used as a visual code and a text feature to perform visual text feature alignment, and then video-level modeling is performed, so that the problem that a traditional model cannot directly capture an accurate time frame when a visual event occurs is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular to a video timestamp event recognition and reasoning method based on a multimodal large model. Background Art

[0002] In the prior art, video event recognition methods usually rely on single-modal information processing technology, such as visual feature extraction based on visual encoders and separate analysis of text information through natural language processing technology. Features of different modalities are usually processed independently and then simple feature splicing or classification reasoning is performed, which fails to effectively utilize the potential associations between modalities. In addition, the marking and logical reasoning of event timestamps mostly use rule-driven methods, and analysis is performed through preset time windows or event sequence templates, which makes it difficult to cope with the diverse event logic in complex scenarios.

[0003] Therefore, existing technologies have made some progress in multimodal event analysis, but their limitations are mainly reflected in the following aspects: (1) They lack the ability to deeply integrate multimodal information, making it difficult to fully capture the complex semantic associations of events. Especially when processing long video sequences or highly dynamic scenes, traditional feature splicing methods are difficult to adapt to the diversity and logical complexity of semantic expressions; (2) They have limited ability to model and reason about the timeline of events, and are unable to accurately mark event timestamps or infer event causal relationships. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a video timestamp event recognition and reasoning method based on a multimodal large model, aiming to solve the problems raised in the background technology.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions: A video timestamp event recognition and reasoning method based on a multimodal large model, the method specifically comprises the following steps: Receiving an input video, and processing the input video through a multimodal information extraction module to extract visual information and text information; Through a multimodal feature fusion module, a multi-head self-attention mechanism is used to interact and fuse the visual information and the text information to generate a semantic representation of the video event; Through the time series analysis module, a recurrent neural network is used to perform time series analysis of event recognition on the semantic representation of the video event to obtain event timestamp tags and event logic reasoning.

[0006] As a further limitation of the technical solution of the embodiment of the present invention, the receiving of the input video, processing the input video through the multimodal information extraction module, and extracting the visual information and text information specifically include the following steps: Receive input video; Using a multimodal information extraction module and a Vision Transformer, the visual features of the input video are extracted to obtain visual information; Extracting the video subtitles and speech of the input video by optical character recognition technology; Using speech recognition technology, converting the video subtitle speech into speech text; Use natural language processing technology to extract text features from the speech text to obtain text information.

[0007] As a further limitation of the technical solution of the embodiment of the present invention, the use of Vision Transformer to extract visual features from the input video and obtain visual information specifically includes the following steps: Processing the input video frame by frame to obtain a plurality of frame-by-frame input images; Dividing the plurality of frame-by-frame input images into a plurality of image blocks; Mapping the plurality of image blocks to a D-dimensional vector space to obtain a corresponding first embedding vector; Add position encoding to retain the spatial relationship of the image blocks and generate multiple second embedding vectors with position encoding; Normalizing the timestamps of the plurality of frame-by-frame input images, and calculating a plurality of normalized timestamps; Creating a corresponding time coding vector according to the plurality of normalized timestamps; Adding the plurality of time coding vectors to the corresponding optimized embedding vectors to generate a plurality of third embedding vectors with time and space information; The plurality of third embedding vectors are introduced into a multi-head self-attention mechanism, focusing on the important influence areas in the plurality of frame-by-frame input images, performing visual feature extraction, and acquiring visual information.

[0008] As a further limitation of the technical solution of the embodiment of the present invention, the expression of the plurality of first embedding vectors is: ; in, For the The first embedding vector of the image block, Representative image blocks, Represents linear change calculation; The calculation formula of the plurality of second embedding vectors is: ; in, For the The second embedding vector of the image block, For the Position encoding of image blocks; The calculation formula for the multiple normalized timestamps is: ; in, For the The normalized timestamp of the image blocks, For the The timestamp of the frame-by-frame input image corresponding to each image block, is the minimum value of the timestamp, The maximum value of the timestamp; The calculation formula of the multiple time coding vectors is: ; in, is the dimension index of the time encoding vector, For the The temporal encoding vector of the image block, is the total dimension of the time encoding vector; The calculation formula of the plurality of third embedding vectors is: ; in, For the The third embedding vector of the image patch.

[0009] As a further limitation of the technical solution of the embodiment of the present invention, the calculation formula of the multi-head self-attention mechanism is: ; in, , and represents the query, key, and value matrices corresponding to the third embedding vector, is the dimension of the key vector.

[0010] As a further limitation of the technical solution of the embodiment of the present invention, the multi-modal feature fusion module uses a multi-head self-attention mechanism to interact and fuse the visual information and the text information to generate a video event semantic representation, which specifically includes the following steps: Constructing a visual feature matrix and a text feature matrix according to the visual information and the text information through a multimodal feature fusion module; Importing the visual feature matrix and the text feature matrix into the corresponding multi-head self-attention mechanism to capture multimodal semantic associations; Based on multimodal semantic association, the visual information and the text information are interacted and fused to generate a semantic representation of a video event.

[0011] As a further limitation of the technical solution of the embodiment of the present invention, the multi-modal feature fusion module uses a multi-head self-attention mechanism to interact and fuse the visual information and the text information to generate a video event semantic representation, which also includes the following steps: Conduct adversarial training for multimodal information understanding and add a modality discriminator; The modal discriminator is used to discriminate the source of features.

[0012] As a further limitation of the technical solution of the embodiment of the present invention, the objective function of the adversarial training is: ; in, is the main task loss of the multimodal feature fusion module, the cross entropy loss of event classification or timestamp annotation, is the balance coefficient, is the feature representation of modality fusion, For all The average value of stands for modality discriminator.

[0013] As a further limitation of the technical solution of the embodiment of the present invention, the time series analysis of the video event semantic representation for event recognition using a recurrent neural network through a time series analysis module to obtain event timestamps and event logic reasoning specifically includes the following steps: Performing time series event representation processing on the video event semantic representation through a time series analysis module; Determine, using a recurrent neural network, a hidden state sequence corresponding to a plurality of the time series events; Calculating the weights of the time steps corresponding to the multiple hidden state sequences through the attention mechanism; Calculating a plurality of weighted sequence representations according to the plurality of hidden state sequences and corresponding weights; Determining a plurality of key event nodes according to the plurality of weighted sequence representations; According to the plurality of key event nodes, they are mapped to the embedding space of a large language model to obtain event timestamp tags and event logic reasoning.

[0014] As a further limitation of the technical solution of the embodiment of the present invention, the calculation formula of the weight is: ; in, is the time step The weight of is the time step The intermediate representation of the hidden state, w is the weight vector, is the total time; The calculation formula for the weighted sequence representation is: ; in, is the time step The weighted sequence representation of is the time step hidden state.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The embodiment of the present invention processes the input video to extract visual information and text information; interacts and fuses the visual information and text information to generate a semantic representation of video events; and performs event recognition on the semantic representation of video events to obtain event timestamp tags and event logic reasoning. It can achieve deep fusion and semantic association understanding of multimodal information such as vision and text, solve the problem of insufficient semantic consistency and feature expression ability between modalities in traditional methods, achieve deep fusion of multimodal features and accurate understanding of semantic association, and can use frame-level time information as visual encoding and text features to align visual and text features, and then perform video-level modeling, solving the problem that traditional models cannot directly capture the precise time frame when visual events occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0017] Figure 1 A flowchart of a video timestamp event recognition and reasoning method based on a multimodal large model provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] It is understandable that existing technologies have made some progress in multimodal event analysis, but their limitations are mainly reflected in the following aspects: (1) They lack the ability to deeply integrate multimodal information, making it difficult to fully capture the complex semantic associations of events. In particular, when processing long video sequences or highly dynamic scenes, traditional feature splicing methods are difficult to adapt to the diversity and logical complexity of semantic expressions; (2) They have limited ability to model and reason about the timeline of events, and are unable to accurately mark event timestamps or infer event causal relationships.

[0020] To solve the above problems, the embodiment of the present invention receives input video, processes the input video through a multimodal information extraction module, extracts visual information and text information; uses a multimodal feature fusion module to interact and fuse visual information and text information using a multi-head self-attention mechanism to generate a video event semantic representation; uses a time series analysis module to perform time series analysis of event recognition on the video event semantic representation using a recurrent neural network to obtain event timestamp tags and event logical reasoning. It can achieve deep fusion and semantic association understanding of multimodal information such as vision and text, solves the problem of insufficient semantic consistency and feature expression ability between modalities in traditional methods, achieves deep fusion of multimodal features and accurate understanding of semantic association, and can use frame-level time information as visual encoding and text features to align visual and text features, and then perform video-level modeling, solving the problem that traditional models cannot directly capture the precise time frame when visual events occur.

[0021] Figure 1 A flowchart of a video timestamp event recognition and reasoning method based on a multimodal large model provided by an embodiment of the present invention is shown.

[0022] Specifically, a video timestamp event recognition and reasoning method based on a multimodal large model includes the following steps: Step S101, receiving an input video, and processing the input video through a multimodal information extraction module to extract visual information and text information.

[0023] In an embodiment of the present invention, after receiving an input video, a multimodal information extraction module is used to extract visual features from the input video using VisionTransformer. Specifically, the input video is processed frame by frame to obtain a plurality of frame-by-frame input images, and then the plurality of frame-by-frame input images are divided into a plurality of image blocks. After that, the plurality of image blocks are mapped to a D-dimensional vector space, and corresponding first embedding vectors are obtained. Position coding is then added to retain the spatial relationship of the image blocks, and a plurality of second embedding vectors with position coding are generated. The plurality of frame-by-frame input images are normalized in time stamps, and a plurality of normalized time stamps are calculated. Then, according to the plurality of normalized time stamps, the plurality of image blocks are mapped to a D-dimensional vector space to obtain corresponding first embedding vectors. The timestamp is normalized to create a corresponding time coding vector, and then multiple time coding vectors are added to the corresponding optimized embedding vector to generate multiple third embedding vectors with time and space information, and the multiple third embedding vectors are introduced into the multi-head self-attention mechanism to focus on the important influence areas in multiple frame-by-frame input images, perform visual feature extraction, obtain visual information, and extract the video subtitles and voice of the input video through optical character recognition technology, use speech recognition technology to convert the video subtitles and voice into speech text, and then use natural language processing technology to extract text features from the speech text to obtain text information. Specifically, the expressions of the multiple first embedding vectors are: ; in, For the The first embedding vector of the image block, Representative image blocks, It stands for linear change computation, specifically a fully connected layer, which maps each image block to a higher-dimensional space, namely a D-dimensional vector space. This process can be regarded as encoding each image block to generate a representation that can better capture the image features; The calculation formula for multiple second embedding vectors is: ; in, For the The second embedding vector of the image block, For the Position encoding of image blocks; The calculation formula for multiple normalized timestamps is: ; in, For the The normalized timestamp of the image blocks, For the The timestamp of the frame-by-frame input image corresponding to each image block, is the minimum value of the timestamp, The maximum value of the timestamp; The calculation formula for multiple time encoding vectors is: ; in, is the dimension index of the time encoding vector, For the The temporal encoding vector of the image block, is the total dimension of the time encoding vector; The calculation formula for multiple third embedding vectors is: ; in, For the The third embedding vector of the image block; The calculation formula of the multi-head self-attention mechanism is: ; in, , and represents the query, key, and value matrices corresponding to the third embedding vector, is the dimension of the key vector.

[0024] It can be understood that the multimodal information extraction module is responsible for separating visual text features from the input video. For visual information, the image feature extraction algorithm is used to extract the visual features of the video frame, including object shape, color, texture, motion trajectory, etc.; for text information, the optical character recognition (OCR) technology is used to extract the video subtitles and speech, and the speech recognition technology is used to convert the video subtitles and speech in the audio into text, and then the natural language processing technology is used to extract text features.

[0025] It is understandable that the multimodal information extraction module uses Vision Transformer (ViT) for visual feature extraction. As a model based on the Transformer architecture, it abandons the traditional convolutional neural network structure and uses pure Transformer to process image data.

[0026] It can be understood that through the multi-head self-attention mechanism, it is possible to automatically focus on the areas in the video frame that have an important impact on the event, the actions and expressions of the main characters, and ignore useless information related to the background, thereby extracting more discriminative visual features and establishing time perception between image frame features.

[0027] It is understandable that for text feature extraction, in addition to traditional OCR and speech recognition technologies, pre-trained models in natural language processing are used to semantically enhance text features. The BERT model uses a bidirectional Transformer architecture to encode the input text in context and generate semantically rich text representations. For the extracted subtitles and speech recognition text, manual annotations are added, and the words are segmented and input into the BERT model. The output of the model can capture the semantic relationships, timing information, causal relationships, etc. in the text, which are crucial for subsequent cross-modal fusion. Importantly, in the process of text feature extraction, the timestamp of each frame is additionally added to the text features as additional text information for each image frame, for example, "This frame is sampled at 1 second." This lays the foundation for establishing the temporal alignment of frame-level visual text features during subsequent modal fusion.

[0028] Step S102, through a multimodal feature fusion module, a multi-head self-attention mechanism is used to interact and fuse the visual information and the text information to generate a semantic representation of the video event.

[0029] In an embodiment of the present invention, a visual feature matrix and a text feature matrix are constructed according to visual information and text information through a multimodal feature fusion module, and then the visual feature matrix and the text feature matrix are introduced into the corresponding multi-head self-attention mechanism to capture multimodal semantic associations, and then based on multimodal semantic associations, the visual information and the text information are interacted and fused to generate a video event semantic representation, and adversarial training of multimodal information understanding is performed, and a modal discriminator is added. The source of the feature is discriminated by the modal discriminator. Specifically, the objective function of the adversarial training is: ; in, is the main task loss of the multimodal feature fusion module, the cross entropy loss of event classification or timestamp annotation, is the balance coefficient, is the feature representation of modality fusion, For all The average value of stands for modality discriminator.

[0030] It is understandable that in the multi-head self-attention mechanism, modality-specific attention heads are designed, that is, different attention heads focus on feature processing of visual and textual modalities respectively. In the specific implementation, for the input features of each modality, the visual feature matrix and the text feature matrix , and are input into their own dedicated attention heads. The calculation process of each attention head still follows the standard Transformer mechanism. Then, in the cross-modal stage, the features between different modalities are interacted and integrated through the cross-modal attention heads to explicitly capture the semantic association between visual and textual modalities. In this way, the layer-by-layer stacking of modality specificity and cross-modal attention mechanism can gradually build a multi-modal deep semantic relationship.

[0031] It is understandable that in order to further enhance the model's ability to understand multimodal information, an adversarial training mechanism is introduced during the training process, and a modal discriminator D is added. The task of the modal discriminator is to distinguish the source modality of the input features based on their distribution, and to determine whether a feature is from the visual or textual modality. The multimodal feature fusion module adjusts the distribution of fused features through adversarial training, making it difficult for the modal discriminator D to correctly classify them, thereby forcing feature learning to have modal invariance and higher semantic consistency. Through adversarial games, the modal discriminator continuously improves its ability to discriminate the source of the modality, while the large model continuously adjusts the feature distribution, ultimately achieving semantic alignment between modalities.

[0032] Step S103, through the time series analysis module, using a recurrent neural network to perform time series analysis of event recognition on the semantic representation of the video event, and obtain event timestamp tags and event logic reasoning.

[0033] In the embodiment of the present invention, the semantic representation of video events is processed by the time series analysis module, and then the hidden state sequences corresponding to the multiple time series events are determined by the recurrent neural network. The weights of the time steps corresponding to the multiple hidden state sequences are calculated by the attention mechanism, and then multiple weighted sequence representations are calculated according to the multiple hidden state sequences and the corresponding weights. Then, multiple key event nodes are determined according to the multiple weighted sequence representations, and mapped to the embedding space of the large language model according to the multiple key event nodes to obtain event timestamp tags and event logic reasoning. Specifically, the weight calculation formula is: ; in, is the time step The weight of is the time step The intermediate representation of the hidden state, w is the weight vector, is the total time; The calculation formula for multiple weighted sequence representations is: ; in, is the time step The weighted sequence representation of is the time step hidden state.

[0034] It is understandable that in the time series analysis module, in order to more accurately model the time distribution and evolution logic of events, combining recurrent neural networks and attention mechanisms can improve the ability to capture key information. Recurrent neural networks use long short-term memory networks (LSTM) to effectively model the temporal dependencies of events in time series, while attention mechanisms can assign dynamic weights to events at each time step to highlight the role of key event nodes.

[0035] It is understandable that by calculating multiple weighted sequence representations, key event nodes (such as starting points, turning points, and ending points) occupy a larger proportion in the sequence representation through this weighted method, which helps the model to more accurately capture important patterns on the timeline. In surveillance videos, sudden sound changes (such as explosions) may be the starting point of an event, while obvious visual movements (such as crowds) may correspond to the climax of the event. The attention mechanism can automatically focus on these nodes, providing a more reliable basis for subsequent reasoning.

[0036] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0037] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0038] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0039] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video timestamp event recognition and reasoning method based on a multimodal large model, characterized in that: The method specifically comprises the following steps: Receiving an input video, and processing the input video through a multimodal information extraction module to extract visual information and text information; Through a multimodal feature fusion module, a multi-head self-attention mechanism is used to interact and fuse the visual information and the text information to generate a semantic representation of the video event; Through the time series analysis module, a recurrent neural network is used to perform time series analysis of event recognition on the semantic representation of the video event to obtain event timestamp tags and event logic reasoning.

2. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 1 is characterized in that: The receiving of the input video, processing the input video through the multimodal information extraction module, and extracting the visual information and text information specifically comprises the following steps: Receive input video; Using a multimodal information extraction module and a Vision Transformer, the visual features of the input video are extracted to obtain visual information; Extracting the video subtitles and speech of the input video by optical character recognition technology; Using speech recognition technology, converting the video subtitle speech into speech text; Use natural language processing technology to extract text features from the speech text to obtain text information.

3. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 2 is characterized in that: The method of using Vision Transformer to extract visual features from the input video and obtain visual information specifically includes the following steps: Processing the input video frame by frame to obtain a plurality of frame-by-frame input images; Dividing the plurality of frame-by-frame input images into a plurality of image blocks; Mapping the plurality of image blocks to a D-dimensional vector space to obtain a corresponding first embedding vector; Add position encoding to retain the spatial relationship of the image blocks and generate multiple second embedding vectors with position encoding; Normalizing the timestamps of the plurality of frame-by-frame input images, and calculating a plurality of normalized timestamps; Creating a corresponding time coding vector according to the plurality of normalized timestamps; Adding the plurality of time coding vectors to the corresponding optimized embedding vectors to generate a plurality of third embedding vectors with time and space information; The plurality of third embedding vectors are introduced into a multi-head self-attention mechanism, focusing on the important influence areas in the plurality of frame-by-frame input images, performing visual feature extraction, and acquiring visual information.

4. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 3 is characterized in that: The expressions of the plurality of first embedding vectors are: ； in, For the The first embedding vector of the image block, Representative image blocks, Represents linear change calculation; The calculation formula of the plurality of second embedding vectors is: ； in, For the The second embedding vector of the image block, For the Position encoding of image blocks; The calculation formula for the multiple normalized timestamps is: ； in, For the The normalized timestamp of the image blocks, For the The timestamp of the frame-by-frame input image corresponding to each image block, is the minimum value of the timestamp, The maximum value of the timestamp; The calculation formula of the multiple time coding vectors is: ； in, is the dimension index of the time encoding vector, For the The temporal encoding vector of the image block, is the total dimension of the time encoding vector; The calculation formula of the plurality of third embedding vectors is: ； in, For the The third embedding vector of the image patch.

5. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 4 is characterized in that: The calculation formula of the multi-head self-attention mechanism is: ； in, , and represents the query, key, and value matrices corresponding to the third embedding vector, is the dimension of the key vector.

6. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 1 is characterized in that: The method of generating a video event semantic representation by interacting and fusing the visual information and the text information using a multi-head self-attention mechanism through a multimodal feature fusion module specifically includes the following steps: Constructing a visual feature matrix and a text feature matrix according to the visual information and the text information through a multimodal feature fusion module; Importing the visual feature matrix and the text feature matrix into the corresponding multi-head self-attention mechanism to capture multimodal semantic associations; Based on multimodal semantic association, the visual information and the text information are interacted and fused to generate a semantic representation of a video event.

7. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 6 is characterized in that: The method of generating a video event semantic representation by interacting and fusing the visual information and the text information using a multi-head self-attention mechanism through a multimodal feature fusion module further includes the following steps: Conduct adversarial training for multimodal information understanding and add a modality discriminator; The modal discriminator is used to discriminate the source of features.

8. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 7 is characterized in that: The objective function of the adversarial training is: ； in, is the main task loss of the multimodal feature fusion module, the cross entropy loss of event classification or timestamp annotation, is the balance coefficient, is the feature representation of modal fusion, For all The average value of stands for modality discriminator.

9. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 1 is characterized in that: The time series analysis module is used to perform time series analysis of event recognition on the semantic representation of video events using a recurrent neural network to obtain event timestamps and event logic reasoning, specifically including the following steps: Performing time series event representation processing on the video event semantic representation through a time series analysis module; Determine, using a recurrent neural network, a hidden state sequence corresponding to a plurality of the time series events; Calculating the weights of the time steps corresponding to the multiple hidden state sequences through the attention mechanism; Calculating a plurality of weighted sequence representations according to the plurality of hidden state sequences and corresponding weights; Determining a plurality of key event nodes according to the plurality of weighted sequence representations; According to the plurality of key event nodes, they are mapped to the embedding space of a large language model to obtain event timestamp tags and event logic reasoning.

10. The video timestamp event recognition and reasoning method based on multimodal large model according to claim 9 is characterized in that: The calculation formula of the weight is: ； in, is the time step The weight of is the time step The intermediate representation of the hidden state, w is the weight vector, is the total time; The calculation formula for the weighted sequence representation is: ； in, is the time step The weighted sequence representation of is the time step The hidden state.

Images