Token data fusion and video data processing method, system, device and medium

Abstract

This invention provides a token data fusion method and system, as well as a video data processing method and system. The token data fusion method includes: acquiring raw token data from the video data to be processed; extracting first token data from the raw token data according to a first frame rate and extracting second token data according to a second frame rate, where the first frame rate is less than the second frame rate; calculating an attention weight matrix between the first token data and the second token data; and generating fused token data for the video data based on the attention weight matrix and the first token data. The token data fusion scheme proposed in this invention effectively alleviates resource consumption issues through hierarchical processing and attention weight fusion, significantly enhancing the model's spatiotemporal awareness and analytical accuracy, thus exhibiting superior performance and applicability when processing long videos and dynamic, complex scenes.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a token data fusion method, a video data processing method, a token data fusion system, a video data processing system, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In recent years, with the development of multimodal technologies, video large language models have become an important direction in multimodal research. These models combine video input with pre-trained large language models (LLMs) to achieve understanding and textual representation of complex video content. However, due to the high spatiotemporal complexity of video data, existing video large language models still face many challenges in terms of computational resource requirements and algorithm accuracy.

[0003] 1. Resource consumption issues

[0004] The high-dimensionality of video data makes visual information processing a crucial step in large video language models. In most existing methods, after video frames undergo feature extraction by a visual encoder, a large number of visual tokens are generated. As the number of video frames increases, the number of these visual tokens grows exponentially, significantly increasing the computational burden of subsequent LLM (Large Video Language Model). Consequently, many existing large video language models only support processing a limited number of video frame inputs, thus limiting their ability to capture fine-grained spatiotemporal information across the entire video. This issue not only affects the computational efficiency of the model but also weakens its performance in long video analysis.

[0005] 2. Insufficient spatiotemporal awareness

[0006] Most current video large language models employ a simple single-frame feature processing strategy, which involves breaking down the video into a series of independent image frames, extracting features from each frame, and directly inputting them into the LLM (Language Model). This approach ignores the temporal relationships inherent in time-series characteristics and lacks effective modeling of continuous dynamic changes within the video content. Furthermore, this method completely entrusts the task of understanding complex spatiotemporal information to the LLM, which itself is not specifically optimized for motion information within the video scene. This approach results in the model's inadequate performance in capturing the complex spatial semantics and long-term temporal context of video content, making it difficult to accurately analyze and deeply understand dynamically changing scenes. Summary of the Invention

[0007] In view of the above problems, embodiments of the present invention are proposed to provide a token data fusion method, a video data processing method, a token data fusion system, a video data processing system, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.

[0008] To address the aforementioned problems, this invention discloses a token data fusion method, the method comprising:

[0009] Obtain the raw token data of the video data to be processed;

[0010] First token data is extracted from the original token data according to a first frame rate, and second token data is extracted according to a second frame rate, wherein the first frame rate is less than the frame rate.

[0011] Calculate the attention weight matrix between the first token data and the second token data;

[0012] The video data is fused into token data based on the attention weight matrix and the first token data.

[0013] Optionally, generating the fused token data of the video data based on the attention weight matrix and the first token data includes:

[0014] The first token data is linearly mapped to obtain the value token data;

[0015] The attention weight matrix is ​​reversed by performing a reverse numerical mapping to obtain the reverse attention weight matrix;

[0016] The fusion token data is generated based on the inverse attention weight matrix and the value token data.

[0017] Optionally, generating the fused token data based on the inverse attention weight matrix and the value token data includes:

[0018] The reverse attention weight matrix is ​​multiplied by the value token data to obtain multiple product results;

[0019] The fusion token data is obtained by weighted summation of multiple product results.

[0020] Optionally, calculating the attention weight matrix between the first token data and the second token data includes:

[0021] The first token data is linearly mapped to obtain key token data, and the second token data is linearly mapped to obtain query token data;

[0022] A similarity score matrix is ​​obtained by performing a dot product operation on the key token data and the query token data;

[0023] The attention weight matrix is ​​obtained by applying the SoftMax function to the similarity score matrix.

[0024] Optionally, extracting the first token data from the original token data according to the first frame rate includes:

[0025] The first token data is obtained by uniformly sampling from the original token data according to the first frame rate, and spatial pooling is performed on the first token data.

[0026] Optionally, extracting the second token data from the original token data according to the second frame rate includes:

[0027] The original token data is spatially downsampled at the second frame rate to obtain the second token data.

[0028] Optionally, the step of obtaining the raw token data of the video data to be processed includes:

[0029] Obtain the original token data after the video data has been mapped through the mapping layer.

[0030] This invention also discloses a video data processing method, the method comprising:

[0031] Obtain the fused token data of the video data according to the token data fusion method described above;

[0032] The fusion token data is input into the large language model, and the task result of the video data is output according to the task instructions of the video data.

[0033] This invention also discloses a token data fusion system, the system comprising:

[0034] The token data acquisition module is used to acquire the raw token data of the video data to be processed.

[0035] The token data extraction module is used to extract first token data from the original token data according to a first frame rate and extract second token data according to a second frame rate, wherein the first frame rate is less than the second frame rate.

[0036] The attention weight matrix calculation module is used to calculate the attention weight matrix between the first token data and the second token data.

[0037] The token data fusion module is used to generate fused token data for the video data based on the attention weight matrix and the first token data.

[0038] Optionally, the token data fusion module includes:

[0039] The first mapping module is used to perform a linear mapping on the first token data to obtain value token data;

[0040] The reverse mapping module is used to perform reverse numerical mapping on the attention weight matrix to obtain a reverse attention weight matrix;

[0041] The fusion generation module is used to generate the fusion token data based on the reverse attention weight matrix and the value token data.

[0042] Optionally, the fusion generation module includes:

[0043] The product module is used to multiply the reverse attention weight matrix with the value token data to obtain multiple product results;

[0044] The summation module is used to perform a weighted summation of multiple product results to obtain the fusion token data.

[0045] Optionally, the attention weight matrix calculation module includes:

[0046] The second mapping module is used to perform a linear mapping on the first token data to obtain key token data, and to perform a linear mapping on the second token data to obtain query token data;

[0047] The dot product operation module is used to perform a dot product operation on the key token data and the query token data to obtain a similarity score matrix;

[0048] The function execution module is used to execute the SoftMax function on the similarity score matrix to obtain the attention weight matrix.

[0049] Optionally, the token data extraction module is used to obtain the first token data by uniformly sampling from the original token data according to the first frame rate, and to perform spatial pooling operation on the first token data.

[0050] Optionally, the token data extraction module is used to perform spatial downsampling on the original token data according to the second frame rate to obtain the second token data.

[0051] Optionally, the token data acquisition module is used to acquire the original token data after the video data has been mapped through the mapping layer.

[0052] This invention also discloses a video data processing system, the system comprising:

[0053] The fusion token data acquisition module is used to acquire the fusion token data of the video data according to the token data fusion method described above;

[0054] The task result output module is used to input the fusion token data into the large language model and output the task result of the video data according to the task instructions of the video data.

[0055] This invention also discloses an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform the token data fusion method as described above, and / or the video data processing method as described above.

[0056] This invention also discloses a computer-readable storage medium storing a computer program that causes a processor to execute the token data fusion method described above, and / or the video data processing method described above.

[0057] The embodiments of the present invention have the following advantages:

[0058] The token data fusion scheme provided in this embodiment of the invention obtains the original token data of the video data to be processed; extracts the first token data from the original token data according to a first frame rate and extracts the second token data according to a second frame rate, wherein the first frame rate is less than the second frame rate; calculates the attention weight matrix between the first token data and the second token data; and generates fused token data of the video data based on the attention weight matrix and the first token data.

[0059] Compared to traditional methods in the background art, the token data fusion scheme proposed in this invention significantly improves the performance of video large language models in terms of resource utilization and spatiotemporal perception through the separation and fusion of multi-level token data. Specifically, this scheme brings the following beneficial effects:

[0060] 1. Reduce computing resource consumption

[0061] In the background technology, because the features of all video frames are processed equally, the number of visual tokens increases exponentially with the number of frames, resulting in excessive consumption of computing resources. This solution extracts first token data (first frame rate, low frame rate) and second token data (second frame rate, high frame rate) from the original token data at different frame rates (first frame rate and second frame rate). The first token data is extracted from the original token data through downsampling, significantly reducing its quantity; however, directly inputting it as fused token data into the LLM may lead to information loss. Therefore, this solution calculates an attention weight matrix that integrates the spatiotemporal information and global semantic information of the second token data. Fusion token data is then generated based on the attention weight matrix and the first token data. This ensures that the fused token data maintains the same data volume as the first token data while compensating for dynamic details and important semantics that may be lost through direct downsampling via weight correction. This significantly reduces the number of token data input to the LLM, lowers subsequent computing resource requirements, and enables the LLM to efficiently process long video content, expanding its applicability.

[0062] 2. Enhance the model's spatiotemporal awareness.

[0063] In the background, simply decomposing video into single-frame features and inputting them into an LLM ignores the temporal relationships between frames, limiting the accurate modeling of dynamic changes in the video. This scheme captures the correlation between global information at low frame rates and local details at high frame rates by calculating an attention weight matrix between the first and second token data. Subsequently, the two types of token data are fused using the attention weight matrix, achieving collaborative modeling of global and local information. This enhances the model's ability to capture spatiotemporal information in video, enabling it to more accurately understand the complex spatial semantics and temporal context in dynamic scenes.

[0064] 3. Improve analytical accuracy and consistency

[0065] Traditional methods for analyzing long videos, limited by frame rate, struggle to capture fine-grained spatiotemporal information, potentially leading to inconsistent or inaccurate results. This approach, by fusing token data from different frame rates, not only preserves global semantic information but also effectively captures local details and dynamic changes. This multi-layered information integration enhances the model's ability to analyze video content, ensuring consistency and accuracy. It enables the model to comprehensively understand video content, particularly accurately capturing dynamically changing scenes and long-term semantic relationships, thus improving its applicability in complex video scenarios.

[0066] 4. Improved ability to adapt to long video processing

[0067] Thanks to reduced computational resource consumption and enhanced spatiotemporal modeling capabilities, this solution can handle longer video content. It performs exceptionally well in scenarios requiring the analysis of long videos (such as surveillance videos and film / television content analysis), resolving the issues of inefficiency and information loss inherent in traditional methods when processing long videos.

[0068] In summary, the token data fusion scheme proposed in this invention effectively alleviates the resource consumption problem through hierarchical processing and attention weight fusion, significantly enhances the model's spatiotemporal perception capability and analysis accuracy, and thus demonstrates superior performance and applicability when processing long videos and dynamic complex scenes. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the steps of a token data fusion method according to an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of the spatiotemporal token fusion scheme based on the reverse attention mechanism according to an embodiment of the present invention.

[0071] Figure 3 This is a schematic diagram of slow frame branch token time downsampling in a spatiotemporal token fusion scheme based on a reverse attention mechanism according to an embodiment of the present invention.

[0072] Figure 4 This is a schematic diagram of fast frame branch token space downsampling in a spatiotemporal token fusion scheme based on a reverse attention mechanism according to an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram of the reverse attention mechanism in a spatiotemporal token fusion scheme based on the reverse attention mechanism according to an embodiment of the present invention;

[0074] Figure 6 This is a structural block diagram of a token data fusion system according to an embodiment of the present invention;

[0075] Figure 7 This is a structural block diagram of a video data processing system according to an embodiment of the present invention. Detailed Implementation

[0076] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] The token data fusion scheme proposed in this invention significantly improves the efficiency and spatiotemporal awareness of video large language models by extracting and fusing token data from video data in a hierarchical manner. Specifically, the scheme acquires the original token data of the video data to be processed, and extracts first token data according to a first frame rate (low frame rate) and second token data according to a second frame rate (high frame rate). Subsequently, by calculating the attention weight matrix between the two types of token data, efficient and information-density fused token data is generated based on this matrix. This scheme reduces computational resource consumption while enhancing the model's ability to perceive spatiotemporal dynamics, thereby achieving efficient processing and accurate analysis of long video content, effectively solving the problems of excessive resource consumption and insufficient spatiotemporal modeling in traditional methods.

[0078] Reference Figure 1 This diagram illustrates a flowchart of a token data fusion method according to an embodiment of the present invention. This token data fusion method can be applied to a token data fusion system, hereinafter referred to as the system. Specifically, the token data fusion method may include the following steps:

[0079] Step 101: Obtain the raw token data of the video data to be processed.

[0080] This step is the starting point for the token data fusion method, laying the foundation for subsequent processing by extracting raw token data from the video data. The raw video data presents dynamically changing scene information in a multi-frame format, containing rich spatial and temporal features. However, to directly adapt to large language models, the video data needs to be transformed into a token format that the model can process. Specifically, the video data is processed by a feature extractor (such as a Transformer or convolutional neural network) to convert it into a high-dimensional feature representation. These features are then subjected to dimensionality reduction processing through a projection layer, forming a series of raw token data. This raw token data represents each frame of the video in the spatial and temporal dimensions and is typically input into subsequent processing units in the form of a fixed-length sequence.

[0081] The key to this step is ensuring that the generated raw token data fully preserves the core feature information of the video while reducing redundant information to optimize subsequent computational efficiency. During feature extraction, video data with different frame rates, resolutions, or encoding methods can be standardized into a unified token sequence, thereby improving the applicability and scalability of the scheme.

[0082] Step 102: Extract the first token data from the original token data according to the first frame rate, and extract the second token data according to the second frame rate.

[0083] In this step, the original token data is decomposed into two distinct token data sets: first token data and second token data, representing low-frame-rate and high-frame-rate video data, respectively. The first token data (corresponding to the relatively low-frame-rate video data) is extracted from the original token data through uniform sampling to reduce computational burden while preserving important global semantic information. For example, a token is extracted every few frames from the high-frame-rate video to form a sparse but globally feature-rich sequence. This process is typically combined with spatial pooling operations to further compress the spatial resolution of the first token data, thereby reducing computational resource requirements. The second token data (corresponding to the relatively high-frame-rate video data) retains the token data from all frames in the video, capturing motion information and dynamically changing temporal context through a larger spatial pooling stride (e.g., downsampling to 4×4 tokens per frame). This step, through the separation of fast and slow frames (low and high frame rates), establishes a multi-temporal-scale feature representation framework, laying the foundation for subsequent fusion operations.

[0084] Step 103: Calculate the attention weight matrix between the first token data and the second token data.

[0085] This step aims to uncover the correlation between the first and second token data using an attention mechanism. First, the second token data is mapped to a query token, and the first token data is mapped to a key token and a value token, respectively. Then, a dot product operation is performed between the query token and the key token, and an attention weight matrix is ​​calculated using the SoftMax function. This matrix represents the correlation between the first and second token data, dynamically capturing the complementary characteristics of global information in slow frames and motion information in fast frames.

[0086] To enhance feature representation capabilities, this scheme introduces a reverse attention mechanism, which performs a reverse numerical mapping on the attention matrix to highlight details in the first token data that were not noticed by the second token data. The reverse attention mechanism effectively compensates for the shortcomings of traditional attention methods in capturing complex spatiotemporal features, enabling the model to more accurately balance the information weights between fast and slow frame features. The final attention weight matrix not only reflects the dependency between the two types of token data but also provides important weighting criteria to guide the fusion operation.

[0087] Step 104: Generate fused token data for the video data based on the attention weight matrix and the first token data.

[0088] In this step, a matrix multiplication operation is performed between the attention weight matrix and the value Token in the first token data, and the results are summed to generate fused token data. This process integrates slowly changing visual semantics and rapidly changing motion features, effectively fusing features across multiple spatiotemporal scales. Specifically, high-attention regions in the attention weight matrix reflect the high correlation between fast and slow frames, and by aggregating information from related regions, important details in dynamic scenes are preserved; while the reverse attention mechanism supplements the high-resolution details of slow frames missed in fast frames.

[0089] The fusion of token data not only comprehensively represents the spatial and temporal information of the video, but also reduces the computational burden on the input LLM through optimized control of the number of tokens. This fusion approach balances the model's modeling capabilities with hardware resource requirements, ensuring that fine-grained features are preserved while improving the overall performance and adaptability of the video large language model.

[0090] The token data fusion scheme provided in this embodiment of the invention obtains the original token data of the video data to be processed; extracts the first token data from the original token data according to a first frame rate and extracts the second token data according to a second frame rate, wherein the first frame rate is less than the second frame rate; calculates the attention weight matrix between the first token data and the second token data; and generates fused token data of the video data based on the attention weight matrix and the first token data.

[0091] Compared to traditional methods in the background art, the token data fusion scheme proposed in this invention significantly improves the performance of video large language models in terms of resource utilization and spatiotemporal perception through the separation and fusion of multi-level token data. Specifically, this scheme brings the following beneficial effects:

[0092] 1. Reduce computing resource consumption

[0093] In the background technology, because the features of all video frames are processed equally, the number of visual tokens increases exponentially with the number of frames, resulting in excessive consumption of computing resources. This solution extracts first token data (first frame rate, low frame rate) and second token data (second frame rate, high frame rate) from the original token data at different frame rates (first frame rate and second frame rate). The first token data is extracted from the original token data through downsampling, significantly reducing its quantity; however, directly inputting it as fused token data into the LLM may lead to information loss. Therefore, this solution calculates an attention weight matrix that integrates the spatiotemporal information and global semantic information of the second token data. Fusion token data is then generated based on the attention weight matrix and the first token data. This ensures that the fused token data maintains the same data volume as the first token data while compensating for dynamic details and important semantics that may be lost through direct downsampling via weight correction. This significantly reduces the number of token data input to the LLM, lowers subsequent computing resource requirements, and enables the LLM to efficiently process long video content, expanding its applicability.

[0094] 2. Enhance the model's spatiotemporal awareness.

[0095] In the background, simply decomposing video into single-frame features and inputting them into an LLM ignores the temporal relationships between frames, limiting the accurate modeling of dynamic changes in the video. This scheme captures the correlation between global information at low frame rates and local details at high frame rates by calculating an attention weight matrix between the first and second token data. Subsequently, the two types of token data are fused using the attention weight matrix, achieving collaborative modeling of global and local information. This enhances the model's ability to capture spatiotemporal information in video, enabling it to more accurately understand the complex spatial semantics and temporal context in dynamic scenes.

[0096] 3. Improve analytical accuracy and consistency

[0097] Traditional methods for analyzing long videos, limited by frame rate, struggle to capture fine-grained spatiotemporal information, potentially leading to inconsistent or inaccurate results. This approach, by fusing token data from different frame rates, not only preserves global semantic information but also effectively captures local details and dynamic changes. This multi-layered information integration enhances the model's ability to analyze video content, ensuring consistency and accuracy. It enables the model to comprehensively understand video content, particularly accurately capturing dynamically changing scenes and long-term semantic relationships, thus improving its applicability in complex video scenarios.

[0098] 4. Improved ability to adapt to long video processing

[0099] Thanks to reduced computational resource consumption and enhanced spatiotemporal modeling capabilities, this solution can handle longer video content. It performs exceptionally well in scenarios requiring the analysis of long videos (such as surveillance videos and film / television content analysis), resolving the issues of inefficiency and information loss inherent in traditional methods when processing long videos.

[0100] In summary, the token data fusion scheme proposed in this invention effectively alleviates the resource consumption problem through hierarchical processing and attention weight fusion, significantly enhances the model's spatiotemporal perception capability and analysis accuracy, and thus demonstrates superior performance and applicability when processing long videos and dynamic complex scenes.

[0101] In an exemplary embodiment of the present invention, one implementation of generating fused token data of video data based on the attention weight matrix and the first token data is as follows: performing a linear mapping on the first token data to obtain value token data; performing a reverse numerical mapping on the attention weight matrix to obtain a reverse attention weight matrix; and generating fused token data based on the reverse attention weight matrix and the value token data.

[0102] In this implementation, by processing the first token data (slow-frame token data) and the attention weight matrix, effective integration of multi-temporal and spatial information is achieved, thereby generating fused token data for subsequent video analysis tasks. The specific operation process is as follows:

[0103] First, the initial token data is linearly mapped to generate value token data. This linear mapping process is typically achieved through a specially designed mapping layer (e.g., a fully connected layer or a projection layer). Its purpose is to transform the slow-frame token data in the feature space, making it compatible with subsequent calculations while ensuring that the global spatial semantic information in the video is effectively represented. The value token data not only preserves the spatial details of the slow-frame token data but also provides the necessary feature carrier for subsequent weighted fusion.

[0104] Subsequently, the attention weight matrix is ​​reversed numerically to generate an inverse attention weight matrix. This inverse numerical mapping aims to highlight aspects of the slow-frame token data that are not adequately addressed by the fast-frame token data by reversing the numerical characteristics of the attention weight matrix. This process captures high-resolution slow-frame information missed in the dynamic motion features of fast frames, thus compensating for the shortcomings of traditional attention mechanisms in capturing long-term static semantics in dynamic video analysis. Through this reverse processing operation, key spatial details in the slow-frame token data are enhanced, providing a foundation for generating more balanced and comprehensive fused token data.

[0105] Finally, the back attention weight matrix and the value token data are multiplied together, and the results are summed to generate fused token data. This process leverages the weight distribution characteristics of the back attention mechanism, ensuring the effective fusion of complementary features between dynamic motion information and static spatial details. The fused token data not only preserves multi-temporal and spatiotemporal scale features in the video but also significantly reduces the introduction of redundant information, thereby optimizing the number of tokens input into the large language model.

[0106] This invention effectively addresses the problem of insufficient attention to static semantics in traditional video language models when capturing dynamic video content by introducing a reverse attention mechanism. The reverse attention weight matrix enhances details in slow-frame token data that are missed by fast-frame token data, enabling the fused token data to more comprehensively express the global semantics and local dynamic features in the video. Furthermore, through linear mapping and refined processing of the weight matrix, the number of tokens is optimized during the fusion process, significantly reducing computational resource requirements and improving the model's ability to process long videos.

[0107] In an exemplary embodiment of the present invention, one way to generate fused token data based on the reverse attention weight matrix and the value token data is as follows: multiply the reverse attention weight matrix and the value token data to obtain multiple product results; and perform a weighted summation of the multiple product results to obtain the fused token data.

[0108] This implementation details the specific process of generating fused token data using the reverse attention weight matrix and value token data, and achieves the integration and optimization of multi-temporal features through matrix multiplication and weighted summation operations.

[0109] First, the inverse attention weight matrix is ​​multiplied with the value token data. The core feature of the inverse attention weight matrix is ​​that it highlights slow-frame token data, which is often overlooked by fast-frame token data, through inverse numerical mapping, thus compensating for the omission of global semantic information about slow frames in the dynamic features of fast frames. The value token data is a feature representation generated by linearly mapping the slow-frame token data; its main function is to carry key static semantic information in the spatial dimension. During matrix multiplication, each weight value in the inverse attention weight matrix serves as an interaction coefficient between dynamic and static features, weighting and adjusting the features of each dimension of the value token data. This process effectively enhances the details of slow frames through the inverse attention mechanism, enabling the fused token data to more comprehensively express the multi-temporal and spatial features of the video.

[0110] Next, a weighted summation operation is performed on the multiple product results. The product results are essentially subsets of features generated after reconstructing each feature dimension in the slow frame according to the weight matrix. By weighted summing of these subsets, the harmonizing effect of the backattention matrix on global semantics and local dynamic features can be further strengthened. Weighted summation not only optimizes the balance of feature representation but also integrates multi-view spatiotemporal information from the video, providing more efficient input for subsequent large-scale language models. In this operation, the weight allocation follows the dependencies between matrices, ensuring that the contributions of different feature dimensions are consistent with the semantic requirements of the video, thereby avoiding the accumulation of redundant information.

[0111] Through the above steps, the generated fused token data retains both dynamically changing temporal contextual features and supplements static semantic details, providing the model with a more complete and compact video representation. This multi-step operation balances the advantages and disadvantages of fast and slow frame features during information integration, providing a solid data foundation for subsequent video analysis and task processing.

[0112] This implementation leverages a reverse attention weight matrix to enhance the focus on global semantic information in slow frames, effectively addressing the problem of insufficient capture of static semantic details in traditional attention mechanisms. Through matrix multiplication and weighted summation operations, features at different time scales are fully integrated, significantly improving the model's ability to understand complex dynamic scenes. Furthermore, the weighted summation process reduces the risk of introducing redundant features, decreasing the number of tokens input to a large language model while ensuring model accuracy. This not only optimizes computational resource requirements but also expands the model's application scope in long video processing and dynamically changing scenarios.

[0113] In an exemplary embodiment of the present invention, one method for calculating the attention weight matrix between the first token data and the second token data is as follows: performing a linear mapping on the first token data to obtain key token data, and performing a linear mapping on the second token data to obtain query token data; performing a dot product operation on the key token data and the query token data to obtain a similarity score matrix; and performing the SoftMax function on the similarity score matrix to obtain the attention weight matrix.

[0114] This implementation calculates the attention weight matrix between the first token data and the second token data through linear mapping, dot product operation, and the SoftMax function, laying an important foundation for the subsequent fusion of multi-temporal features.

[0115] First, the first token data is linearly mapped to generate key token data. The first token data, serving as a feature representation of slow frames at low frame rates, primarily contains global semantic information and key static features in the spatial dimension. Through linear mapping, the first token data can be transformed into a vector representation of a specific dimension, i.e., the key token data. The generation of key token data aims to characterize the dependency and contribution of slow frame information within the context of fast frames in subsequent steps, while maintaining the compactness and efficiency of feature representation.

[0116] Simultaneously, query token data is generated by linearly mapping the second token data. The second token data, as a feature of high frame rate fast frames, primarily captures dynamically changing motion information in the video. Through linear mapping, the query token data can map the motion information of fast frames to a specific vector dimension, making it suitable for subsequent dot product calculations. The generation of query token data aims to characterize the contextual relationship of fast frame dynamic features within the global semantics of slow frames, thereby enabling efficient querying and utilization of dynamic information.

[0117] Next, a dot product operation is performed on the key token data and query token data to obtain a similarity score matrix. The essence of the dot product operation is to measure the correlation between two token data sets; the similarity score matrix reflects the semantic matching degree between slow-frame and fast-frame features. For each query token, the dot product result can capture its dependency relationship with all key tokens in the feature space, thus dynamically representing the interaction pattern between fast and slow-frame features.

[0118] Finally, the SoftMax function is applied to the similarity score matrix to generate the attention weight matrix. The SoftMax function normalizes the similarity scores, transforming them into a probability distribution; larger weight values ​​indicate a higher degree of attention paid by fast-frame features to slow-frame features. The attention weight matrix provides an effective weight foundation for subsequent feature fusion by finely adjusting the interaction weights between fast-frame and slow-frame features.

[0119] This implementation achieves efficient correlation between slow-frame static features and fast-frame dynamic features by calculating the attention weight matrix, enhancing the model's ability to understand multi-temporal and spatiotemporal features. The generation of key tokens and query tokens enables the model to extract global semantic information and temporal contextual information respectively, providing complementary feature representations for dynamically changing scenes and static spatial details. The use of the similarity score matrix and the SoftMax function not only improves computational efficiency but also effectively avoids information redundancy, making the attention distribution more accurate. The final generated attention weight matrix significantly improves the quality of feature fusion, enabling the model to understand video content more comprehensively while reducing computational burden. Therefore, this implementation achieves a good balance between modeling capability and computational efficiency, possessing significant technical value and practical application prospects.

[0120] In an exemplary embodiment of the present invention, one way to extract the first token data from the original token data according to the first frame rate is as follows: the first token data is obtained by uniformly sampling from the original token data according to the first frame rate, and the first token data is subjected to spatial pooling operation.

[0121] This implementation extracts the first token data from the original token data through uniform sampling and spatial pooling operations, providing an efficient feature representation for subsequent computation and fusion. The specific process consists of two steps: uniform sampling and spatial pooling. Each step plays an important role in reducing computational complexity, improving model efficiency, and preserving key information.

[0122] First, the first token data is extracted from the original token data through uniform sampling. The original token data contains information from all frames of the video; the information from each frame is converted into multiple tokens, forming a high-dimensional feature set. When processing long videos, directly using all the token data for computation would result in significant computational resource consumption. To optimize computational efficiency, representative token data is selected through uniform sampling. This process is achieved by sampling video frames at regular intervals, ensuring that important temporal information is not lost while reducing data redundancy. Uniform sampling effectively preserves the key information of the video while significantly reducing the computational burden in subsequent processing, providing a more efficient data input for further spatiotemporal information modeling.

[0123] Next, spatial pooling is performed on the first token data. Spatial pooling is a dimensionality reduction method that aggregates token data along spatial dimensions, compressing high-dimensional features into lower-dimensional representations, thereby reducing computational complexity. In this implementation, spatial pooling not only preserves important spatial semantic information in the first token data but also effectively compresses the data volume, improving the model's response speed when processing complex videos. Spatial pooling typically employs strategies such as max pooling or average pooling, selecting the most important or average features within the pooling region for representation. This reduces the data volume while ensuring that no key information is lost.

[0124] This implementation optimizes the use of computational resources and processing efficiency when extracting the first token data through uniform sampling and spatial pooling operations. Uniform sampling allows the model to selectively retain key time frame data when processing video, avoiding over-computation of all frame data and thus effectively reducing computational complexity. Spatial pooling, through dimensionality reduction, reduces the size of the token data while retaining sufficient spatial semantic information, enabling subsequent processing to more efficiently focus on the key features of the video.

[0125] Furthermore, this implementation method also helps improve the model's spatiotemporal modeling capabilities. Through uniform sampling and pooling, the model can better balance the extraction of temporal and spatial features, ensuring that effective information is captured at different time scales, while reducing the negative impact of redundant data on model performance. This lays a solid foundation for further spatiotemporal information fusion and analysis, enabling the video large language model to achieve a better balance between computational efficiency and analytical accuracy.

[0126] In one exemplary embodiment of the present invention, one method of extracting the second token data from the original token data according to the second frame rate is as follows: spatial downsampling is performed on the original token data according to the second frame rate to obtain the second token data.

[0127] This implementation extracts the second token data by performing spatial downsampling on the original token data. The core of this process is to compress the original token data through downsampling, thereby reducing the data's dimensionality and size while preserving important spatiotemporal information. Downsampling is a common dimensionality reduction technique that effectively reduces computational complexity while maintaining key data information.

[0128] First, when processing video data, the raw token data often contains a wealth of spatial information and details, not all of which may be crucial for capturing the dynamic changes in the video. By performing spatial downsampling, the spatial information in the raw token data is compressed to a lower resolution, removing unnecessary details while preserving the most important features of the video. Spatial downsampling is typically achieved by aggregating surrounding neighboring pixels (or tokens). This can be done through average pooling, max pooling, or other types of aggregation operations. The aggregated data retains important spatial information rather than performing detailed calculations on every single pixel, thus reducing computational overhead.

[0129] Secondly, the goal of spatial downsampling is to generate second token data, which will be presented at a lower resolution but still effectively characterize motion changes and temporal dynamics in the video. Because downsampling reduces the use of computational resources, the generated second token data can focus on capturing more critical and rapidly changing parts of the video content, especially dynamic changes over time. In this way, the second token data becomes a more efficient feature representation, facilitating subsequent spatiotemporal fusion and analysis tasks.

[0130] This implementation reduces the dimensionality and computational complexity of video data through spatial downsampling. Raw token data typically contains a wealth of detail, while downsampling removes redundant information, focusing on preserving the most important spatiotemporal features of the video. This not only effectively reduces the demand for computing resources but also improves processing efficiency, especially when processing long-duration video data, significantly reducing computation time and memory consumption.

[0131] Furthermore, spatial downsampling can optimize the information extraction process in video understanding tasks. By reducing spatial dimensions, the second token data can more centrally represent dynamic information, enabling the model to more efficiently identify and analyze rapidly changing motion features when processing videos. This method is particularly suitable for large-scale video analysis tasks, as it can reduce data redundancy and optimize computational paths, allowing the model to maintain high accuracy when processing dynamic scenes while simultaneously achieving computational optimization.

[0132] In one exemplary embodiment of the present invention, one way to obtain the original token data of the video data to be processed is to obtain the original token data of the video data after it has been mapped by the mapping layer.

[0133] This implementation converts raw video data into raw token data through a mapping layer, thus laying the foundation for subsequent feature extraction and analysis. The role of the mapping layer is to transform video data from its original pixel-level or video frame representation into token data format suitable for model processing. This transformation allows for more efficient processing of video data and facilitates subsequent spatiotemporal feature modeling and analysis.

[0134] First, video data typically exists as a sequence of images at a certain number of frames per second. Each frame contains rich visual information, usually including various elements such as background, people, objects, and actions, which need to be extracted in subsequent analysis. To facilitate subsequent feature extraction, the raw video data needs to be transformed into token data through a mapping layer.

[0135] The mapping layer plays a crucial role in this process. Its function is to transform the pixel data of each frame into a set of high-dimensional feature vectors, which typically represent the main information of each frame more compactly. Traditional image processing methods usually use the image's pixel data directly for computation. However, in video language models, due to the need to process large-scale data, directly using pixel data is often too cumbersome and redundant. By extracting and transforming features from the original image, the mapping layer can compress video data into a smaller, highly abstract set of tokens. These tokens not only retain the key information in the video but also effectively reduce computational complexity.

[0136] In the mapping layer, methods such as Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs) are typically used to extract and map image features. These networks can identify elements such as objects, scenes, and actions in an image and convert them into vector form. Each token represents a local or global piece of information in the video, and the mapped token data has a high semantic information density, which can better support subsequent model training and inference.

[0137] This implementation transforms video data into token data through a mapping layer, significantly improving computational efficiency and feature extraction capabilities. By mapping video data into a more concise token representation, the mapping layer reduces redundant information in subsequent processing, enabling the model to process video content more efficiently. This transformation not only reduces the data scale processed by the model but also improves the model's ability to represent video content, especially when processing long videos, better preserving key spatiotemporal features.

[0138] Furthermore, the introduction of the mapping layer makes video data processing more aligned with the input requirements of deep learning models. Token data, as an input format for deep learning models, can be widely used in various model architectures, such as Transformer. This enables video large language models to utilize advanced natural language processing techniques for video analysis, improving the model's generalization ability and multi-task processing capabilities.

[0139] Based on the above description of an embodiment of a token data fusion method, a video data processing method is introduced below. This method obtains fused token data of video data according to the above embodiment of the token data fusion method, then inputs the fused token data into a large language model, and outputs the task result of the video data according to the task instructions of the video data.

[0140] The core of the aforementioned video data processing method lies in optimizing the extraction and representation of video features through token data fusion and combining it with a large language model to complete the processing of task instructions. Specifically, this method can be divided into the following key steps:

[0141] The stages of token data fusion: (For details, please refer to the above implementation example of the token data fusion method, which will not be repeated here)

[0142] Input fusion token data into the large language model:

[0143] The generated fused token data is passed as input to the large language model. Since the amount of fused token data has been significantly reduced, the computational resource consumption of the large language model is reduced compared to directly inputting the original token data. At the same time, the input data still retains key spatiotemporal information, thus ensuring the model's ability to understand the video content.

[0144] Process task instructions and generate task results:

[0145] After receiving the fusion token data, the large language model analyzes and processes it based on the user-provided video data task instructions (such as video content description, event recognition, question answering, plot summary, etc.) using its powerful natural language processing capabilities. Ultimately, the large language model generates task results corresponding to the task instructions, such as an accurate description of the video content, a detailed summary of the video plot, or an answer to a specific question.

[0146] Based on the above description of an embodiment of a token data fusion method, a spatiotemporal token fusion scheme based on a reverse attention mechanism is introduced below. (Refer to...) Figure 2 The diagram illustrates a schematic of the spatiotemporal token fusion scheme based on a reverse attention mechanism according to an embodiment of the present invention.

[0147] First, the video data is processed by a uniform sampler, which samples frames evenly across the temporal dimension. This reduces redundancy while preserving key temporal information, thereby lowering the computational complexity of subsequent processing. The sampled frames are then processed by a visual encoder to extract features, generating preliminary visual tokens that capture the spatial semantic information and basic temporal context features of the frames. To ensure that the input features of the two-stream network have a uniform dimension and representation, the extracted features are further mapped to a consistent feature space through a projector layer.

[0148] Next, the processing flow is divided into two paths: the Slow Pathway and the Fast Pathway. The Slow Pathway performs temporal downsampling on the input, extracting low-frame-rate tokens while preserving the spatial semantic information of the frames. It also uses pooling operations to compress the number of tokens, reducing computational complexity. The Fast Pathway retains high-frame-rate tokens and focuses on capturing rapidly changing motion information in the video through spatial downsampling, enhancing the ability to model the temporal context. These two paths generate slow-frame tokens and fast-frame tokens respectively, forming a multi-temporal and spatiotemporal scale feature representation.

[0149] Subsequently, these tokens are fed into an inverse attention mechanism for further spatiotemporal feature fusion. Fast frame tokens serve as queries, and slow frame tokens as keys and values. The inverse attention mechanism calculates the slow frame information not covered by the fast frames, generating fused tokens containing complementary features. This mechanism achieves dynamic integration of fast and slow frame features, not only compressing the number of visual tokens but also preserving crucial spatiotemporal information.

[0150] The fused tokens undergo a flattening operation, flattening the multi-dimensional features into a one-dimensional sequence. This sequence is then transformed into text tokens acceptable to the language model using a tokenizer. Finally, these tokens are input into the Large Language Model, which generates corresponding natural language descriptions, completing the conversion from video content to text output.

[0151] Multi-temporal-scale feature extraction techniques based on dual-stream networks:

[0152] Reference Figure 3 The diagram illustrates a slow-frame branch token time downsampling in a spatiotemporal token fusion scheme based on a reverse attention mechanism according to an embodiment of the present invention.

[0153] The slow-frame branch optimizes computational resources and improves efficiency by uniformly sampling lower frame rate tokens from the original video tokens mapped by the projector layer; this is known as temporal downsampling. Simultaneously, pooling operations are applied to aggregate the spatial dimensionality information of the slow-frame tokens, preserving spatial details at higher resolution. This process reduces computational burden while ensuring the retention of key information necessary for effective feature representation.

[0154] Reference Figure 4 The diagram illustrates a fast frame branch token space downsampling in a spatiotemporal token fusion scheme based on a reverse attention mechanism according to an embodiment of the present invention.

[0155] The fast frame branch performs computation at high frame rates. It retains all frame tokens from the original video tokens mapped by the Projector to capture the temporal context information in the video as completely as possible. At the same time, it applies a larger spatial pooling stride (e.g., downsampling each frame to a 4×4 token) to downsample in the spatial dimension, focusing on motion information in the video.

[0156] These two paths integrate their respective tokens, combining the complementary characteristics of slowly changing visual semantics and rapidly changing motion information to provide comprehensive video understanding. The tokens from both paths collectively serve as multi-scale spatiotemporal feature outputs, providing rich multi-view support for representing video information.

[0157] Spatiotemporal Token Fusion Based on Reverse Attention Mechanism:

[0158] Reference Figure 5 The diagram illustrates a schematic of the reverse attention mechanism in a spatiotemporal token fusion scheme based on the reverse attention mechanism according to an embodiment of the present invention.

[0159] Fast frame tokens are mapped to query tokens (Q) through different linear mapping layers, and slow frame tokens are mapped to key tokens (K) and value tokens (V) respectively.

[0160] By performing a dot product operation between the query token and the key token, and calculating the attention weight matrix between the two sets of token sequences using the SoftMax function, this matrix represents the attention dependency between different token sequences that contain slowly changing visual semantics and rapidly changing motion information.

[0161] By performing a reverse numerical mapping on the attention weight matrix, we obtain an inverse attention matrix. We then query which parts of the key token (slow frame token) were not addressed in the query token (fast frame), thus calculating and preserving spatial detail information at a higher resolution.

[0162] Multiplying the inverse attention weight matrix with the token value obtained from the slow frame mapping and summing the results yields a fused token that integrates "slow" and "fast" information, combining complementary features of slowly changing visual semantics and rapidly changing motion information. This token serves as an effective representation for various video tasks. It effectively fuses the tokens obtained from the two-stream network design, balancing modeling capability and computational efficiency, enabling the model to input more video frames to retain sufficient visual information details.

[0163] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0164] ReferenceFigure 6 The diagram illustrates a structural block diagram of a token data fusion system according to an embodiment of the present invention. Specifically, the token data fusion system may include the following modules.

[0165] Token data acquisition module 61 is used to acquire the raw token data of the video data to be processed;

[0166] The token data extraction module 62 is used to extract first token data from the original token data according to a first frame rate and extract second token data according to a second frame rate, wherein the first frame rate is less than the second frame rate.

[0167] Attention weight matrix calculation module 63 is used to calculate the attention weight matrix between the first token data and the second token data;

[0168] The token data fusion module 64 is used to generate fused token data of the video data based on the attention weight matrix and the first token data.

[0169] In an exemplary embodiment of the present invention, the token data fusion module 64 includes:

[0170] The first mapping module is used to perform a linear mapping on the first token data to obtain value token data;

[0171] The reverse mapping module is used to perform reverse numerical mapping on the attention weight matrix to obtain a reverse attention weight matrix;

[0172] The fusion generation module is used to generate the fusion token data based on the reverse attention weight matrix and the value token data.

[0173] In an exemplary embodiment of the present invention, the fusion generation module includes:

[0174] The product module is used to multiply the reverse attention weight matrix with the value token data to obtain multiple product results;

[0175] The summation module is used to perform a weighted summation of multiple product results to obtain the fusion token data.

[0176] In an exemplary embodiment of the present invention, the attention weight matrix calculation module 63 includes:

[0177] The second mapping module is used to perform a linear mapping on the first token data to obtain key token data, and to perform a linear mapping on the second token data to obtain query token data;

[0178] The dot product operation module is used to perform a dot product operation on the key token data and the query token data to obtain a similarity score matrix;

[0179] The function execution module is used to execute the SoftMax function on the similarity score matrix to obtain the attention weight matrix.

[0180] In an exemplary embodiment of the present invention, the token data extraction module 62 is used to obtain the first token data by uniformly sampling from the original token data according to the first frame rate, and to perform spatial pooling operation on the first token data.

[0181] In an exemplary embodiment of the present invention, the token data extraction module 62 is used to perform spatial downsampling on the original token data according to the second frame rate to obtain the second token data.

[0182] In an exemplary embodiment of the present invention, the token data acquisition module 61 is used to acquire the original token data after the video data is mapped through the mapping layer.

[0183] Reference Figure 7 The diagram illustrates a structural block diagram of a video data processing system according to an embodiment of the present invention. Specifically, the video data processing system may include the following modules.

[0184] The fusion token data acquisition module 71 is used to acquire fusion token data of video data according to the token data fusion method described above;

[0185] The task result output module 72 is used to input the fusion token data into the large language model and output the task result of the video data according to the task instructions of the video data.

[0186] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0188] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0193] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0194] The foregoing has provided a detailed description of a token data fusion method, a video data processing method, a token data fusion system, and a video data processing system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A token data fusion method, characterized in that, The method includes: Obtain the raw token data of the video data to be processed; First token data is extracted from the original token data according to a first frame rate, and second token data is extracted according to a second frame rate, wherein the first frame rate is less than the second frame rate; Calculate the attention weight matrix between the first token data and the second token data; The video data is fused into token data based on the attention weight matrix and the first token data.

2. The method according to claim 1, characterized in that, The step of generating fused token data for the video data based on the attention weight matrix and the first token data includes: The first token data is linearly mapped to obtain the value token data; The attention weight matrix is ​​reversed by performing a reverse numerical mapping to obtain the reverse attention weight matrix; The fusion token data is generated based on the inverse attention weight matrix and the value token data.

3. The method according to claim 2, characterized in that, The step of generating the fused token data based on the inverse attention weight matrix and the value token data includes: The reverse attention weight matrix is ​​multiplied by the value token data to obtain multiple product results; The fusion token data is obtained by weighted summation of multiple product results.

4. The method according to claim 1, characterized in that, The calculation of the attention weight matrix between the first token data and the second token data includes: The first token data is linearly mapped to obtain key token data, and the second token data is linearly mapped to obtain query token data; A similarity score matrix is ​​obtained by performing a dot product operation on the key token data and the query token data; The attention weight matrix is ​​obtained by applying the SoftMax function to the similarity score matrix.

5. The method according to claim 1, characterized in that, The step of extracting the first token data from the original token data according to the first frame rate includes: The first token data is obtained by uniformly sampling from the original token data according to the first frame rate, and spatial pooling is performed on the first token data.

6. The method according to claim 1, characterized in that, The step of extracting the second token data from the original token data according to the second frame rate includes: The original token data is spatially downsampled at the second frame rate to obtain the second token data.

7. The method according to claim 1, characterized in that, The raw token data for obtaining the video data to be processed includes: Obtain the original token data after the video data has been mapped through the mapping layer.

8. A video data processing method, characterized in that, The method includes: The token data fusion method according to any one of claims 1 to 7 is used to obtain the fused token data of the video data; The fusion token data is input into the large language model, and the task result of the video data is output according to the task instructions of the video data.

9. A token data fusion system, characterized in that, The system includes: The token data acquisition module is used to acquire the raw token data of the video data to be processed. The token data extraction module is used to extract first token data from the original token data according to a first frame rate and extract second token data according to a second frame rate, wherein the first frame rate is less than the second frame rate. The attention weight matrix calculation module is used to calculate the attention weight matrix between the first token data and the second token data. The token data fusion module is used to generate fused token data for the video data based on the attention weight matrix and the first token data.

10. A video data processing system, characterized in that, The system includes: A token data acquisition module is used to acquire the token data fusion method of video data according to any one of claims 1 to 7; The task result output module is used to input the fusion token data into the large language model and output the task result of the video data according to the task instructions of the video data.

11. An electronic device, characterized in that, include: One or more processors; and One or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform the token data fusion method as described in any one of claims 1 to 7, and / or the video data processing method as described in claim 8.

12. A computer-readable storage medium, characterized in that, The stored computer program causes the processor to execute the token data fusion method as described in any one of claims 1 to 7, and / or the video data processing method as described in claim 8.

Images