A video understanding method and device based on a double-layer hybrid expert model

Abstract

The application provides a video understanding method and device based on a double-layer mixed expert model, which comprises the following steps: performing text augmentation on an obtained text description, and generating a learnable text prompt; encoding the augmented text description obtained through the text augmentation and the learnable text prompt to obtain text features; dividing a video into multiple video sequence blocks, inputting the video and the video sequence blocks into a video encoder after splicing to obtain visual features; each video sequence block contains complete video information at the same position and a learnable visual prompt; generating visual tags and text tags based on the visual features and the text features respectively by using a pre-trained large language model; and obtaining a video content expression by using a double-layer mixed expert model to learn the visual tags and the text tags. The application can more accurately understand the video content by combining the text description and the visual features, and using the pre-trained large language model and the double-layer mixed expert model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video understanding method and device based on a double-layer hybrid expert model. Background Art

[0002] Video understanding is a hot research topic in computer vision and a key research direction in vision-based pattern recognition applications. It aims to automatically capture video features and model spatiotemporal information through techniques such as deep learning to achieve tasks such as video recognition and action detection. Because the text descriptions in video datasets are relatively simple, the text information extracted by the model is not detailed enough, providing limited support for video understanding. Video data is rich in contextual information, but existing methods do not fully utilize this information. Many methods focus on extracting superficial information, generating a lot of redundant information and failing to fully understand the contextual relationships within the video. Summary of the Invention

[0003] The present invention provides a video understanding method and device based on a two-layer hybrid expert model to address the shortcomings of existing technologies in insufficiently utilizing and understanding contextual information, and to accurately extract important information and contextual information from videos. The technical solutions proposed in the present invention are as follows:

[0004] In a first aspect, the present invention provides a video understanding method based on a two-layer hybrid expert model, comprising:

[0005] Obtaining a video and its corresponding text description, as well as a pre-established composite attention module, wherein the composite attention module includes a pre-trained large language model and a two-layer mixture of experts model;

[0006] Performing text amplification on the text description and generating a learnable text prompt at the same time, encoding the amplified text description obtained by the text amplification and the learnable text prompt to obtain text features;

[0007] Divide the video into multiple video sequence blocks, concatenate the video with each of the video sequence blocks, and input the concatenated video into a video encoder for encoding to obtain visual features; wherein each video sequence block contains complete video information at the same position and learnable visual cues;

[0008] Generate visual tags and text tags based on the visual features and the text features respectively using the pre-trained large language model;

[0009] The two-layer hybrid expert model is used to learn the visual tags and text tags to obtain a video content description.

[0010] Optionally, the text augmentation on the text description is performed while generating a learnable text prompt, and the augmented text description obtained through the text augmentation is encoded with the learnable text prompt to obtain the text feature, including:

[0011] The text description is augmented by using a generative pre-training transformer model to obtain an augmented text description.

[0012] A learnable text prompt is generated for the text expression.

[0013] The augmented text description and the text expression are input into a pre-established text encoder to obtain a first feature and a second feature, respectively.

[0014] The first feature and the second feature are fused to obtain the text feature.

[0015] Optionally, the video and each video sequence block are spliced to obtain a spliced video block sequence.

[0016] The video and each video sequence block are spliced to obtain a spliced video block sequence.

[0017] The self-attention is applied to the spliced video block sequence to obtain an interactive video sequence.

[0018] The pre-established feedforward network is used to process the interactive video sequence to obtain the visual feature.

[0019] Optionally, the pre-trained large language model is used to generate a visual mark and a text mark based on the visual feature and the text feature, respectively, including:

[0020] The pre-trained large language model is used to align the visual feature with the feature space of the large language model while learning the text feature to obtain the visual mark and the text mark.

[0021] Optionally, the pre-trained large language model includes an aligner and a KAN module.

[0022] The visual mark is obtained by the following method:

[0023] The aligner aligns the visual feature with the feature space of the large language model following the same mapping relationship as the text mark to obtain an aligned visual feature.

[0024] The KAN module learns the visual feature based on the aligned visual feature to obtain the visual mark.

[0025] Optionally, the pre-trained large language model includes a sampler and a cross-attention module, and the text mark is obtained by the following method:

[0026] The visual features are spliced with the text features to obtain new visual features.

[0027] The new visual features are resampled by using the sampler to obtain resampled visual features.

[0028] The resampled visual features are taken as keys and values of the cross-attention module, and the text features are taken as query vectors to be input into the cross-attention module to obtain the text labels.

[0029] In a second aspect, the present application further provides a video understanding device based on a double-layer hybrid expert model, comprising the following modules:

[0030] A data acquisition module is configured to acquire a video and a corresponding text description, and a pre-established compound attention module, wherein the compound attention module comprises a pre-trained large language model and a double-layer hybrid expert model.

[0031] A text encoding module is configured to perform text augmentation on the text description, generate a learnable text prompt, encode the augmented text description obtained through the text augmentation and the learnable text prompt to obtain text features.

[0032] A video encoding module is configured to divide the video into a plurality of video sequence blocks, splice the video and each video sequence block, and input the spliced video and video sequence block into a video encoder to obtain visual features, wherein each video sequence block contains complete video information at the same position and a learnable visual prompt.

[0033] A feature learning module is configured to use the pre-trained large language model to generate visual labels and text labels based on the visual features and the text features, respectively.

[0034] An expression generation module is configured to use the double-layer hybrid expert model to learn the visual labels and the text labels to obtain a video content expression.

[0035] In a third aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video understanding method based on the double-layer hybrid expert model according to the first aspect.

[0036] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the video understanding method based on the double-layer hybrid expert model according to the first aspect.

[0037] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the video understanding method based on the two-layer hybrid expert model as described in the first aspect above.

[0038] Based on the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0039] The present invention provides a video understanding method and device based on a two-layer hybrid expert model. By amplifying the text description and using learnable text prompts to learn specific text information related to the video content, more detailed task-related information and contextual information are provided, which can help the model understand the video content more accurately. Using learnable visual prompts, we focus on task-related areas and dynamically changing parts in the video, improve the analysis ability of long time series data, and can also share learned features and information to improve the synergy between different tasks. By using a two-layer hybrid expert model, the bottom-level experts focus on low-level feature extraction, and the high-level experts can handle more complex tasks. The model can capture a variety of information of the data at different levels, which can help the model build a more comprehensive and powerful feature representation. By combining text descriptions and visual features, and using a pre-trained large language model and a two-layer hybrid expert model, the present invention can more accurately understand the video content and accurately extract important information and contextual information in the video.

[0040] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a flowchart of the video understanding method based on the two-layer hybrid expert model provided by the present invention.

[0044] Figure 2 It is a schematic diagram of the overall network structure provided by the present invention.

[0045] Figure 3 It is a structural diagram of the composite attention module provided by the present invention.

[0046] Figure 4 It is a schematic diagram of the network structure of the double-layer hybrid expert model provided by the present invention.

[0047] Figure 5 It is a structural diagram of a video understanding device based on a two-layer hybrid expert model provided by the present invention.

[0048] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0050] The following combination Figure 1-Figure 5 The present invention describes a video understanding method and device based on a two-layer hybrid expert model.

[0051] Reference Figure 1 As shown, the video understanding method based on the two-layer hybrid expert model is Figure 2 The network structure shown is implemented. The network includes a video encoder, a text encoder, and a composite attention module.

[0052] The video encoder described above is responsible for taking the raw video and converting it into a numerical representation that a computer can process. Through operations such as multi-layer convolution and pooling, the video encoder is able to extract spatial and temporal features from the video, which are crucial for subsequent video understanding tasks.

[0053] The text encoder is responsible for receiving the text description associated with the video (such as the title, introduction, etc.) and converting it into a numerical representation. Through the encoding process, the text encoder is able to extract key information from the text, such as vocabulary, phrases, and sentence structure, which helps to understand the content of the video.

[0054] The composite attention module receives the outputs of the video encoder and the text encoder and fuses the features using an attention mechanism. It captures the correlation between video and text, generating more accurate and comprehensive video understanding results. The composite attention module can process data from different modalities (such as video and text) and effectively fuse them. By introducing the attention mechanism, the composite attention module focuses on key information in both video and text, improving the accuracy of video understanding.

[0055] The video understanding method based on the two-layer hybrid expert model provided by the present invention includes the following:

[0056] Step S110: Obtain the video and the corresponding text description, as well as the pre-established composite attention module.

[0057] First, you need to collect or obtain a series of video data and their corresponding text descriptions. These text descriptions can be the video title, introduction, or any text information related to the video content.

[0058] Specifically, based on the dataset's objectives, videos that meet the requirements are screened. This involves screening video content (such as action type and scene type) and video quality (such as resolution and clarity). During the screening process, the integrity and consistency of the video data must be ensured. Text descriptions can come from the video's own title, description, subtitles, etc., or from manually annotated or machine-generated text. When selecting sources, the quality, accuracy, and diversity of the text descriptions must be considered. The collected text descriptions are cleaned to remove irrelevant information (such as watermarks) and redundant information (such as repeated descriptions and irrelevant vocabulary). The text descriptions are standardized, such as by standardizing the format and language style. The processed text descriptions are matched with the corresponding videos to ensure that each video has a corresponding text description. During the matching process, the accuracy and consistency of the text descriptions with the video content must be ensured. Finally, the collected videos and text descriptions are organized according to specific rules to form a dataset.

[0059] Reference Figure 3 As shown in Figure 2, the composite attention module includes a pre-trained large language model and a two-layer hybrid expert model. The large language model is used to generate visual and textual tags, while the two-layer hybrid expert model is used to specifically understand the meaning of visual and textual tags to generate a more accurate representation of the video content.

[0060] Step S120: perform text amplification on the text description and generate a learnable text prompt at the same time, and encode the amplified text description obtained by the text amplification and the learnable text prompt to obtain text features.

[0061] To improve the richness and diversity of text descriptions, the original text descriptions are augmented. This can be done by adding synonyms, phrases, or sentences to increase the information content of the text. In addition to augmenting the text, some learnable text prompts are generated. These prompts can be adjusted by the model during training to help the model better understand the text content. The augmented text descriptions and learnable text prompts are input into the text encoder for encoding to generate text features. These features are used for subsequent video content understanding.

[0062] This method augments text descriptions to generate detailed and semantically rich descriptions of video content. By analyzing video frames, the model generates natural language text containing scenes, actions, and objects. This textual description provides additional semantic information for the video understanding model. Furthermore, learnable textual cues are used to generate specific textual information related to the video content, tailored to specific tasks. By learning these cues, the model can better focus on specific goals or tasks.

[0063] Step S130: Divide the video into a plurality of video sequence blocks, splice the video with the video sequence blocks, and input the spliced ​​video into a video encoder for encoding to obtain visual features.

[0064] The video is divided into multiple video sequence blocks. Each video sequence block contains complete video information at a given location (such as a frame or a period of time), as well as learnable visual cues. The video is concatenated with each video sequence block and then fed into a video encoder for encoding. The video encoder can be a convolutional neural network (CNN) or any other neural network architecture suitable for processing video data. The encoded output is visual features that reflect the content of the video.

[0065] Step S140: Generate visual tags and text tags based on the visual features and the text features using the pre-trained large language model.

[0066] Visual features and text features are input into a pre-trained large language model. The large language model generates visual and textual tags based on these features. These tags are the model's abstract representation of the video and text content. The pre-trained large language model can generate detailed and semantically rich descriptions of the video content. By analyzing video frames, the model can generate natural language text containing scenes, actions, and objects. These textual descriptions can provide additional semantic information to the video understanding model. With this additional information, the model can better understand the video content, especially when processing complex or ambiguous video data, enhancing video understanding accuracy. The generated textual description provides additional context for the model, helping to distinguish similar visual features or handle dynamically changing scenes. Learnable textual cues learn specific textual information related to the video content and can be adjusted according to specific tasks. Through these cues, the model can better focus on specific targets or tasks. Learnable cues can also be dynamically adjusted based on training data to improve the model's performance in different video scenarios.

[0067] Step S150: Use the two-layer hybrid expert model to learn the visual tags and text tags to obtain a video content description.

[0068] Visual and textual tags are fed into a two-layer mixture of experts model. The model uses an expert mechanism (such as a mixture of experts network) to process these tags and generate a final representation of the video content. These representations can be text, labels, vectors, or other forms that describe the video's content, theme, or sentiment.

[0069] This invention uses a two-layer expert network, enabling the model to process information at different levels, thereby capturing more complex video content. By combining the knowledge and capabilities of multiple expert networks, the hybrid expert model can better capture complex patterns and relationships. This architecture allows each expert network to focus on different aspects of the data or specific subtasks, thereby improving the overall expressive power of the model.

[0070] In step S130, the complete time series of a variable is extracted as a marker, rather than the method used by most previous models that treats multiple variables at the same time as markers. This method allows the video encoder to focus on representation learning and adaptive association of multivariate sequences. The added learnable visual cue module can automatically learn and optimize cue information, automatically adjusting the cue information based on the characteristics of different videos to adapt to different scenarios. Compared with traditional manually designed cueing, this method can more accurately capture key features in the video, thereby improving the accuracy and efficiency of video comprehension.

[0071] The video understanding method based on the two-layer hybrid expert model provided by the present invention amplifies the text description, uses learnable text prompts to learn specific text information related to the video content, provides more detailed task-related information and contextual information, and can help the model understand the video content more accurately. Using learnable visual prompts, focus on task-related areas and dynamically changing parts in the video, improve the analysis ability of long time series data, and can also share learned features and information to improve the synergy between different tasks. By using a two-layer hybrid expert model, the bottom-level experts focus on low-level feature extraction, and high-level experts can handle more complex tasks. The model can capture a variety of information of the data at different levels, which can help the model build a more comprehensive and powerful feature representation. By combining text descriptions and visual features, and using a pre-trained large language model and a two-layer hybrid expert model, the present invention can more accurately understand the video content.

[0072] Furthermore, due to its use of learnable textual and visual cues, as well as a pre-trained model, this method demonstrates enhanced generalization across different types of videos and text descriptions. It can generate diverse representations of video content to meet diverse application needs. By more accurately understanding video content, this method can provide users with more precise video recommendations, search, and classification services, thereby enhancing the user experience.

[0073] In an optional embodiment, a learnable visual cue module can improve the accuracy of the target detection algorithm by learning the contextual information and target features in the video. The learnable visual cue module can establish a temporal association by analyzing the information of the previous and next frames, helping the system understand the target's motion trajectory and behavior pattern in the video. The learnable visual cue module can also improve the robustness of the detection algorithm to different environmental conditions by continuously learning and adapting to changes in lighting, viewing angle, occlusion, etc. in the scene from the video. The learnable visual cue module can reduce the dependence on a large amount of labeled data through adaptive learning. The learnable visual cue module can also improve the analysis ability of long time series data through long-term learning and adjustment.

[0074] The text description is amplified in step S120, and a learnable text prompt is generated. The amplified text description obtained by the text amplification and the learnable text prompt are encoded to obtain text features, including:

[0075] S1201. Use a generative pre-trained transformer (GPT) model to perform text amplification on the text description to obtain an amplified text description.

[0076] The generative pre-trained transformer model described above is based on the Transformer architecture. Trained on large amounts of text data, it generates coherent, grammatically correct text. When the original text description is input into the GPT model, the model generates an augmented text description that is relevant and rich in information based on the input text content and context. Text augmentation increases the diversity of text data and improves the model's understanding of text content. By introducing new vocabulary, phrases, or sentence structures, it provides more information for subsequent text encoding.

[0077] S1202: Generate a learnable text prompt for the text representation.

[0078] A text prompt is a short sentence or phrase that guides the model's understanding of the text content. The generated text prompts are learnable, meaning the model adjusts the prompt content during training based on the text content and task requirements.

[0079] Text prompts can be generated using fixed prompts, context-based prompts, or creative prompts. Fixed prompts are predefined words or phrases that guide the model's thinking. Context-based prompts are words or phrases generated based on the current context, more accurately reflecting the text content. Creative prompts are words or phrases generated for specific scenarios and needs, inspiring the model's creativity.

[0080] S1203: Input the augmented text description and the text expression into a pre-established text encoder to obtain a first feature and a second feature respectively.

[0081] During this process, the augmented text description and the original text description (i.e., the aforementioned text representation) are encoded using a pre-established text encoder. The augmented text description and the text representation are each input into the text encoder. The text encoder performs preprocessing on the input text, such as word segmentation, stop word removal, and word embedding, before converting it into a numerical representation (i.e., a feature vector). After processing by the text encoder, the augmented text description is converted into the first feature. The original text description (i.e., the text representation) is converted into the second feature.

[0082] S1204: Fuse the first feature and the second feature to obtain the text feature.

[0083] The final text feature is obtained by concatenating, weighted summing, or using other fusion methods. By fusing the features of the amplified text description with the original text description, a more comprehensive and rich text feature representation is obtained. This helps improve the model's ability to understand and accurately interpret text content.

[0084] Specifically, the present invention uses the GPT model to amplify the text description in the data set, mainly adding a description of the environment and the appearance, position, and motion trajectory of the detection object to obtain the amplified text description. . Amplify text description Feed into the text encoder to get the first feature , designing learnable text prompts , learn relevant prompt information for specific tasks, and convert the learnable text prompts Feed into the text encoder to get the second feature :

[0085]

[0086] in, Represents the operation of a text encoder.

[0087] The above two features are combined to obtain new text features ,

[0088]

[0089] The present invention increases the diversity and information content of text data through text amplification and learnable text prompts, improves the model's ability to understand text content, optimizes the model's performance, and thus improves the model's efficiency and accuracy in processing text data. By fusing the features of the amplified text description and the original text description, a more comprehensive and accurate text feature representation is obtained. The generated amplified text description and learnable text prompts are both trained based on a large amount of text data, which enables the model to have stronger generalization capabilities when processing different types of text data. The text features generated by this process can be used in a variety of natural language processing tasks such as text classification, text generation, and sentiment analysis, supporting a variety of application scenarios.

[0090] In an optional embodiment, the step S130 of splicing the video with the video sequence blocks and inputting the spliced ​​video with the video sequence blocks into a video encoder for encoding to obtain visual features includes:

[0091] S1301: Splice the video with each of the video sequence blocks to obtain a spliced ​​video block sequence.

[0092] The original video is segmented into multiple video sequence blocks. These blocks can be based on consecutive frames in time or image regions in space. The original video is then concatenated with these video sequence blocks to produce a concatenated video block sequence. This step aims to combine global and local information in the video, providing rich context for subsequent processing.

[0093] S1302, apply self-attention to the spliced video block sequence to obtain an interactive video sequence.

[0094] The self-attention mechanism is a deep learning technique that allows the model to consider all other elements in the sequence when processing each element in the sequence. In video processing, the use of the self-attention mechanism can capture long-range dependencies between video frames, thereby extracting more global and coherent visual features. Through the self-attention mechanism, each part of the spliced video block sequence can interact with each other, enhancing the model's understanding of video content.

[0095] S1303, processing the interactive video sequence using a pre-established feedforward network to obtain the visual features.

[0096] The video sequence after the self-attention mechanism is input into the pre-established feedforward network. The feedforward network is a multi-layer neural network that can perform nonlinear transformation on input data, thereby extracting higher-level features. In this step, the feedforward network processes the interactive video sequence and finally outputs visual features. These visual features can be used for subsequent video understanding, classification, or recognition tasks.

[0097] Specifically, referring to Figure 2 , the original video is divided into N video sequence blocks of the same size, each sequence block has a size of HxWxT, and each sequence block contains all visual information at the same position. N learnable visual cues with a size of HxWx3 are added to the original video , and the video sequence block is spliced to obtain a spliced video block sequence . The self-attention mechanism is applied to the video block sequence for mutual interaction, and then processed separately through the feedforward network for sequence representation to obtain visual features .

[0098]

[0099]

[0100] wherein, represents the splicing operation, represents the operation of the feedforward network, represents the self-attention operation.

[0101] The present invention uses a self-attention mechanism to capture long-range dependencies between video frames, enabling the model to better understand the video content. This helps the model achieve better performance in subsequent tasks such as video understanding, classification, or recognition. The feedforward network can perform nonlinear transformations on the video sequence after interaction and extract higher-level visual features. These features are more abstract and representative, helping the model to better identify and understand the video content in subsequent tasks. By splicing video and video sequence blocks and combining them with the self-attention mechanism for mutual interaction, the model can process multiple parts of the video data in parallel, thereby improving the efficiency of video processing.

[0102] Those skilled in the art can adjust the segmentation method of the video sequence blocks, the specific implementation method of the self-attention mechanism, and the architecture of the feedforward network to adapt to different video processing tasks and data sets.

[0103] In an optional embodiment, aligning visual and language features can help the model better understand video content, especially in complex or ambiguous scenes, and improve the accuracy of results. By integrating language information, the model can more accurately extract important information and context from the video, enhancing object detection and behavior recognition capabilities. Aligning visual and language features can help the model better handle potential differences between vision and language, improving robustness in diverse scenarios.

[0104] The step S140 described above of using the pre-trained large language model to generate visual tags and text tags based on the visual features and the text features, respectively, includes:

[0105] S1401: Use the pre-trained large language model to align the visual features with the feature space of the large language model, and learn the text features at the same time to obtain the visual markup. and the text mark .

[0106]

[0107] in, represents the operation of a pre-trained large language model, Represents visual features, Represents text features.

[0108] A visual encoder (such as a convolutional neural network (CNN) or Transformer-based model) processes image data to extract key visual features from the image. These visual features are typically represented as high-dimensional vectors that contain information such as color, shape, and texture. The extracted visual features are aligned with the feature space of the large language model. Specifically, the visual features are mapped into a semantic space shared with the language features, allowing features from different modalities to be compared and interact in the same space. This alignment process can utilize a variety of techniques, such as contrastive learning, attention mechanisms, and cross-modal attention layers. Simultaneously, the large language model processes the input text data and learns text features. Text features can include semantic information, syntactic structure, and contextual relationships of vocabulary. In this aligned feature space, the large language model generates corresponding visual and textual tags based on the visual and textual features. Visual tags represent specific objects, scenes, or attributes in the image, while textual tags represent descriptions, labels, or explanations related to the image.

[0109] The present invention aligns visual features with the feature space of a large language model. The aligned feature space provides a basis for cross-modal applications, enabling the model to better understand and process the relationship between images and text. The model can convert and generate between different modalities, such as generating text from images or generating images from text. This helps to improve the performance of the model in tasks such as visual question answering and image caption generation. Learning text features at the same time enables the model to generate more accurate, fluent and natural text output. Since the large language model pre-trained by the present invention has learned rich language knowledge and patterns. By incorporating visual features into it, the model can adapt and generalize faster on new tasks or new fields.

[0110] In an alternative embodiment, referring to Figure 3 The pre-trained large language model in the present invention includes an aligner, a KAN module, a sampler, and a cross-attention module.

[0111] The above-mentioned aligner is responsible for aligning the visual features with the feature space of the large language model through an alignment mechanism.

[0112] The Kolmogorov-Arnold Network (KAN) module is a neural network architecture with strong fitting capabilities and interpretability. It learns based on aligned visual features, capturing complex patterns and relationships within them through a combination of multiple layers of nodes and activation functions (such as square, cubic, sine, and cosine). The KAN module achieves higher accuracy with fewer parameters while maintaining model simplicity and interpretability.

[0113] The sampler is used to extract more critical or representative information from the output visual and text features. The sampler can adopt various strategies, such as greedy search, random sampling, and beam search, to generate high-quality text.

[0114] The cross-attention module is used to process the interactive information between two different sequences of visual features and textual features to enhance the model's ability to understand the relationship between the input sequence and the context, thereby generating an output vector that combines visual and textual information.

[0115] The present invention enhances the modal consistency between vision and language by reusing the weights of a pre-trained large language model, which can help the model better handle the potential differences between vision and language, more accurately extract important information and context in the video, and improve robustness in diverse scenarios. In addition, aligning the visual and language modalities by reusing weights does not require additional modules, thereby improving data efficiency. The use of the KAN module improves the interpretability of the model, uses fewer parameters and higher computational efficiency, and completes the task of fitting large-scale data in a shorter time. It can also flexibly adjust the activation mechanism according to the characteristics of different tasks to improve the adaptability and accuracy of the model.

[0116] The process of learning visual features through a pre-trained large language model to obtain visual tags is as follows:

[0117] S210 : The aligner aligns the visual features with the feature space of the large language model according to the same mapping relationship as the text tags to obtain aligned visual features.

[0118] This process follows the same mapping relationship as text tags, ensuring that visual features and text features can be compared and interacted in the same semantic space. The aligned visual features will have a similar representation to the text features, making them easier to process in subsequent modules.

[0119] S220 , the KAN module learns visual features based on the aligned visual features to obtain the visual markers.

[0120] After learning the KAN module, the model is able to generate corresponding visual tags based on the aligned visual features. Visual tags represent specific objects, scenes, attributes, or relationships in the image, depending on the training data and task requirements.

[0121] Specifically, the visual features obtained in step S130 are Following the same mapping relationship as text tags, it is aligned with the feature space of the large language model to obtain the aligned visual features. .

[0122]

[0123] in, The weights of the pre-trained large language model.

[0124] The present invention leverages the existing weights of a pre-trained large language model to enhance the modal alignment between vision and language, helping the model better handle the potential differences between vision and language without incurring additional overhead.

[0125] Then, the KAN model is used to further learn visual features and obtain visual tags .

[0126]

[0127] in, express Operations of the model.

[0128] By aligning visual features with the feature space of a large language model and introducing a KAN module for learning, the model can better understand and process the relationship between images and text. This helps to improve the performance of the model in cross-modal tasks such as visual question answering and image caption generation. The KAN module has strong fitting capabilities and interpretability, and can achieve higher accuracy with fewer parameters. This enables the model to maintain high quality when generating visual tags while reducing computational complexity and storage requirements. The pre-trained large language model has learned rich language knowledge and patterns. By introducing components such as aligners and KAN modules to process visual features, the model can adapt and generalize faster on new tasks or new domains.

[0129] In an optional embodiment, the process of obtaining text tags using a pre-trained large language model is as follows:

[0130] S310: Combine the visual features and the text features to obtain new visual features.

[0131] Concatenating visual and textual features involves merging the two feature vectors along a certain dimension to form a new visual feature vector that incorporates information from both. This concatenation allows the model to consider both visual and textual information simultaneously, leading to a more accurate understanding of the overall content of the input data.

[0132] S320: Resample the new visual features using the sampler to obtain resampled visual features.

[0133] The resampling process can generate new feature representations by randomly selecting feature vectors, performing weighted averaging, or applying a transformation (such as an attention mechanism). Resampling extracts more critical or representative information from the original features (i.e., the new visual features mentioned above), thereby reducing noise and enhancing the model's sensitivity to important features.

[0134] S330 , using the resampled visual features as the key and value of the cross-attention module, and sending the text features as the query vector into the cross-attention module to obtain the text tag.

[0135] The resampled visual features are used as the key and value of the cross-attention module. At the same time, the text features are fed into the cross-attention module as the query vector. The cross-attention module calculates the similarity between the query vector and the key (such as using dot product or cosine similarity), and then weights the values ​​according to the similarity to generate an output vector that combines visual and textual information. This output vector can be regarded as a text tag. The text tag can be a word, phrase, or sentence, depending on the requirements of the task and the training method of the model. Those skilled in the art can also further process the output of the cross-attention module (such as applying a fully connected layer, softmax function, etc.) to convert it into the final text tag.

[0136] Specifically, the present invention splices visual features With text features As a new visual feature , and resample to obtain the resampled visual features :

[0137]

[0138] in, Represents a splicing operation, Represents a sampler.

[0139] The resampled visual features As the key and value of the cross attention module, the text features Feed it into the cross-attention module as a query vector to get the text tag :

[0140]

[0141] in, Represents the operation of the cross-attention module.

[0142] The method divides the video into N video blocks of equal size in step S120, and splices the video block sequence with the original video of reduced size after encoding, facilitating learning of global information. Downsampling is performed using a resampler, which can reduce the sequence length and alleviate the computational pressure.

[0143] The application can extract more critical or representative information from the original features by resampling new visual features, thereby reducing noise and enhancing the sensitivity of the model to important features. This enables the model to more efficiently and accurately extract useful information when processing complex inputs. By splicing and resampling visual and textual features, the model can better integrate information from both modalities, thereby more accurately understanding the overall content of the input data. This helps to improve the performance of the model in cross-modal tasks (i.e., image description generation). The cross-attention module integrates information by calculating the similarity between the query vector and the key, which enables the model to better handle the relevance and differences between different inputs. This helps to improve the generalization ability of the model on unseen data or tasks. The output of the cross-attention module can reflect the similarity distribution between the query vector and the key, thereby providing additional information about the model's decision-making process. This helps to enhance the explainability of the model, enabling users to better understand the output results and decision basis of the model.

[0144] In an optional embodiment, the above-mentioned double-layer mixed expert model (Mixture of Experts, MoE) has two layers, namely a first layer of expert networks and a second layer of expert networks. The first layer of expert networks is composed of N experts (i.e., neural network modules), and the second layer of expert networks is composed of N x M experts.

[0145] The above-mentioned step S150 uses the double-layer mixed expert model to learn the visual and textual features, obtaining a video content representation, as described in Figure 4 The method comprises the following steps:

[0146] S1501, splice the visual features and the textual features to form a unified sequence. Then, the spliced features are fused with the pre-initialized embedding vectors to obtain the fused features.

[0147] First, the visual features and the textual features are spliced to form a unified sequence. Then, the spliced features are fused with the pre-initialized embedding vectors to obtain the fused features. It is used to map data from different modalities into the same vector space for subsequent processing and analysis. The fused markup contains both visual and textual information, providing rich input for subsequent processing.

[0148] S1502: Process the fused markup through the self-attention module to obtain an updated markup;

[0149] The self-attention module first calculates the attention weights between each fused token and all other tokens. These "other tokens" refer to all other fused tokens except the one currently being processed. Each fused token interacts with all other tokens to calculate attention weights. These weights reflect the importance or relevance of other tokens to the current token. When calculating the attention weights, the current token is compared with all other tokens (e.g., using some similarity metric). After obtaining the attention weights, the self-attention module multiplies these weights by the corresponding value vector, namely the fused token. Through weighted summation, each fused token receives a new representation that incorporates information from all other fused tokens, namely the updated token described above. This representation not only contains information about the token itself but also incorporates the attention information from other tokens. This fused representation is more comprehensive and rich because it takes into account the correlations between different tokens.

[0150] S1503: Use the updated mark as an input token, select an expert with a routing weight greater than a predefined confidence from the first-layer expert network to process the input token, and obtain a processed token.

[0151] The fused tokens are fed into the first-layer expert network as input tokens. Each expert in the first-layer expert network excels at processing different types of input tokens. By processing these input tokens in parallel, the first-layer expert network is able to extract preliminary features of the video content. A softmax predictor is used to calculate routing weights, which reflect the applicability of each expert to the current input token. Based on the routing weights, experts with routing weights greater than a predefined confidence level are selected from the first-layer expert network to process each input token. This selection mechanism ensures that only the most relevant experts are activated, thereby improving the efficiency and performance of the model. The selected experts further process the input tokens to obtain processed tokens. These processed tokens contain higher-level features of the video content.

[0152] S1504. Select an expert whose routing weight is greater than a predefined confidence level from among the experts under each expert selected from the first-layer expert network, process the processed tokens, and perform weighted summation of the processing results of each expert to obtain the video content representation.

[0153] For each expert selected from the first-layer expert network, a softmax predictor is again used to calculate routing weights. Experts with routing weights greater than a predefined confidence level are selected to process the processed tokens. The second-layer expert network further extracts and refines features of the video content, performing a weighted sum of the outputs of each expert to ultimately generate a video content representation. This representation contains comprehensive and in-depth information about the video content and can be used for subsequent video analysis, comprehension, or generation tasks.

[0154] Specifically, splicing visual markers With text mark Get the spliced ​​mark , the concatenated mark With embedding vector After fusion, the updated mark is obtained by processing through the self-attention module . Updated markup The embedding vector can provide modal information for routing. , text markup and the updated markup As input tokens, each input token is processed by k experts selected from N experts to obtain processed tokens. Among the experts under the first selection of experts, k experts are selected again, totaling k The k experts process the tokens after the above processing again. The present invention uses a Softmax predictor to calculate the routing weight and selects appropriate experts based on the weight.

[0155]

[0156]

[0157]

[0158] in, Represents a splicing operation, The video content representation output by the MoE model, represents the operation of the Softmax predictor, represents the output of the KAN module in the MoE model, For the updated mark, is the operation of the self-attention module.

[0159] The hybrid expert model can better capture complex patterns and relationships by combining the knowledge and capabilities of multiple expert networks. This architecture allows each expert network to focus on different aspects of the data or specific subtasks, thereby improving the overall expressiveness of the model. The hybrid expert model selectively activates some expert networks through a gating mechanism, thereby reducing computational and storage requirements. By learning how to select the most appropriate experts, it dynamically adjusts the contributions of different experts to ensure that each expert's capabilities are fully utilized during the training process, allowing the model to adapt to different tasks or domains. Each expert may perform well on a specific subtask or feature. By combining the predictions of multiple experts, the model can better handle complex, unseen samples and reduce the risk of overfitting. The hybrid expert model achieves knowledge modularity, which not only makes model training and tuning more flexible, but also makes model expansion and maintenance more efficient.

[0160] By using a softmax predictor to calculate routing weights and selecting experts for processing based on these weights, the two-layer hybrid expert model dynamically adjusts the use of experts, thereby reducing unnecessary computational overhead. This design enables the model to maintain high efficiency when handling complex tasks while improving resource utilization. Furthermore, the two-layer hybrid expert model integrates different expert networks, allowing each expert to function within their area of ​​expertise. This modular design allows the model to be easily expanded and adapted to new tasks or data. Furthermore, since each expert is an independent module, it can be trained and optimized independently, improving the model's flexibility and maintainability.

[0161] By combining multiple specialized experts, the two-layer hybrid expert model can learn richer video content features, thereby improving the model's generalization capabilities. This design enables the model to better handle unseen data or tasks, enhancing its practicality and robustness. The two-layer hybrid expert model uses a gating mechanism and an expert selection mechanism to ensure that the model selects appropriate experts for prediction from a large number of parameters, rather than simply relying on all experts. This approach effectively reduces the risk of overfitting and improves the model's generalization performance and stability.

[0162] In the above step S140, the present invention mainly calculates the self-attention of text tags and the cross-attention between text and video, because the video encoder has already learned the video features very well and does not need to learn the self-attention features of the video again. The use of a hybrid expert model of different modalities can help align the visual modality with the language modality. Therefore, in the above step S150, the present invention uses modality routing technology to capture information of a specific modality, and at the same time enhances the model's ability to understand the task by selectively switching to different experts for fusion. In addition, the use of a modality-aware two-layer hybrid expert model can process different semantic levels or visual features separately, selectively activate part of the expert network through a gating mechanism, and dynamically adjust the importance of each expert and knowledge according to different input situations, so that the model has strong adaptability. At the same time, it can reduce the amount of calculation and storage requirements, thereby improving training and reasoning efficiency.

[0163] The present invention provides a video understanding method based on a two-layer hybrid expert model. The method uses a video encoder to encode a full sequence of video blocks to obtain video information more completely. Adding amplified text descriptions and learnable text descriptions provides more contextual information and target information for video understanding; aligning visual features with the feature space of a pre-trained large language model helps the model better handle the potential differences between vision and language, and more accurately extract important information and context from the video. Use a modality-aware two-layer hybrid expert model to learn visual tags and text tags, selectively switch to different experts for fusion based on modal information, reduce the misaligned learning problem caused by modal differences, better utilize the capabilities of the expert model, and thus improve the model's generalization ability in different data sets or scenarios.

[0164] The video understanding device based on the double-layer hybrid expert model provided by the present invention is described below. The video understanding device based on the double-layer hybrid expert model described below and the video understanding method based on the double-layer hybrid expert model described above can refer to each other.

[0165] The video understanding device based on the double-layer hybrid expert model provided by the present invention refers to Figure 5 Shown, including:

[0166] A data acquisition module 410 is used to acquire videos and corresponding text descriptions, as well as a pre-established composite attention module, wherein the composite attention module includes a pre-trained large language model and a two-layer hybrid expert model;

[0167] A text encoding module 420 is configured to perform text amplification on the text description and generate a learnable text prompt, and encode the amplified text description obtained by the text amplification and the learnable text prompt to obtain text features;

[0168] The video encoding module 430 is configured to divide the video into a plurality of video sequence blocks, concatenate the video with the video sequence blocks, and input the concatenated video into a video encoder for encoding to obtain visual features; wherein each video sequence block contains complete video information at the same position and learnable visual cues;

[0169] A feature learning module 440 is configured to generate a visual tag and a text tag based on the visual features and the text features using the pre-trained large language model;

[0170] The description generation module 450 is configured to use the two-layer hybrid expert model to learn the visual tags and text tags to obtain a video content description.

[0171] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. The processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute the video understanding method based on the two-layer hybrid expert model provided above.

[0172] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0173] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video understanding method based on the two-layer hybrid expert model provided by the above methods.

[0174] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video understanding method based on the two-layer hybrid expert model provided by the above methods.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0176] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A video understanding method based on a two-layer hybrid expert model, characterized in that: include: Obtaining a video and its corresponding text description, as well as a pre-established composite attention module, wherein the composite attention module includes a pre-trained large language model and a two-layer mixture of experts model; Performing text amplification on the text description and generating a learnable text prompt at the same time, encoding the amplified text description obtained by the text amplification and the learnable text prompt to obtain text features; Divide the video into multiple video sequence blocks, concatenate the video with each of the video sequence blocks, and input the concatenated video into a video encoder for encoding to obtain visual features; wherein each video sequence block contains complete video information at the same position and learnable visual cues; Generate visual tags and text tags based on the visual features and the text features respectively using the pre-trained large language model; The two-layer hybrid expert model is used to learn the visual tags and text tags to obtain a video content description.

2. The video understanding method based on the two-layer hybrid expert model according to claim 1 is characterized in that The text description is amplified, and a learnable text prompt is generated. The amplified text description obtained by the text amplification and the learnable text prompt are encoded to obtain text features, including: Use the generative pre-trained transformer model to perform text amplification on the text description to obtain the amplified text description; generating a learnable textual prompt for the textual representation; Inputting the augmented text description and the text representation into a pre-established text encoder to obtain a first feature and a second feature respectively; The first feature and the second feature are fused to obtain the text feature.

3. The video understanding method based on the two-layer hybrid expert model according to claim 1 is characterized in that The step of splicing the video with the video sequence blocks and inputting the spliced ​​video with the video sequence blocks into a video encoder for encoding to obtain visual features includes: splicing the video with each of the video sequence blocks to obtain a spliced ​​video block sequence; Applying self-attention to the spliced ​​video block sequence to interact with each other to obtain an interacted video sequence; The video sequence after the interaction is processed using a pre-established feedforward network to obtain the visual features.

4. The video understanding method based on the two-layer hybrid expert model according to claim 1 is characterized in that Generating visual tags and text tags based on the visual features and the text features using the pre-trained large language model, respectively, includes: The pre-trained large language model is used to align the visual features with the feature space of the large language model, and text features are learned simultaneously to obtain the visual tags and the text tags.

5. The video understanding method based on the two-layer hybrid expert model according to claim 1 is characterized in that: The pre-trained large language model includes an aligner and a KAN module; The visual mark is obtained by: The aligner aligns the visual features with the feature space of the large language model using the same mapping relationship as the text tags to obtain aligned visual features; The KAN module learns visual features based on the aligned visual features to obtain the visual markers.

6. The video understanding method based on the two-layer hybrid expert model according to claim 1 is characterized in that: The pre-trained large language model includes a sampler and a cross-attention module; the text token is obtained by: Combine visual features with text features to obtain new visual features; Resampling the new visual features using the sampler to obtain resampled visual features; The resampled visual features are used as keys and values ​​of the cross-attention module, and the text features are sent as query vectors into the cross-attention module to obtain the text tags.

7. A video understanding device based on a two-layer hybrid expert model, characterized in that: include: A data acquisition module for acquiring videos and corresponding text descriptions, as well as a pre-established composite attention module, wherein the composite attention module includes a pre-trained large language model and a two-layer hybrid expert model; A text encoding module is used to perform text amplification on the text description and generate a learnable text prompt, and encode the amplified text description obtained by text amplification and the learnable text prompt to obtain text features; A video encoding module is used to divide the video into multiple video sequence blocks, splice the video with each of the video sequence blocks, and input the video encoder to encode and obtain visual features; wherein each video sequence block contains complete video information at the same position and learnable visual cues; a feature learning module, configured to generate a visual tag and a text tag based on the visual features and the text features, respectively, using the pre-trained large language model; The description generation module is used to use the two-layer hybrid expert model to learn the visual tags and text tags to obtain the video content description.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video understanding method based on the two-layer hybrid expert model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video understanding method based on the two-layer hybrid expert model as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video understanding method based on the two-layer hybrid expert model as claimed in any one of claims 1 to 6 is implemented.

Images