Video modal feature processing method and system based on BPE

Abstract

The invention discloses a BPE-based video modal feature processing method and system, and the method comprises the steps: merging video frames through a BPE algorithm, carrying out the video coding after a set number of video frames are reached, and obtaining a video feature vector representation, or carrying out the video coding of the video frames, and then carrying out the merging through the BPE algorithm, and obtaining a video frame feature vector; encoding the text to obtain text feature vector representation; performing linear transformation on the text feature vector representation and the video feature vector representation, or performing linear transformation on the text feature vector representation and the video frame feature vector to obtain multi-modal feature vector representation; and processing the multi-modal feature vector representation through a large language model to generate a multi-dimensional video content representation. The method can adapt to video contents with different lengths through the BPE algorithm, ensures that key information of the videos can be effectively reserved regardless of the lengths of the videos, provides a data basis for video LLMs, and enables the video LLMs to comprehensively understand the video contents.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and more particularly to a video modal feature processing method and system based on BPE. Background Art

[0002] In today's digital age, the rapid development of artificial intelligence technology and the booming rise of online video platforms complement each other and jointly promote the explosive growth of video content creation and sharing. This phenomenon has not only greatly enriched our information acquisition channels, but also put forward unprecedented requirements for the efficiency and intelligence level of video understanding tools. As an important branch of the vast field of computer vision, video understanding is becoming increasingly important and is rapidly becoming a hot spot at the forefront of scientific research and technological innovation. Thanks to the outstanding ability of large language models (LLMs) in processing language and multimodal tasks, the scope of video understanding has expanded from traditional basic classification and recognition tasks to more complex and in-depth fields. Current video understanding tasks not only require models to accurately annotate and identify video content, but also expect them to perform advanced functions such as title generation, content summarization, scene association, information retrieval, and complex question and answer. These advanced tasks place higher requirements on models: they must be able to recognize and understand various objects and scenes appearing in the video, and capture actions, events, emotions, and deeper semantic information in the video. In addition, the model needs to be able to parse the temporal continuity of the video, apply logical reasoning, and combine rich contextual knowledge to achieve near-human-level understanding of video content. To achieve this level of intelligence, the video understanding model must integrate multiple information sources and advanced algorithms, including but not limited to deep learning, natural language processing, computer vision, and machine learning techniques. In this way, the model can extract rich features from the video data and build a comprehensive, multi-dimensional representation of the video content, thereby achieving in-depth analysis and accurate interpretation of the video content.

[0003] However, although current video LLMs (Large Language Models with Video Understanding Capabilities) have made significant progress in short video understanding, long video understanding remains a challenging research area. Compared to short videos, long videos place higher demands on the model's understanding capabilities due to their large number of frames, longer time span, and redundant information mixed in them. These challenges require the model to not only have excellent memory and understanding capabilities to filter out irrelevant information and extract key content, but also to be able to work under the limited GPU memory and context length limitations of large language models (LLMs), which means that the model can usually only process a limited number of video frames. One of the coping strategies of the existing technology is to use a uniform sampling method to process video modalities. This method uniformly samples a fixed number of video frames, such as 8 or 16 frames, at a fixed time interval as input data regardless of the original length of the video. Although this method can retain a relatively sufficient amount of information when processing videos with relatively short durations, the limitations of its sampling strategy are clearly revealed when facing long videos. As the length of the video increases, the amount of information that can be captured by uniform sampling decreases relatively, which makes it difficult for the model to fully capture and understand the overall content of the video.

[0004] Therefore, how to provide a processing method and system that can fully and deeply understand the content of long videos is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a video modal feature processing method and system based on BPE, which can adapt to video contents of different lengths through the BPE algorithm, ensuring that the key information of the video can be effectively retained regardless of its length, and providing a data basis for video LLMs, enabling it to fully understand the video content.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A video modality feature processing method based on BPE, comprising:

[0008] S1: Merge the video frames through the BPE algorithm, and after reaching a set number of video frames, perform video encoding to obtain a video feature vector representation, or merge the video frames through the BPE algorithm after video encoding to obtain a merged video frame feature vector;

[0009] Encode the text to obtain a text feature vector representation;

[0010] S2: linearly transform the text feature vector representation and the video feature vector representation, or linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation;

[0011] S3: Process the multimodal feature vector representation through a large language model to generate a multi-dimensional video content representation.

[0012] Preferably, the BPE algorithm specifically includes:

[0013] S101: Acquire all video frames of the video;

[0014] S102: segmenting all video frames or segmenting video frame feature vectors obtained by encoding the video frames, that is, merging and iterating video frames or video frame feature vectors in the same field, and video frames or video frame feature vectors in different fields are not merged;

[0015] S103: Calculating the similarity of feature vectors of consecutive frames or consecutive video frames;

[0016] S104: merging the two frames or frame feature vectors with the highest similarity, and averaging and merging them into a new video frame or a new video frame feature vector;

[0017] S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames, and merging all sub-segments;

[0018] S106: Merge the iterative process and repeat S103 and S104 until a predetermined number of frames is reached.

[0019] Preferably, when merging video frames through the BPE algorithm, the similarity of consecutive frames is calculated by the histogram overlap, and the calculation formula is as follows:

[0020]

[0021] Among them, I(H1,H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

[0022] Preferably, when the video frames are combined by the BPE algorithm after being encoded by the video, the similarity of the feature vectors of consecutive frames is calculated by the cosine similarity:

[0023]

[0024] Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

[0025] Preferably, if the number of video frames is less than a predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

[0026] A video modality feature processing system based on BPE, comprising:

[0027] Video BPE module: used to merge video frames through the BPE algorithm to reach a set number of video frames, or to merge video frames after video encoding through the BPE algorithm to obtain a merged video frame feature vector;

[0028] Multimodal feature encoding module: used to encode the video frames through the BPE algorithm to obtain a video feature vector representation or directly encode the video frames, and encode the text to obtain a text feature vector representation;

[0029] Feature alignment module: used to linearly transform the text feature vector representation and the video feature vector representation, or linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation;

[0030] Cognitive module: used to process multimodal feature vector representations through a large language model to generate multi-dimensional video content representations.

[0031] Preferably, the BPE algorithm specifically includes:

[0032] S101: Acquire all video frames of the video;

[0033] S102: segmenting all video frames or segmenting video frame feature vectors obtained by encoding the video frames, that is, merging and iterating video frames or video frame feature vectors in the same field, and video frames or video frame feature vectors in different fields are not merged;

[0034] S103: Calculating the similarity of consecutive frames or consecutive frame feature vectors;

[0035] S104: merging the two frames or frame feature vectors with the highest similarity, and averaging and merging them into a new video frame or a new video frame feature vector;

[0036] S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames, and merging all sub-segments;

[0037] S106: Merge the iterative process and repeat S103 and S104 until a predetermined number of frames is reached.

[0038] Preferably, when merging video frames through the BPE algorithm, the similarity of consecutive frames is calculated by the histogram overlap, and the calculation formula is as follows:

[0039]

[0040] Among them, I(H1, H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

[0041] Preferably, when the video frames are combined by the BPE algorithm after being encoded by the video, the similarity of the feature vectors of consecutive frames is calculated by the cosine similarity:

[0042]

[0043] Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

[0044] Preferably, if the number of video frames is less than a predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

[0045] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a video modal feature processing method and system based on BPE, which effectively filters out irrelevant information and extracts key content, providing strong support for the deep intelligent analysis and understanding of video content. It can not only meet the needs of rapid analysis of short videos, but also maintain the integrity and accuracy of information in the broad scenes of long videos, so that video LLMs can achieve a comprehensive and in-depth understanding of video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A flow chart of a video modal feature processing method based on BPE provided by the present invention.

[0048] Figure 2 This is a flow chart of the BPE algorithm provided by the present invention.

[0049] Figure 3 A flowchart of a video modality feature processing method based on BPE is provided for another embodiment of the present invention.

[0050] Figure 4 A structural block diagram of a BPE-based video modal feature processing system provided by the present invention.

[0051] Figure 5 A structural block diagram of a video modality feature processing system based on BPE is provided as another embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The embodiment of the present invention discloses a video modality feature processing method based on BPE, such as Figure 1 As shown, including:

[0054] S1: Merge video frames using the BPE algorithm, and after reaching a set number of video frames, perform video encoding to obtain a video feature vector representation; encode the text to obtain a text feature vector representation;

[0055] S2: linearly transform the text feature vector representation and the video feature vector representation to obtain a multimodal feature vector representation;

[0056] S3: Process the multimodal feature vector representation through a large language model to generate a multi-dimensional video content representation.

[0057] In this embodiment, if Figure 2 As shown, the BPE algorithm specifically includes:

[0058] S101: Acquire all video frames of the video;

[0059] S102: segmenting all video frames, that is, merging and iterating video frames in the same field, and video frames in different fields are not merged;

[0060] S103: Calculate the similarity of consecutive frames;

[0061] S104: merging the two frames with the highest similarity and averaging them into a new video frame;

[0062] S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames (generally not more than 10 times), and merging all sub-segments; S102 After the video frame is segmented, each video frame is processed in S103 and S104 respectively, and S105 processes each video frame segment at the same time until the total number of frames of all video frame segments reaches twice the predetermined number of frames;

[0063] S106: Merge iterative process, repeat S103 and S104 until the predetermined number of frames is reached. S106 is to merge all video frame segments into one segment and then iterate.

[0064] At the same time, in S105 and S106, if the number of video frames is less than the predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

[0065] The present invention retains video information to the greatest extent by iteratively merging similar consecutive frames in a video while only being able to process a limited number of video frames. Through a carefully designed multi-step processing flow, it can adapt to video content of different lengths, ensuring that key information can be effectively retained regardless of the length of the video.

[0066] In this embodiment, when the merged object of the video frame BPE is a video frame, that is, when the video frame is merged by the BPE algorithm, the similarity of the continuous frames is calculated by the histogram overlap, and the calculation formula is as follows:

[0067]

[0068] Among them, I(H1,H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

[0069] The embodiment of the present invention discloses a video modality feature processing method based on BPE, such as Figure 3 As shown, including:

[0070] S1: After encoding the video frame, merge it through the BPE algorithm to obtain the video frame feature vector; encode the text to obtain the text feature vector representation;

[0071] S2: linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation;

[0072] S3: Process the multimodal feature vector representation through a large language model to generate a multi-dimensional video content representation.

[0073] In this embodiment, if Figure 2As shown, the BPE algorithm specifically includes:

[0074] S101: Acquire all video frames of the video;

[0075] S102: segmenting the video frame feature vectors obtained after video encoding of all video frames, that is, merging and iterating the video frame feature vectors in the same field, and the video frame feature vectors in different fields are not merged;

[0076] S103: Calculating the similarity of feature vectors of consecutive video frames;

[0077] S104: merging the two video frame feature vectors with the highest similarity, and averaging and merging them into a new video frame feature vector;

[0078] S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames (generally not more than 10 times), and merging all sub-segments; after S102 segmentation, each segment of the video frame feature vector is processed by S103 and S104 respectively, and S105 is to process each video frame feature vector segment at the same time until the total number of frames of all video frame feature vector segments reaches twice the predetermined number of frames;

[0079] S106: Merge iterative process, repeat S103 and S104 until the predetermined number of frames is reached. S106 is to merge all video frame feature vector segments into one segment and then iterate.

[0080] At the same time, in S105 and S106, if the number of video frames is less than the predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

[0081] The present invention retains video information to the greatest extent by iteratively merging similar consecutive frames in a video while only being able to process a limited number of video frames. Through a carefully designed multi-step processing flow, it can adapt to video content of different lengths, ensuring that key information can be effectively retained regardless of the length of the video.

[0082] In this embodiment, when the merged object is a video frame that is merged through video encoding and then through the BPE algorithm, the similarity of the feature vectors of consecutive frames is calculated by cosine similarity:

[0083]

[0084] Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

[0085] This embodiment provides a video modality feature processing system based on BPE, such as Figure 4 As shown, including:

[0086] Video BPE module: used to merge video frames through the BPE algorithm to reach a set number of video frames;

[0087] Multimodal feature encoding module: used to encode the video frames through the BPE algorithm to obtain the video feature vector representation, and to encode the text to obtain the text feature vector representation;

[0088] Feature alignment module: used to linearly transform the text feature vector representation and the video feature vector representation to obtain a multimodal feature vector representation;

[0089] Cognitive module: used to process multimodal feature vector representations through a large language model to generate multi-dimensional video content representations.

[0090] In this embodiment, the BPE algorithm specifically includes:

[0091] S101: Acquire all video frames of the video;

[0092] S102: segmenting all video frames, that is, merging and iterating video frames in the same field, and video frames in different fields are not merged;

[0093] S103: Calculate the similarity of consecutive frames;

[0094] S104: merging the two frames with the highest similarity and averaging them into a new video frame;

[0095] S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames, and merging all sub-segments; S102 After the video frame is segmented, each video frame is processed in S103 and S104 respectively, and S105 processes each video frame segment at the same time until the total number of frames of all video frame segments reaches twice the predetermined number of frames;

[0096] S106: Merge iterative process, repeat S103 and S104 until the predetermined number of frames is reached. S106 is to merge all video frame segments into one segment and then iterate.

[0097] In this embodiment, the video BPE module is one of the key components of the system, and its function is to process video information by merging similar video frames, while only a limited number of video frames can be processed, and to capture the most effective video information for input to subsequent modules.

[0098] The merged object of the video frame BPE is the video frame, which is located before the video encoder. The similarity of consecutive frame pairs is calculated by histogram overlap. The histogram overlap calculation method is as follows:

[0099]

[0100] Among them, I(H1, H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

[0101] Multimodal feature encoding module, in the field of multimodal information processing, the role of multimodal feature encoding module is crucial. It is responsible for converting different forms of data in the human world into feature vector representations so that the model can perform further analysis and understanding. The multimodal information encoding module of the present invention includes a video encoder and a text encoder, which provides accurate input for the feature alignment module. The video encoder adopts the VIT-L / 14 model of CLIP, which is a Transformer structure with a depth of 14 layers, and stands out in video feature extraction with its excellent performance. VIT-L / 14 can not only deeply explore the details of each frame in the video and accurately capture the essence of the visual content, but also effectively process the temporal correlation between frames and frames, and convert complex video information into a series of high-dimensional feature vectors. These feature vectors not only capture the essence of the visual content, but also retain the temporal correlation between frames, providing a data basis for subsequent video analysis tasks such as action recognition, scene understanding or content retrieval. The text encoder selects LLMs as the text encoder to extract the feature vector representation of the text. The power of LLMs lies in its ability to understand and process complex language structures, capture semantic information and contextual relationships in text, and thus transform text into feature vectors with rich expressiveness.

[0102] The feature vector representation of the text (what does this video talk about) and the video is added with special tokens (characters) and then concatenated and input into the feature alignment module. The format of the aligned data is <text>text<text\> <video>video<video\> ,The purpose of special tokens is to distinguish text features from video features.

[0103] The feature alignment module is responsible for unifying the representations from different modal encoders into a shared space to achieve effective integration of multimodal information. Specifically, the feature alignment module adopts a concise and efficient linear layer structure. This design enables the module to achieve accurate alignment and fusion of different modal features while maintaining computational efficiency. The role of the linear layer is to linearly transform the feature vectors from the video encoder and the text encoder and map them into a common feature space. In this space, features of different modalities can not only complement each other, but also eliminate the representation differences between different modalities, enhancing the model's overall understanding of multimodal data.

[0104] The cognitive module is responsible for receiving and processing the feature representations after multiple modal alignments, making full use of the powerful understanding ability of the pre-trained LLM, and can deeply analyze and integrate the information of each modality, so as to provide accurate and comprehensive responses. The LLM selected by existing videoLLMs is often a dense architecture model, with only one MLP in each layer, and different modal features are processed by the same MLP. The LLM adopted in the present invention is a model of MOE architecture, and each layer of the MOE architecture model has multiple MLPs. After LLM is trained, different MLPs have the ability to process different modal features. Compared with the model of dense architecture LLM, the LLM of MOE architecture can better understand the information of different modal features. According to special tokens, different MLPs in LLM can process and understand information of different modalities and capture the complex semantic relationship between modalities. Through this module, LLM can generate an accurate summary of the video content and give a response in the form of natural language.

[0105] This embodiment provides a video modality feature processing system based on BPE, such as Figure 5 As shown, including:

[0106] Multimodal feature encoding module: used to directly encode video frames and encode text to obtain text feature vector representation;

[0107] Video BPE module: used to merge the video frames after video encoding through the BPE algorithm to obtain the video frame feature vector;

[0108] Feature alignment module: used to linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation;

[0109] Cognitive module: used to process multimodal feature vector representations through a large language model to generate multi-dimensional video content representations.

[0110] In this embodiment, the BPE algorithm specifically includes:

[0111] S101: Acquire all video frames of the video;

[0112] S102: segmenting the video frame feature vectors obtained after video encoding of all video frames, that is, merging and iterating the video frame feature vectors in the same field, and the video frame feature vectors in different fields are not merged;

[0113] S103: Calculating the similarity of feature vectors of consecutive video frames;

[0114] S104: merging the two video frame feature vectors with the highest similarity, and averaging and merging them into a new video frame feature vector;

[0115] S105: Segmented iterative process, repeating S103 and S104 until the number of frames reaches twice the predetermined number, and merging all sub-segments; after S102 segmentation, each video frame feature vector is processed in S103 and S104 respectively, and S105 processes each video frame feature vector segment at the same time until the total number of frames of all video frame feature vector segments reaches twice the predetermined number of frames;

[0116] S106: Merge iterative process, repeat S103 and S104 until the predetermined number of frames is reached. S106 is to merge all video frame feature vector segments into one segment and then iterate.

[0117] In this embodiment,

[0118] Multimodal feature encoding module, in the field of multimodal information processing, the role of multimodal feature encoding module is crucial. It is responsible for converting different forms of data in the human world into feature vector representations so that the model can perform further analysis and understanding. The multimodal information encoding module of the present invention includes a video encoder and a text encoder, which provides accurate input for the feature alignment module. The video encoder adopts the VIT-L / 14 model of CLIP, which is a Transformer structure with a depth of 14 layers, and stands out in video feature extraction with its excellent performance. VIT-L / 14 can not only deeply explore the details of each frame in the video and accurately capture the essence of the visual content, but also effectively process the temporal correlation between frames and frames, and convert complex video information into a series of high-dimensional feature vectors. These feature vectors not only capture the essence of the visual content, but also retain the temporal correlation between frames, providing a data basis for subsequent video analysis tasks such as action recognition, scene understanding or content retrieval. The text encoder selects LLMs as the text encoder to extract the feature vector representation of the text. The power of LLMs lies in its ability to understand and process complex language structures, capture semantic information and contextual relationships in text, and thus transform text into feature vectors with rich expressiveness.

[0119] The video BPE module is one of the key components of the system. Its function is to process video information by merging similar video frames and capturing the most effective video information to input to subsequent modules when only a limited number of video frames can be processed.

[0120] The merged object of the video frame BPE is the feature vector of the video frame, which is located after the video encoder. The similarity of the feature vectors of consecutive frames is calculated by cosine similarity:

[0121]

[0122] Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

[0123] The processed text (what does this video say) and the feature vector representation of the video are concatenated with special tokens and input into the feature alignment module, such as <text>text<text\> <video>video<video\> ,The purpose of special tokens is to distinguish text features from video features.

[0124] The feature alignment module is responsible for unifying the representations from different modal encoders into a shared space to achieve effective integration of multimodal information. Specifically, the feature alignment module adopts a concise and efficient linear layer structure. This design enables the module to achieve accurate alignment and fusion of different modal features while maintaining computational efficiency. The role of the linear layer is to linearly transform the feature vectors from the video encoder and the text encoder and map them into a common feature space. In this space, features of different modalities can not only complement each other, but also eliminate the representation differences between different modalities, enhancing the model's overall understanding of multimodal data.

[0125] The cognitive module is responsible for receiving and processing the feature representations after multiple modal alignments, making full use of the powerful understanding ability of the pre-trained LLM, and can deeply analyze and integrate the information of each modality, so as to provide accurate and comprehensive responses. The LLM selected by existing videoLLMs is often a dense architecture model, with only one MLP in each layer, and different modal features are processed by the same MLP. The LLM adopted in the present invention is a model of MOE architecture, and each layer of the MOE architecture model has multiple MLPs. After LLM is trained, different MLPs have the ability to process different modal features. Compared with the model of dense architecture LLM, the LLM of MOE architecture can better understand the information of different modal features. According to special tokens, different MLPs in LLM can process and understand information of different modalities and capture the complex semantic relationship between modalities. Through this module, LLM can generate an accurate summary of the video content and give a response in the form of natural language.

[0126] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0127] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / video> < / text> < / video> < / text>

Claims

1. A video modality feature processing method based on BPE, characterized in that: include: S1: Merge the video frames through the BPE algorithm, and after reaching a set number of video frames, perform video encoding to obtain a video feature vector representation, or merge the video frames through the BPE algorithm after video encoding to obtain a merged video frame feature vector; Encode the text to obtain a text feature vector representation; S2: linearly transform the text feature vector representation and the video feature vector representation, or linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation; S3: Process the multimodal feature vector representation through a large language model to generate a multi-dimensional video content representation.

2. The video modality feature processing method based on BPE according to claim 1, characterized in that: The BPE algorithm specifically includes: S101: Acquire all video frames of the video; S102: segmenting all video frames or segmenting video frame feature vectors obtained by encoding the video frames, that is, merging and iterating video frames or video frame feature vectors in the same field, and video frames or video frame feature vectors in different fields are not merged; S103: Calculating the similarity of consecutive frames or consecutive frame feature vectors; S104: merging the two frames or frame feature vectors with the highest similarity, and averaging and merging them into a new video frame or a new video frame feature vector; S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames, and merging all sub-segments; S106: Merge the iterative process and repeat S103 and S104 until a predetermined number of frames is reached.

3. The video modality feature processing method based on BPE according to claim 2 is characterized in that: When merging video frames through the BPE algorithm, the similarity of consecutive frames is calculated by the histogram overlap. The calculation formula is as follows: Among them, I(H1,H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

4. The video modality feature processing method based on BPE according to claim 2 is characterized in that: When the video frames are encoded and then merged by the BPE algorithm, the similarity of the feature vectors of consecutive frames is calculated by cosine similarity: Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

5. The video modality feature processing method based on BPE according to claim 2 is characterized in that: If the number of video frames is less than the predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

6. A video modality feature processing system based on BPE, characterized in that: include: Video BPE module: used to merge video frames through the BPE algorithm to reach a set number of video frames, or to merge video frames after video encoding through the BPE algorithm to obtain a merged video frame feature vector; Multimodal feature encoding module: used to encode the video frames through the BPE algorithm to obtain a video feature vector representation or directly encode the video frames, and encode the text to obtain a text feature vector representation; Feature alignment module: used to linearly transform the text feature vector representation and the video feature vector representation, or linearly transform the text feature vector representation and the video frame feature vector to obtain a multimodal feature vector representation; Cognitive module: used to process multimodal feature vector representations through a large language model to generate multi-dimensional video content representations.

7. The BPE-based video modality feature processing system according to claim 6, characterized in that: The BPE algorithm specifically includes: S101: Acquire all video frames of the video; S102: segmenting all video frames or segmenting video frame feature vectors obtained by encoding the video frames, that is, merging and iterating video frames or video frame feature vectors in the same field, and video frames or video frame feature vectors in different fields are not merged; S103: Calculating the similarity of consecutive frames or consecutive frame feature vectors; S104: merging the two frames or frame feature vectors with the highest similarity, and averaging and merging them into a new video frame or a new video frame feature vector; S105: Segmented iterative process, repeating S103 and S104 until reaching twice the predetermined number of frames, and merging all sub-segments; S106: Merge the iterative process and repeat S103 and S104 until a predetermined number of frames is reached.

8. The BPE-based video modality feature processing system according to claim 7, characterized in that: When merging video frames through the BPE algorithm, the similarity of consecutive frames is calculated by the histogram overlap. The calculation formula is as follows: Among them, I(H1, H2) represents the overlap between the two histograms H1 and H2, H 1i and H 2i They represent the value of the i-th interval in histograms H1 and H2 respectively, and m is the number of intervals in the histogram.

9. The BPE-based video modality feature processing system according to claim 7, characterized in that: When the video frames are encoded and then merged by the BPE algorithm, the similarity of the feature vectors of consecutive frames is calculated by cosine similarity: Among them, A and B are two consecutive frame feature vectors, A i represents the i-th element of the eigenvector of A, B i represents the i-th element of the B feature vector, and n represents the dimension of the feature vector.

10. The BPE-based video modality feature processing system according to claim 7, characterized in that: If the number of video frames is less than the predetermined number of frames, blank frames are added at the end to make it consistent with the predetermined number of frames.

Images