Video understanding method, system and equipment based on instruction condition compression and medium

Abstract

The invention discloses a video understanding method, system, equipment and medium based on instruction condition compression, which are one-to-one corresponding schemes, in the scheme, from the perspective of conditional compression, instruction content is introduced as a condition, and targeted compression is performed, that is, instructions are injected at two levels of local and global mixing, so that the compression efficiency is improved; visual information associated with an instruction is reserved as much as possible in the compression process, irrelevant information loss is allowed, conditional compression is realized, high compression rate and low information loss of visual features can be well considered during compression, visual details required for completing an instruction task can be reserved as much as possible, and the compression efficiency is improved. Therefore, a video understanding task is better completed, and excellent video understanding ability is obtained while more efficient compression is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of video content understanding based on a video large language model, and in particular to a video understanding method, system, device and medium based on instruction conditional compression. Background Art

[0002] The video language model is an autoregressive text generation model with a large number of parameters, trained with massive data, and able to understand video content. It supports multimodal data input including video. Based on its understanding of video content, the video language model can respond to input text commands and answer command questions. It has great application potential in the fields of intelligent human-computer interaction, augmented reality, autonomous driving, robotics, and smart medical care, and is also an important step in achieving general artificial intelligence.

[0003] The video large language model mainly consists of a visual encoder, a visual connector, and a large language model (LLM). Video data consists of multiple frames of images, which will generate a large number of visual tokens after visual encoding, which brings a heavy computational burden to the training and reasoning of the large language model. Therefore, an efficient video token compression method is an urgent solution strategy for all current video large language models to balance the computational overhead. Compression often inevitably loses information details, resulting in a decrease in the model's video comprehension ability.

[0004] Existing compression methods include pooling, convolution, clustering in space and time, and the use of attention-based mechanisms such as Q-Former (a lightweight Transformer architecture designed for multimodal models, Transformer is a neural network based on attention mechanism) or memory pool. However, their compression methods lack explicit guidance and are therefore unconditional compression, making it difficult to balance high compression rate and low information loss for video tokens.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a video understanding method, system, device and medium based on instruction conditional compression. From the perspective of conditional compression, instruction injection is performed at two mixed levels, local and global, and conditional compression is completed using an attention mechanism, thereby achieving excellent video understanding capabilities while achieving more efficient compression.

[0007] The objective of the present invention is achieved through the following technical solutions: A video understanding method based on instruction conditional compression, combined with a video large language model, includes: Visually encode the input video to obtain a visual feature sequence; At the local level, the visual feature sequence is divided into several groups, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions, and the local visual features corresponding to all groups are integrated as the locally compressed visual features; at the global level, the visual feature sequence is positionally embedded, and the learnable visual features are introduced and the instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism; where the instruction conditions are the text features of the text instructions; The answer result is outputted by combining the input text instruction, the locally compressed visual features and the globally compressed visual features.

[0008] A video understanding system based on instruction conditional compression, wherein a video large language model is configured in the system to implement the aforementioned method, wherein the video large language model comprises: A visual encoder is used to visually encode the input video to obtain a visual feature sequence; The visual connector is used to divide the visual feature sequence into several groups at the local level, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions, and the local visual features corresponding to all groups are integrated as the local compressed visual features; at the global level, the visual feature sequence is positionally embedded, and the learnable visual features are introduced and the instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism; wherein the instruction conditions are the text features of the text instructions; The large language model is used to combine the input text instructions, the locally compressed visual features and the globally compressed visual features to output an answer result.

[0009] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that the instruction content is introduced as a condition in the compression stage to perform targeted compression, that is, the instructions are injected at two levels of local and global mixing. During the compression process, the visual information associated with the instructions is retained as much as possible, and irrelevant information loss is allowed to achieve conditional compression. During compression, it can well balance the high compression rate and low information loss of visual features, and can retain as many visual details as possible required to complete the instruction task, thereby better completing the video understanding task. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic diagram of a video understanding method based on instruction conditional compression provided by an embodiment of the present invention; Figure 2 A schematic diagram of a direct injection method of instruction conditions provided by an embodiment of the present invention; Figure 3 A schematic diagram of a coarse-grained injection method of instruction conditions provided in an embodiment of the present invention; Figure 4 A schematic diagram of a fine-grained injection method of instruction conditions provided in an embodiment of the present invention; Figure 5 A schematic diagram of attention in mixed-level instruction conditional compression provided by an embodiment of the present invention; Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0015] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0016] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0017] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0018] The following is a detailed description of a video understanding method, system, device and medium based on command conditional compression provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. The instruments used in the embodiments of the present invention, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0019] Embodiment 1 The embodiment of the present invention provides a video understanding method based on instruction conditional compression, which is implemented in combination with a video large language model and mainly includes the following steps: Step 1: Visually encode the input video to obtain a visual feature sequence.

[0020] Step 2: At the local level, the visual feature sequence is divided into several groups, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions (i.e., the text features of the text instructions), and the local visual features corresponding to all groups are integrated as the local compressed visual features; at the global level, the visual feature sequence is positionally embedded, and learnable visual features are introduced and instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism.

[0021] Step 3: Combine the input text instruction, the locally compressed visual features and the globally compressed visual features to output the answer result. Specifically, the locally compressed visual features and the globally compressed visual features are respectively mapped to the embedding space of the large language model and concatenated, and then the large language model combines the concatenated embedded features with the text instruction to output the answer result.

[0022] like Figure 1 As shown in the figure, the overall framework process of the method is shown; Figure 1 As shown at the bottom, the input information includes: visual and text instructions. The visual encoder is responsible for executing the above step 1 and outputting a visual feature sequence. The visual encoder is a fully pre-trained model. The above step 2 is executed by the visual connector of the video large language model. The visual connector is trained with randomly initialized parameters to achieve the compression of visual features and project the visual features to the embedding layer of the large language model. The present invention is designed in the visual connector part, and instructions are injected from two mixed levels, local and global. The attention mechanism (referred to as local attention and global attention, respectively) is used to complete conditional compression. Specifically, at the local level, the visual feature sequence is divided into several groups, and each group of visual features is compressed into one visual feature according to the injected instruction conditions. Unlike Q-Former, the grouped local attention can explicitly maintain the spatiotemporal structural visual features of the video, while highlighting the relevant parts within each group. At the global level, the instruction conditions are injected into a small number of learnable visual features, and then the attention processing is performed on the flattened visual features without grouping. Global compression focuses more on searching for information related to the instruction in the entire video, so it can be expressed with a small number of tags as an auxiliary; the instruction conditions input in this part are text features extracted from the text instructions by the text encoder. The above step 3 is implemented by a large language model, which is also a fully pre-trained model, responsible for outputting the corresponding answer results, such as Figure 1 As shown in the right part, examples of input text instructions and answer results are provided. In this part, in addition to the above locally compressed visual features and globally compressed visual features, the large language model also needs to input text instructions. Specifically, in order for the large language model to be able to recognize the content of the text instructions, the text instructions need to be embedded and then the embedded text instructions are input into the large language model.

[0023] The above-mentioned scheme provided by the embodiment of the present invention can be loaded on a computer or a server and used for intelligent question and answering, and accurate answers are given according to the video and questions input by the user to help the user understand the video.

[0024] The above scheme provided by the embodiment of the present invention mainly has the following advantages: the existing methods usually adopt unconditional compression, that is, compression is not performed under the guidance of instruction information, which often makes it difficult to balance high compression rate and low information loss of visual features during compression. The visual details required to complete the instruction task may be ignored or suppressed, making it impossible for the large language model to complete the video understanding task of the given instruction. Starting from the perspective of conditional compression, the present invention performs instruction injection at the local and global mixed levels, and uses the attention mechanism to complete conditional compression, thereby achieving excellent video understanding capabilities while achieving more efficient compression.

[0025] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the detailed process of the method provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0026] In many practical application scenarios, the visual area that the user is interested in completing the instruction may be ignored or suppressed by the unconditional compression strategy. Therefore, we explore introducing the instruction content that the user cares about as a condition in the compression stage to perform targeted compression, that is, injecting instructions at two levels of local and global mixing, retaining the visual information associated with the instruction as much as possible during the compression process, allowing irrelevant information to be lost, and realizing conditional compression. When processing complex visual information, a coarse-to-fine approach is usually adopted, first considering the relevant parts in the overall rough impression, and then looking for detailed information locally in the video. Inspired by this perception process, the present invention compresses at the local and global levels, retaining as much information related to the instruction as possible, while maintaining the spatiotemporal structure with a smaller number of features, thereby achieving a better balance between computational burden and video understanding.

[0027] 1. Local level instruction condition compression.

[0028] In order to achieve effective conditional compression while maintaining the spatiotemporal structure of visual features, the present invention performs instruction conditional compression at the local level. The visual feature sequence is denoted as ,in, is a real number set symbol, T represents the length of the visual feature sequence, H and W represent the height and width of each visual feature (i.e., visual token), and D represents the dimension of the visual feature; at the same time, the downsampling rate parameter is given ; Divide the visual feature sequence into Group, where the number ,quantity ,quantity ,symbol Indicates rounding up.

[0029] Each group contains visual features, represented as ,in, are three index symbols, , , . Each group Average pooling (downsampling) into a visual feature , and combined with the injected instruction condition C (i.e., the text features of the text instruction), the corresponding local visual features are calculated through the attention mechanism , expressed as: ; Among them, Inj represents the injection process of instruction conditions, Attn represents the attention mechanism; the input of Attn is query, key and value (that is, the three parts in the brackets) in sequence. Therefore, the visual features in each group are compressed into only one visual feature, while conditionally highlighting the part related to the instruction. The local visual features corresponding to all groups of visual features are spliced ​​to obtain the locally compressed visual features. After that, we use an MLP (Multi-layer Perceptron) layer to transform Mapped into the embedding space of a large language model.

[0030] 2. Compression of global level instruction conditions.

[0031] Although the spatiotemporal structure can be preserved at the local level, attention is forced to focus on small sub-regions. However, each sub-region may contain information that is not equally valid for the problem, and some sub-regions are even completely irrelevant. Therefore, the embodiment of the present invention also implements conditional compression at the global level to highlight the most relevant parts of the entire video. Specifically: Initialize a small set of learnable visual features ,in Indicates the number of tokens, for example, 32; then the instruction condition C is injected into it. Different from grouping the video frame features, in this part, it is a sequence of visual features Apply the three-dimensional position embedding Pos and directly flatten them when compressing at the global level, and then calculate the visual features after global compression through the attention mechanism, which is expressed as: ; in, Represents the visual features after global compression, and Flat represents the flattening operation.

[0032] Similar to the local level, an MLP layer is also used to transform Mapped into the embedding space of the large language model. Finally, and The mapped results are stitched together for further understanding by the large language model.

[0033] Considering that the training process of the learnable visual features can be implemented with reference to conventional techniques, for example, they are randomly initialized at the beginning of training and set to be gradient-updatable, so that the learnable visual features in the model training process will also be optimized to appropriate parameters under the supervision of the loss function, so that they have the potential to represent visual features. The model training process described here refers to the training process of the video large language model, which can also be implemented with reference to conventional techniques, so it will not be described in detail.

[0034] 3. Instruction conditional injection.

[0035] In order to realize the guiding role of instruction conditions, the present invention explores three different types of injection methods and defines them as direct injection, coarse-grained injection and fine-grained injection respectively.

[0036] (1) Direct injection.

[0037] like Figure 2 As shown, the instruction condition C is injected directly, and the steps include: (1.1) inputting the text instruction into the text encoder to obtain the pooled global text features , taking it as instruction condition C; (1.2) the global text is transformed into Processing is done to complete the injection of instruction conditions. In this way, the global text is transformed by the linear layer Converts to injected output, not with or L for any interaction.

[0038] (2) Coarse-grained injection.

[0039] like Figure 3 As shown, the instruction condition C is injected in a coarse-grained manner, and the steps include: (2.1) inputting the text instruction into the text encoder to obtain the pooled global text features , and use it as the instruction condition C; (2.2) Calculate the product coefficient and offset coefficient through linear layer regression; (2.3) At the local level, after layer normalization of each group of visual features, add the multiplication coefficient and offset coefficient to complete the injection of the instruction condition; at the global level, after layer normalization of the learnable visual features, add the multiplication coefficient and offset coefficient to complete the injection of the instruction condition.

[0040] Use A to unify the representation Or L, the above process can be expressed as: ; in, , The corresponding representation is from the global text The product coefficient and offset coefficient are calculated in , and LN is the layer normalization processing.

[0041] (3) Fine-grained injection.

[0042] like Figure 4 As shown, the instruction condition C is injected in a fine-grained manner. The steps include: (4.1) inputting the text instruction into the text encoder to obtain the last hidden layer feature of the text encoder, which is called the fine-grained text feature , which is used as the instruction condition C; (4.2) At the local level, the visual features of each group are layer-normalized and compared with the fine-grained text Attention operations (e.g., cross-attention operations) are performed to complete the injection of instruction conditions; at the global level, the learnable visual features are layer-normalized and compared with the fine-grained text Perform cross-attention operations to complete the injection of instruction conditions.

[0043] Use A to unify the representation Or L, the above process can be expressed as: ; Among them, Attn represents the attention mechanism.

[0044] Since direct injection directly converts conditions into the query of the attention mechanism at both the local and global levels, the length of the query limits it to local level compression only, as the local branches compress each group into only one visual feature. In contrast, coarse-grained and fine-grained injection is more flexible and can adapt to various situations because conditions are added to A.

[0045] above Figure 2~Figure 4 In the figure, the rectangular box filled with horizontal lines on the right represents the corresponding instruction condition C, and the white rectangular box on the left represents the corresponding visual feature. or L.

[0046] In order to verify the effectiveness of the present invention, verification and evaluation were carried out on five general evaluation benchmark datasets (Video MME, MV-Bench, Ego-Schema, Activity-Net, and VCG-Bench). The training results of the Qwen2.5 model (an open source large model launched by Alibaba Cloud) with 7B (7 billion) parameters implemented by the present invention all achieved advanced performance, as shown in Table 1.

[0047] Table 1: Video understanding performance on various benchmark datasets

[0048] In Table 1, VideoMME w / o sub and VideoMMEw / sub are two subsets of the benchmark dataset Video MME. The former subset consists of videos without subtitle information, and the latter subset consists of videos with time-synchronized subtitles.

[0049] In addition, the attention mechanism during compression is visualized, e.g. Figure 5 As shown, it is intuitively demonstrated that the video large language model pays attention to the areas related to instructions during the compression process, thereby better retaining relevant information.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0051] Embodiment 2 The present invention also provides a video understanding system based on instruction conditional compression, in which a video large language model is configured, which is mainly used to implement the method provided in the above embodiment, and the video large language model includes: A visual encoder is used to visually encode the input video to obtain a visual feature sequence; The visual connector is used to divide the visual feature sequence into several groups at the local level, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions, and the local visual features corresponding to all groups are integrated as the local compressed visual features; at the global level, the visual feature sequence is positionally embedded, and the learnable visual features are introduced and the instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism; wherein the instruction conditions are the text features of the text instructions; The large language model is used to combine the input text instructions, the locally compressed visual features and the globally compressed visual features to output an answer result.

[0052] Considering that the technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0053] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0054] Embodiment 3 The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0055] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0056] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0057] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0058] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0059] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A video understanding method based on instruction conditional compression, combined with a video large language model, characterized in that: include: Visually encode the input video to obtain a visual feature sequence; At the local level, the visual feature sequence is divided into several groups, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions, and the local visual features corresponding to all groups are integrated as the locally compressed visual features; at the global level, the visual feature sequence is positionally embedded, and the learnable visual features are introduced and the instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism; where the instruction conditions are the text features of the text instructions; The answer result is outputted by combining the input text instruction, the locally compressed visual features and the globally compressed visual features.

2. The video understanding method based on instruction conditional compression according to claim 1, characterized in that: At the local level, the visual feature sequence is divided into several groups, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions. The local visual features corresponding to all groups are integrated as the local compressed visual features including: The visual feature sequence is recorded as ,in, is a real number set symbol, T represents the length of the visual feature sequence, H and W represent the height and width of each visual feature, and D represents the dimension of the visual feature; Divide the visual feature sequence into groups, each containing visual features, represented as , where the number ,quantity ,quantity , For a given downsampling rate parameter, the symbol Indicates rounding up; Each group Average pooling as a visual feature , and combined with the injected instruction condition C, the corresponding local visual features are calculated through the attention mechanism , expressed as: ； Among them, Inj represents the injection process of instruction conditions, and Attn represents the attention mechanism; Concatenate the local visual features corresponding to all groups of visual features to obtain the locally compressed visual features .

3. The video understanding method based on instruction conditional compression according to claim 1, characterized in that: At the global level, the visual feature sequence is positionally embedded, and learnable visual features are introduced and instruction conditions are injected. Then, the globally compressed visual features are calculated through the attention mechanism, which is expressed as: ； in, represents the visual feature sequence, Pos represents the position embedding, Flat represents the flattening operation, L represents the learnable visual feature, C represents the instruction condition, Inj represents the injection process of the instruction condition, Attn represents the attention mechanism, Represents the visual features after global compression.

4. A video understanding method based on instruction conditional compression according to any one of claims 1 to 3, characterized in that: It also includes: injecting instruction condition C by direct injection, the steps include: Input the text instruction into the text encoder to obtain the pooled global text features , take it as instruction condition C; Through the linear layer, the global text Processing is performed to complete the injection of instruction conditions.

5. A video understanding method based on instruction conditional compression according to any one of claims 1 to 3, characterized in that: It also includes: injecting instruction condition C in the following manner, the steps include: Input the text instruction into the text encoder to obtain the pooled global text features , take it as instruction condition C; The product coefficient and the offset coefficient are calculated by linear layer regression; At the local level, after the layer normalization processing is performed on the visual features of each group, the product coefficient and the offset coefficient are added to complete the injection of the instruction condition; at the global level, after the layer normalization processing is performed on the learnable visual features, the product coefficient and the offset coefficient are added to complete the injection of the instruction condition; The method of injecting instruction condition C using the above steps is called a coarse-grained method.

6. A video understanding method based on instruction conditional compression according to any one of claims 1 to 3, characterized in that: It also includes: injecting instruction condition C in the following manner, the steps include: Input the text instruction into the text encoder to obtain the last hidden layer feature of the text encoder, which is called fine-grained text feature , take it as instruction condition C; At the local level, the visual features of each group are layer-normalized and compared with the fine-grained text Attention is calculated to complete the injection of instruction conditions; at the global level, the learnable visual features are layer-normalized and compared with the fine-grained text Perform cross-attention operations to complete the injection of instruction conditions; The method of injecting instruction condition C using the above steps is called a fine-grained method.

7. The video understanding method based on instruction conditional compression according to claim 1, characterized in that: The combined input text instruction, the locally compressed visual features and the globally compressed visual features, output answer results include: The locally compressed visual features and the globally compressed visual features are respectively mapped to the embedding space of the large language model and spliced; the large language model then combines the spliced ​​embedded features with the text instructions to output an answer result.

8. A video understanding system based on instruction conditional compression, characterized in that: The system is configured with a video language model for implementing the method described in any one of claims 1 to 7, and the video language model includes: A visual encoder is used to visually encode the input video to obtain a visual feature sequence; The visual connector is used to divide the visual feature sequence into several groups at the local level, each group is compressed into a visual feature, and the local visual features are calculated through the attention mechanism in combination with the injected instruction conditions, and the local visual features corresponding to all groups are integrated as the local compressed visual features; at the global level, the visual feature sequence is positionally embedded, and the learnable visual features are introduced and the instruction conditions are injected, and then the globally compressed visual features are calculated through the attention mechanism; wherein the instruction conditions are the text features of the text instructions; The large language model is used to combine the input text instructions, the locally compressed visual features and the globally compressed visual features to output an answer result.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images