A long video understanding method, device, equipment and medium based on large model

Through the second-level frame extraction and batch processing technology, the problem of the context length limitation of large language models in long video understanding is solved, and efficient understanding and accurate analysis of long videos is achieved.

CN118840697BActive Publication Date: 2024-11-26BEIJING SINOITS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411310740.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-26
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

Existing long video understanding large models cannot effectively process videos that are too long because the context length of large language models is limited, resulting in information loss or inefficient processing.

Method used

The video frame sequence is obtained by drawing frames in seconds, and divided according to the frame sequence length and the context length of the large language model. Each sub-sequence is processed in batches. Finally, the video target understanding result is obtained through fusion processing, reducing the large language model pair Limitation of video understanding.

Benefits of technology

Without losing important information, the video duration that large language models can process is significantly increased, avoiding information loss, and improving the accuracy of video comprehension.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118840697B_ABST
    Figure CN118840697B_ABST
Patent Text Reader

Abstract

The present invention relates to a method, device, equipment and medium for understanding long videos based on a large model, the method comprising: obtaining a video to be processed; extracting frames of the video to be processed at the second level to obtain a preliminary video frame sequence; dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length to obtain at least one subsequence, batch processing each of the subsequences through the first large language model to obtain an understanding result corresponding to each subsequence; fusing multiple understanding results to obtain a target understanding result corresponding to the video to be processed. The method of the present invention effectively reduces the limitation of the context length of the large language model on video understanding, and greatly increases the duration of the video that can be processed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models. Specifically, the present invention relates to a long video understanding method, device, equipment and medium based on a large model. Background Art

[0002] In the prior art, a large model for understanding long videos is one that can provide a textual understanding of a long video and give answers to the user's questions. The main challenge of a large model for understanding long videos is that the model has a limited context length and cannot process videos that are too long.

[0003] The current mainstream long video understanding large models usually use methods such as video information compression or increasing the context length of large language models.

[0004] See also Figure 1 The video information compression method refers to sending the frame sequence after video extraction into the visual encoder for encoding, and then using a model to compress the encoded information of all input frames. Finally, the large language model is connected through the projection layer.

[0005] See also Figure 2 ,The method of increasing the context length of the large language model refers to ,sending the video frame by frame into the visual encoder for encoding, and then connecting it to the large language model through the ,projection layer, and finally using the long context capability of the large language model to ,understand the video.

[0006] Among the existing long video understanding methods, the video information compression method is the most common. When the input video is short and the content is simple, not much information will be lost. However, if the input video is long or densely populated, the video compression method will lose information, making it impossible for the large language model to accurately understand the video.

[0007] The method of increasing the context length of the large language model is a popular method recently. This method processes each frame of a long video and inputs it into a large language model with a long context. The advantage of this method over video information compression is that it does not cause information loss. As long as the context length of the large language model is infinite, it can theoretically process videos of any length. However, the context length of the current large language models is limited. The length of the video that this method can process depends entirely on the context length of the large language model. If the context length of the large language model LLM is not enough, it will not be able to process videos that are too long. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a long video understanding method, device, equipment and medium based on a large model, aiming to solve at least one of the above technical problems.

[0009] In the first aspect, the technical solution of the present invention to solve the above technical problem is as follows: a long video understanding method based on a large model, the method comprising:

[0010] Get the video to be processed;

[0011] Extract frames at the second level on the video to be processed to obtain a preliminary video frame sequence;

[0012] Dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and a first set length to obtain at least one subsequence, wherein the first set length is a length reserved for user input questions, and the length of each of the subsequences is a maximum length processed by the first large language model at one time;

[0013] Processing each of the subsequences in batches by using the first large language model to obtain an understanding result corresponding to each of the subsequences;

[0014] The multiple understanding results are fused to obtain a target understanding result corresponding to the video to be processed.

[0015] The beneficial effect of the present invention is that: for the video to be processed, firstly, frame extraction is performed at the second level to obtain a preliminary video frame sequence, so that there will be neither too much redundant information nor loss of important information. Then, according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length, the preliminary video frame sequence is divided to obtain at least one subsequence, and then each of the subsequences is processed in batches by the first large language model to obtain the understanding result corresponding to each subsequence, and finally all the batches of understanding results are analyzed and summarized as a whole to obtain the target understanding result, which effectively reduces the limitation of the context length of the large language model on video understanding and greatly increases the duration of the video that can be processed.

[0016] Based on the above technical solution, the present invention can also be improved as follows.

[0017] Furthermore, after extracting frames at the second level on the video to be processed to obtain a preliminary video frame sequence, the method further includes:

[0018] Extracting features from the preliminary video frame sequence to obtain a feature sequence;

[0019] Performing dimensionality reduction processing on the feature sequence to obtain a target feature sequence, wherein the dimension of the target feature sequence is the processing dimension corresponding to the first largest language model;

[0020] The step of dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and the first set length to obtain at least one subsequence includes:

[0021] The target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and a first set length to obtain at least one subsequence.

[0022] The beneficial effect of adopting the above further scheme is that the feature extraction is first performed on the preliminary video frame sequence, which can effectively extract useful information, and then the dimension reduction processing is performed on the feature sequence to ensure the processing accuracy of the first language model.

[0023] Further, if the at least one subsequence is at least two subsequences, there is at least one overlapping video frame between each two adjacent subsequences in the at least two subsequences.

[0024] The beneficial effect of adopting the above further solution is that, in order to ensure the connection between the subsequences of each batch, there is at least one overlapping video frame between two adjacent subsequences.

[0025] Furthermore, the target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and the first set length to obtain at least one subsequence, including:

[0026] Determining a maximum number of video frames that the first large language model processes at one time according to the length of the target feature sequence, the context length of the first large language model, and a first set length;

[0027] The target feature sequence is divided according to the maximum number of frames to obtain at least one subsequence.

[0028] The beneficial effect of adopting the above further solution is that the maximum number of frames is determined first, and then the sub-sequences are divided, which can further ensure the division accuracy and fully utilize the processing capacity of the first language model.

[0029] Furthermore, the above-mentioned fusion processing of the multiple understanding results to obtain the target understanding result corresponding to the video to be processed includes:

[0030] The plurality of understanding results are fused through the second largest language model to obtain a target understanding result corresponding to the video to be processed, wherein the length of each understanding result is no greater than the length corresponding to a subsequence processed by the second largest language model.

[0031] The beneficial effect of adopting the above further scheme is that the length of the understanding result output by the first largest language model is limited based on the corresponding length when the second largest language model processes a subsequence, which can ensure that the second largest language model has sufficient context length to process multiple subsequences.

[0032] In a second aspect, in order to solve the above technical problems, the present invention also provides a long video understanding device based on a large model, the device comprising:

[0033] An acquisition module is used to acquire the video to be processed;

[0034] A frame extraction module, used for extracting frames of the video to be processed at the second level to obtain a preliminary video frame sequence;

[0035] a division module, configured to divide the preliminary video frame sequence into at least one subsequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and a first set length, wherein the first set length is a length reserved for a user to input a question, and the length of each of the subsequences is a maximum length processed by the first large language model at one time;

[0036] A batch processing module, used for performing batch processing on each of the subsequences through the first large language model to obtain an understanding result corresponding to each of the subsequences;

[0037] The summary module is used to fuse the multiple understanding results to obtain the target understanding result corresponding to the video to be processed.

[0038] In the third aspect, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the large model-based long video understanding method of the present application is implemented.

[0039] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the large model-based long video understanding method of the present application is implemented.

[0040] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0042] Figure 1A schematic diagram of a flow chart of a video information compression method in the prior art provided by an embodiment of the present invention;

[0043] Figure 2 A flowchart of a method for increasing the context length of a large language model in the prior art provided by one embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a process for understanding a long video based on a large model provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of another process of long video understanding based on a large model provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a long video understanding device based on a large model provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0049] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0050] The solution provided by the embodiment of the present invention can be applied to any application scenario that requires accurate understanding of the video. The solution provided by the embodiment of the present invention can be executed by any electronic device, for example, it can be a user's terminal device, including at least one of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0051] The embodiment of the present invention provides a possible implementation method, such as Figure 1 As shown, a flowchart of a long video understanding method based on a large model is provided. The solution can be executed by any electronic device, for example, a terminal device, or a terminal device and a server. For the convenience of description, the method provided by the embodiment of the present invention will be described below using a terminal device as an example of the execution subject. Figure 1 The method may include the following steps:

[0052] Step S10, obtaining the video to be processed;

[0053] Step S20, extracting frames at the second level for the video to be processed to obtain a preliminary video frame sequence;

[0054] Step S30, dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and a first set length to obtain at least one subsequence, wherein the first set length is a length reserved for user input questions, and the length of each of the subsequences is a maximum length processed by the first large language model at one time;

[0055] Step S40, batch processing each of the subsequences using the first large language model to obtain an understanding result corresponding to each of the subsequences;

[0056] Step S50, fusing the plurality of understanding results to obtain a target understanding result corresponding to the video to be processed.

[0057] According to the method of the present invention, for the video to be processed, firstly, frame extraction is performed at the second level to obtain a preliminary video frame sequence, so that there will be neither too much redundant information nor loss of important information. Then, according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length, the preliminary video frame sequence is divided to obtain at least one subsequence, and then the first large language model is used to process each of the subsequences in batches to obtain the understanding result corresponding to each subsequence, and finally the understanding results of all batches are analyzed and summarized as a whole to obtain the target understanding result, which effectively reduces the limitation of the context length of the large language model on video understanding and greatly increases the duration of the video that can be processed.

[0058] The scheme of the present invention is further described below in conjunction with the following specific embodiments. In this embodiment, see Figure 3 and Figure 4 , a long video understanding method based on a large model provided may include the following steps:

[0059] Step S10, obtaining the video to be processed;

[0060] The video to be processed refers to a video that needs to be understood, which can be a video longer than a set length, which can be called a long video. The video to be processed can also be a short video, that is, a video shorter than a set length. The video to be processed in the solution of this application is not limited in length, but based on the solution of this application, the processing effect of long videos is better than that of short videos.

[0061] Step S20, extracting frames at the second level for the video to be processed to obtain a preliminary video frame sequence;

[0062] One possible implementation of the above S20 is:

[0063] Extract the middle frame of each video frame corresponding to each second in the video to be processed to obtain a preliminary video frame sequence. In this way, the extracted frames are separated by only 1 second, so there will not be too much redundant information and important information will not be lost.

[0064] After S20, the method further includes:

[0065] S21, extracting features from the preliminary video frame sequence to obtain a feature sequence;

[0066] S22, performing dimensionality reduction processing on the feature sequence to obtain a target feature sequence, wherein the dimension of the target feature sequence is a processing dimension corresponding to the first largest language model.

[0067] Among them, one implementation of the above S21 is: extract features of each frame image in the preliminary video frame sequence through a video encoder, which can also be called encoding, to obtain features corresponding to each frame image, which can be expressed as , where index represents the frame index to which each feature belongs. Then the encoded features are sorted according to the time axis to obtain the feature sequence.

[0068] In order to enable the first language model to recognize video features, the feature sequence needs to be projected into the embedding space of the first language model LLM, that is, the feature sequence needs to be reduced in dimension. Then, one implementation of S22 is as follows:

[0069] The feature sequence is projected into the embedding space of the first language model LLM through the projection layer to obtain the target feature sequence , the length of the target feature sequence is the same as the length of the preliminary feature sequence. The specific process can be expressed as:

[0070] (1)

[0071] Among them, Dim ( ) indicates obtaining the target feature sequence The dimension of [-1] is the last dimension obtained, and D represents the processing dimension corresponding to the first language model, that is, the dimension of the embedding space. Formula (1) represents The last dimension of is equal to the dimension of the embedding space.

[0072] Step S30, dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and a first set length to obtain at least one subsequence, wherein the first set length is a length reserved for user input questions, and the length of each of the subsequences is a maximum length processed by the first large language model at one time;

[0073] After the processing from S21 to S22, the above S30 can be expressed as:

[0074] The target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and a first set length to obtain at least one subsequence.

[0075] Each subsequence includes features corresponding to at least one video frame. The context length of the first largest language model refers to the maximum length of features that can be processed by the first largest language model at one time.

[0076] Optionally, a possible implementation manner of dividing the target feature sequence according to the length of the target feature sequence, the context length of the first large language model and the first set length to obtain at least one subsequence is:

[0077] S31, determining a maximum number of video frames that the first large language model processes at one time according to the length of the target feature sequence, the context length of the first large language model, and a first set length;

[0078] The specific implementation process of the above S31 can be expressed based on formula (2):

[0079] (2)

[0080] in, Indicates the maximum number of frames. represents the context length of the first language model, It represents the first set length. When the first language model LLM understands the processed video in batches, each batch needs to input user questions at the same time so that the first language model LLM can output answers related to the questions. Represents the target feature sequence The length of , that is, the number of tokens occupied by each frame after passing through the projection layer. Express Round down.

[0081] S32: Divide the target feature sequence according to the maximum number of frames to obtain at least one subsequence.

[0082] The number of at least one subsequence, that is, the total number of subsequences obtained by division, is represented by BN, then:

[0083] (3)

[0084] in, Indicates the number of video input frames to be processed. When the number of video input frames is less than or equal to When the number of video input frames is too large, that is, the number of video input frames is greater than It needs to be processed in batches. and Express the same meaning.

[0085] Optionally, in order to ensure the connection between each batch, batches should overlap by one frame. That is, if at least one subsequence is at least two subsequences, there is at least one overlapping video frame between each adjacent subsequence in the at least two subsequences. Therefore, the total number of frames processed by the first largest language model should be .

[0086] Step S40, batch processing is performed on each of the subsequences through the first large language model to obtain an understanding result corresponding to each of the subsequences; as a result, the understanding result corresponding to each subsequence can be represented as output 1, output 2, and so on.

[0087] Step S50, fusing the plurality of understanding results to obtain a target understanding result corresponding to the video to be processed.

[0088] Optionally, a plurality of the understanding results are fused to obtain a target understanding result corresponding to the video to be processed, including:

[0089] The plurality of understanding results are fused through the second largest language model to obtain a target understanding result corresponding to the video to be processed, wherein the length of each understanding result is no greater than the length corresponding to a subsequence processed by the second largest language model.

[0090] After being processed by large language model A (the first large language model), BN understanding results will be obtained. If BN is greater than 1, large language model B (the second largest language model) is needed to make a final analysis and summary of the BN understanding results. Among them, large language model A and large language model B can be the same model or different models. It can be flexibly configured according to needs.

[0091] The specific implementation process of the above analysis and summary (fusion processing) is: input the prompt words of analysis and summary to the large language model B, for example: "Please analyze and summarize the following paragraphs: Output 1 of large language model A. Output 2 of large language model A... Output BN of large language model A". The large language model B will analyze and summarize all the inputs (understanding results) according to the prompt words and output the target understanding results.

[0092] In order to ensure that the large language model B has enough context length to process BN understanding results, it is necessary to limit the output length of the large language model A, that is, the length of each understanding result. Suppose the length of each understanding result is , the context length of B of the large language model is , then there should be:

[0093] (4)

[0094] In general, a pre-set prompt can be used to limit the length of the understanding result output by the large language model A.

[0095] As an example, see Figure 4 , the above-mentioned large language model B finally obtains the target understanding result based on the processing of the word segmenter in the process of fusing the multiple understanding results. The user question is: describe this video, and the target understanding result is: there are two kangaroos in this video. They are in the jungle. One is eating grass, and the other is sitting in a daze.

[0096] The solution of the present invention has the following beneficial effects:

[0097] This solution can understand and analyze long videos without losing information. Compared with the video information compression method, it avoids information loss and enables the large language model to obtain rich video features.

[0098] Compared with the method of increasing the length of the large language model context, the present invention adopts second-level frame extraction, which can effectively reduce information redundancy. Secondly, the present invention adopts a batch processing method, which effectively reduces the dependence of long videos on the length of the LLM context and greatly increases the length of long videos that can be processed.

[0099] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides a long video understanding device 20 based on a large model, such as Figure 5 As shown in , the large model-based long video understanding device 20 may include an acquisition module 210, a frame extraction module 220, a division module 230, a batch processing module 240 and a summary module 250, wherein:

[0100] An acquisition module 210 is used to acquire a video to be processed;

[0101] The frame extraction module 220 is used to extract frames of the video to be processed at the second level to obtain a preliminary video frame sequence;

[0102] a division module 230, configured to divide the preliminary video frame sequence into at least one subsequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and a first set length, wherein the first set length is a length reserved for a user to input a question, and the length of each of the subsequences is a maximum length processed by the first large language model at one time;

[0103] A batch processing module 240 is used to perform batch processing on each of the subsequences using the first large language model to obtain an understanding result corresponding to each of the subsequences;

[0104] The summarizing module 250 is used to fuse the multiple understanding results to obtain a target understanding result corresponding to the video to be processed.

[0105] Optionally, after extracting frames at the second level on the video to be processed to obtain a preliminary video frame sequence, the device further includes:

[0106] A dimension reduction module is used to extract features from the preliminary video frame sequence to obtain a feature sequence; perform dimension reduction processing on the feature sequence to obtain a target feature sequence, wherein the dimension of the target feature sequence is the processing dimension corresponding to the first language model;

[0107] The division module is specifically used to divide the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length to obtain at least one subsequence:

[0108] The target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and a first set length to obtain at least one subsequence.

[0109] Optionally, if the at least one subsequence is at least two subsequences, there is at least one overlapping video frame between each two adjacent subsequences in the at least two subsequences.

[0110] Optionally, when the division module 230 divides the target feature sequence according to the length of the target feature sequence, the context length of the first large language model and the first set length to obtain at least one subsequence, it is specifically used to:

[0111] Determining a maximum number of video frames that the first large language model processes at one time according to the length of the target feature sequence, the context length of the first large language model, and a first set length;

[0112] The target feature sequence is divided according to the maximum number of frames to obtain at least one subsequence.

[0113] Optionally, when the summarizing module 250 performs fusion processing on the plurality of the understanding results to obtain a target understanding result corresponding to the video to be processed, it is specifically used to:

[0114] The plurality of understanding results are fused through the second largest language model to obtain a target understanding result corresponding to the video to be processed, wherein the length of each understanding result is no greater than the length corresponding to a subsequence processed by the second largest language model.

[0115] The long video understanding device based on a big model in the embodiment of the present invention can execute the long video understanding method based on a big model provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the long video understanding device based on a big model in each embodiment of the present invention correspond to the steps in the long video understanding method based on a big model in each embodiment of the present invention. For the detailed functional description of each module of the long video understanding device based on a big model, please refer to the description of the corresponding long video understanding method based on a big model shown in the previous text, which will not be repeated here.

[0116] Among them, the above-mentioned long video understanding device based on a large model can be a computer program (including program code) running in a computer device, for example, the long video understanding device based on a large model is an application software; the device can be used to execute the corresponding steps in the method provided in an embodiment of the present invention.

[0117] In some embodiments, the long video understanding device based on a large model provided in an embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the long video understanding device based on a large model provided in an embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the long video understanding method based on a large model provided in an embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0118] In other embodiments, the long video understanding device based on a large model provided by the embodiment of the present invention can be implemented in a software manner. Figure 5 A long video understanding device based on a large model stored in a memory is shown, which can be software in the form of a program and a plug-in, and includes a series of modules, including an acquisition module 210, a frame extraction module 220, a division module 230, a batch processing module 240 and a summary module 250, which are used to implement the long video understanding method based on a large model provided in an embodiment of the present invention.

[0119] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0120] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0121] In an alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0122] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0123] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0124] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0125] The memory 4003 is used to store application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0126] The electronic device may also be a terminal device. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0127] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0128] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0129] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0132] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0133] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A long video understanding method based on a large model, characterized in that: The following steps are involved: Get the video to be processed; Extract frames at the second level on the video to be processed to obtain a preliminary video frame sequence; Dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and a first set length to obtain at least one subsequence, wherein the first set length is a length reserved for user input questions, and the length of each of the subsequences is a maximum length processed by the first large language model at one time; Processing each of the subsequences in batches by using the first large language model to obtain an understanding result corresponding to each of the subsequences; Performing fusion processing on the multiple understanding results to obtain a target understanding result corresponding to the video to be processed; The plurality of understanding results are BN understanding results, and the fusion processing of the plurality of understanding results to obtain a target understanding result corresponding to the video to be processed includes: The plurality of the understanding results are fused by the second largest language model to obtain a target understanding result corresponding to the video to be processed, wherein the length of each understanding result is no greater than the length corresponding to a subsequence processed by the second largest language model; If BN is greater than 1, the specific implementation process of using the second largest language model to fuse the BN understanding results is as follows: input the analyzed and summarized prompt words to the second largest language model, so that the second largest language model analyzes and summarizes all inputs according to the prompt words and then outputs the target understanding result; Among them, let the length of each understanding result be LenOutA, and the context length of the second largest language model be , then there should be: Among them, a preset prompt is used to limit the length of the understanding result output by the first language model.

2. The method according to claim 1, characterized in that After extracting frames at the second level on the video to be processed to obtain a preliminary video frame sequence, the method further includes: Extracting features from the preliminary video frame sequence to obtain a feature sequence; Performing dimensionality reduction processing on the feature sequence to obtain a target feature sequence, wherein the dimension of the target feature sequence is the processing dimension corresponding to the first largest language model; The step of dividing the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length to obtain at least one subsequence includes: The target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and a first set length to obtain at least one subsequence.

3. The method according to claim 1, characterized in that If the at least one subsequence is at least two subsequences, there is at least one overlapping video frame between each two adjacent subsequences in the at least two subsequences.

4. The method according to claim 2, characterized in that: The step of dividing the target feature sequence according to the length of the target feature sequence, the context length of the first large language model and the first set length to obtain at least one subsequence includes: Determining a maximum number of video frames that the first large language model processes at one time according to the length of the target feature sequence, the context length of the first large language model, and a first set length; The target feature sequence is divided according to the maximum number of frames to obtain at least one subsequence.

5. The method according to claim 1, characterized in that: The fusing of the plurality of understanding results to obtain a target understanding result corresponding to the video to be processed includes: The plurality of understanding results are fused through the second largest language model to obtain a target understanding result corresponding to the video to be processed, wherein the length of each understanding result is no greater than the length corresponding to a subsequence processed by the second largest language model.

6. A long video understanding device based on a large model, characterized in that: The method for understanding a long video based on a large model as described in claim 1 is adopted, and the device comprises: An acquisition module is used to acquire the video to be processed; A frame extraction module, used for extracting frames of the video to be processed at the second level to obtain a preliminary video frame sequence; a division module, configured to divide the preliminary video frame sequence into at least one subsequence according to the length of the preliminary video frame sequence, the context length of the first large language model, and a first set length, wherein the first set length is a length reserved for a user to input a question, and the length of each of the subsequences is a maximum length processed by the first large language model at one time; A batch processing module, used for performing batch processing on each of the subsequences through the first large language model to obtain an understanding result corresponding to each of the subsequences; The summary module is used to fuse the multiple understanding results to obtain the target understanding result corresponding to the video to be processed.

7. The device according to claim 6, characterized in that After extracting frames at the second level on the video to be processed to obtain a preliminary video frame sequence, the device further includes: A dimension reduction module is used to extract features from the preliminary video frame sequence to obtain a feature sequence; perform dimension reduction processing on the feature sequence to obtain a target feature sequence, wherein the dimension of the target feature sequence is the processing dimension corresponding to the first language model; The division module is specifically used to divide the preliminary video frame sequence according to the length of the preliminary video frame sequence, the context length of the first large language model and the first set length to obtain at least one subsequence: The target feature sequence is divided according to the length of the target feature sequence, the context length of the first large language model and a first set length to obtain at least one subsequence.

8. The device according to claim 6, characterized in that If the at least one subsequence is at least two subsequences, there is at least one overlapping video frame between each two adjacent subsequences in the at least two subsequences.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.