Video understanding method and related device

Abstract

The invention discloses a video understanding method and a related device. The method comprises the steps that a to-be-understood video is subjected to frame extraction and cut into a plurality of first video blocks, the total time period of the first video blocks is the same as the video time period of the to-be-understood video, and even if the video time period of the to-be-understood video is long, the to-be-understood video can be subjected to frame extraction so as to compress the number of to-be-understood video frames in the to-be-understood video frames; cutting the to-be-understood video to reduce the sequence length of each first video block of the to-be-understood video as an input sequence of the model; inputting each first video block into a video description generator for text description, and outputting a first description text of each first video block so as to quickly, accurately, interpretively and independently describe each first video block; inputting a plurality of first description texts corresponding to the plurality of first video blocks into a first language model for overall description, and outputting an overall description text of the to-be-understood video; the video to be understood can be quickly, accurately, interpretively and completely described.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video understanding method and related devices. Background Art

[0002] With the rapid development of video processing technology, video understanding is the basis of video content review, video content operation, video content recommendation and other links, and is of great significance to tasks such as video content classification, video tag recognition, and video title generation.

[0003] In related technologies, the video understanding method refers to: extracting the visual features of each video frame in the video to be understood, splicing multiple visual features corresponding to multiple video frames, inputting them into the video understanding model for video understanding, and outputting the video understanding results of the video to be understood.

[0004] However, when the video time period of the video to be understood is long, the above method needs to extract a large number of visual features corresponding to a large number of video frames, resulting in a longer sequence length of the input sequence of the video understanding model, which leads to slower video understanding speed and poorer video understanding effect. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides a video understanding method and related devices, which can extract frames of the video to be understood to compress the number of video frames to be understood in the video to be understood, and cut the video to be understood into blocks to reduce the sequence length of each first video block of the video to be understood as the input sequence of the model; a video description generator is used to represent each first video block separately in language to obtain a first description text, and the first language model is used to aggregate the first description text into the overall description text of the video to be understood. On the basis of quickly, accurately and explanatoryly describing each first video block separately, the video to be understood is quickly, accurately and explanatoryly described in its entirety; thereby improving the speed and effect of video understanding.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In one aspect, an embodiment of the present application provides a video understanding method, the method comprising:

[0008] Extract frames and cut the video to be understood into blocks to obtain a plurality of first video blocks; the total time period of the plurality of first video blocks is the same as the video time period of the video to be understood, and each first video block includes a plurality of video frames;

[0009] Performing a text description on each of the first video blocks by a video description generator to obtain a first description text for each of the first video blocks;

[0010] The first language model is used to perform an overall description of the plurality of first description texts corresponding to the plurality of first video blocks to obtain an overall description text of the video to be understood.

[0011] On the other hand, an embodiment of the present application provides a video understanding device, the device comprising: a frame extraction and slicing unit, a text description unit, and an overall description unit;

[0012] The frame extraction and segmentation unit is configured to extract frames and segment the video to be understood to obtain a plurality of first video segments; the total time period of the plurality of first video segments is the same as the video time period of the video to be understood, and each first video segment includes a plurality of video frames;

[0013] The text description unit is configured to perform a text description on each first video block through a video description generator to obtain a first description text for each first video block;

[0014] The overall description unit is configured to perform an overall description of the plurality of first description texts corresponding to the plurality of first video blocks using a first language model to obtain an overall description text of the video to be understood.

[0015] In another aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory.

[0016] The memory is used to store a computer program and transmit the computer program to the processor;

[0017] The processor is configured to execute the method described in any one of the preceding aspects according to instructions in the computer program.

[0018] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is run on a computer device, the computer device executes the method described in any one of the above aspects.

[0019] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which, when executed on a computer device, enables the computer device to execute the method described in any one of the aforementioned aspects.

[0020] As can be seen from the above technical solution, first, the video to be understood is divided into multiple first video blocks by extracting frames. The total time period of these multiple first video blocks is the same as the video time period of the video to be understood, and each first video block includes multiple video frames. This frame extraction and segmentation method not only reduces the high redundancy between adjacent video frames in the video to be understood by extracting frames, thereby compressing the number of video frames to be understood in the video to be understood, but also shortens the video to be understood with a relatively long time period into multiple first video blocks with relatively short time periods, thereby reducing the sequence length of each first video block of the video to be understood as the input sequence of the model. Then, each first video block is input into the video description generator for text description, and the first description text of each first video block is output. This text description method, based on each first video block as the input sequence of the video description generator with a relatively short sequence length, uses the first description text to represent the first video block in language separately, and can quickly, accurately, and explanatoryly describe each first video block separately. Finally, the multiple first description texts corresponding to the multiple first video blocks are input into the first language model for overall description, and the overall description text of the video to be understood is output; this overall description method aggregates the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood, which can quickly, accurately and explanatoryally fully describe the video to be understood.

[0021] Based on this, even when the video time period of the video to be understood is long, the method can extract frames of the video to be understood to compress the number of video frames to be understood in the video to be understood, and cut the video to be understood into blocks to reduce the sequence length of each first video block of the video to be understood as the input sequence of the model; a video description generator is used to represent each first video block separately in language to obtain a first description text, and the first language model is used to aggregate the first description text into the overall description text of the video to be understood. On the basis of quickly, accurately and explanatoryly describing each first video block separately, the method quickly, accurately and explanatoryly describes the video to be understood in its entirety, thereby improving the speed and effect of video comprehension. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technical members in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A system diagram of a video understanding method provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a video understanding method provided in an embodiment of the present application;

[0025] Figure 3 A framework diagram of a video understanding method provided in an embodiment of the present application;

[0026] Figure 4 A framework diagram of another video understanding method provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of text similarity between each group of preceding description text and following description text provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of redundantly merging a plurality of first description texts corresponding to a plurality of first video blocks into a plurality of second description texts corresponding to a plurality of second video blocks provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of a video processing method provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a video processing link provided in an embodiment of the present application;

[0031] Figure 9 A structural diagram of a video understanding device provided in an embodiment of the present application;

[0032] Figure 10 A structural diagram of a server provided in an embodiment of the present application;

[0033] Figure 11 A structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described below with reference to the accompanying drawings.

[0035] Currently, video understanding lays the foundation for tasks such as video content classification, video tag recognition, and video title generation. Video understanding methods typically extract visual features from each frame in the video to be understood, concatenate multiple visual features corresponding to multiple frames, and input them into a video understanding model for video understanding, outputting the video understanding results for the video to be understood. However, research has found that when the video time period of the video to be understood is long, it is necessary to extract visual features corresponding to a large number of video frames. This results in a longer input sequence for the video understanding model, resulting in slower video understanding speed and poorer video understanding results.

[0036] An embodiment of the present application provides a video understanding method. Even when the video time period of the video to be understood is long, frames of the video to be understood can be extracted to compress the number of video frames to be understood in the video to be understood, and the video to be understood can be segmented to reduce the sequence length of each first video block of the video to be understood as an input sequence of the model; a video description generator is used to represent each first video block separately in language to obtain a first description text, and the first language model is used to aggregate the first video block into an overall description text of the video to be understood. On the basis of quickly, accurately, and explanatoryly describing each first video block separately, the video to be understood is quickly, accurately, and explanatoryly described in its entirety, thereby improving the speed and effect of video understanding.

[0037] Next, we will introduce the system architecture of the video understanding method. Figure 1 , Figure 1 A system diagram of a video understanding method provided in an embodiment of the present application, the system includes a computer device 100, and the computer device 100 is used to execute the video understanding method.

[0038] The computer device 100 extracts frames and cuts the video to be understood into blocks to obtain a plurality of first video blocks; the total time period of the plurality of first video blocks is the same as the video time period of the video to be understood, and each first video block includes a plurality of video frames.

[0039] As an example, the video to be understood is video X; the computer device 100 extracts frames from video X and cuts it into multiple first video blocks including video block 1, video block 2, ..., video block n. The total time period of video block 1, video block 2, ..., video block n is the same as the video time period of video X, and video block 1, video block 2, ..., video block n each includes multiple video frames.

[0040] The computer device 100 uses the video description generator to perform a text description on each first video block to obtain a first description text for each first video block.

[0041] As an example, based on the above example, the computer device 100 inputs video block 1, video block 2, ..., video block n into the video description generator for text description, and outputs the first description text of video block 1 as c1, the first description text of video block 2 as c2, ..., the first description text of video block n as c n .

[0042] The computer device 100 uses the first language model to perform an overall description of the multiple first description texts corresponding to the multiple first video blocks to obtain an overall description text of the video to be understood.

[0043] As an example, based on the above example, the first language model is a large language model Y; the computer device 100 converts c1 of video block 1, c2 of video block 2, ..., c of video block n inton Input the large language model Y for overall description, and output the overall description text of video X as c X .

[0044] That is to say, the frame extraction and segmentation method not only reduces the high redundancy between adjacent video frames in the video to be understood by extracting frames, thereby compressing the number of video frames to be understood in the video to be understood, but also shortens the video to be understood with a relatively long time period into multiple first video blocks with relatively short time periods, thereby reducing the sequence length of each first video block of the video to be understood as an input sequence of the model; the text description method uses the first description text to represent the first video block separately in language based on each first video block as an input sequence with a shorter sequence length for the video description generator, and can quickly, accurately and explanatoryly describe each first video block separately; the overall description method aggregates multiple first description texts corresponding to multiple first video blocks into an overall description text of the video to be understood, and can quickly, accurately and explanatoryally describe the video to be understood in its entirety.

[0045] It should be noted that in the embodiments of the present application, the computer device may be a server or a terminal, and the method provided in the embodiments of the present application may be executed by the terminal or the server alone, or by the terminal and the server in combination. Figure 1 The corresponding embodiments are similar, primarily replacing the computer device with a terminal or server. Furthermore, when the methods provided in the embodiments of the present application are executed in conjunction with a terminal and a server, the steps that need to be displayed on the front-end interface can be executed by the terminal, while steps that require background computing and do not need to be displayed on the front-end interface can be executed by the server.

[0046] Among them, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, vehicle-mounted terminal, extended reality device or aircraft, etc., but is not limited to these. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. For example, the terminal and the server can be connected via a network, which can be a wired or wireless network.

[0047] Next, the video understanding method provided in the embodiment of the present application will be introduced in detail with reference to the accompanying drawings, taking a computer device executing the method provided in the embodiment of the present application as an example.

[0048] See also Figure 2 , Figure 2This is a flowchart of a video understanding method provided in an embodiment of the present application, and the method includes the following S201-S203.

[0049] S201: Extract frames and cut blocks from a video to be understood to obtain a plurality of first video blocks; the total time period of the plurality of first video blocks is the same as the video time period of the video to be understood, and each first video block includes a plurality of video frames.

[0050] Related technologies typically extract visual features from each frame of the video to be understood, then combine the visual features corresponding to multiple frames and input them into a video understanding model for video understanding. This process then outputs the video understanding results for the video to be understood. However, research has found that when the video time period of the video to be understood is long, it is necessary to extract visual features corresponding to a larger number of frames. This results in a longer input sequence for the video understanding model, which in turn slows down video understanding and produces poorer video understanding results.

[0051] Therefore, in an embodiment of the present application, considering the high redundancy between adjacent video frames in the video to be understood, the high redundancy between adjacent video frames in the video to be understood can be reduced by frame extraction. Furthermore, considering the long time period of the video to be understood, the video to be understood with a relatively long time period can be shortened by slicing the video to be understood into multiple video blocks with relatively short time periods. Based on this, the video to be understood is sliced ​​by frame extraction into multiple first video blocks, the total time period of the multiple first video blocks being the same as the video time period of the video to be understood, and each first video block includes multiple video frames.

[0052] The video to be understood refers to the video that needs to be understood; frame extraction and segmentation refers to extracting some video frames from the video and segmenting the video into video blocks; and the first video block refers to a video block obtained by extracting some video frames from the video to be understood and segmenting the video.

[0053] In a specific implementation, extracting frames of the video to be understood and cutting them into multiple first video blocks can be: retaining a preset number of video frames of the video to be understood according to the first time period and cutting them into multiple first video blocks according to the second time period; or retaining a preset number of video frames of the video to be understood according to the first time period and cutting them into multiple first video blocks according to the shot scenes.

[0054] The above S201 reduces the high redundancy between adjacent video frames in the video to be understood by extracting frames, thereby compressing the number of video frames to be understood in the video to be understood, and shortens the video to be understood with a relatively long time period into multiple first video blocks with relatively short time periods by slicing, thereby reducing the sequence length of each first video block of the video to be understood as an input sequence of the model; laying the foundation for the subsequent rapid, accurate and explanatory individual description of each first video block based on the model.

[0055] As an example of S201, the video to be understood is video X; the computer device extracts frames from video X and cuts it into multiple first video blocks including video block 1, video block 2, ..., video block n. The total time period of video block 1, video block 2, ..., video block n is the same as the video time period of video X. Video block 1, video block 2, ..., video block n each includes multiple video frames. The video block time period of video block 1 is t1, the video block time period of video block 2 is t2, ..., and the video block time period of video block n is t n .

[0056] S202: Performing a text description on each first video block by a video description generator to obtain a first description text for each first video block.

[0057] In an embodiment of the present application, after the video to be understood is extracted and cut into multiple first video blocks in the above S201, since the number of video frames to be understood is reduced after each first video block is extracted and the time period is relatively short after cutting, that is, the sequence length of each first video block as the input sequence of the model is short, considering the natural advantages of language representation, the first video block can be represented separately by language; based on this, each first video block is input into the video description generator for text description, and the first description text of each first video block is output, so that each first video block is represented separately by language.

[0058] The video description generator refers to a model that represents a video in language; the text description refers to describing the video by representing the video in language; and the first description text refers to the representation language of the first video block.

[0059] The above S202 uses each first video block as an input sequence with a shorter sequence length for the video description generator, and expresses each first video block individually in language to obtain a first description text for each first video block. This can quickly, accurately, and explanatoryly describe each first video block individually, and provide a basis for subsequently quickly, accurately, and explanatoryally describing the video to be understood completely based on the model.

[0060] As an example of S202, based on the example of S201 above, the computer device inputs video block 1, video block 2, ..., video block n into the video description generator for text description, and outputs the first description text of video block 1 as c1, the first description text of video block 2 as c2, ..., the first description text of video block n as c n That is, the text description results of video block 1, video block 2, ..., video block n are C = {(c1, t1), (c2, t2) ..., (c n , t n )}.

[0061] S203: Using a first language model, a plurality of first description texts corresponding to the plurality of first video blocks are comprehensively described to obtain an overall description text of the video to be understood.

[0062] In the embodiment of the present application, after each first video block is input into the video description generator for text description and the first description text of each first video block is output in the above S202, since the first description text of each first video block represents each first video block separately in language, the video to be understood can be fully represented in language based on the multiple first description texts corresponding to the multiple first video blocks; based on this, the multiple first description texts corresponding to the multiple first video blocks are input into the first language model for overall description, and the overall description text of the video to be understood is output.

[0063] The first language model refers to a model that aggregates multiple representation languages ​​into one representation language; the overall description refers to a model that aggregates multiple representation languages ​​into one representation language; and the overall description text refers to the representation language of the video to be understood.

[0064] The above S203 aggregates the multiple first description texts corresponding to the multiple first video blocks into an overall description text of the video to be understood, and fully represents the video to be understood in language, which can quickly, accurately and explanatorily describe the video to be understood; thereby improving the video understanding speed and video understanding effect.

[0065] As an example of S203, based on the example of S202 above, the first language model is a large language model Y; the computer device converts c1 of video block 1, c2 of video block 2, ..., c of video block n, and n Input the large language model Y for overall description, and output the overall description text of video X as c X .

[0066] See also Figure 3 , Figure 3 A framework diagram of a video understanding method provided in an embodiment of the present application. The method includes: first, extracting frames from a video to be understood and cutting it into a plurality of first video blocks, for example, extracting frames from video X and cutting it into video block 1, video block 2, ..., video block n; then, inputting each first video block into a video description generator for text description, and outputting a first description text of each first video block, for example, inputting video block 1, video block 2, ..., video block n into a video description generator for text description, and outputting a first description text c1 of video block 1, a first description text c2 of video block 2, ..., a first description text c of video block n. n Finally, multiple first description texts corresponding to multiple first video blocks are input into the first language model for overall description, and the overall description text of the video to be understood is output. For example, c1 of video block 1, c2 of video block 2, ..., c of video block n are input into the first language model for overall description, and the overall description text of the video to be understood is output. nInput the large language model Y for overall description and output the overall description text c of video X X .

[0067] It can be seen from the above technical solutions that the frame extraction and segmentation method not only reduces the high redundancy between adjacent video frames in the video to be understood by extracting frames, thereby compressing the number of video frames to be understood in the video to be understood, but also shortens the video to be understood with a relatively long time period into multiple first video blocks with relatively short time periods, thereby reducing the sequence length of each first video block of the video to be understood as an input sequence of the model; the text description method uses the first description text to represent the first video block separately in language based on each first video block as an input sequence with a shorter sequence length for the video description generator, and can quickly, accurately and explanatoryly describe each first video block separately; the overall description method aggregates multiple first description texts corresponding to multiple first video blocks into an overall description text of the video to be understood, which can quickly, accurately and explanatoryally describe the video to be understood completely. Even when the video time period of the video to be understood is long, the method can extract frames of the video to be understood to compress the number of video frames to be understood in the video to be understood, and cut the video to be understood into blocks to reduce the sequence length of each first video block of the video to be understood as the input sequence of the model; a video description generator is used to represent each first video block separately in language to obtain a first description text, and the first language model is used to aggregate the first description text into the overall description text of the video to be understood. On the basis of quickly, accurately and explanatoryly describing each first video block separately, the method quickly, accurately and explanatoryly fully describes the video to be understood, thereby improving the video comprehension speed and video comprehension effect.

[0068] In an embodiment of the present application, when the above-mentioned S203 is specifically implemented by inputting the multiple first description texts corresponding to the multiple first video blocks into the first language model for overall description and outputting the overall description text of the video to be understood, it is considered that the first language model needs to aggregate the multiple first description texts corresponding to the multiple first video blocks into a complete description text under the overall description prompt as the overall description text of the video to be understood; therefore, it is necessary to input the multiple first description texts corresponding to the multiple first video blocks into the first language model, and perform an overall description of the multiple first description texts corresponding to the multiple first video blocks according to the overall description prompt, and output the overall description text of the video to be understood. Based on this, the present application provides a possible implementation method, and the above-mentioned S203 may include S203a (not shown in the figure): the first language model performs an overall description of the multiple first description texts corresponding to the multiple first video blocks according to the overall description prompt to obtain the overall description text of the video to be understood; the overall description prompt is used to guide the first language model to aggregate the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood.

[0069] The overall description prompt refers to the overall description text provided to the first language model, which is used to guide the first language model to aggregate multiple first description texts corresponding to multiple first video blocks into the overall description text of the video to be understood.

[0070] The above S203a guides the first language model to aggregate the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood through the overall description prompt, so as to fully represent the video to be understood in language, thereby more accurately and completely describing the video to be understood.

[0071] As an example of S203a, based on the above-mentioned example of S203, the overall description prompt is promptY; the computer device inputs the description text 1 of video block 1, the description text 2 of video block 2, ..., the description text n of video block n into the large language model Y, and performs an overall description of the description text 1 of video block 1, the description text 2 of video block 2, ..., the description text n of video block n according to promptY, and outputs the description text x of video X.

[0072] Here, promptY is as follows:

[0073] "Prompt: Generate overall description text

[0074] You are an experienced video understanding assistant. Now you are given multiple first description texts corresponding to multiple first video blocks. Each line includes the video block time period and the first description text of the first video block, as follows:

[0075] Video block time period: [t1], first description text: [c1]

[0076] Video block time period: [t2], first description text: [c2]

[0077] …

[0078] Video block time period: [t n 】, first description text:

c n

[0079] Based on the above description, please directly output the overall description text of the video to be understood. The description should be a complete sentence and grammatically correct. The description should be rich in information and as detailed as possible, including the main elements, characters, scenes, style, shots, etc.

[0080] In an embodiment of the present application, when the above-mentioned S202 is specifically implemented by inputting each first video block into the video description generator for text description and outputting the first description text of each first video block, the video description generator includes an encoder and a decoder, the encoder is used to extract the visual information of the video, and the decoder is used to represent the visual information in language; therefore, for each first video block, the first video block is first input into the encoder of the video description generator for feature encoding, outputting the visual features of the first video block, and then the visual features of the first video block are input into the decoder of the video description generator for feature decoding, and outputting the first description text of the first video block. Based on this, the present application provides a possible implementation method, and the above-mentioned S202 may, for example, include the following S202a-S202b (not shown in the figure).

[0081] S202a: For each first video block, perform feature encoding on the first video block by an encoder in the video description generator to obtain visual features of the first video block.

[0082] S202b: Decoding the visual features of the first video block by a decoder in the video description generator to obtain a first description text of the first video block.

[0083] Among them, the encoder in the video description generator refers to the module that extracts the visual information of the video; feature encoding refers to the module that extracts the visual information of the video; the visual features of the first video block represent the visual information of the first video block; the decoder in the video description generator refers to the module that represents visual information in language; and feature decoding refers to the module that represents visual information in language.

[0084] The above S202a-S202b uses each first video block as an input sequence with a shorter sequence length for the video description generator, extracts the visual information of each first video block through the encoder to obtain the visual features of each first video block, and can quickly and accurately extract the visual information of each first video block separately; and uses the decoder to represent each first video block separately in language to obtain the first description text of each first video block, so that each first video block can be described separately and quickly, accurately and explanatoryly.

[0085] As an example of S202a-S202b, based on the example of S202 above, the computer device inputs video block 1, video block 2, ..., video block n into the encoder of the video description generator for feature decoding, and outputs the visual feature of video block 1 as f1, the visual feature of video block 2 as f2, ..., the visual feature of video block n as f n ; The computer equipment will f1, f2, ..., f n Input the decoder in the video description generator to decode the features and output c1, c2, ..., c n .

[0086] In addition, in an embodiment of the present application, after each first video block is input into the video description generator for text description and the first description text of each first video block is output in the above S202, considering that similar redundancy may still exist between adjacent upper and lower video blocks in the multiple first video blocks, the similar redundancy between adjacent upper and lower video blocks in the multiple first video blocks can be further reduced by redundant merging, so that the multiple first description texts corresponding to the multiple first video blocks after redundant merging are subsequently input into the first language model for overall description, and the overall description text of the video to be understood is output.

[0087] Therefore, in an embodiment of the present application, after S202 and before S203, the multiple first description texts corresponding to the multiple first video blocks can be redundantly merged into multiple second description texts corresponding to the multiple second video blocks, so that the number of blocks of the multiple second video blocks is less than the number of blocks of the multiple first video blocks, that is, the number of texts of the multiple second description texts is less than the number of texts of the multiple first description texts; in this case, when S203 is specifically implemented, the multiple second description texts corresponding to the multiple second video blocks are input into the first language model for overall description, and the overall description text of the video to be understood is output.

[0088] Based on this, the present application provides a possible implementation method, and the method may also include S1 (not shown in the figure): redundantly merging multiple first description texts corresponding to multiple first video blocks to obtain multiple second description texts corresponding to multiple second video blocks; the number of blocks of the multiple second video blocks is less than the number of blocks of the multiple first video blocks; correspondingly, the above-mentioned S203 may include S203b (not shown in the figure): using the first language model to perform an overall description of the multiple second description texts corresponding to the multiple second video blocks to obtain an overall description text of the video to be understood.

[0089] Among them, redundant merging refers to merging a first number of description texts corresponding to a first number of video blocks into a second number of description texts corresponding to a second number of video blocks that are not redundant, and the second number is smaller than the first number; multiple second video blocks refer to multiple first video blocks after redundant merging; and the second description text refers to the representation language of the second video blocks.

[0090] The above S1 and S203b redundantly merge the multiple first description texts corresponding to the multiple first video blocks into the multiple second description texts corresponding to the multiple second video blocks, which can reduce the similarity redundancy between the adjacent upper video blocks and the following video blocks in the multiple first video blocks, so that there is no similarity redundancy between the adjacent upper video blocks and the following video blocks in the multiple second video blocks, and reduce the sequence length of the input sequence of the first language model; aggregate the multiple second description texts corresponding to the multiple second video blocks into the overall description text of the video to be understood, and fully represent the video to be understood in language, which can fully describe the video to be understood more quickly, more accurately, and without redundancy; thereby further improving the video comprehension speed and video comprehension effect.

[0091] As an example of S1, based on the example of S202 above, the computer device converts c1 of video block 1, c2 of video block 2, ..., c n Redundancy is merged into c1' of video block 1', c2' of video block 2', ..., c of video block m' m ', wherein the number m' of blocks of video block 1', video block 2', ..., video block m' is less than the number n of blocks of video block 1, video block 2, ..., video block n; as an example of S203b, based on the example of S1, the computer device converts c1' of video block 1', c2' of video block 2', ..., c of video block m' into m ' Input the large language model Y for overall description, and output the overall description text of video X as c X '.

[0092] See also Figure 4 , Figure 4 A framework diagram of another video understanding method provided in an embodiment of the present application. The method includes: first, extracting frames from the video to be understood and cutting it into multiple first video blocks, for example, extracting frames from video X and cutting it into video block 1, video block 2, ..., video block n; second, inputting each first video block into a video description generator for text description, and outputting a first description text of each first video block, for example, inputting video block 1, video block 2, ..., video block n into a video description generator for text description, and outputting a first description text c1 of video block 1, a first description text c2 of video block 2, ..., a first description text c of video block n. n Then, the first description texts corresponding to the first video blocks are redundantly merged into the second description texts corresponding to the second video blocks, for example, c1 of video block 1, c2 of video block 2, ..., c of video block n. n Redundancy is merged into c1' of video block 1', c2' of video block 2', ..., c of video block m' mFinally, multiple second description texts corresponding to multiple second video blocks are input into the first language model for overall description, and the overall description text of the video to be understood is output. For example, c1' of video block 1', c2' of video block 2', ..., c of video block m' m ' Input the large language model Y for overall description, and output the overall description text c of video X X '.

[0093] In the embodiment of the present application, when redundantly merging the plurality of first description texts corresponding to the plurality of first video blocks into the plurality of second description texts corresponding to the plurality of second video blocks in the above S1 is specifically implemented, it is considered that the redundant merging is actually performed by determining whether each group of upper description text and lower description text in the plurality of first description texts corresponding to the plurality of first video blocks is similar, so as to determine at least two first video blocks to be redundantly merged and independent first video blocks that do not need to be redundantly merged in the plurality of first video blocks, and further implement redundant merging between the at least two first description texts corresponding to the at least two first video blocks that may need to be redundantly merged; thereby determining the plurality of second video blocks and the plurality of second description texts corresponding to the plurality of second video blocks.

[0094] Therefore, in an embodiment of the present application, first, based on the text similarity of each group of upper description text and lower description text in the multiple first description texts corresponding to the multiple first video blocks, at least two first video blocks to be redundantly merged in the multiple first video blocks are determined as candidate merged video blocks, and independent first video blocks that do not need to be redundantly merged in the multiple first video blocks are determined as independent video blocks; then, at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks are input into the second language model for redundant merging, and the merged description text of the merged video block is output; finally, based on the first description texts corresponding to the independent video blocks and the merged description text of the merged video block, multiple second description texts corresponding to the multiple second video blocks are determined. Based on this, the present application provides a possible implementation method, and the above S1 may, for example, include S11-S13 (not shown in the figure).

[0095] S11: Determine candidate merged video blocks and independent video blocks from the plurality of first video blocks based on text similarity between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks; the candidate merged video blocks include at least two first video blocks.

[0096] S12: Redundantly merge at least two first description texts corresponding to at least two first video blocks in the candidate merged video block using a second language model to obtain a merged description text of the merged video block.

[0097] S13: Determine a plurality of second description texts corresponding to a plurality of second video blocks according to the first description text corresponding to the independent video block and the merged description text of the merged video block.

[0098] The text similarity between each group of preceding description text and following description text refers to the textual similarity between each group of preceding description text and following description text; the candidate merged video block refers to at least two first video blocks to be redundantly merged among the multiple first video blocks; the independent video block refers to an independent first video block that does not need to be redundantly merged among the multiple first video blocks; the second language model refers to redundantly merging a third number of language representations into a fourth number of language representations, where the fourth number is less than the third number; the merged video block refers to at least two first video blocks to be redundantly merged among the multiple first video blocks after redundant merging; and the merged description text refers to the representation language of the merged video block.

[0099] In the above-mentioned S11-S13, based on the text similarity between each group of preceding description text and following description text in the multiple first description texts corresponding to the multiple first video blocks, whether each group of preceding description text and following description text is textually similar can be determined accurately. This can accurately determine textually similar candidate merged video blocks and textually independent independent video blocks in the multiple first video blocks. Furthermore, at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks are redundantly merged into a merged description text of the merged video block through the second language model. This allows accurate determination of multiple second description texts corresponding to the multiple second video blocks based on the first description texts corresponding to the independent video blocks and the merged description text of the merged video block.

[0100] As an example of S11-S13, based on the example of S1 above, the second language model is a large language model Z; the computer device uses c1 of video block 1, c2 of video block 2, ..., c of video block n. n Description text c i and each following description text c i+1 The computer device inputs at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks into the large language model Z for redundant merging, and outputs the merged description text of the merged video blocks; the computer device determines c1' of video block 1', c2' of video block 2', ..., c of video block m' through the first description texts corresponding to the independent video blocks and the merged description text of the merged video blocks. m '.

[0101] In an embodiment of the present application, the method for calculating the text similarity between each group of the above description text and the following description text in the above S11 can be: first, each first description text is input into the feature extractor for feature extraction, and the description text features of each first description text are output; then, for each group of the above description text and the following description text, the text similarity between each group of the above description text and the following description text is calculated through the description text features of each group of the above description text and the description text features of the following description text. Based on this, the present application provides a possible implementation method, and the steps for obtaining the text similarity between each group of the above description text and the following description text in the above S11 can, for example, include the following S2-S3 (not shown in the figure).

[0102] S2: Perform feature extraction on each first description text through a feature extractor to obtain description text features of each first description text.

[0103] S3: performing similarity calculation based on the description text features of each group of upper description texts and the description text features of the lower description texts in the multiple first description texts corresponding to the multiple first video blocks to obtain the text similarity of each group of upper description texts and the lower description texts.

[0104] Among them, the feature extractor refers to a model that represents text as features; feature extraction refers to representing text as features; description text features refer to the representation features of the first description text; and similarity calculation refers to calculating the similarity between two features.

[0105] The above S2-S3 extracts the descriptive text features of each first descriptive text through the feature extractor, which can accurately represent the semantic information of each first descriptive text, so that through the descriptive text features of each group of previous description texts and the descriptive text features of the following description texts, the text similarity of each group of previous description texts and the following description texts can be accurately calculated at the semantic level.

[0106] As an example of S2-S3, based on the above examples of S11-S13, the feature extractor is CLIP_Text; the computer device extracts c1 of video block 1, c2 of video block 2, ..., c of video block n. n Input CLIP_Text for feature extraction, and output the description text feature of c1 as f1, the description text feature of c2 as f2, ..., c n The description text feature is f n ; Then target c i and c i+1 , through c i The description text feature f i and c i+1 The description text feature f i+1 , calculate ci and c i+1 s(i, i+1); where f i =CLIP_Text(c i ), f i+1 =CLIP_Text(c i+1 ), s(i, i+1)=sim(f i , f i+1 ).

[0107] See also Figure 5 , Figure 5 A schematic diagram of the text similarity between each group of the above description text and the following description text provided in the embodiment of the present application. k-1 and the following description text c k The text similarity is s(k-1, k); the above description text c k and the following description text c k+1 The text similarity is s(k, k+1); the above description text c k+1 and the following description text c k+2 The text similarity is s(k+1, k+2) and so on.

[0108] In an embodiment of the present application, in the above S11, determining candidate merged video blocks and independent video blocks in the plurality of first video blocks by using the text similarity between each group of preceding and following description texts in the plurality of first description texts corresponding to the plurality of first video blocks is specifically implemented. For each group of preceding and following description texts in the plurality of first description texts corresponding to the plurality of first video blocks, if the text similarity between two adjacent first video blocks is greater than or equal to a preset similarity, they are the two first video blocks to be redundantly merged. In this case, the two adjacent first video blocks with the text similarity greater than or equal to the preset similarity are considered similar video blocks. If the text similarity between first video blocks is less than the preset similarity, they are independent first video blocks that do not need to be redundantly merged. In this case, the first video blocks with the text similarity less than the preset similarity are considered independent video blocks. If there are repeated first video blocks in different similar video blocks, this indicates that three first video blocks in the different similar video blocks are the three first video blocks to be redundantly merged, and the different similar video blocks need to be aggregated into a candidate merged video block. If there are no repeated first video blocks in the different similar video blocks, the candidate merged video block can be determined directly based on the different similar video blocks. Based on this, the present application provides a possible implementation method, and the above S11 may include, for example, the following S11a-S11c (not shown in the figure).

[0109] S11a: Determine, based on the text similarity between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks, similar video blocks corresponding to which the text similarity is greater than or equal to a preset similarity, and independent video blocks corresponding to which the text similarity is less than the preset similarity; the similar video blocks include two adjacent first video blocks.

[0110] S11b: If there are repeated first video blocks in different similar video blocks, perform aggregation processing on the different similar video blocks to obtain candidate merged video blocks.

[0111] S11c: If there is no repeated first video block among the different similar video blocks, determine a candidate merged video block according to the similar video blocks.

[0112] The preset similarity refers to the lower limit similarity between the two first video blocks in terms of text; similar video blocks refer to two adjacent first video blocks whose text similarity is greater than or equal to the preset similarity.

[0113] The above S11a-S11c judge whether each group of upper description text and lower description text in the multiple first description texts corresponding to the multiple first video blocks is textually similar based on the textual similarity between each group of upper description text and lower description text. This can accurately determine textually similar similar video blocks and textually independent independent video blocks in the multiple first video blocks. It can further judge whether there are repeated first video blocks in different similar video blocks, and can accurately aggregate the largest number of textually similar similar video blocks as candidate merged video blocks.

[0114] As an example of S11a-S11c, based on the above examples of S11-S13, the similarity is preset to θ; the computer device uses c1 of video block 1, c2 of video block 2, ..., c of video block n. n Description text c i and each following description text c i+1 The text similarity s(i, i+1) of the two adjacent first video blocks with text similarity greater than or equal to θ is determined as a similar video block, and the first video block with text similarity less than θ is determined as an independent video block; when there are repeated first video blocks in different similar video blocks, the different similar video blocks are aggregated as candidate merged video blocks; when there are no repeated first video blocks in different similar video blocks, the candidate merged video blocks are directly determined by different similar video blocks, wherein θ does not need to be set too high, and is only used to roughly determine similar video blocks and independent video blocks. Figure 5On the basis of , when s(k, k+1)≥θ, similar video blocks include video block k and video block k+1, when s(k+1, k+2)≥θ, similar video blocks include video block k+1 and video block k+2, (video block k, video block k+1) and (video block k+1, video block k+2) have repeated video block k+1, and (video block k, video block k+1, video block k+2) are aggregated as a candidate merged video block, that is, (video block k, video block k+1, video block k+2).

[0115] In an embodiment of the present application, when the above-mentioned S12 is specifically implemented by inputting the at least two first description texts corresponding to the at least two first video blocks in the candidate merged video block into the second language model for redundant merging and outputting the merged description text of the merged video block, considering that the second language model needs to merge the at least two first description texts corresponding to the at least two first video blocks in the candidate merged video block into the merged description text of the merged video block under the redundant merging prompt, considering that the second language model needs to merge the at least two first description texts corresponding to the at least two first video blocks in the candidate merged video block into the merged description text of the merged video block under the redundant merging prompt; therefore, it is necessary to input the at least two first description texts corresponding to the at least two first video blocks in the candidate merged video block into the merged description text of the merged video block. Based on this, the present application provides a possible implementation method, in which the above-mentioned S11 may include S12a (not shown in the figure): redundantly merging the at least two first description texts corresponding to the at least two first video blocks in the candidate merged video block according to the redundant merging prompt by the second language model to obtain the merged description text of the merged video block.

[0116] The redundant merging prompt refers to redundant merging text provided to the second language model, and is used to guide the second language model to redundantly merge the third number of language representations into a fourth number of language representations.

[0117] The above-mentioned S12a guides the second language model through the redundant merging prompt to redundantly merge at least two first description texts corresponding to at least two first video blocks in the candidate merged video block into the merged description text of the merged video block, thereby achieving redundant merging of the candidate merged video blocks and providing a basis for subsequent more accurate and non-redundant description of the video to be understood.

[0118] As an example of S12a, based on the examples of S11-S13 above, the redundant merging prompt is promptZ; the computer device inputs at least two first description texts corresponding to at least two first video blocks in the candidate merged video block into the language model Z, redundantly merges the at least two first description texts corresponding to at least two first video blocks in the candidate merged video block according to promptZ, and outputs the merged description text of the merged video block.

[0119] Among them, promptZ is as follows:

[0120] "Prompt: Redundant merging candidate merged video blocks

[0121] You are an experienced video understanding assistant. Now give the three first description texts corresponding to the three first video segments in the candidate merged video block. Each line includes the video block time period and the first description text of the first video block, as follows:

[0122] Video block time period: [t k 】, first description text:

c k

[0123] Video block time period: [t k+1 】, first description text:

c k+1

[0124] Video block time period: [t k+2 】, first description text:

c k+2

[0125] Based on the above description, please analyze and identify the similar redundancies in the three consecutive first video segments, then reorganize and merge them while maintaining the above format, and output a more concise and cohesive description text of the merged video block. Be sure to express it as detailed as possible and do not omit any information.

[0126] As a specific example of S12a, at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks, redundant merge hints, and merged description texts of the merged video blocks are as follows:

[0127] "Prompt: Redundant merging candidate merged video blocks

[0128] You are an experienced video understanding assistant. Now give the six first description texts corresponding to the six first video segments in the candidate merged video block. Each line includes the video block time period and the first description text of the first video block, as follows:

[0129] Video block time: [00:00-00:04], first description text: [User u walks into the garage]

[0130] Video block time period: [00:04-00:08], first description text: [User u opens the car door]

[0131] Video block time period: [00:08-00:12], first description text: [User u enters the vehicle]

[0132] Video block time period: [00:12-00:16], first description text: [User u starts the vehicle]

[0133] Video block time period: [00:16-00:20], first description text: [User u drives away]

[0134] Video block time: [00:20-00:24], first description text: [User u arrives at the valley and starts setting up a tent]

[0135] Based on the above description, please analyze and identify the similar redundancies in the six consecutive first video segments, then reorganize and merge them while maintaining the above format, and output a more concise and cohesive merged description text of the merged video block. Be sure to express it as detailed as possible and do not omit any information.

[0136] "Video block time period: [00:00-00:12], merged description text: [User u walks into the garage, opens the door and enters the vehicle]

[0137] Video block time period: [00:12-00:20], description text merged: [User u starts the car and drives away]

[0138] Video block time period: [00:20-00:24], description text merged: [User u arrives at the valley and starts setting up a tent]"

[0139] In summary, see S1, S11-S13, etc. Figure 6 , Figure 6 A schematic diagram of an embodiment of the present application provides a method for redundantly merging multiple first description texts corresponding to multiple first video blocks into multiple second description texts corresponding to multiple second video blocks. When n=12, through c1 of video block 1, c2 of video block 2, ..., c of video block 12, 12 Description text c i and each following description text c i+1The text similarity s(i, i+1) of the merged video blocks is determined to be (video block 1, video block 2, video block 3) and (video block 7, video block 8, video block 9, video block 10) in video block 1, video block 2, ..., video block 12, and the five independent video blocks in video block 1, video block 2, ..., video block 12 are determined to be video block 4, video block 5, video block 6, video block 11 and video block 12. The c1 of video block 1, c2 of video block 2 and c3 of video block 3 are input into the large language model Z for redundant merging, and the merged description text of the merged video blocks is output as c1' of video block 1'; the c7 of video block 7, c8 of video block 8, c9 of video block 9 and c10 of video block 10 ... 10 Input the large language model Z for redundant merging, and output the merged description text of the merged video blocks including c5' of video block 5' and c6' of video block 6'; through c1' of video block 1', c4 of video block 4, c5 of video block 5, c6 of video block 6, c5' of video block 5', c6' of video block 6', c1' of video block 11 11 and c of video block 12 12 , determine c1' of video block 1', c2' of video block 2', ..., c of video block m' m ' includes c1' of video block 1', c2' of video block 2', c3' of video block 3', c4' of video block 4', c5' of video block 5', c6' of video block 6', c7' of video block 7' and c8' of video block 8', that is, m=8.

[0140] In an embodiment of the present application, when the above-mentioned S201 is specifically implemented to extract frames of the video to be understood and cut into multiple first video blocks, the video to be understood can be first extracted into a video to be processed, so that the video time period of the video to be processed is the same as the video time period of the video to be understood; then the video to be processed is cut into multiple first video blocks, and the total time period of the multiple first video blocks is the same as the video time period of the video to be processed, that is, the total time period of the multiple first video blocks is the same as the video time period of the video to be understood, and each first video block includes multiple video frames. Based on this, the present application provides a possible implementation method, and the above-mentioned S201 can, for example, include the following S201a-S201b (not shown in the figure).

[0141] S201a: Extract frames from the video to be understood to obtain a video to be processed; the video time period of the video to be processed is the same as the video time period of the video to be understood.

[0142] S201b: Slice the video to be processed into blocks to obtain a plurality of first video blocks.

[0143] Among them, video frame extraction refers to extracting some video frames from a video; the video to be processed refers to the video obtained by extracting some video frames from the video to be processed; and video segmentation refers to segmenting a video into video blocks.

[0144] The above S201a-S201b first extracts the video to be understood into a video to be processed by frame extraction, reducing the high redundancy between adjacent video frames in the video to be understood, so as to compress the number of video frames to be understood in the video to be understood and reduce the sequence length of the input sequence of the model; and then further processes the video to be processed by block cutting, shortening the video to be processed with a relatively long time period into multiple first video blocks with relatively short time periods, so as to further reduce the sequence length of each first video block of the video to be processed as the input sequence of the model.

[0145] As an example of S201a-S201b, based on the example of S201 above, the computer device first extracts frames from video X to be processed, namely video X', where the video time period of video X' is the same as the video time period of video X; and then blocks video X' into video block 1, video block 2, ..., video block n.

[0146] For example, video X is obtained by retaining 4 video frames per second, and video X' is divided into video block 1, video block 2, ..., video block n at 4-second intervals; or video X is obtained by retaining 4 video frames per second, and video X' is divided into video block 1, video block 2, ..., video block n according to shot scenes.

[0147] In an embodiment of the present application, when the above-mentioned S201 is specifically implemented to extract frames and cut the video to be understood into multiple first video blocks, the video to be understood can be cut into multiple video blocks to be processed first, so that the total time period of the multiple video blocks to be processed is the same as the video time period of the video to be understood, and each video block to be processed includes multiple video frames; then, each of the multiple video blocks to be processed is extracted frames to obtain multiple first video blocks, the total time period of the multiple first video blocks is the same as the total time period of the multiple video blocks to be processed, and the video block time period of each first video block is the same as the video block time period of the corresponding video block to be processed, that is, the total time period of the multiple first video blocks is the same as the video time period of the video to be understood. Based on this, the present application provides a possible implementation method, and the above-mentioned S201 can, for example, include the following S201c-S201d (not shown in the figure).

[0148] S201c: Slice the video to be understood to obtain multiple video blocks to be processed; the total time period of the multiple video blocks to be processed is the same as the video time period of the video to be understood, and each video block to be processed includes multiple video frames.

[0149] S201d: Perform video frame extraction on each of the multiple video blocks to be processed to obtain multiple first video blocks.

[0150] Among them, video segmentation refers to segmenting a video into video blocks; to-be-processed video blocks refer to video blocks obtained by segmenting the to-be-processed video; and video frame extraction refers to extracting some video frames from a video.

[0151] The above S201a-S201b first processes the video to be understood by cutting the video to be understood into blocks, shortening the video to be processed with a relatively long time period into multiple video blocks to be processed with relatively short time periods, thereby reducing the sequence length of the input sequence of the model; and then further extracting each video block to be processed into a first video block by extracting frames, thereby reducing the high redundancy between adjacent video frames in the video block to be processed, so as to compress the number of video frames to be understood in the video block to be processed, and further reducing the sequence length of each first video block as the input sequence of the model.

[0152] As an example of S201c-S201d, based on the example of S201 above, the computer device first divides video X into multiple video blocks to be processed, including video block 1', video block 2', ..., video block n'. The total time period of video block 1', video block 2', ..., video block n' is the same as the video time period of video X, and video block 1', video block 2', ..., video block n' each includes multiple video frames. Then, each of the video blocks to be processed in video block 1', video block 2', ..., video block n' is extracted by frame to obtain video block 1, video block 2, ..., video block n.

[0153] In addition, in an embodiment of the present application, after executing the above S201-S203 to perform video understanding on the video to be understood to obtain the first description text of each first video block extracted from the frame of the video to be understood, as well as the overall description text of the video to be understood; considering that video understanding lays the foundation for tasks such as video content classification, video tag recognition, and video title generation, when the task is related to the global video content of the video to be understood, the overall description text of the video to be understood can be processed by the task-related model task to obtain the task result; therefore, when the first task is associated with the global video content of the video to be understood, the overall description text of the video to be understood is input into the first task model for task processing, and the first task result is output. Based on this, the present application provides a possible implementation method, after the above S201-S203, the method also includes S4 (not shown in the figure): if the first task is associated with the global video content of the video to be understood, the overall description text is task processed according to the first task model to obtain the first task result.

[0154] Among them, the first task refers to a task associated with the global video content of the video to be understood, such as a video content classification task, a video tag recognition task, or a video title generation task; the first task model is a model for processing the first task; task processing is based on a text processing task; the first task result refers to the task result obtained by the first task model based on the overall description text of the video to be understood to process the first task.

[0155] When the task is related to the global video content of the video to be understood, the above S4 flexibly selects the overall description text of the video to be understood, and accurately obtains the task result by processing the overall description text through the task-related model task.

[0156] As an example of S4, based on the above examples of S201-S203, the first task is Task 1, and the first task model is Task Model 1; when Task 1 is associated with the global video content of Video X, the computer device converts the c X Input task model 1 for task processing, and output the first task result as task result 1.

[0157] When the first task is a video tag recognition task, the video tag recognition prompt of the first task model is as follows:

[0158] “Prompt: Video tag recognition

[0159] You are an experienced video understanding assistant. Now give the overall description text of the video to be understood:

[0160]

Overall description text of the video to be understood

[0161] Based on the above description, please extract the main tags of the video to be understood, which should include key information such as the video theme, style, scene, characters and roles.

[0162] When the first task is a video title generation task, the video title generation prompts of the first task model are as follows:

[0163] "Prompt: Video title generation

[0164] You are an experienced video understanding assistant. Now give the overall description text of the video to be understood:

[0165]

Overall description text of the video to be understood

[0166] Based on the above description and the video content presented in the video to be understood, please generate an attractive video title of no more than 20 words.

[0167] In addition, in an embodiment of the present application, after executing the above S201-S203 to perform video understanding on the video to be understood to obtain the first description text of each first video block extracted from the frame of the video to be understood, as well as the overall description text of the video to be understood; considering that video understanding lays the foundation for tasks such as video content classification, video tag recognition, and video title generation, when the task is related to the local video content of the video to be understood, the task result can be obtained by processing the first description text associated with the second task through the task-related model task; therefore, when the second task is associated with the local video content of the video to be understood, the first description text of the first video block associated with the second task is input into the second task model for task processing, and the second task result is output. Based on this, the present application provides a possible implementation method, after the above S201-S203, the method also includes S5 (not shown in the figure): if the second task is associated with the local video content of the video to be understood, the first description text of the first video block associated with the second task is task processed according to the second task model to obtain the second task result.

[0168] The second task is a task associated with the local video content of the video to be understood, for example, a task analyzing a portion of a video period. The second task model is a model for processing the second task; task processing is based on text processing; and the second task result is the result obtained by the second task model processing the second task based on the first descriptive text of the first video segment associated with the second task.

[0169] In the above S5, when the task is related to the local video content of the video to be understood, the first description text of the first video block associated with the second task is flexibly selected, and the task result is accurately obtained by processing the first description text through the task-related model task.

[0170] As an example of S5, based on the above examples of S201-S203, the second task is task 2, the second task model is task model 2, and the first description text of the first video block associated with task 2 is c of video block j. j , j∈1, 2, ..., n; when task 2 associates the local video content of video X, the computer device will c j Input task model 2 for task processing, and output the second task result as task result 2.

[0171] In addition, in an embodiment of the present application, after executing the above S1 to redundantly merge the multiple first description texts corresponding to the multiple first video blocks into the multiple second description texts corresponding to the multiple second video blocks; considering that video understanding lays the foundation for tasks such as video content classification, video tag recognition, and video title generation, when the task is related to the local video content of the video to be understood, the task result can be obtained by processing the second description text associated with the second task through the task-related model task; therefore, when the second task is associated with the local video content of the video to be understood, the second description text of the second video block associated with the second task is input into the second task model for task processing, and the second task result is output. Based on this, the present application provides a possible implementation method. After the above S201-S203, the method also includes S6 (not shown in the figure): if the second task is associated with the local video content of the video to be understood, the second description text of the second video block associated with the second task is task processed according to the second task model to obtain the second task result.

[0172] The second task is a task associated with the local video content of the video to be understood, for example, a task analyzing a portion of a video period. The second task model is a model for processing the second task; task processing is based on text processing; and the second task result is the result of the second task model processing the second task based on the second descriptive text of the second video segment associated with the second task.

[0173] In the above S6, when the task is related to the local video content of the video to be understood, the second description text of the second video block associated with the second task is flexibly selected, and the task result is accurately obtained by processing the second description text through the task-related model task.

[0174] As an example of S6, based on the example of S1 above, the second task is task 2, the second task model is task model 2, and the second description text of the second video block associated with task 2 is c of video block p'. p ', p∈1, 2, ..., m; when task 2 is associated with the local video content of video X, the computer device will c p 'Input task model 2 for task processing, and output the second task result as task result 2'.

[0175] For all the above, see Figure 7 , Figure 7 This is a flowchart of a video processing method provided in an embodiment of the present application, and the method includes the following S701-S713.

[0176] S701: Extract frames from the video to be understood to obtain a video to be processed; the video time period of the video to be processed is the same as the video time period of the video to be understood.

[0177] S702: Slice the video to be processed to obtain a plurality of first video blocks; the total time period of the plurality of first video blocks is the same as the video time period of the video to be processed, and each first video block includes a plurality of video frames.

[0178] S703: Perform a text description on each first video block by using a video description generator to obtain a first description text for each first video block.

[0179] S704: Perform feature extraction on each first description text by a feature extractor to obtain description text features of each first description text.

[0180] S705: Calculate similarity based on the description text features of each group of preceding description texts and the description text features of each group of following description texts in the plurality of first description texts corresponding to the plurality of first video blocks to obtain text similarity between each group of preceding description texts and following description texts.

[0181] S706: Determine, based on the text similarity between each group of preceding description text and following description text, similar video blocks corresponding to text similarities greater than or equal to a preset similarity and independent video blocks corresponding to text similarities less than the preset similarity; the similar video blocks include two adjacent first video blocks.

[0182] S707: If there are repeated first video blocks in different similar video blocks, perform aggregation processing on the different similar video blocks to obtain candidate merged video blocks.

[0183] S708: If there is no repeated first video block among the different similar video blocks, determine a candidate merged video block according to the similar video blocks.

[0184] S709: Redundantly merge at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks according to the redundant merging prompt using the second language model to obtain a merged description text of the merged video block.

[0185] S710: Determine a plurality of second description texts corresponding to a plurality of second video blocks according to the first description text corresponding to the independent video block and the merged description text of the merged video block.

[0186] S711: Performing an overall description of the plurality of second description texts corresponding to the plurality of second video blocks using a first language model to obtain an overall description text of the video to be understood.

[0187] S712: If the first task is associated with the global video content of the video to be understood, perform task processing on the overall description text according to the first task model to obtain a first task result.

[0188] S713: If the second task is associated with the partial video content of the video to be understood, perform task processing on the first description text of the first video block associated with the second task according to the second task model to obtain a second task result.

[0189] It should be noted that the video understanding method provided in the embodiment of the present application is simple and effective, does not require training, is highly versatile, and can be applied to multiple video scenarios, for example, short videos, long videos, video streams (live videos), and other scenarios.

[0190] See also Figure 8 , Figure 8 This is a schematic diagram of a video processing chain provided in an embodiment of the present application. The video processing chain refers to the process by which a video enters the content processing stage from the upstream content production stage, undergoes human-machine collaborative manual and machine tagging stages to obtain corresponding content features, and then enters the downstream content distribution stage. The video understanding method provided in an embodiment of the present application belongs to the machine tagging stage of the content processing stage.

[0191] It should be noted that, based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0192] based on Figure 2 Corresponding to the video understanding method provided in the embodiment, the embodiment of the present application also provides a video understanding device, see Figure 9 , Figure 9 This is a structural diagram of a video understanding device provided in an embodiment of the present application. The video understanding device 900 includes: a frame extraction and segmentation unit 901, a text description unit 902, and an overall description unit 903;

[0193] The frame extraction and segmentation unit 901 is configured to extract frames and segment the video to be understood to obtain a plurality of first video segments; the total time period of the plurality of first video segments is the same as the video time period of the video to be understood, and each first video segment includes a plurality of video frames;

[0194] A text description unit 902 is configured to perform a text description on each first video block through a video description generator to obtain a first description text for each first video block;

[0195] The overall description unit 903 is configured to perform an overall description on the plurality of first description texts corresponding to the plurality of first video blocks using a first language model to obtain an overall description text of the video to be understood.

[0196] In a possible implementation, the overall description unit 903 is configured to:

[0197] The first language model performs an overall description of the multiple first description texts corresponding to the multiple first video blocks according to the overall description prompt to obtain the overall description text of the video to be understood; the overall description prompt is used to guide the first language model to aggregate the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood.

[0198] In a possible implementation, the text description unit 902 is configured to:

[0199] For each first video block, encoding the first video block by an encoder in the video description generator to obtain visual features of the first video block;

[0200] The decoder in the video description generator performs feature decoding on the visual features of the first video block to obtain a first description text of the first video block.

[0201] In a possible implementation, the video understanding apparatus 900 further includes: a redundancy merging unit;

[0202] a redundant merging unit, configured to redundantly merge the plurality of first description texts corresponding to the plurality of first video blocks to obtain a plurality of second description texts corresponding to the plurality of second video blocks; the number of blocks in the plurality of second video blocks is less than the number of blocks in the plurality of first video blocks;

[0203] The overall description unit 903 is used to:

[0204] The first language model is used to perform an overall description of the plurality of second description texts corresponding to the plurality of second video blocks to obtain an overall description text of the video to be understood.

[0205] In a possible implementation, the redundant merging unit is configured to:

[0206] determining candidate merged video blocks and independent video blocks from the plurality of first video blocks based on text similarity between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks; the candidate merged video blocks including at least two first video blocks;

[0207] Redundantly merging at least two first description texts corresponding to at least two first video blocks in the candidate merged video block using a second language model to obtain a merged description text of the merged video block;

[0208] A plurality of second description texts corresponding to the plurality of second video blocks are determined according to the first description text corresponding to the independent video block and the merged description text of the merged video block.

[0209] In a possible implementation, the video understanding apparatus 900 further includes: a feature extraction unit and a similarity calculation unit;

[0210] A feature extraction unit, configured to extract features from each first description text using a feature extractor to obtain description text features of each first description text;

[0211] The similarity calculation unit is used to perform similarity calculation based on the description text features of each group of upper description texts and the description text features of the lower description texts in the multiple first description texts corresponding to the multiple first video blocks, so as to obtain the text similarity of each group of upper description texts and the lower description texts.

[0212] In a possible implementation, the redundant merging unit is configured to:

[0213] Determining, based on the text similarity between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks, similar video blocks corresponding to which the text similarity is greater than or equal to a preset similarity, and independent video blocks corresponding to which the text similarity is less than the preset similarity; the similar video blocks include two adjacent first video blocks;

[0214] If there are repeated first video blocks in different similar video blocks, aggregating the different similar video blocks to obtain candidate merged video blocks;

[0215] If the different similar video blocks do not have a repeated first video block, a candidate merged video block is determined according to the similar video blocks.

[0216] In a possible implementation, the redundant merging unit is configured to:

[0217] Redundant merging is performed on at least two first description texts corresponding to at least two first video blocks in the candidate merged video block according to the redundant merging prompt using the second language model to obtain a merged description text of the merged video block.

[0218] In a possible implementation, the frame extraction and slicing unit 901 is configured to:

[0219] Extract frames from the video to be understood to obtain a video to be processed; the video time period of the video to be processed is the same as the video time period of the video to be understood;

[0220] The video to be processed is segmented to obtain a plurality of first video blocks.

[0221] In a possible implementation, the frame extraction and slicing unit 901 is configured to:

[0222] Slice the video to be processed to obtain multiple video blocks to be processed; the total time period of the multiple video blocks to be processed is the same as the video time period of the video to be processed, and each video block to be processed includes multiple video frames;

[0223] Perform video frame extraction on each of the multiple video blocks to be processed to obtain multiple first video blocks.

[0224] In a possible implementation, the video understanding apparatus 900 further includes: a first task processing unit;

[0225] The first task processing unit is configured to, if the first task is associated with the global video content of the video to be understood, perform task processing on the overall description text according to the first task model to obtain a first task result.

[0226] In a possible implementation, the video understanding apparatus 900 further includes: a second task processing unit;

[0227] The second task processing unit is configured to, if the second task is associated with a partial video content of the video to be understood, perform task processing on the first description text of the first video block associated with the second task according to the second task model to obtain a second task result.

[0228] As can be seen from the above technical solution, the video understanding device includes a frame extraction and slicing unit, a text description unit, and an overall description unit. The frame extraction and slicing unit extracts frames from the video to be understood and slices it into multiple first video blocks. The total time period of these multiple first video blocks is the same as the video time period of the video to be understood, and each first video block includes multiple video frames. This frame extraction and slicing unit not only reduces the high redundancy between adjacent video frames in the video to be understood by extracting frames, thereby compressing the number of video frames to be understood in the video to be understood, but also shortens the video to be understood with relatively long time periods into multiple first video blocks with relatively short time periods, thereby reducing the sequence length of each first video block of the video to be understood as an input sequence for the model. The text description unit inputs each first video block into a video description generator for text description, and outputs a first description text for each first video block. Based on each first video block as the input sequence of the video description generator with a relatively short sequence length, this text description unit uses the first description text to represent the first video block in language, enabling a fast, accurate, and explanatory description of each first video block. The overall description unit inputs multiple first description texts corresponding to multiple first video blocks into the first language model for overall description, and outputs the overall description text of the video to be understood; the overall description unit aggregates the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood, and can quickly, accurately and explanatoryally fully describe the video to be understood.

[0229] Based on this, the device can extract frames of the video to be understood to compress the number of video frames to be understood in the video to be understood, and cut the video to be understood into blocks to reduce the sequence length of each first video block of the video to be understood as the input sequence of the model even when the video time period of the video to be understood is long; the video description generator uses language to represent each first video block separately to obtain a first description text, and aggregates it into an overall description text of the video to be understood through the first language model; on the basis of quickly, accurately and explanatoryly describing each first video block separately, the device quickly, accurately and explanatoryly describes the video to be understood in its entirety; thereby improving the speed and effect of video comprehension.

[0230] The present application also provides a computer device, which may be a server. Figure 10 , Figure 10 This is a structural diagram of a server provided in an embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors, such as a central processing unit CPU 1022, and a memory 1032, one or more storage media 1030 (such as one or more massive storage devices) storing application programs 1042 or data 1044. Among them, the memory 1032 and the storage medium 1030 can be temporary storage or persistent storage. The program stored in the storage medium 1030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1022 can be configured to communicate with the storage medium 1030 to execute a series of instruction operations in the storage medium 1030 on the server 1000.

[0231] The server 1000 may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server 2003. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM etc.

[0232] In this embodiment, the central processing unit 1022 in the server 1000 can execute the methods provided in various optional implementations of the above embodiments.

[0233] The computer device provided in the embodiment of the present application may also be a terminal. Figure 11 , Figure 11A structural diagram of a terminal provided in an embodiment of the present application. Taking a smartphone as an example, the smartphone includes components such as a radio frequency (RF) circuit 1110, a memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a processor 1180, and a power supply 1190. The input unit 1130 may include a touch panel 1131 and other input devices 1132, the display unit 1140 may include a display panel 1141, and the audio circuit 1160 may include a speaker 1161 and a microphone 1162. Those skilled in the art will understand that Figure 11 The structure of the smartphone shown in the figure does not constitute a limitation on the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0234] The memory 1120 can be used to store software programs and modules. The processor 1180 executes the various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 1120. The memory 1120 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the smartphone (such as audio data, a phone book, etc.). In addition, the memory 1120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0235] Processor 1180 is the control center of the smartphone, connecting all components of the smartphone using various interfaces and circuits. It executes software programs and / or modules stored in memory 1120 and accesses data stored in memory 1120 to perform various smartphone functions and process data. Optionally, processor 1180 may include one or more processing units. Preferably, processor 1180 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1180.

[0236] In this embodiment, the processor 1180 in the smartphone may execute the methods provided in various optional implementations of the above embodiments.

[0237] According to one aspect of the present application, a computer-readable storage medium is provided, which is used to store a computer program. When the computer program is run on a computer device, the computer device executes the methods provided in various optional implementations of the above embodiments.

[0238] According to one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional implementations of the above-described embodiments.

[0239] The descriptions of the processes or structures corresponding to the above figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0240] The terms "first", "second" etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0243] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0244] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk.

[0245] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0246] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, ordinary technical members in this field should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video understanding method, characterized in that: The method comprises: Extract frames and cut the video to be understood into blocks to obtain a plurality of first video blocks; the total time period of the plurality of first video blocks is the same as the video time period of the video to be understood, and each first video block includes a plurality of video frames; Performing a text description on each of the first video blocks by a video description generator to obtain a first description text for each of the first video blocks; The first language model is used to perform an overall description of the plurality of first description texts corresponding to the plurality of first video blocks to obtain an overall description text of the video to be understood.

2. The method according to claim 1, characterized in that The step of using a first language model to overall describe the plurality of first description texts corresponding to the plurality of first video blocks to obtain an overall description text of the video to be understood includes: The first language model performs an overall description of the multiple first description texts corresponding to the multiple first video blocks according to the overall description prompt to obtain the overall description text of the video to be understood; the overall description prompt is used to guide the first language model to aggregate the multiple first description texts corresponding to the multiple first video blocks into the overall description text of the video to be understood.

3. The method according to claim 1, characterized in that The step of performing a text description on each first video block by a video description generator to obtain a first description text for each first video block includes: For each of the first video blocks, performing feature encoding on the first video block by an encoder in the video description generator to obtain visual features of the first video block; The decoder in the video description generator performs feature decoding on the visual features of the first video block to obtain a first description text of the first video block.

4. The method according to claim 1, wherein The method further comprises: Redundantly merging the plurality of first description texts corresponding to the plurality of first video blocks to obtain a plurality of second description texts corresponding to the plurality of second video blocks; the number of blocks in the plurality of second video blocks is less than the number of blocks in the plurality of first video blocks; The step of using a first language model to overall describe the plurality of first description texts corresponding to the plurality of first video blocks to obtain an overall description text of the video to be understood includes: The first language model is used to perform an overall description of the plurality of second description texts corresponding to the plurality of second video blocks to obtain an overall description text of the video to be understood.

5. The method according to claim 4, characterized in that The redundantly merging the plurality of first description texts corresponding to the plurality of first video blocks to obtain the plurality of second description texts corresponding to the plurality of second video blocks includes: determining candidate merged video blocks and independent video blocks from the plurality of first video blocks based on text similarities between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks; the candidate merged video blocks including at least two first video blocks; Redundantly merging at least two first description texts corresponding to at least two first video blocks in the candidate merged video block using a second language model to obtain a merged description text of the merged video block; A plurality of second description texts corresponding to the plurality of second video blocks are determined according to the first description text corresponding to the independent video block and the merged description text of the merged video block.

6. The method according to claim 5, characterized in that The step of obtaining the text similarity between each group of the preceding description text and the following description text comprises: Performing feature extraction on each of the first description texts by a feature extractor to obtain description text features of each of the first description texts; Similarity calculation is performed based on the description text features of each group of upper description texts and the description text features of the lower description texts in the multiple first description texts corresponding to the multiple first video blocks to obtain text similarity of each group of upper description texts and the lower description texts.

7. The method according to claim 5, characterized in that The determining, based on the text similarity between each group of upper description text and lower description text in the plurality of first description texts corresponding to the plurality of first video blocks, candidate merged video blocks and independent video blocks in the plurality of first video blocks includes: Determining, based on the text similarity between each group of preceding description text and following description text in the plurality of first description texts corresponding to the plurality of first video blocks, similar video blocks corresponding to which the text similarity is greater than or equal to a preset similarity, and independent video blocks corresponding to which the text similarity is less than the preset similarity; the similar video blocks include two adjacent first video blocks; If there is a repeated first video block in different similar video blocks, aggregating the different similar video blocks to obtain the candidate merged video block; If the different similar video blocks do not have a repeated first video block, the candidate merged video block is determined according to the similar video blocks.

8. The method according to claim 5, characterized in that The redundantly merging at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks using a second language model to obtain a merged description text of the merged video block includes: Redundant merging of at least two first description texts corresponding to at least two first video blocks in the candidate merged video blocks is performed using the second language model according to the redundant merging prompt to obtain a merged description text of the merged video block.

9. The method according to claim 1, characterized in that The step of extracting frames and slicing the video to be understood to obtain a plurality of first video blocks includes: Extracting frames from the video to be understood to obtain a video to be processed; the video time period of the video to be processed is the same as the video time period of the video to be understood; The video to be processed is segmented to obtain the plurality of first video blocks.

10. The method according to claim 1, characterized in that The step of extracting frames and slicing the video to be understood to obtain a plurality of first video blocks includes: Slice the video to be understood to obtain a plurality of video blocks to be processed; the total time period of the plurality of video blocks to be processed is the same as the video time period of the video to be understood, and each video block to be processed includes a plurality of video frames; Perform video frame extraction on each of the multiple video blocks to be processed to obtain the multiple first video blocks.

11. The method according to claim 1, wherein The method further comprises: If the first task is associated with the global video content of the video to be understood, the overall description text is processed according to the first task model to obtain a first task result; or If the second task is associated with the local video content of the video to be understood, task processing is performed on the first description text of the first video block associated with the second task according to the second task model to obtain a second task result.

12. A video understanding device, characterized in that: The device comprises: a frame extraction and block cutting unit, a text description unit and an overall description unit; The frame extraction and segmentation unit is configured to extract frames and segment the video to be understood to obtain a plurality of first video segments; the total time period of the plurality of first video segments is the same as the video time period of the video to be understood, and each first video segment includes a plurality of video frames; The text description unit is configured to perform a text description on each first video block through a video description generator to obtain a first description text for each first video block; The overall description unit is configured to perform an overall description of the plurality of first description texts corresponding to the plurality of first video blocks using a first language model to obtain an overall description text of the video to be understood.

13. A computer device, characterized in that: The computer device includes a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the method according to any one of claims 1 to 11 according to instructions in the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and when the computer program is run on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is run on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 11.

Images