Video processing method and device, computer device, storage medium and program product

Abstract

This application relates to a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product. The method includes: acquiring multiple ordered sub-videos corresponding to a video to be processed; based on the processing order of the ordered sub-videos, acquiring the current sub-video from the sub-videos and obtaining information features of the current sub-video; fusing the memory content of the preceding sub-videos and the information features of the current sub-video to obtain fused information features; performing memory content generation processing based on the fused information features to obtain the memory content of the current sub-video; returning to the step of acquiring the current sub-video from the sub-videos based on the processing order of the ordered sub-videos, until a processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition; and obtaining global video understanding information of the video to be processed based on the memory content of the target sub-video. Using this method can improve the accuracy of video understanding.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of artificial intelligence technology, video understanding technology has made significant progress, but processing long videos remains a huge challenge.

[0003] In traditional technologies, the understanding of long videos is generally achieved through a hierarchical summarization method. For example, the entire long video is divided into segments, and then a summary is generated for each video segment. These summaries are then summarized, and so on, to obtain the video understanding information of the long video. However, information is easily lost during the layer-by-layer compression, and errors accumulate and amplify at each level, leading to deviations in video understanding and ultimately resulting in low accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of video understanding in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a video processing method, including:

[0006] The video to be processed is obtained, and the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed.

[0007] Based on the ordered processing order of the sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video;

[0008] Obtain the memory content corresponding to the preceding sub-video of the current sub-video, and fuse the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to characterize the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video;

[0009] Based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video. Then, the process returns to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained.

[0010] Based on the memory content of the target sub-video, global video understanding information corresponding to the video to be processed is obtained.

[0011] In one embodiment, obtaining the predicted answer information of the sample question information under different processing sessions based on the global video understanding information of the sample video under different processing sessions includes:

[0012] Obtain the semantic information of the sample problem information;

[0013] The global video understanding information of the sample video under different processing sessions is queried respectively to obtain the matching information of the semantic information in the global video understanding information under different processing sessions;

[0014] Based on the semantic information and its matching information in the global video understanding information under different processing sessions, predictive answer information for the sample question information under different processing sessions is generated.

[0015] In one embodiment, obtaining the reward information corresponding to each processing session of the sample video based on the predicted answer information under different processing sessions and the sample answer information, includes:

[0016] When the sample question information belongs to an objective question answering task, the consistency comparison results between the predicted answer information and the sample answer information under different processing sessions are obtained respectively.

[0017] Based on the consistency comparison results between the predicted answer information and the sample answer information under different processing sessions, the reward information corresponding to each processing session of the sample video is determined.

[0018] In one embodiment, obtaining the reward information corresponding to each processing session of the sample video based on the predicted answer information under different processing sessions and the sample answer information, includes:

[0019] When the sample question information belongs to a generative task, the semantic similarity between the predicted answer information and the sample answer information under different processing sessions is obtained respectively;

[0020] Based on the semantic similarity between the predicted answer information and the sample answer information under different processing sessions, the reward information corresponding to each processing session of the sample video is determined.

[0021] In one embodiment, after obtaining the global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video, the method further includes:

[0022] From the global video understanding information corresponding to the video to be processed, obtain candidate information corresponding to the summary information generation task for the video to be processed;

[0023] The candidate information is fused to obtain the summary information corresponding to the summary information generation task.

[0024] Secondly, this application provides another video processing method, including:

[0025] The video to be processed is displayed on the video analysis page;

[0026] In response to a global video understanding event for the video to be processed, global video understanding information corresponding to the video to be processed is displayed on the video analysis page; the global video understanding information is obtained according to the method described in the first aspect.

[0027] Thirdly, this application provides yet another video processing method, including:

[0028] The video to be processed is displayed on the video analysis page;

[0029] During the processing of the video to be processed, or before the processing of the video to be processed, or after the processing of the video to be processed is completed, in response to the query content input event for the video to be processed, the query content corresponding to the query content input event is displayed on the video analysis page;

[0030] After the video to be processed is completed, the response content corresponding to the query content is displayed on the video analysis page; the response content is obtained by querying the global video understanding information corresponding to the video to be processed.

[0031] Fourthly, this application also provides a video processing apparatus, comprising:

[0032] The segmentation processing module is used to acquire the video to be processed, segment the video to be processed, and obtain multiple ordered sub-videos corresponding to the video to be processed.

[0033] The feature extraction module is used to obtain the current sub-video from the sub-videos based on the ordered processing order of the sub-videos, perform feature extraction processing on the current sub-video, and obtain the information features of the current sub-video;

[0034] The fusion processing module is used to obtain the memory content corresponding to the previous sub-video of the current sub-video, and to fuse the memory content corresponding to the previous sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video;

[0035] The content generation module is used to perform memory content generation processing based on the fused information features to obtain the memory content corresponding to the current sub-video, return to the processing order based on the ordered sub-videos, and obtain the current sub-video from the sub-videos until the processing completion condition is met, and obtain the memory content of the target sub-video corresponding to the processing completion condition.

[0036] The information determination module is used to obtain global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video.

[0037] Fifthly, this application also provides another video processing apparatus, including:

[0038] The first display module is used to display the video to be processed on the video analysis page;

[0039] The second display module is configured to, in response to a global video understanding event for the video to be processed, display global video understanding information corresponding to the video to be processed on the video analysis page; the global video understanding information is obtained according to the method described in the first aspect.

[0040] Sixthly, this application also provides yet another video processing apparatus, comprising:

[0041] The video display module is used to display the video to be processed on the video analysis page;

[0042] An event response module is used to display the query content corresponding to the query content input event on the video analysis page in response to a query content input event for the video to be processed, either during the processing of the video to be processed, before the processing of the video to be processed, or after the processing of the video to be processed is completed.

[0043] The content display module is used to display the response content corresponding to the query content on the video analysis page after the video to be processed is completed; the response content is obtained by querying the global video understanding information corresponding to the video to be processed.

[0044] In a seventh aspect, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0045] The video to be processed is obtained, and the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed.

[0046] Based on the ordered processing order of the sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video;

[0047] Obtain the memory content corresponding to the preceding sub-video of the current sub-video, and fuse the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to characterize the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video;

[0048] Based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video. Then, the process returns to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained.

[0049] Based on the memory content of the target sub-video, global video understanding information corresponding to the video to be processed is obtained.

[0050] Eighthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0051] The video to be processed is obtained, and the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed.

[0052] Based on the ordered processing order of the sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video;

[0053] Obtain the memory content corresponding to the preceding sub-video of the current sub-video, and fuse the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to characterize the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video;

[0054] Based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video. Then, the process returns to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained.

[0055] Based on the memory content of the target sub-video, global video understanding information corresponding to the video to be processed is obtained.

[0056] Ninthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0057] The video to be processed is obtained, and the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed.

[0058] Based on the ordered processing order of the sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video;

[0059] Obtain the memory content corresponding to the preceding sub-video of the current sub-video, and fuse the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to characterize the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video;

[0060] Based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video. Then, the process returns to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained.

[0061] Based on the memory content of the target sub-video, global video understanding information corresponding to the video to be processed is obtained.

[0062] The aforementioned video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product first acquire the video to be processed, then segment the video to be processed to obtain multiple ordered sub-videos corresponding to the video to be processed; next, based on the processing order of the ordered sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video; then, the memory content corresponding to the preceding sub-video of the current sub-video is obtained, and the memory content corresponding to the preceding sub-video and the information features of the current sub-video are fused to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and its historical sub-videos; next, based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video, and the process returns to the step of obtaining the current sub-video from the sub-videos based on the processing order of the ordered sub-videos, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained; finally, based on the memory content of the target sub-video, the global video understanding information corresponding to the video to be processed is obtained. In this way, when understanding the video to be processed, the entire video is first divided into multiple ordered sub-videos. Then, for each current sub-video, the memory content corresponding to the preceding sub-video is combined with the information features of the current sub-video to generate the memory content corresponding to the current sub-video. This process is iterated until the processing completion condition is met. This method fully considers the temporal dependency between the current sub-video and the preceding sub-videos. When generating the memory content of the current sub-video, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are integrated, which can effectively alleviate the problems of detail loss and error accumulation in traditional hierarchical summarization, thereby improving the accuracy of the generated memory content. At the same time, using the memory content corresponding to the preceding sub-video of the current sub-video to constrain the understanding of the current sub-video can eliminate the ambiguity of understanding individual sub-videos and avoid contextual fragmentation. This is conducive to restoring the temporal logic of the entire video, making the memory content generated segment by segment more accurate and reliable, thereby improving the accuracy of the memory content of the final target sub-video. This can effectively reduce video understanding bias, thereby improving the accuracy of determining global video understanding information and thus improving the overall video understanding accuracy. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is an application environment diagram of a video processing method in one embodiment;

[0065] Figure 2 This is a flowchart illustrating a video processing method in one embodiment;

[0066] Figure 3 This is an architecture diagram of a long video understanding system based on a reinforcement learning memory agent in one embodiment.

[0067] Figure 4 This is a flowchart illustrating the steps for obtaining the information features of the current sub-video in one embodiment;

[0068] Figure 5 This is a flowchart illustrating the training steps of a pre-trained video processing model in one embodiment.

[0069] Figure 6 This is a flowchart illustrating the steps for obtaining global video understanding information of a sample video under different processing sessions in one embodiment.

[0070] Figure 7 This is a flowchart illustrating the steps for obtaining predicted answer information for sample question information under different processing sessions in one embodiment.

[0071] Figure 8 This is a flowchart illustrating the steps for obtaining the model loss of the video processing model to be trained in one embodiment.

[0072] Figure 9 This is a flowchart illustrating the steps for obtaining the generation loss of each memory content generation step in each processing session of a sample video in one embodiment.

[0073] Figure 10 This is a flowchart illustrating a video processing method in another embodiment;

[0074] Figure 11 This is a schematic diagram illustrating the display of the video to be processed on a video analysis page in one embodiment;

[0075] Figure 12 This is a schematic diagram illustrating the display of global video understanding information corresponding to the video to be processed on a video analysis page in one embodiment.

[0076] Figure 13 This is a schematic diagram illustrating the display of query content on a video analytics page in one embodiment.

[0077] Figure 14 This is a schematic diagram illustrating the display of the response content corresponding to the query content on the video analysis page in one embodiment;

[0078] Figure 15 This is a schematic diagram illustrating the display of target information corresponding to generated memory content on a video analysis page in one embodiment.

[0079] Figure 16 This is a flowchart illustrating the video processing method in yet another embodiment;

[0080] Figure 17 This is a structural block diagram of a video processing device in one embodiment;

[0081] Figure 18 This is a structural block diagram of the video processing apparatus in another embodiment;

[0082] Figure 19 This is a structural block diagram of the video processing device in yet another embodiment;

[0083] Figure 20 This is an internal structural diagram of a computer device in one embodiment;

[0084] Figure 21 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0086] With the development of large multimodal models (such as Video-LLM), video understanding technology has made significant progress; however, most of these models perform well in processing short videos (usually a few seconds to a few minutes), and processing infinitely long videos remains a huge challenge; currently, the main technical solutions for processing long videos are as follows:

[0087] 1. Sliding Window: The fixed context window of the model is slid across the entire timeline of the video. Although this method can handle long videos, there is a lack of effective information interaction between windows, making it difficult for the model to understand long-term dependencies that span more than the length of a single window.

[0088] 2. Hierarchical Summarization: The video is divided into segments, a summary is generated for each segment, and then these summaries are summarized again, and so on recursively. The disadvantage of this method is that information is easily lost in the layer compression, resulting in the "information bottleneck" problem, and errors will accumulate and amplify at each level.

[0089] 3. Extended Context Window Method: By improving the model structure (such as sparse attention, linear attention) or positional encoding method, the length of the context that the model can handle can be directly extended. Although this type of method can handle longer videos, it usually brings huge computational and memory overhead. Its scalability still has an upper limit, and its performance often drops significantly when processing videos that exceed the training length.

[0090] 4. External memory database method: Video information (such as identified objects and events) is stored in an external vector database; during processing, relevant information is retrieved according to the current task; this method relies on precise information extraction and retrieval modules, the system design is complex, and the organization and correlation of information are not as flexible as the end-to-end model.

[0091] Current technical solutions for processing long videos suffer from the following drawbacks: 1. High computational complexity: Models based on the standard Transformer attention mechanism have a computational complexity proportional to the square of the video length (number of frames or tokens) (O(N²)). This results in extremely high computational costs when processing long videos, rendering them impractical. 2. Context length bottleneck: All models have a maximum context window determined by hardware and algorithms, making them unable to handle video streams of arbitrary length. 3. Performance cliff: When the video length exceeds the model's training length, the model's understanding ability typically drops sharply, resulting in the so-called "performance cliff" phenomenon, making effective long-distance extrapolation impossible. 4. Information loss and error accumulation: Methods such as hierarchical summarization inevitably lose details during information transmission, leading to deviations in the final understanding.

[0092] In order to solve the above problems, this application proposes a video processing method, specifically a long video understanding method based on reinforcement learning memory agents, which can improve the accuracy of video understanding and reduce computational complexity. It can be applied to various video understanding scenarios, intelligent video analysis scenarios, video-based question answering scenarios, and video-based summary generation scenarios.

[0093] The video processing method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The data storage system can store data that server 104 needs to process, such as the memory content corresponding to the preceding sub-video (e.g., the previous sub-video). The data storage system can be integrated on server 104 or placed on the cloud or other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Furthermore, terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0094] It should be noted that both terminal 102 and server 104 can be used independently to execute the video processing method provided in the embodiments of this application.

[0095] For example, refer to Figure 1 First, terminal 102 acquires the video to be processed and segments it to obtain multiple ordered sub-videos. Next, based on the processing order of the ordered sub-videos, terminal 102 acquires the current sub-video and performs feature extraction to obtain its information features. Then, terminal 102 acquires the memory content corresponding to the preceding sub-videos and fuses the memory content with the information features of the current sub-video to obtain fused information features. The memory content corresponding to the sub-videos represents the video understanding information of the sub-videos and their historical sub-videos. Next, terminal 102 generates memory content based on the fused information features to obtain the memory content corresponding to the current sub-video, and returns to the step of acquiring the current sub-video from the sub-videos based on the processing order until the processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition. Finally, based on the memory content of the target sub-video, terminal 102 obtains the global video understanding information corresponding to the video to be processed.

[0096] For example, refer to Figure 1First, server 104 acquires the video to be processed, segments it to obtain multiple ordered sub-videos. Next, based on the processing order of the ordered sub-videos, server 104 acquires the current sub-video, extracts its features, and obtains its information features. Then, server 104 acquires the memory content corresponding to the preceding sub-videos, fuses the memory content with the information features of the current sub-video, and obtains fused information features. The memory content corresponding to the sub-videos represents the video understanding information of the sub-videos and their historical sub-videos. Next, server 104 generates memory content based on the fused information features, obtains the memory content corresponding to the current sub-video, and returns to the step of acquiring the current sub-video from the sub-videos based on the processing order until the processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition. Finally, server 104 obtains the global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video.

[0097] It should be noted that terminal 102 and server 104 can also be used in conjunction to execute the video processing method provided in the embodiments of this application.

[0098] For example, refer to Figure 1 Terminal 102 acquires the video to be processed and sends it to server 104. Server 104 segments the video to be processed, obtaining multiple ordered sub-videos corresponding to the video to be processed. Next, based on the processing order of the ordered sub-videos, server 104 retrieves the current sub-video from the sub-videos, performs feature extraction processing on the current sub-video, and obtains the information features of the current sub-video. Then, server 104 retrieves the memory content corresponding to the preceding sub-video of the current sub-video, and fuses the memory content corresponding to the preceding sub-video with the information features of the current sub-video to obtain fused information features. The memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and its historical sub-videos. Next, server 104 performs memory content generation processing based on the fused information features to obtain the memory content corresponding to the current sub-video, and returns to the step of retrieving the current sub-video from the sub-videos based on the processing order, until the processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition. Finally, based on the memory content of the target sub-video, server 104 obtains the global video understanding information corresponding to the video to be processed and sends the global video understanding information to terminal 102, which displays the global video understanding information on the terminal interface.

[0099] Before introducing the specific embodiments of this application, the technical terms involved in this application will be explained:

[0100] Unlimited long video: refers to video data whose duration exceeds the single-processing capacity limit of a conventional model, such as long movies, TV series, conference recordings, monitoring videos, live sports events, or live games, whether continuous or segmented.

[0101] Multimodal Large Language Model (MLLM): refers to a large neural network model that can simultaneously understand and process multiple types of information (such as video, audio, and text).

[0102] In one exemplary embodiment, such as Figure 2 As shown, a video processing method is provided, illustrated by applying it to a computer device, which can be a terminal or a server. It is understood that this method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (Reference) Figure 2 The video processing method includes the following steps S201 to S205. Wherein:

[0103] Step S201: Obtain the video to be processed, segment the video to be processed, and obtain multiple ordered sub-videos corresponding to the video to be processed.

[0104] The videos to be processed can be of any type, such as game videos, animal videos, travel videos, food videos, etc.; they can also be of any length (i.e., any duration), such as short videos, long videos, and infinitely long videos. Infinitely long videos refer to videos whose duration exceeds the single-processing capacity limit of the conventional model, such as TV dramas, feature films, meeting recordings, monitoring videos, live sports broadcasts, and live game broadcasts. The single-processing capacity limit refers to the maximum video duration that the model can process in a single run, such as 1 minute, 2 minutes, 5 minutes, 10 minutes, etc.

[0105] In this context, a sub-video refers to a video segment whose duration is shorter than the total video duration after segmenting the video to be processed, such as a video block. For example, a 1-minute video to be processed can be divided into 6 sub-videos: sub-videos of 1-10 seconds, 11-20 seconds, 21-30 seconds, 31-40 seconds, 41-50 seconds, and 51-60 seconds; a 20-minute video to be processed can be divided into 4 sub-videos: sub-videos of 1-5 minutes, 6-10 minutes, 11-15 minutes, and 16-20 minutes. It should be noted that the sub-videos obtained from the segmented processing are temporally continuous, meaning that adjacent sub-videos are sequentially connected, without time gaps or content overlap, and all sub-videos can be stitched together to form the original video to be processed. In other words, all sub-videos, when stitched together in sequence, can completely cover the entire duration of the video to be processed. Furthermore, the video durations corresponding to each sub-video can be the same or different, but all must be less than the upper limit of the model's single-processing capacity. For example, they must all be less than the upper limit of the single-processing capacity of a pre-trained video processing model (i.e., a multimodal large model). Ordered sub-videos refer to multiple sub-video segments obtained by dividing the video to be processed according to the original playback time sequence. Each sub-video segment maintains the original temporal sequence relationship of the video and is arranged sequentially, for example, sub-video 1 (1-5 minutes) → sub-video 2 (6-10 minutes) → sub-video 3 (11-15 minutes) → sub-video 4 (16-20 minutes).

[0106] The segmentation of the video to be processed refers to dividing the entire video into multiple ordered sub-videos. Specifically, it means dividing the entire video into a series of consecutive sub-videos, such as sub-video 1, sub-video 2, sub-video 3, and sub-video 4. It should be noted that this application uses a streaming processing mechanism to divide a video of arbitrary length into multiple sub-videos of fixed duration. The model (such as a multimodal large model) does not need to load the entire video at once, but rather "watches" it segment by segment, like a human, thus breaking the model's own context window limitation. That is, only one sub-video needs to be processed at a time, instead of processing the entire video at once. It should also be noted that this application processes each sub-video sequentially; for example, sub-video 1 is processed first, then sub-video 2, and so on.

[0107] For example, the computer device retrieves the video to be processed from a local database, or receives a video understanding request for the video to be processed, and parses the video understanding request to obtain the video to be processed. Then, the computer device segments the video to be processed according to video segmentation instructions, obtaining a series of consecutive sub-videos, which serve as multiple ordered sub-videos corresponding to the video to be processed; for example, the video to be processed is segmented according to a preset duration, obtaining a series of consecutive sub-videos whose duration is less than or equal to the preset duration; for example, a 1-minute video to be processed is segmented according to a preset duration of 10 seconds, resulting in six 10-second sub-videos, namely sub-video 1 (1-10 seconds) → Sub-video 2 (11-20 seconds) → Sub-video 3 (21-30 seconds) → Sub-video 4 (31-40 seconds) → Sub-video 5 (41-50 seconds) → Sub-video 6 (51-60 seconds); For example, according to the preset duration of 10 seconds, the 55-second video to be processed is segmented to obtain 6 sub-videos, namely Sub-video 1 (1-10 seconds) → Sub-video 2 (11-20 seconds) → Sub-video 3 (21-30 seconds) → Sub-video 4 (31-40 seconds) → Sub-video 5 (41-50 seconds) → Sub-video 6 (51-55 seconds).

[0108] Step S202: Based on the ordered processing order of sub-videos, obtain the current sub-video from the sub-videos, perform feature extraction processing on the current sub-video, and obtain the information features of the current sub-video.

[0109] The processing order refers to the ordered arrangement of the sub-videos. For example, sub-video 1 (1-10 seconds) → sub-video 2 (11-20 seconds) → sub-video 3 (21-30 seconds) → sub-video 4 (31-40 seconds) → sub-video 5 (41-50 seconds) → sub-video 6 (51-60 seconds). Sub-video 1 is processed first, followed by sub-video 2, and so on. Furthermore, the current sub-video retrieved from the sub-videos is different each time it is processed. For instance, the first time it is processed, the current sub-video retrieved is the first sub-video; the second time, the second sub-video; the third time, the third sub-video, and so on. In addition, based on the ordered processing order of the sub-videos and the processed current sub-video, a new current sub-video (i.e., the next sub-video after the processed current sub-video) can be retrieved. For example, if the processed current sub-video is sub-video 3, and sub-video 4 follows sub-video 3, then the new current sub-video is sub-video 4.

[0110] Feature extraction of the current sub-video refers to extracting its information features. Information features are feature representations used to characterize the visual content and semantic information of the current sub-video, reflecting the scene content, target objects, actions, scene changes, and visual semantic information within the current sub-video. In practical scenarios, the information features of the current sub-video refer to the multimodal feature vector obtained after feature extraction of the visual content of the current sub-video.

[0111] For example, the computer device determines the next sub-video after the current sub-video has been processed, based on the ordered processing sequence of the sub-videos, and designates it as the new current sub-video. For instance, if the current sub-video is empty, the new current sub-video is the first sub-video; if the current sub-video is the first sub-video, the new current sub-video is the second sub-video, and so on. Next, the computer device performs feature extraction processing on the current sub-video using feature extraction instructions to obtain the information features of the current sub-video; alternatively, the computer device acquires the visual content of the current sub-video and performs feature extraction processing on the visual content to obtain the multimodal feature vector of the current sub-video, which serves as the information features of the current sub-video; or, the computer device inputs the current sub-video into a pre-trained video processing model (i.e., a multimodal large model), acquires the visual content of the current sub-video through the pre-trained video processing model, and performs feature extraction processing on the visual content to obtain the multimodal feature vector of the current sub-video, which serves as the information features of the current sub-video.

[0112] For example, a computer device, based on an ordered processing sequence of sub-videos, obtains the current sub-video from the sub-videos. It then inputs the current sub-video into a pre-trained video processing model, which performs frame segmentation on the current sub-video to obtain its video frames. Next, key video frames are extracted from these frames. Then, visual content extraction processing is performed on the key video frames to obtain their visual content. The visual content of each key video frame is then fused to obtain the visual content of the current sub-video. Next, feature encoding processing is performed on the visual content of the current sub-video to obtain its visual features. Then, feature mapping processing is performed on the visual features of the current sub-video to obtain mapped features, which serve as the multimodal feature vector corresponding to the current sub-video. Finally, the multimodal feature vector corresponding to the current sub-video is confirmed as the information features of the current sub-video.

[0113] Step S203: Obtain the memory content corresponding to the previous sub-video of the current sub-video, and fuse the memory content corresponding to the previous sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video.

[0114] Here, the preceding sub-video of the current sub-video refers to the previous sub-video of the current sub-video; for example, if the current sub-video is sub-video 2, then the preceding sub-video of the current sub-video is sub-video 1; if the current sub-video is sub-video 3, then the preceding sub-video of the current sub-video is sub-video 2. It should be noted that if the preceding sub-video of the first sub-video is empty, it means that the memory content corresponding to the preceding sub-video of the first sub-video is empty.

[0115] In this context, the historical sub-videos of a sub-video refer to all sub-videos preceding the current sub-video. For example, if the sub-video is sub-video 3, then its historical sub-videos include sub-videos 1 and 2; if the sub-video is sub-video 4, then its historical sub-videos include sub-videos 1, 2, and 3. Video understanding information refers to all semantic content identified by the model (such as a pre-trained video processing model, i.e., a multimodal large model) from the video, including but not limited to: what objects, people, and scenes are in the frame; what actions, events, and processes occurred; the sequence and causal relationships of events; and the video's theme, content, details, and logic. The video understanding information of a sub-video refers to all semantic content identified by the model from the sub-video. The video understanding information corresponding to a sub-video and its historical sub-videos refers to all semantic content identified by the model from the sub-video and its historical sub-videos.

[0116] The memory content corresponding to a sub-video refers to the video understanding information corresponding to the sub-video and its historical sub-videos. Specifically, it refers to the information obtained after understanding the sub-video and its historical sub-videos. For example, the memory content corresponding to sub-video 3 refers to the video understanding information corresponding to sub-videos 1, 2, and 3. Furthermore, the memory content corresponding to a sub-video is represented in the form of a feature vector. This vector integrates the visual features and semantic understanding information of the sub-video and its historical sub-videos, and can be directly fused and calculated by the model. Simultaneously, this vector can generate text understanding content in natural language form to represent the semantic information of the video. In practical scenarios, the memory content corresponding to a sub-video refers to the summative understanding result of the sub-video and its historical sub-videos. It is a content understanding of the sub-video and its historical sub-videos, including information such as time sequence, events, characters, scenes, and logical relationships. It is structured semantic information generated by the model itself (such as a pre-trained video processing model, i.e., a multimodal large model), which can be transmitted, and it accumulates and updates continuously as sub-videos are processed.

[0117] Among them, the memory content corresponding to the previous sub-video of the current sub-video is used to represent the video understanding information corresponding to the previous sub-video (i.e., the previous sub-video) and the historical sub-video of the previous sub-video. It can refer to the video understanding information corresponding to the previous sub-video and the historical sub-video of the previous sub-video, specifically the information obtained after understanding the previous sub-video and the historical sub-video of the previous sub-video.

[0118] Here, fusion processing can refer to splicing processing. Fusion information features refer to the information features obtained after fusing the memory content corresponding to the preceding sub-video and the information features of the current sub-video; specifically, it refers to the spliced ​​information features of the memory content corresponding to the preceding sub-video and the information features of the current sub-video. Furthermore, fusion information features can also refer to the fusion information features corresponding to the current sub-video.

[0119] For example, the computer device acquires video understanding information corresponding to the preceding sub-video and the historical sub-video of the preceding sub-video, as the memory content corresponding to the preceding sub-video. For instance, it acquires video understanding information corresponding to the previous sub-video and the historical sub-video of the previous sub-video, as the memory content corresponding to the previous sub-video. Then, through a splicing processing instruction, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are spliced ​​together to obtain spliced ​​information features, which are used as fusion information features. Alternatively, the memory content corresponding to the previous sub-video and the information features of the current sub-video are input into a pre-trained video processing model, and the pre-trained video processing model splices together the memory content corresponding to the previous sub-video and the information features of the current sub-video to obtain spliced ​​information features, which are used as fusion information features.

[0120] For example, when the current sub-video is sub-video 3, the computer device obtains the memory content corresponding to sub-video 2, and concatenates the memory content corresponding to sub-video 2 with the multimodal feature vector of sub-video 3 to obtain a concatenated feature vector, which is used as the fused feature information corresponding to the current sub-video.

[0121] Step S204: Based on the fused information features, perform memory content generation processing to obtain the memory content corresponding to the current sub-video. Return to the processing order based on the ordered sub-videos and obtain the current sub-video from the sub-videos until the processing completion condition is met, and obtain the memory content of the target sub-video corresponding to the processing completion condition.

[0122] Among them, memory content generation processing refers to generating memory content corresponding to the fused information features. The memory content corresponding to the current sub-video is used to represent the video understanding information corresponding to the current sub-video and its historical sub-videos. Specifically, it refers to the information obtained after understanding the current sub-video and its historical sub-videos.

[0123] The step of returning to the processing order based on the ordered sub-videos and obtaining the current sub-video from the sub-videos indicates that the computer device processes the next sub-video of the current sub-video as the updated current sub-video until the processing completion condition is met.

[0124] The processing completion condition refers to the memory content corresponding to the last sub-video of the video to be processed. For example, if the video to be processed includes 10 sub-videos, then the processing completion condition refers to the memory content corresponding to the 10th sub-video (i.e., sub-video 10) of the video to be processed. The memory content corresponding to the last sub-video is used to represent the video understanding information corresponding to the last sub-video and its historical sub-videos. It can refer to the video understanding information corresponding to the last sub-video and its historical sub-videos, specifically the information obtained after understanding the last sub-video and its historical sub-videos.

[0125] The target sub-video corresponding to the processing completion condition refers to the last sub-video of the video to be processed. The memory content of the target sub-video refers to the memory content corresponding to the last sub-video of the video to be processed.

[0126] For example, the computer device generates memory content based on fused information features by generating processing instructions, and obtains the generated memory content as the memory content corresponding to the current sub-video; or, the computer device generates memory content based on fused information features by using a pre-trained video processing model (i.e., a multimodal large model), and outputs the memory content as the memory content corresponding to the current sub-video; then, based on the ordered processing order of the sub-videos, the computer device sets the next sub-video of the current sub-video as the updated current sub-video for processing, and so on, until the memory content corresponding to the last sub-video of the video to be processed is obtained, that is, the processing completion condition is met, and the memory content corresponding to the last sub-video of the video to be processed is used as the memory content of the target sub-video.

[0127] Step S205: Based on the memory content of the target sub-video, obtain the global video understanding information corresponding to the video to be processed.

[0128] The global video understanding information corresponding to the video to be processed refers to the information obtained after performing a global understanding of the video. Specifically, it refers to all semantic content identified from the video, including but not limited to: what objects, people, and scenes are in the frame; what actions, events, and processes occurred; the sequence and causal relationships of events; and the video's theme, content, details, and logic. Specifically, the global video understanding information corresponding to the video to be processed refers to the memory content of the target sub-video, such as the memory content corresponding to the last sub-video of the video to be processed.

[0129] It should be noted that in traditional technologies, when understanding long videos, the video is typically segmented, a summary is generated for each segment, and then these summaries are summarized. However, information is easily lost during layer-by-layer compression, and errors accumulate and amplify at each level, ultimately leading to low accuracy in video understanding. The core improvement of this application is a long video understanding process based on dynamic memory. First, the long video is acquired and segmented into blocks to obtain a series of consecutive video blocks. Then, the first video block is input into a multimodal large model to obtain its multimodal feature vector. Based on this feature vector, a memory block corresponding to the first video block is generated. Next, the second video block is input into the multimodal large model to obtain its multimodal feature vector, and combined with the previous memory block (i.e., the memory block corresponding to the first video block), a memory block corresponding to the second video block is generated. This process continues until the target memory block corresponding to the last video block is obtained. Finally, the target memory block is used to answer user questions, generate summaries, or perform other downstream tasks.

[0130] For example, the computer device uses the memory content of the target sub-video as the global video understanding information corresponding to the video to be processed; for instance, it uses the memory content corresponding to the last sub-video of the video to be processed as the global video understanding information corresponding to the video to be processed.

[0131] For example, see reference. Figure 3 The computer device receives the input video stream and cuts it into a series of continuous video blocks C1, C2, C3...C1, C2, C3, C4, each with a preset duration of 10 seconds. N Next, for each video block C k It uses a pre-trained multimodal large model to extract its visual content and transforms it into a compact multimodal feature vector, which serves as the current video block C. k The characteristics, then the current video block C k The characteristics, and the memory M from the previous step k-1 The features are concatenated to obtain concatenated features, and a new memory M is generated based on these features. k This process continues until the last video block C is obtained. N Corresponding memory MN After processing all N video blocks, based on the user's specific task (such as answering questions or generating summaries), and using memory M... N This is used to generate the final results, such as answer information and summary information.

[0132] In the above video processing method, the video to be processed is first acquired and segmented to obtain multiple ordered sub-videos. Then, based on the processing order of the ordered sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction is performed on it to obtain its information features. Next, the memory content corresponding to the preceding sub-videos of the current sub-video is acquired, and the memory content and the information features of the current sub-video are fused to obtain fused information features. The memory content corresponding to the sub-videos is used to represent the video understanding information corresponding to the sub-videos and their historical sub-videos. Then, based on the fused information features, memory content generation is performed to obtain the memory content corresponding to the current sub-video. The process returns to the step of obtaining the current sub-video from the sub-videos based on the processing order, until the processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition. Finally, based on the memory content of the target sub-video, the global video understanding information corresponding to the video to be processed is obtained. In this way, when understanding the video to be processed, the entire video is first divided into multiple ordered sub-videos. Then, for each current sub-video, the memory content corresponding to the preceding sub-video is combined with the information features of the current sub-video to generate the memory content corresponding to the current sub-video. This process is iterated until the processing completion condition is met. This method fully considers the temporal dependency between the current sub-video and the preceding sub-videos. When generating the memory content of the current sub-video, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are integrated, which can effectively alleviate the problems of detail loss and error accumulation in traditional hierarchical summarization, thereby improving the accuracy of the generated memory content. At the same time, using the memory content corresponding to the preceding sub-video of the current sub-video to constrain the understanding of the current sub-video can eliminate the ambiguity of understanding individual sub-videos and avoid contextual fragmentation. This is conducive to restoring the temporal logic of the entire video, making the memory content generated segment by segment more accurate and reliable, thereby improving the accuracy of the memory content of the final target sub-video. This can effectively reduce video understanding bias, thereby improving the accuracy of determining global video understanding information and thus improving the overall video understanding accuracy.

[0133] In an exemplary embodiment, step S201 above, which involves segmenting the video to be processed to obtain multiple ordered sub-videos corresponding to the video to be processed, specifically includes the following: segmenting the video to be processed according to a preset video segmentation method to obtain multiple ordered sub-videos corresponding to the video to be processed; the preset video segmentation method includes at least one of the following: a segmentation method based on a preset duration, a segmentation method based on a preset number of frames, a segmentation method based on scene switching, a segmentation method based on key events, and a segmentation method based on semantic content consistency; the video data length corresponding to each sub-video is less than or equal to the input data length of the pre-trained video processing model.

[0134] The preset video segmentation method refers to the method of segmenting the video to be processed, including but not limited to segmentation based on preset duration, segmentation based on preset frame count, segmentation based on scene switching, segmentation based on key events, and segmentation based on semantic content consistency. Segmentation based on preset duration means segmenting the video to be processed according to a preset duration, such as each sub-video having a duration less than or equal to the preset duration. Segmentation based on preset frame count means segmenting the video to be processed according to a preset number of frames, such as each sub-video having a frame count less than or equal to the preset number of frames. Segmentation based on scene switching means segmenting the video to be processed according to a scene, such as one scene corresponding to one sub-video. Segmentation based on key events means segmenting the video to be processed according to key events, such as one key event corresponding to one sub-video. Segmentation based on semantic content consistency means segmenting the video to be processed according to semantic content consistency, such as segmenting video content with consistent semantic content as one sub-video.

[0135] The input data length of the pre-trained video processing model refers to the maximum input data length of the pre-trained video processing model, specifically the upper limit of the single-processing capability of the pre-trained video processing model, such as the maximum number of frames or tokens processed in a single session.

[0136] The video data length corresponding to a sub-video refers to the number of frames or tokens in that sub-video. It should be noted that regardless of the preset video segmentation method used, the video data length of each sub-video is less than or equal to the input data length of the pre-trained video processing model, and the sub-videos are continuous.

[0137] Among them, the pre-trained video processing model refers to the model used to extract the information features of sub-videos and output the memory content corresponding to the sub-video based on the information features of the sub-video and the memory content corresponding to the previous sub-video, such as the multimodal large model.

[0138] For example, when the preset video segmentation method is a segmentation method based on a preset duration, the computer device segments the video to be processed according to the preset duration to obtain multiple ordered sub-videos corresponding to the video to be processed, and the video duration of each sub-video is less than or equal to the preset duration.

[0139] For example, when the preset video segmentation method is a segmentation method based on a preset number of frames, the computer device segments the video to be processed according to the preset number of frames to obtain multiple ordered sub-videos corresponding to the video to be processed, and the number of video frames of each sub-video is less than or equal to the preset number of frames.

[0140] For example, when the preset video segmentation method is a scene-switching segmentation method, the computer device performs scene recognition on the video to be processed to obtain various scenes of the video to be processed, and segments the video to be processed according to each scene to obtain multiple ordered sub-videos corresponding to the video to be processed, and each sub-video corresponds to one scene.

[0141] For example, when the preset video segmentation method is based on key events, the computer device identifies key events in the video to be processed, obtains each key event of the video to be processed, and segments the video to be processed according to each key event to obtain multiple ordered sub-videos corresponding to the video to be processed, and each sub-video corresponds to one key event.

[0142] For example, when the preset video segmentation method is based on semantic content consistency, the computer device performs semantic content consistency recognition processing on the video to be processed, obtains the semantic content consistency recognition result of the video to be processed, and segments the video to be processed according to the semantic content consistency recognition result, resulting in multiple ordered sub-videos corresponding to the video to be processed, and the semantic content within each sub-video is consistent. For example, for a product introduction video, it is divided into sub-video 1 "Product Function Introduction" and sub-video 2 "Product Usage Introduction".

[0143] In this embodiment, the video to be processed is segmented according to a preset video segmentation method, resulting in multiple ordered sub-videos corresponding to the video to be processed. This facilitates the subsequent processing of each sub-video individually, eliminating the need to process the entire video to be processed directly. This avoids the high computational complexity that results from directly processing the entire video to be processed, thus reducing computational complexity. At the same time, the length of the video data corresponding to each sub-video obtained from the segmentation is less than or equal to the input data length of the pre-trained video processing model. This helps to overcome the context window limitation of the model, achieving the goal of processing videos of arbitrary length, and avoiding the defect that performance drops sharply as the video length increases.

[0144] In one exemplary embodiment, such as Figure 4 As shown, step S202 above involves feature extraction processing of the current sub-video to obtain the information features of the current sub-video, including the following steps S401 to S404. Wherein:

[0145] Step S401: Input the current sub-video into the pre-trained video processing model.

[0146] Step S402: Using a pre-trained video processing model, perform visual content extraction processing on the current sub-video to obtain the visual content of the current sub-video.

[0147] Step S403: Perform feature extraction processing on the visual content of the current sub-video to obtain the multimodal feature vector corresponding to the current sub-video.

[0148] Step S404: Based on the multimodal feature vector, obtain the information features of the current sub-video.

[0149] The visual content of the current sub-video refers to the image information extracted from the current sub-video, including but not limited to: objects, people, and scenes in the image; actions, behaviors, and postures; colors, textures, and target positions; changes and motion information between frames; and specific information used to characterize the image targets, scenes, actions, motion changes, and visual semantic information in the current sub-video.

[0150] Among them, the multimodal feature vector refers to the high-dimensional vector representation obtained after feature encoding of the visual content of the current sub-video. It integrates visual features and semantic features and can be used by the model for subsequent feature fusion and understanding generation.

[0151] Here, the information feature of the current sub-video refers to the multimodal feature vector corresponding to the current sub-video.

[0152] For example, the computer device inputs the current sub-video into a pre-trained video processing model. The pre-trained model extracts key video frames from the current sub-video's video frames, then performs visual content extraction processing on the key video frames to obtain their visual content. Next, it fuses the visual content of each key video frame to obtain fused visual content, which serves as the visual content of the current sub-video. Then, it performs feature encoding processing on the visual content of the current sub-video to obtain its visual features. Next, it performs feature mapping processing on the visual features of the current sub-video to obtain mapped features, which serve as the multimodal feature vector corresponding to the current sub-video. Finally, it confirms the multimodal feature vector corresponding to the current sub-video as the information features of the current sub-video.

[0153] In this embodiment, the visual content of the current sub-video is first acquired, and then feature extraction processing is performed on the visual content of the current sub-video to obtain the multimodal feature vector corresponding to the current sub-video. This vector serves as the information feature of the current sub-video, achieving the goal of transforming the visual content of the current sub-video into a multimodal feature vector. This effectively preserves the key visual and semantic information of the current sub-video, providing an accurate and stable feature foundation for the subsequent fusion processing of the memory content corresponding to the preceding sub-video. This is beneficial for improving the reliability and accuracy of the subsequent memory content generation.

[0154] In an exemplary embodiment, step S402 above, which involves extracting visual content from the current sub-video using a pre-trained video processing model to obtain the visual content of the current sub-video, specifically includes the following: performing frame segmentation on the current sub-video using the pre-trained video processing model to obtain video frames of the current sub-video; extracting key video frames from the video frames of the current sub-video; performing visual content extraction on the key video frames to obtain the visual content of the key video frames; and fusing the visual content of each key video frame to obtain the visual content of the current sub-video.

[0155] Among them, key video frames refer to the more important video frames in the current sub-video, specifically those with an importance greater than a preset importance.

[0156] The visual content of key video frames refers to the image information extracted from key video frames, specifically used to represent the image targets, scenes, actions, motion changes, and visual semantic information in key video frames.

[0157] The visual content of the current sub-video refers to the fused visual content obtained after fusing the visual content of each key video frame.

[0158] For example, the computer device performs frame segmentation on the current sub-video using a pre-trained video processing model to obtain video frames of the current sub-video; then, it acquires the video information of the video frames of the current sub-video, performs importance identification processing on the video information of the video frames of the current sub-video to obtain the importance of the video frames of the current sub-video, for example, based on the video information of the video frames of the current sub-video, it queries the correspondence between video information and importance to obtain the importance of the video frames of the current sub-video; next, it identifies video frames from the video frames of the current sub-video whose corresponding importance is greater than a preset importance as key video frames; then, it performs visual content extraction processing on the key video frames to obtain the visual content of the key video frames; finally, it performs fusion processing on the visual content of each key video frame to obtain fused visual content, which is used as the visual content of the current sub-video.

[0159] In this embodiment, a pre-trained video processing model is used to extract key video frames from the video frames of the current sub-video, and the visual content of each key video frame is fused to obtain the visual content of the current sub-video. In this way, when obtaining the visual content of the current sub-video, the visual content of the key frames in the video frames of the current sub-video is comprehensively considered, avoiding information interference from non-key frames, which helps to improve the accuracy of determining the visual content of the current sub-video.

[0160] In an exemplary embodiment, step S403 above, which involves performing feature extraction processing on the visual content of the current sub-video to obtain a multimodal feature vector corresponding to the current sub-video, specifically includes the following: performing feature encoding processing on the visual content of the current sub-video to obtain the visual features of the current sub-video; performing feature mapping processing on the visual features of the current sub-video to obtain the mapped features; and obtaining the multimodal feature vector corresponding to the current sub-video based on the mapped features.

[0161] The visual features of the current sub-video refer to the features obtained after feature encoding of the visual content of the current sub-video.

[0162] In this context, feature mapping of the visual features of the current sub-video refers to mapping the visual features of the current sub-video to a multimodal feature space, so that the mapped features simultaneously integrate visual and semantic features.

[0163] The multimodal feature vector corresponding to the current sub-video refers to the mapped features, specifically the features obtained after feature mapping processing of the visual features of the current sub-video, which integrates visual and semantic features.

[0164] For example, the computer device performs feature encoding on the visual content of the current sub-video using a pre-trained video processing model to obtain the encoded features corresponding to the visual content of the current sub-video, which are used as the visual features of the current sub-video. Then, the visual features of the current sub-video are mapped to a multimodal feature space to obtain the mapped features that simultaneously fuse visual and semantic features. Finally, the mapped features are used as the multimodal feature vector corresponding to the current sub-video.

[0165] In this embodiment, the visual content of the current sub-video is first processed by feature encoding to obtain the visual features of the current sub-video. Then, the visual features of the current sub-video are processed by feature mapping to obtain the multimodal feature vector corresponding to the current sub-video. In this way, by obtaining the multimodal feature vector corresponding to the current sub-video, the key visual and semantic information of the current sub-video can be effectively preserved, providing an accurate and stable feature foundation for the subsequent fusion processing of the memory content corresponding to the preceding sub-video, which is conducive to improving the accuracy of subsequent memory content generation.

[0166] In an exemplary embodiment, step S203 above, obtaining the memory content corresponding to the previous sub-video of the current sub-video, specifically includes the following: obtaining the memory content corresponding to the previous sub-video of the current sub-video, thus obtaining the memory content corresponding to the previous sub-video of the current sub-video.

[0167] Step S203 above involves fusing the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features. Specifically, this includes: inputting the memory content corresponding to the preceding sub-video and the information features of the current sub-video into a pre-trained video processing model; using the pre-trained video processing model to concatenate the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain concatenated information features; and obtaining fused information features based on the concatenated information features.

[0168] Here, the preceding sub-video refers to the previous sub-video; the memory content corresponding to the preceding sub-video of the current sub-video refers to the memory content corresponding to the previous sub-video of the current sub-video.

[0169] Among them, fused information features refer to spliced ​​information features, specifically the information features obtained after splicing the memory content corresponding to the previous sub-video and the information features of the current sub-video. For example, the information features obtained after splicing the memory content corresponding to the previous sub-video and the information features of the current sub-video.

[0170] For example, the computer device obtains the memory content corresponding to the previous sub-video of the current sub-video as the memory content corresponding to the preceding sub-video of the current sub-video; then, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are input together into a pre-trained video processing model. The pre-trained video processing model concatenates the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain concatenated information features, which serve as fused information features; for example, the computer device inputs the memory content corresponding to the previous sub-video of the current sub-video and the information features of the current sub-video together into a pre-trained video processing model. The pre-trained video processing model concatenates the memory content corresponding to the previous sub-video of the current sub-video and the information features of the current sub-video to obtain fused information features.

[0171] In this embodiment, a pre-trained video processing model is used to concatenate the memory content corresponding to the previous sub-video of the current sub-video and the information features of the current sub-video to obtain fused information features. In this way, when obtaining fused information features, the memory content corresponding to the previous sub-video of the current sub-video and the information features of the current sub-video are comprehensively considered, which helps to improve the accuracy of determining fused information features. This makes the memory content generated based on the fused information features more accurate, thereby improving the accuracy of generating memory content corresponding to the current sub-video.

[0172] In an exemplary embodiment, step S204 above, which generates memory content based on fused information features to obtain the memory content corresponding to the current sub-video, specifically includes the following: using a pre-trained video processing model, generating memory content based on fused information features to obtain updated memory content corresponding to the previous sub-video; the content length of the updated memory content is the same as the content length of the memory content corresponding to the previous sub-video; and obtaining the memory content corresponding to the current sub-video based on the updated memory content.

[0173] Step S204 above returns to the step of obtaining the current sub-video from the sub-videos based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained. Specifically, it includes the following: based on the ordered sub-video processing order, the next sub-video of the current sub-video is set as the updated current sub-video for processing, until the memory content corresponding to the last sub-video of the video to be processed is obtained; based on the memory content corresponding to the last sub-video, the memory content of the target sub-video is obtained.

[0174] The updated memory content corresponding to the forward sub-video refers to the memory content obtained by updating the memory content corresponding to the forward sub-video. Specifically, it refers to a completely new, refined, and summarized memory content, and its content length (such as feature dimension) is the same as the content length of the memory content corresponding to the forward sub-video. It should be noted that the content length of the memory content corresponding to each sub-video is the same.

[0175] The memory content corresponding to the current sub-video refers to the updated memory content corresponding to the previous sub-video. Specifically, it is generated based on the fused information features. That is, the memory content corresponding to the current sub-video replaces the memory content corresponding to the previous sub-video, thus achieving the purpose of dynamic memory and overwriting.

[0176] Each time the current sub-video is processed, the next sub-video is set as the updated current sub-video based on the ordered processing order of the sub-videos, and this process continues until the memory content corresponding to the last sub-video is obtained.

[0177] For example, the computer device uses a pre-trained video processing model to perform memory content generation processing based on fused information features, obtaining updated memory content corresponding to the preceding sub-video of the current sub-video, and the content length of the updated memory content is the same as the content length of the memory content corresponding to the preceding sub-video; then, this updated memory content is used as the memory content corresponding to the current sub-video; then, based on the ordered processing order of sub-videos, the next sub-video of the current sub-video is set as the updated current sub-video for processing, until the memory content corresponding to the last sub-video of the video to be processed is obtained; for example, if the current sub-video is the second sub-video, then the third sub-video is used as the updated current sub-video. The video is processed in this manner until the memory content corresponding to the last sub-video of the video to be processed is generated. For example, using the information features of the second sub-video and the memory content corresponding to the first sub-video, the memory content corresponding to the second sub-video is generated; using the information features of the third sub-video and the memory content corresponding to the second sub-video, the memory content corresponding to the third sub-video is generated; using the information features of the fourth sub-video and the memory content corresponding to the third sub-video, the memory content corresponding to the fourth sub-video is generated, and so on, until the memory content corresponding to the last sub-video of the video to be processed is generated; finally, the memory content corresponding to the last sub-video is used as the memory content of the target sub-video.

[0178] In this embodiment, memory content generation processing is performed based on fused information features to obtain updated memory content corresponding to the preceding sub-video, achieving the purpose of dynamic memory and overwriting. Moreover, the content length of the updated memory content is the same as the content length of the memory content corresponding to the preceding sub-video, ensuring that the model's single-step computation and memory usage remain constant when processing videos of arbitrary length, which helps reduce the overall computational complexity. In addition, through iterative processing, the memory content corresponding to the last sub-video is obtained as the memory content of the target sub-video, which helps improve the accuracy of determining the memory content of the target sub-video.

[0179] In one exemplary embodiment, the memory content corresponding to the current sub-video is obtained through processing by a pre-trained video processing model; such as Figure 5 As shown, the video processing method provided in this application also includes a training step for a pre-trained video processing model, specifically including steps S501 to S507. Wherein:

[0180] Step S501: Obtain training data; the training data includes sample videos, sample question information corresponding to the sample videos, and sample answer information corresponding to the sample question information.

[0181] Step S502: Using the video processing model to be trained, global video understanding information of the sample video under different processing sessions is obtained based on multiple ordered sample sub-videos corresponding to the sample video; each processing session is used to represent the process of obtaining a kind of global video understanding information of the sample video.

[0182] Step S503: Based on the global video understanding information of the sample video under different processing sessions, obtain the predicted answer information of the sample question information under different processing sessions.

[0183] Step S504: Based on the predicted answer information and sample answer information under different processing sessions, obtain the reward information corresponding to each processing session of the sample video; the reward information is used to characterize the similarity between the predicted answer information and the sample answer information.

[0184] Step S505: Based on the reward information corresponding to each processing session of the sample video, obtain the generation loss of each memory content generation step in each processing session of the sample video; the memory content generation step is used to characterize the step of generating memory content corresponding to the sample sub-video.

[0185] Step S506: Based on the generation loss of each memory content generation step in each processing session of the sample video, obtain the model loss of the video processing model to be trained.

[0186] Step S507: Adjust the model parameters of the video processing model to be trained based on the model loss until the convergence condition is met, and obtain the pre-trained video processing model.

[0187] Here, sample question information refers to any question information specific to the sample video. Sample answer information refers to the standard answer corresponding to the sample question information.

[0188] Among them, the video processing model to be trained refers to the multimodal large model to be trained.

[0189] The entire process, from initial memory to multiple iterations and updates, until the memory content corresponding to the last sample sub-video of the sample video is generated, constitutes a processing session. Furthermore, different processing sessions correspond to different memory content generation processes. For the same sample video, multiple different processing sessions are generated (e.g., 5 different processing sessions), each used to generate a global video understanding information for the sample video.

[0190] The global video understanding information of the sample video under different processing sessions includes the global video understanding information corresponding to the sample video under each processing session.

[0191] Among them, the predicted answer information of the sample question information in a single processing session refers to the answer information corresponding to the sample question information obtained based on the global video understanding information of the sample video in a single processing session.

[0192] Among them, the reward information corresponding to the processing session is used to characterize the similarity between the predicted answer information and the sample answer information under the processing session. For example, the more similar they are, the higher the value of the reward information.

[0193] The memory content generation step refers to the step of generating the memory content corresponding to the sample sub-video; each sample sub-video corresponds to one memory content generation step; for example, if a sample video includes 5 sample sub-videos, then each processing session of the sample video includes 5 memory content generation steps.

[0194] Among them, the generation loss of the memory content generation step is used to characterize the accuracy of the memory content generated by the memory content generation step, specifically referring to the core loss.

[0195] The model loss of the video processing model to be trained refers to the loss value of the video processing model to be trained, such as the optimization value of the multi-session decoupling asynchronous strategy.

[0196] The convergence condition can be either the total number of training iterations reaching a preset number of training iterations, or the model loss being less than a preset loss value.

[0197] For example, the computer device obtains training data for the video processing model to be trained from a local database, including sample videos, sample question information corresponding to the sample videos, and sample answer information corresponding to the sample question information. Then, using the video processing model to be trained, based on multiple ordered sample sub-videos corresponding to the sample videos, it obtains global video understanding information for the sample videos in a single processing session. Repeating this process multiple times, it obtains global video understanding information for the sample videos in multiple processing sessions. Next, it obtains the semantic information of the sample question information. It then queries the global video understanding information of the sample videos in different processing sessions to obtain matching information of the semantic information in the global video understanding information of different processing sessions. Based on the semantic information... The system generates predicted answer information for sample question information in different processing sessions by matching information from global video understanding information in different processing sessions. Then, based on the similarity between the predicted answer information and the sample answer information in different processing sessions, the system determines the reward information corresponding to each processing session of the sample video. For example, the similarity between the predicted answer information and the sample answer information in a processing session is used as the reward information for that processing session. Next, based on the reward information corresponding to each processing session of the sample video, the system obtains the reward information advantage value for each memory content generation step in each processing session of the sample video. Finally, the system obtains the reward information advantage value for each memory content generation step in each processing session of the sample video. The weights of each memory content generation step are determined, and the reward information advantage value of each memory content generation step in each processing session of the sample video is updated using these weights to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video. The divergence between the current model parameters and the initial model parameters of the video processing model to be trained is obtained. Based on the target reward information advantage value and divergence of each memory content generation step in each processing session of the sample video, the generation loss of each memory content generation step in each processing session of the sample video is obtained. Finally, the words of the memory content generated by each memory content generation step in each processing session of the sample video are obtained. The number of lexical units is fused to obtain a fused lexical unit count. The generation loss of each memory content generation step in each processing session of the sample video is fused to obtain a fused generation loss. Based on the ratio between the fused generation loss and the fused lexical unit count, the model loss of the video processing model to be trained is obtained. Finally, the model parameters of the video processing model to be trained are adjusted based on the model loss to obtain the adjusted video processing model. The adjusted video processing model is then trained again according to the above training process until the total number of training iterations reaches the preset number of training iterations or the model loss is less than the preset loss value. At this point, training is stopped, and the video processing model obtained from the final training is used as the pre-trained video processing model.

[0198] In this embodiment, the video processing model to be trained is iteratively trained using training data, which can improve the accuracy of the memory content output by the trained video processing model, thereby improving the accuracy of memory content generation and ultimately improving the accuracy of video understanding.

[0199] In one exemplary embodiment, such as Figure 6 As shown, step S502 above, through the video processing model to be trained, obtains global video understanding information of the sample video under different processing sessions based on multiple ordered sample sub-videos corresponding to the sample video. Specifically, this includes steps S601 to S606. Wherein:

[0200] Step S601: Based on the ordered processing order of sample sub-videos, obtain the current sample sub-video from the sample sub-videos and input the current sample sub-video into the video processing model to be trained.

[0201] Step S602: The current sample sub-video is processed by the video processing model to be trained to extract features and obtain the information features of the current sample sub-video.

[0202] Step S603: Obtain the memory content corresponding to the preceding sample sub-video of the current sample sub-video, and fuse the memory content corresponding to the preceding sample sub-video and the information features of the current sample sub-video to obtain the sample fusion information features.

[0203] Step S604: Based on the sample fusion information features, perform corresponding memory content generation processing to obtain the memory content corresponding to the current sample sub-video. Return to the processing order based on the ordered sample sub-videos and obtain the current sample sub-video from the sample sub-videos until the memory content corresponding to the last sample sub-video of the sample video is obtained; different processing sessions correspond to different memory content generation processes.

[0204] Step S605: Based on the memory content corresponding to the last sample sub-video, obtain the global video understanding information of the sample video in a single processing session.

[0205] Step S606: Based on the global video understanding information of the sample video in a single processing session, obtain the global video understanding information of the sample video in different processing sessions.

[0206] The information features of the current sample sub-video can refer to the multimodal feature vector of the current sample sub-video.

[0207] The preceding sample sub-video of the current sample sub-video can refer to the previous sample sub-video of the current sample sub-video.

[0208] Among them, the sample fusion information feature refers to the information feature obtained by fusing the memory content corresponding to the forward sample sub-video and the information feature of the current sample sub-video.

[0209] Different processing sessions correspond to different memory content generation processes. This means that each processing session adopts a different memory content generation strategy, and different processing sessions have different memory content generation strategies. The memory content generation strategy is used to characterize the degree of retention of important information and the degree of deletion of irrelevant information when generating memory content. Specifically, it refers to the rules that control how much to retain and how much to delete. These memory content generation strategies can be set randomly.

[0210] Among them, the global video understanding information of the sample video in a single processing session refers to the memory content corresponding to the last sample sub-video of the sample video.

[0211] For example, the computer device obtains the current sample sub-video from the sample sub-videos based on the ordered processing sequence of the sample sub-videos, and inputs the current sample sub-video into the video processing model to be trained; then, the current sample sub-video is segmented into frames by the video processing model to be trained, resulting in sample video frames of the current sample sub-video; key sample video frames are extracted from the sample video frames of the current sample sub-video; visual content extraction processing is performed on the key sample video frames to obtain the visual content of the key sample video frames; the visual content of each key sample video frame is fused to obtain the visual content of the current sample sub-video; then, feature encoding processing is performed on the visual content of the current sample sub-video to obtain the visual features of the current sample sub-video; feature mapping processing is performed on the visual features of the current sample sub-video to obtain the mapped features, which serve as the multimodal feature vector corresponding to the current sample sub-video, and the multimodal feature vector corresponding to the current sample sub-video is confirmed as the information features of the current sample sub-video. Next, the computer device acquires the memory content corresponding to the previous sample sub-video of the current sample sub-video, and inputs the memory content corresponding to the previous sample sub-video of the current sample sub-video, as well as the information features of the current sample sub-video, into the video processing model to be trained. Through the video processing model to be trained, the memory content corresponding to the previous sample sub-video and the information features of the current sample sub-video are fused to obtain the sample fusion information features. Using the corresponding memory content generation strategy, the corresponding memory content generation process is performed based on the sample fusion information features to obtain the memory content corresponding to the current sample sub-video. The process returns to the step of obtaining the current sample sub-video from the sample sub-videos based on the ordered processing order, and processes the next sample sub-video of the current sample sub-video until the memory content corresponding to the last sample sub-video of the sample video is obtained. The memory content corresponding to the last sample sub-video of the sample video is used as the global video understanding information of the sample video in the first processing session. Similarly, by using different memory content generation strategies, the global video understanding information of the sample video in the second processing session can be obtained... the global video understanding information of the sample video in the Gth (G is a positive integer greater than or equal to 2)th processing session; finally, the global video understanding information of the sample video in a single processing session is summarized to obtain the global video understanding information of the sample video in different processing sessions.

[0212] In this embodiment, the video processing model to be trained obtains global video understanding information of the sample video under different processing sessions based on multiple ordered sample sub-videos corresponding to the sample video. This is beneficial for comprehensively considering the global video understanding information of the sample video under different processing sessions when iterating and updating the video processing model in the future, thereby improving the accuracy of the memory content output by the trained video processing model.

[0213] In one exemplary embodiment, such as Figure 7 As shown, step S503 above, based on the global video understanding information of the sample video under different processing sessions, obtains the predicted answer information of the sample question information under different processing sessions, specifically including the following steps S701 to S703. Wherein:

[0214] Step S701: Obtain the semantic information of the sample problem information.

[0215] Step S702: Query the global video understanding information of the sample video under different processing sessions to obtain the matching information of semantic information in the global video understanding information under different processing sessions.

[0216] Step S703: Based on the matching information in the global video understanding information under different processing sessions, generate the predicted answer information of the sample question information under different processing sessions.

[0217] Among them, the semantic information of the sample question information refers to the information obtained after semantic recognition of the sample question information.

[0218] Specifically, querying the global video understanding information of a sample video within a single processing session can yield matching information of semantic information within that single processing session's global video understanding information.

[0219] Among them, the predicted answer information of the sample question information in a single processing session refers to the answer information generated based on the matching information in the global video understanding information in the single processing session, which is based on the semantic information.

[0220] For example, the computer device performs semantic recognition processing on the sample question information to obtain the semantic information of the sample question information; then, based on the semantic information of the sample question information, it queries the global video understanding information of the sample video in the first processing session to obtain the matching information of the semantic information in the global video understanding information in the first processing session, and based on the matching information of the semantic information in the global video understanding information in the first processing session, it obtains the predicted answer information of the sample question information in the first processing session; next, based on the semantic information of the sample question information, it queries the global video understanding information of the sample video in the second processing session to obtain the matching information of the semantic information in the global video understanding information in the second processing session, and based on the matching information of the semantic information in the global video understanding information in the second processing session, it obtains the predicted answer information of the sample question information in the second processing session; and so on, the predicted answer information of the sample question information in the last processing session can be obtained, and the predicted answer information of the sample question information in a single processing session is summarized to obtain the predicted answer information of the sample question information in different processing sessions.

[0221] In this embodiment, based on the global video understanding information of the sample video under different processing sessions, the predicted answer information of the sample question information under different processing sessions is obtained. This is beneficial for the subsequent iteration and update of the video processing model to comprehensively consider the predicted answer information of the sample question information under different processing sessions, thereby improving the accuracy of the memory content output by the trained video processing model and further improving the accuracy of video understanding.

[0222] In an exemplary embodiment, step S504 above, which obtains reward information corresponding to each processing session of the sample video based on the predicted answer information and sample answer information of the sample question information under different processing sessions, specifically includes the following: when the sample question information belongs to an objective question answering task, obtaining the consistency comparison results between the predicted answer information and sample answer information of the sample question information under different processing sessions; and determining the reward information corresponding to each processing session of the sample video based on the consistency comparison results between the predicted answer information and sample answer information of the sample question information under different processing sessions.

[0223] Objective question-and-answer tasks refer to those with clear and unique answer information and no subjective interpretation, such as what colors the protagonist wears at the beginning and end of a movie, what the protagonist's position is in a TV series, how many championships the protagonist wins in a movie, and how many people appear in a video.

[0224] The consistency comparison result indicates whether the predicted answer information of the sample question information is consistent with the sample answer information in a single processing session. If they are consistent, the reward information corresponding to the single processing session of the sample video is 1; if they are inconsistent, the reward information corresponding to the single processing session of the sample video is 0.

[0225] For example, the computer device performs task type identification processing on the sample question information to obtain the task type corresponding to the sample question information; then, it determines whether the task type corresponding to the sample question information belongs to an objective question-answering task; if the task type of the sample question information belongs to an objective question-answering task, the predicted answer information and the sample answer information of the sample question information under different processing sessions are compared to obtain the consistency comparison result between the predicted answer information and the sample answer information under different processing sessions; then, based on the consistency comparison result between the predicted answer information and the sample answer information under different processing sessions, the reward information corresponding to each processing session of the sample video is determined; for example, if the sample question information is "How many championships did the protagonist win in the movie?" and the sample answer information is "3", if the predicted answer information of the sample question information under the first processing session is "3", which is consistent with the sample answer information, then the reward information corresponding to the first processing session of the sample video is 1; if the predicted answer information of the sample question information under the second processing session is "2", which is inconsistent with the sample answer information, then the reward information corresponding to the second processing session of the sample video is 0.

[0226] In this embodiment, when the sample question information belongs to an objective question-answering task, the reward information corresponding to each processing session of the sample video is determined based on the consistency comparison results between the predicted answer information and the sample answer information under different processing sessions. In this way, when determining the reward information corresponding to each processing session of the sample video, the task type of the sample question information is comprehensively considered, that is, different reward information determination methods are adopted based on different task types, which is conducive to improving the accuracy of determining the reward information corresponding to the processing session of the sample question information.

[0227] In an exemplary embodiment, step S504 above, which obtains reward information corresponding to each processing session of the sample video based on the predicted answer information and sample answer information of the sample question information under different processing sessions, specifically includes the following: when the sample question information belongs to a generative task, the semantic similarity between the predicted answer information and sample answer information of the sample question information under different processing sessions is obtained respectively; based on the semantic similarity between the predicted answer information and sample answer information of the sample question information under different processing sessions, the reward information corresponding to each processing session of the sample video is determined.

[0228] Generative tasks refer to tasks that require generating a piece of natural language text as the output, such as what the cat in the video is doing; please describe the main activities of the protagonist in the video today; what are the three improvements of the new product mentioned in the video compared to the old model.

[0229] Semantic similarity is used to represent the degree of semantic similarity between the predicted answer and the sample answer in a single processing session, such as 0.9, 0.5, or 0.3. The reward information for a single processing session of the sample video can refer to the semantic similarity between the predicted answer and the sample answer in that session. For example, if the semantic similarity between the predicted answer and the sample answer in a single processing session is 0.9, then the reward information for that single processing session of the sample video is 0.9.

[0230] For example, the computer device performs task type identification processing on the sample question information to obtain the task type corresponding to the sample question information; then, it determines whether the task type corresponding to the sample question information belongs to a generative task; if the task type of the sample question information belongs to a generative task, it obtains the semantic features of the predicted answer information of the sample question information under different processing sessions, and obtains the semantic features of the sample answer information; then, based on the semantic features of the predicted answer information of the sample question information under different processing sessions, and the semantic features of the sample answer information, it calculates the cosine similarity between the predicted answer information of the sample question information and the sample answer information under different processing sessions, as the semantic similarity between the predicted answer information of the sample question information and the sample answer information under different processing sessions; finally, the semantic similarity between the predicted answer information of the sample question information and the sample answer information under different processing sessions is used as the reward information corresponding to each processing session of the sample video. For example, if the semantic similarity between the predicted answer and the sample answer in the first processing session is 0.9, it means that the reward information for the first processing session of the sample video is 0.9; if the semantic similarity between the predicted answer and the sample answer in the second processing session is 0.6, it means that the reward information for the second processing session of the sample video is 0.6.

[0231] In this embodiment, when the sample question information belongs to a generative task, the reward information corresponding to each processing session of the sample video is determined based on the semantic similarity between the predicted answer information and the sample answer information under different processing sessions. In this way, when determining the reward information corresponding to each processing session of the sample video, the task type of the sample question information is comprehensively considered, that is, different reward information determination methods are adopted based on different task types, which is conducive to improving the accuracy of determining the reward information corresponding to the processing session of the sample question information.

[0232] In one exemplary embodiment, such as Figure 8As shown, step S506 above, based on the generation loss of each memory content generation step in each processing session of the sample video, yields the model loss of the video processing model to be trained, specifically including the following steps S801 to S803. Wherein:

[0233] Step S801: Obtain the number of lexical units of the memory content generated in each memory content generation step in each processing session of the sample video.

[0234] Step S802: Perform fusion processing on the number of lexical units to obtain the fused number of lexical units. Perform fusion processing on the generation loss of each memory content generation step in each processing session of the sample video to obtain the fused generation loss.

[0235] Step S803: Based on the ratio between the fusion generation loss and the number of fusion tokens, the model loss of the video processing model to be trained is obtained.

[0236] The number of tokens in the memory content refers to the number of tokens corresponding to the memory content. It should be noted that for the same processing session, the number of tokens in the memory content generated by each memory content generation step within that session is the same. However, the number of tokens in the memory content generation steps differs across different processing sessions.

[0237] The number of fused lexical units refers to the sum of the number of lexical units in each memory content generation step in each processing session of the sample video. Specifically, it refers to the sum of the number of lexical units in each memory content generation step in each processing session of all sample videos.

[0238] Among them, the fusion generation loss refers to the sum of the generation loss of each memory content generation step in each processing session of the sample video, specifically the sum of the generation loss of each memory content generation step in each processing session of all sample videos.

[0239] In this context, the model loss of the video processing model to be trained refers to the ratio between the fusion generation loss and the number of fusion tokens.

[0240] For example, the computer device performs word count recognition processing on the memory content generated in each memory content generation step in each processing session of the sample video to obtain the word count of the memory content generated in each memory content generation step in each processing session of the sample video. Then, the word counts of the memory content generated in each memory content generation step in each processing session of the sample video are added together to obtain the fused word count. Next, the generation loss of each memory content generation step in each processing session of the sample video is added together to obtain the fused generation loss. Finally, the ratio between the fused generation loss and the fused word count is obtained, and this ratio is used as the model loss of the video processing model to be trained.

[0241] For example, the key to this application lies in how to train the memory agent module (i.e., the multimodal large model) to learn to intelligently update the memory; this application uses a reinforcement learning framework optimized for agent workflow for end-to-end training.

[0242] Among them, the policy: the memory agent module itself is the policy. , by parameters Decision; its function is based on the current state. Generate an action ,Right now .

[0243] Wherein, State: at step k, the state is... From the current video block The multimodal feature vectors and the memory of the previous step (i.e., k-1 steps). Composition, that is .

[0244] Wherein, Action: At step k, the action... It is to generate new memories (i.e., the memory of step k), that is .

[0245] Among them, Processing Conversation: from initial memory Initially, after N iterations and updates, until the final memory is generated. The complete process of (i.e., the memory of step N, representing the memory of the final step) constitutes a processing session. Each session contains a series of states and actions.

[0246] Among them, the reward function is the reward. It is based on the final memory after the session ends. The utility is calculated. Given a question about the global content of the video. and standard answer The model utilizes final memory Generate predicted answers Reward function Defined as: ;in, For objective question answering, the function is a binary function (1 for correct and 0 for incorrect); for generative tasks, it is a continuous score based on semantic similarity (such as BLEU).

[0247] Among them, the policy optimization algorithm: To efficiently optimize the policy, this application adopts an improved policy optimization algorithm—Multi-Conversation Decoupled Asynchronous Policy Optimization (Multi-Conv DAPO). The core idea of ​​this algorithm is to generate a set (G) of different processing sessions for the same input video and question during training, and calculate the advantage based on the final reward of the entire set of sessions. Then, this advantage signal is backpropagated to optimize each memory update step. Its optimization objective is... (The model loss of the video processing model to be trained) is defined as follows:

[0248] ;

[0249] in, Represents the mathematical expectation; This represents a training sample i (video and corresponding question and answer); This represents the training dataset, where training sample i belongs to this training dataset; This indicates the old strategy; Indicating the old strategy Below, the memory content sequence of each processing session obtained from sampling, that is, the memory content sequence of the j-th processing session of the i-th training sample, for example, the memory content sequence of the first processing session of the first training sample is (memory content 111, memory content 112, memory content 113, memory content 114, memory content 115), a total of 5 memory contents, provided that the video is segmented into 5 sub-videos (i.e., 5 video blocks); a training sample i will generate G groups of processing sessions (G is a positive integer greater than or equal to 2), each processing session includes Each token-level output corresponds to the j-th processing session of the i-th training sample. It equals the sum of the number of lexical units in the memory content generated by each memory content generation step in the processing session; The core loss function at token t (i.e., memory content generation step t) is defined as the generation loss corresponding to the t-th memory content generation step in the j-th processing session of the i-th training sample:

[0250] ;

[0251] in, The KL divergence (i.e., the divergence between the current model parameters of the video processing model to be trained and the initial model parameters of the video processing model to be trained) is used to penalize the current policy. (That is, the current model parameters of the video processing model to be trained, specifically the current model parameters of a multimodal large model) relative to a reference policy (i.e., the excessive deviation of the initial model parameters of the video processing model to be trained, specifically the initial model parameters of the pre-trained model, i.e., the initial model parameters of the multimodal large model); This represents the KL penalty coefficient.

[0252] in, The pruned advantage (i.e., the advantage value of the target reward information corresponding to the t-th memory content generation step in the j-th processing session of the i-th training sample) is defined as:

[0253] ;

[0254] in, The Importance Sampling Weight represents the weight corresponding to the t-th memory content generation step in the j-th processing session of the i-th training sample. It measures the probability ratio of the old and new strategies in generating a specific memory token, and is defined as follows:

[0255] ;

[0256] in, This represents the memory content corresponding to the t-th memory content generation step in the j-th processing session of the i-th training sample; This represents the memory content corresponding to all memory content generation steps preceding the t-th memory content generation step in the j-th processing session of the i-th training sample.

[0257] in, The group-normalized advantage, also known as the advantage value of reward information corresponding to the t-th memory content generation step in the j-th processing session of the i-th training sample, is equal to the difference between the reward information corresponding to the j-th processing session of the i-th training sample and the average reward information corresponding to all processing sessions of the i-th training sample. It is defined as follows:

[0258] ;

[0259] in, It is the reward finally obtained by training sample i (containing G processing sessions), used to represent the reward corresponding to a single processing session in training sample i, that is, the reward corresponding to the j-th processing session in training sample i, and j is less than or equal to G; This represents the average reward across the G processing sessions for training sample i. The advantage function is calculated independently of the quality of individual sessions; instead, it compares the final reward of each processing session in the sample (containing G processing sessions) with the average reward of all G sessions in the same batch (i.e., the same video and question). This advantage value... This will be uniformly applied to all processing sessions j and all memory update steps t generated by sample i. This approach can more stably evaluate the merits of the policy and encourages the model to explore memory update paths with higher rewards.

[0260] in, and All represent preset coefficients; This represents the pruning function, used to adjust the importance sampling weights. Limit to a preset range This design prevents excessively large step sizes in a single update, thus ensuring training stability. It is a key design feature of the PPO (Proximal Policy Optimization) series of algorithms.

[0261] Thus, by maximizing Memory agents can gradually learn an efficient memory management strategy, that is, to make the memory update decision that is most conducive to obtaining high rewards at each step, thereby achieving the best balance between retaining key information and forgetting secondary information.

[0262] In this embodiment, the number of lexical units generated in each memory content generation step of each processing session of the sample video is fused to obtain a fused lexical unit count. The generation loss of each memory content generation step in each processing session of the sample video is also fused to obtain a fused generation loss. Based on the ratio between the fused generation loss and the fused lexical unit count, the model loss of the video processing model to be trained is obtained. Thus, when determining the model loss, considering both the number of lexical units generated in each memory content generation step of each processing session of the sample video and the generation loss of each memory content generation step in each processing session of the sample video improves the accuracy of model loss determination and facilitates accurate updating of the model parameters of the video processing model, thereby improving the accuracy of the memory content output by the trained video processing model.

[0263] In one exemplary embodiment, such as Figure 9 As shown, step S505 above, based on the reward information corresponding to each processing session of the sample video, obtains the generation loss for each memory content generation step in each processing session of the sample video, specifically including the following steps S901 to S903. Wherein:

[0264] Step S901: Based on the reward information corresponding to each processing session of the sample video, obtain the reward information advantage value of each memory content generation step in each processing session of the sample video; the reward information advantage value of the memory content generation step is used to characterize the quality of the memory content generated by the memory content generation step.

[0265] Step S902: Obtain the weight of each memory content generation step in each processing session of the sample video, and update the reward information advantage value of each memory content generation step in each processing session of the sample video using the weights to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video.

[0266] Step S903: Obtain the divergence of the current model parameters of the video processing model to be trained relative to the initial model parameters of the video processing model to be trained. Based on the target reward information advantage value and divergence of each memory content generation step in each processing session of the sample video, obtain the generation loss of each memory content generation step in each processing session of the sample video.

[0267] The reward information advantage value refers to the group normalized advantage, used to characterize the quality of the memory content generated in the memory content generation step. Furthermore, the reward information advantage value is the same for each memory content generation step within the same processing session of the sample video.

[0268] The weight of the memory content generation step refers to the importance sampling weight, which is used to measure the importance of the memory content generation step.

[0269] Among them, the target reward information advantage value refers to the updated reward information advantage value, specifically the pruned advantage item, that is, the pruned reward information advantage value.

[0270] The initial model parameters of the video processing model to be trained refer to the model parameters of the video processing model to be trained before training.

[0271] Here, divergence refers to KL (Kullback-Leibler) divergence, also known as relative entropy, which characterizes the degree of deviation of the current model parameters (corresponding to the current policy) of the video processing model to be trained from the initial model parameters (corresponding to the reference policy).

[0272] In this context, the generation loss for each memory content generation step in each processing session of the sample video can refer to the difference between the target reward information advantage value and the divergence for each memory content generation step in each processing session of the sample video.

[0273] For example, the computer device acquires the average reward information of the reward information corresponding to each processing session of the sample video; acquires the difference between the reward information corresponding to each processing session of the sample video and the average reward information; uses the difference between the reward information corresponding to each processing session of the sample video and the average reward information as the reward information advantage value of each memory content generation step in each processing session of the sample video; then, under the current model parameters of the video processing model to be trained, based on the sample question information and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, determines the first generation probability of the memory content corresponding to each memory content generation step in each processing session; under the historical model parameters of the video processing model to be trained, based on the sample question information and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, determines the second generation probability of the memory content corresponding to each memory content generation step in each processing session; based on the ratio between the first generation probability and the second generation probability, obtains the reward information advantage value of each memory content generation step in each processing session of the sample video. The weights of the memory content generation steps are determined. Then, the weights are adjusted using preset coefficients to obtain the adjusted weights for each memory content generation step in each processing session of the sample video. Using the weights and adjusted weights, the reward information advantage value of each memory content generation step in each processing session of the sample video is updated first and second, respectively, to obtain the first and second reward information advantage values ​​for each memory content generation step in each processing session of the sample video. The minimum of the first and second reward information advantage values ​​is taken as the target reward information advantage value for each memory content generation step in each processing session of the sample video. Finally, the KL divergence calculation method is used to calculate the divergence between the current model parameters of the video processing model to be trained and the initial model parameters, and the preset divergence penalty coefficient is multiplied by the divergence to obtain the target divergence. The difference between the target reward information advantage value and the target divergence for each memory content generation step in each processing session of the sample video is obtained as the generation loss for each memory content generation step in each processing session of the sample video.

[0274] In this embodiment, when determining the generation loss of each memory content generation step in each processing session of the sample video, the divergence of the current model parameters of the video processing model to be trained relative to the initial model parameters of the video processing model to be trained, as well as the advantage value of the target reward information of each memory content generation step in each processing session of the sample video, are comprehensively considered. This is beneficial to improving the accuracy of determining the generation loss, facilitating the accurate updating of the model parameters of the video processing model in the future, thereby improving the accuracy of the memory content output by the trained video processing model, and further improving the accuracy of video understanding.

[0275] In an exemplary embodiment, step S901, which obtains the reward information advantage value of each memory content generation step in each processing session of the sample video based on the reward information corresponding to each processing session of the sample video, specifically includes the following: obtaining the average reward information of the reward information corresponding to each processing session of the sample video; obtaining the difference between the reward information corresponding to each processing session of the sample video and the average reward information; and obtaining the reward information advantage value of each memory content generation step in each processing session of the sample video based on the difference between the reward information corresponding to each processing session of the sample video and the average reward information.

[0276] The average reward information refers to the average reward information for each processing session of the sample video.

[0277] In this context, the reward information advantage value for each memory content generation step in the processing session of the sample video is equal to the difference between the reward information for that processing session and the average reward information. For example, the reward information advantage value for each memory content generation step in the first processing session of the sample video is equal to the difference between the reward information for the first processing session and the average reward information.

[0278] For example, the computer device averages the reward information corresponding to each processing session of the sample video to obtain the average reward information for each processing session of the sample video. Then, it obtains the difference between the reward information corresponding to each processing session of the sample video and the average reward information, and uses the difference between the reward information corresponding to each processing session of the sample video and the average reward information as the reward information advantage value of each memory content generation step in each processing session of the sample video. For example, if the reward information corresponding to the first processing session of the sample video is 0.9 and the average reward information is 0.6, then the reward information advantage value of each memory content generation step in the first processing session of the sample video is equal to 0.3 (i.e., 0.9-0.6); if the reward information corresponding to the second processing session of the sample video is 0.8 and the average reward information is 0.6, then the reward information advantage value of each memory content generation step in the second processing session of the sample video is equal to 0.2 (i.e., 0.8-0.6).

[0279] In this embodiment, based on the reward information corresponding to each processing session of the sample video, the reward information advantage value of each memory content generation step in each processing session of the sample video is obtained. This facilitates the comprehensive consideration of the reward information advantage value of each memory content generation step in each processing session of the sample video when determining the generation loss of each memory content generation step in each processing session of the sample video. This helps to improve the accuracy of determining the generation loss and facilitates the accurate updating of the model parameters of the video processing model, thereby improving the accuracy of the memory content output by the trained video processing model.

[0280] In an exemplary embodiment, step S902, obtaining the weight of each memory content generation step in each processing session of the sample video, specifically includes the following: under the current model parameters of the video processing model to be trained, based on the sample question information and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, a first generation probability of the memory content corresponding to each memory content generation step in each processing session is determined; under the historical model parameters of the video processing model to be trained, based on the sample question information and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, a second generation probability of the memory content corresponding to each memory content generation step in each processing session is determined; based on the ratio between the first generation probability and the second generation probability, the weight of each memory content generation step in each processing session of the sample video is obtained.

[0281] The historical model parameters of the video processing model to be trained refer to the old parameters of the model, such as the model parameters obtained in the previous training round. During the same training round (which involves multiple model parameter iterations), the historical model parameters of the video processing model to be trained are fixed, and initially, the current model parameters are equal to the historical model parameters. However, after each iteration, the current model parameters are updated and differ from the historical model parameters.

[0282] The weights of each memory content generation step in each processing session of the sample video are used to measure the probability ratio of the current model parameters and the historical model parameters of the video processing model to be trained in generating specific memory content. Specifically, this measures the probability ratio of the new and old strategies in generating specific memory tokens. The new strategy corresponds to the current model parameters of the video processing model to be trained, and the old strategy corresponds to the historical model parameters of the video processing model to be trained.

[0283] In this context, the weight of each memory content generation step in each processing session of the sample video is equal to the ratio between the first generation probability and the second generation probability.

[0284] Among them, the historical memory content generation steps refer to all memory content generation steps that precede the memory content generation steps.

[0285] Wherein, the first generation probability represents the conditional probability of generating the memory content corresponding to each memory content generation step in each processing session, given the sample problem information and the memory content corresponding to the historical memory content generation step in each memory content generation step in each processing session, under the current model parameters of the video processing model to be trained.

[0286] The second generation probability represents the conditional probability of generating the memory content corresponding to each memory content generation step in each processing session, given the sample problem information and the memory content corresponding to each memory content generation step in each processing session, under the historical model parameters of the video processing model to be trained.

[0287] For example, the computer device obtains the current model parameters of the video processing model to be trained, sample problem information, and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, and generates a conditional probability of the memory content corresponding to each memory content generation step in each processing session, as a first generation probability; then, it obtains the historical model parameters of the video processing model to be trained, sample problem information, and the memory content corresponding to the historical memory content generation steps of each memory content generation step in each processing session, and generates a conditional probability of the memory content corresponding to each memory content generation step in each processing session, as a second generation probability; finally, it obtains the ratio between the first generation probability and the second generation probability as the weight of each memory content generation step in each processing session of the sample video.

[0288] In this embodiment, the weight of each memory content generation step in each processing session of the sample video is obtained. This facilitates the comprehensive consideration of the weight of each memory content generation step in each processing session of the sample video when determining the generation loss of each memory content generation step in each processing session. This helps to improve the accuracy of determining the generation loss and facilitates the accurate updating of the model parameters of the video processing model, thereby improving the accuracy of the memory content output by the trained video processing model.

[0289] In an exemplary embodiment, step S902, which updates the reward information advantage value of each memory content generation step in each processing session of the sample video using weights to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video, specifically includes the following: adjusting the weights using preset coefficients to obtain the adjusted weights of each memory content generation step in each processing session of the sample video; performing a first update process and a second update process on the reward information advantage value of each memory content generation step in each processing session of the sample video using the weights and the adjusted weights respectively to obtain the first reward information advantage value and the second reward information advantage value of each memory content generation step in each processing session of the sample video; and obtaining the target reward information advantage value of each memory content generation step in each processing session of the sample video based on the first reward information advantage value and the second reward information advantage value.

[0290] The preset coefficients include the first preset coefficient and the second preset coefficient; adjusting the weights using the preset coefficients means limiting the weights to a preset weight range [1 - the first preset coefficient, 1 + the second preset coefficient].

[0291] The adjusted weight of each memory content generation step in each processing session of the sample video refers to the weight adjusted using preset coefficients.

[0292] The first update process involves multiplying the weight by the advantage value of the reward information; the second update process involves multiplying the adjusted weight by the advantage value of the reward information.

[0293] The first reward information advantage value refers to the product of the weight and the reward information advantage value; the second reward information advantage value refers to the product of the adjusted weight and the reward information advantage value.

[0294] In this context, the target reward information advantage value for each memory content generation step in each processing session of the sample video refers to the minimum value between the first reward information advantage value and the second reward information advantage value.

[0295] For example, the computer device uses a first preset coefficient and a second preset coefficient to determine a preset weight range [1 - first preset coefficient, 1 + second preset coefficient]. Then, it restricts the weights within this preset weight range to obtain the adjusted weights for each memory content generation step in each processing session of the sample video. For instance, if the weight is less than (1 - first preset coefficient), the adjusted weight = 1 - first preset coefficient; if the weight is greater than (1 + second preset coefficient), the adjusted weight = 1 + second preset coefficient; if the weight is within the preset weight range [1 - first preset coefficient, 1 + second preset coefficient], the adjusted weight = the original weight (i.e., unchanged). Then, each processing session of the sample video... The reward information advantage value of each memory content generation step in each processing session of the sample video is multiplied by its corresponding weight to obtain the first reward information advantage value of each memory content generation step in each processing session of the sample video. Next, the reward information advantage value of each memory content generation step in each processing session of the sample video is multiplied by its corresponding adjusted weight to obtain the second reward information advantage value of each memory content generation step in each processing session of the sample video. Finally, the minimum value between the first reward information advantage value and the second reward information advantage value of each memory content generation step in each processing session of the sample video is obtained as the target reward information advantage value (i.e., the pruned advantage) of each memory content generation step in each processing session of the sample video.

[0296] In this embodiment, the reward information advantage value of each memory content generation step in each processing session of the sample video is updated using weights to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video. This facilitates the comprehensive consideration of the target reward information advantage value of each memory content generation step in each processing session of the sample video when determining the generation loss of each memory content generation step in each processing session of the sample video. This helps to improve the accuracy of determining the generation loss and facilitates the accurate updating of the model parameters of the video processing model, thereby improving the accuracy of the memory content output by the trained video processing model.

[0297] In an exemplary embodiment, step S205 above, after obtaining the global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video, further includes a step of obtaining summary information corresponding to the summary information generation task. Specifically, it includes the following: obtaining candidate information corresponding to the summary information generation task for the video to be processed from the global video understanding information corresponding to the video to be processed; performing fusion processing on the candidate information to obtain the summary information corresponding to the summary information generation task.

[0298] The summary information generation task refers to the task of generating summary information for the video to be processed. Specifically, it refers to the task of generating summary information that summarizes the core content of the video based on the global video understanding information. For example, generating a content synopsis for news videos ("extract key content from the global video understanding information and generate a 1-2 sentence news synopsis") and generating a summary of main events for documentary videos ("summarize the main events that occurred in the video and generate a brief summary"). It should be noted that the computer device can acquire the summary information generation task for the video to be processed during the processing of the video, before the processing of the video to be processed, or after the processing of the video to be processed is completed.

[0299] Candidate information refers to information in the global video understanding information corresponding to the task of generating summary information for the video to be processed, such as key information.

[0300] The summary information refers to the text information that is output in the final product and can concisely summarize the core content of the video to be processed, such as "Today, our city held a science and technology innovation conference, at which a number of new technological achievements were released and outstanding scientific research teams were commended" or "The blogger went to the supermarket to buy ingredients in the morning and made and tasted a homemade cake at home in the afternoon. The process was easy and enjoyable."

[0301] For example, in a video-based summary information generation scenario, a computer device acquires a video to be processed, segments the video to be processed, and obtains multiple ordered sub-videos corresponding to the video to be processed; then, based on the processing order of the ordered sub-videos, the current sub-video is obtained from the sub-videos; next, the current sub-video is processed into frames using a pre-trained video processing model to obtain video frames of the current sub-video; then, key video frames are extracted from the video frames of the current sub-video; then, visual content extraction processing is performed on the key video frames to obtain the visual content of the key video frames; then, the visual content of each key video frame is fused to obtain the visual content of the current sub-video; then, feature encoding processing is performed on the visual content of the current sub-video to obtain the visual features of the current sub-video; then, feature mapping processing is performed on the visual features of the current sub-video to obtain the mapped features, which serve as the multimodal feature vector corresponding to the current sub-video, and the multimodal feature vector corresponding to the current sub-video is confirmed as the information features of the current sub-video. Next, the computer device obtains the memory content corresponding to the previous sub-video of the current sub-video, and inputs the memory content of the previous sub-video and the information features of the current sub-video into a pre-trained video processing model. The pre-trained video processing model fuses the memory content of the previous sub-video and the information features of the current sub-video to obtain fused information features. Based on the fused information features, memory content generation is performed to obtain the memory content corresponding to the current sub-video. The process then returns to the step of obtaining the current sub-video from the sub-videos in the ordered sub-video processing order, until the memory content corresponding to the last sub-video of the video to be processed is obtained. Finally, the last sub-video of the video to be processed is... The memory content corresponding to a sub-video is used as the global video understanding information for the video to be processed. Finally, a summary information generation task for the video to be processed is obtained, such as "extracting key content from the global video understanding information and generating a news summary of 1-2 sentences". Based on the summary information generation task, the global video understanding information corresponding to the video to be processed is queried to obtain information that matches the summary information generation task for the video to be processed, which is used as candidate information. Then, the candidate information is fused to obtain fused candidate information, which is used as the summary information for the summary information generation task, such as "Today, our city held a science and technology innovation conference, at which many new technological achievements were released and outstanding scientific research teams were commended".

[0302] For example, in a video-based question-and-answer scenario, a computer device acquires a video to be processed, segments the video to be processed, and obtains multiple ordered sub-videos corresponding to the video to be processed. Then, based on the processing order of the ordered sub-videos, the current sub-video is obtained from the sub-videos. Next, the current sub-video is processed into frames using a pre-trained video processing model to obtain video frames of the current sub-video. Key video frames are extracted from the video frames of the current sub-video. Visual content extraction processing is performed on the key video frames to obtain the visual content of the key video frames. The visual content of each key video frame is fused to obtain the visual content of the current sub-video. Next, feature encoding processing is performed on the visual content of the current sub-video to obtain the visual features of the current sub-video. Feature mapping processing is performed on the visual features of the current sub-video to obtain the mapped features, which serve as the multimodal feature vector corresponding to the current sub-video. The multimodal feature vector corresponding to the current sub-video is then identified as the information feature of the current sub-video. Next, the computer device acquires the memory content corresponding to the previous sub-video of the current sub-video, and inputs the memory content of the previous sub-video and the information features of the current sub-video into a pre-trained video processing model. The pre-trained video processing model fuses the memory content of the previous sub-video and the information features of the current sub-video to obtain fused information features. Based on the fused information features, memory content generation is performed to obtain the memory content corresponding to the current sub-video. The process then returns to the step of retrieving the current sub-video from the sub-videos based on the ordered sub-video processing sequence, and processes the next sub-video of the current sub-video until the memory content corresponding to the last sub-video of the video to be processed is obtained. This last sub-video's memory content is used as the global video understanding information for the video to be processed. Finally, the question information for the video to be processed, such as "What are the colors of the protagonist's clothing at the beginning and end of the movie?", is obtained, along with the target semantic information of the question. The global video understanding information corresponding to the video to be processed is queried to obtain the target matching information of the target semantic information within the global video understanding information. Based on the target matching information, the answer information corresponding to the question information, such as "red and blue", is generated.

[0303] For example, in a video classification scenario, a computer device acquires a video to be processed, segments the video to be processed, and obtains multiple ordered sub-videos corresponding to the video to be processed. Then, based on the processing order of the ordered sub-videos, the current sub-video is obtained from the sub-videos. Next, the current sub-video is processed into frames using a pre-trained video processing model to obtain video frames of the current sub-video. Key video frames are extracted from the video frames of the current sub-video. Visual content extraction processing is performed on the key video frames to obtain the visual content of the key video frames. The visual content of each key video frame is fused to obtain the visual content of the current sub-video. Next, feature encoding processing is performed on the visual content of the current sub-video to obtain the visual features of the current sub-video. Feature mapping processing is performed on the visual features of the current sub-video to obtain the mapped features, which serve as the multimodal feature vector corresponding to the current sub-video. The multimodal feature vector corresponding to the current sub-video is then identified as the information feature of the current sub-video. Next, the computer device acquires the memory content corresponding to the previous sub-video of the current sub-video, and inputs the memory content of the previous sub-video and the information features of the current sub-video into a pre-trained video processing model. The pre-trained video processing model fuses the memory content of the previous sub-video and the information features of the current sub-video to obtain fused information features. Based on the fused information features, memory content generation is performed to obtain the memory content corresponding to the current sub-video. The process then returns to the step of retrieving the current sub-video from the sub-videos based on the ordered sub-video processing sequence, and processes the next sub-video of the current sub-video until the memory content corresponding to the last sub-video of the video to be processed is obtained. This last sub-video's memory content is used as the global video understanding information for the video to be processed. Finally, a video classification task for the video to be processed is obtained, such as "Please determine the category of the video," and the global video understanding information corresponding to the video to be processed is queried based on the video classification task to obtain the category information of the video to be processed, such as animal videos. For example, after obtaining global video understanding information for multiple videos to be processed, video category prediction or classification is performed on each video based on the global video understanding information for each video to be processed, and the video category to which each video to be processed belongs is obtained.

[0304] In this embodiment, summary information corresponding to the summary information generation task is generated based on the global video understanding information corresponding to the video to be processed. By comprehensively considering the global video understanding information corresponding to the video to be processed, the generated summary information can be more accurate, thereby improving the accuracy of summary information generation. At the same time, there is no need to rescan and analyze the entire video to be processed, which helps to simplify the summary information generation process and thus improves the efficiency of summary information generation.

[0305] In one exemplary embodiment, such as Figure 10As shown, another video processing method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps S1001 to S1002. Wherein:

[0306] Step S1001: Display the video to be processed on the video analysis page.

[0307] Step S1002: In response to the global video understanding event for the video to be processed, display the global video understanding information corresponding to the video to be processed on the video analysis page.

[0308] The global video understanding information is obtained according to the first video processing method. For details, please refer to the relevant embodiments of the first video processing method, which will not be repeated here.

[0309] The video analysis page refers to the page that performs intelligent analysis on videos, such as generating global video understanding information for the video to be processed, answering questions, generating summary information, and executing downstream tasks (such as video classification tasks and video review tasks).

[0310] The videos to be processed can be uploaded by users or obtained from the internet.

[0311] Among them, the global video understanding event refers to the global video understanding operation of the video to be processed. By triggering the global video understanding operation for the video to be processed, the global video understanding information corresponding to the video to be processed can be requested to be generated.

[0312] For example, in response to an upload operation for a video to be processed, the terminal displays the uploaded video on the video analysis page; then, in response to a global video understanding operation for the video to be processed, it generates global video understanding information corresponding to the video to be processed, or requests the server to generate global video understanding information corresponding to the video to be processed, and displays the global video understanding information corresponding to the video to be processed on the video analysis page. For further details, refer to... Figure 11 The user selects the video to be uploaded (A) in the message sending area and clicks the "Send" button, triggering the upload operation for video A. The terminal responds to this upload operation by displaying video A on the video analysis page; then, referring to... Figure 12 After detecting the uploaded video A to be processed, a global video understanding operation for video A is automatically triggered (or can be triggered by the user). The terminal responds to the global video understanding operation by generating global video understanding information for video A, or requests the server to generate global video understanding information for video A, and displays the global video understanding information B corresponding to video A on the video analysis page.

[0313] In the aforementioned video processing method, when understanding the video to be processed, the entire video is first divided into multiple ordered sub-videos. Then, for each current sub-video, the memory content corresponding to the preceding sub-video is combined with the information features of the current sub-video to generate the corresponding memory content. This process is iteratively executed until the processing completion condition is met. This method fully considers the temporal dependency between the current sub-video and the preceding sub-videos. When generating the memory content of the current sub-video, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are integrated, which can effectively alleviate the problems of detail loss and error accumulation in traditional hierarchical summarization, thereby improving the accuracy of the generated memory content. At the same time, using the memory content corresponding to the preceding sub-video of the current sub-video to constrain the understanding of the current sub-video can eliminate the ambiguity of understanding a single sub-video, while avoiding contextual fragmentation. This is conducive to restoring the temporal logic of the entire video, making the memory content generated segment by segment more accurate and reliable, thereby improving the accuracy of the memory content of the final target sub-video. This can effectively reduce video understanding bias, thereby improving the accuracy of determining global video understanding information and thus improving the overall video understanding accuracy.

[0314] In one exemplary embodiment, yet another video processing method is provided, which is applied to... Figure 1 Taking the terminal as an example, the specific content includes the following: displaying the video to be processed on the video analysis page; during the processing of the video to be processed, or before the processing of the video to be processed, or after the processing of the video to be processed is completed, responding to the query content input event for the video to be processed, displaying the query content corresponding to the query content input event on the video analysis page; after the processing of the video to be processed is completed, displaying the response content corresponding to the query content on the video analysis page; the response content is obtained by querying the global video understanding information corresponding to the video to be processed.

[0315] Among them, the query content input event refers to the query content input operation; by triggering the query content input event, query content for the video to be processed can be entered.

[0316] The query content refers to the content that needs to be queried to obtain the response, specifically the question information, such as "What are the colors of the protagonist's clothes at the beginning and end of the movie?"

[0317] The response content refers to the response to the query content based on the global video understanding information of the video to be processed. Specifically, it refers to the answer to the question, such as "the main character's clothing colors at the beginning and end of the movie are red and blue, respectively".

[0318] For example, the terminal displays the video to be processed on the video analysis page; during the processing of the video to be processed, or before the processing of the video to be processed, or after the processing of the video to be processed is completed, the terminal responds to a query input operation for the video to be processed, obtains the query content corresponding to the query input operation, and displays the query content on the video analysis page; then, after the processing of the video to be processed is completed, based on the query content, it queries the global video understanding information corresponding to the video to be processed, obtains the response content corresponding to the query content, and displays the response content corresponding to the query content on the video analysis page. For example, refer to... Figure 11 The user selects the video to be uploaded (A) in the message sending area and clicks the "Send" button, triggering the upload operation for video A. The terminal responds to this upload operation by displaying video A on the video analysis page. (Reference) Figure 13 After obtaining the global video understanding information B corresponding to the video to be processed A, the user enters query content C in the message sending area and clicks the "Send" button, triggering the sending operation for query content C; the terminal responds to this sending operation, obtains the user-input query content C, and displays the query content C on the video analysis page; then, referring to Figure 14 The terminal queries the global video understanding information B corresponding to the video to be processed, based on the query content C, obtains the response content D corresponding to the query content C, and displays the response content D corresponding to the query content C on the video analysis page.

[0319] For example, this application can be applied to an intelligent video analytics platform. User workflow: 1. The user uploads a long video file (such as a movie or a complete sports game recording), or provides a real-time video stream link (such as a live stream). 2. The system's background "memory agent" begins streaming the video; during processing, the user interface (such as the video analytics page) can display a summary of the currently processed portion or key event nodes in real time. 3. After processing is complete (or during processing), the user can ask the system any question about the entire video through a natural language dialog box. For example: For a movie: "What colors are the main character's clothes at the beginning and end of the movie?" For a sports game: "Please summarize all the goals scored by Team A in the second half." For a monitoring video: "Tracerify the path of the person wearing the red shirt from their appearance to their departure." 4. After receiving the question, the system uses its highly condensed global memory to quickly generate an accurate answer without rescanning and analyzing the entire video file.

[0320] For example, after the video to be processed is completed, the terminal queries the global video understanding information corresponding to the video to be processed based on the query content, obtains the response content corresponding to the query content, and displays the response content corresponding to the query content on the video analysis page; for example, it obtains the target semantic information of the query content, queries the global video understanding information corresponding to the video to be processed based on the target semantic information, obtains the target matching information of the target semantic information in the global video understanding information, and generates the response content corresponding to the query content based on the target matching information. It should be noted that the global video understanding information corresponding to the video to be processed is obtained according to the first video processing method, and for details, please refer to the relevant embodiments of the first video processing method (such as...). Figure 2 For example, the process involves acquiring a video to be processed, segmenting it to obtain multiple ordered sub-videos; based on the processing order of the ordered sub-videos, retrieving the current sub-video from the sub-videos, performing feature extraction on the current sub-video to obtain its information features; acquiring the memory content corresponding to the preceding sub-videos of the current sub-video, fusing the memory content of the preceding sub-videos with the information features of the current sub-video to obtain fused information features; where the memory content corresponding to the sub-videos is used to represent the video understanding information corresponding to the sub-videos and their historical sub-videos; generating memory content based on the fused information features to obtain the memory content corresponding to the current sub-video, returning to the step of retrieving the current sub-video from the sub-videos based on the processing order of the ordered sub-videos, until the processing completion condition is met, obtaining the memory content of the target sub-video corresponding to the processing completion condition; and obtaining the global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video.

[0321] In this embodiment, after the video to be processed is completed, the corresponding query content can be obtained quickly based on the query content input event. There is no need to rescan and analyze the entire video to be processed, which simplifies the process of generating the response content and improves the efficiency of response content generation.

[0322] In an exemplary embodiment, before displaying the response content corresponding to the query content on the video analysis page, the method further includes a step of generating the response content corresponding to the query content. Specifically, this includes: after the video to be processed is completed, obtaining the target semantic information of the query content; querying the global video understanding information corresponding to the video to be processed to obtain the target matching information of the target semantic information in the global video understanding information; and generating the response content corresponding to the query content based on the target matching information.

[0323] Among them, target semantic information refers to the semantic information corresponding to the query content.

[0324] Among them, target matching information refers to the matching information of target semantic information in global video understanding information.

[0325] The response content corresponding to the query content refers to the response content generated based on the target semantic information in the global video understanding information.

[0326] For example, after the video to be processed is completed, the terminal performs semantic recognition processing on the query content through semantic recognition instructions to obtain the target semantic information corresponding to the query content; then, it queries the global video understanding information corresponding to the video to be processed to obtain the matching information of the target semantic information in the global video understanding information, which serves as the target matching information; finally, based on the target matching information, it generates the corresponding response content, which serves as the response content corresponding to the query content. For example, for the query content "What are the colors of the protagonist's clothes at the beginning and end of the movie?", the corresponding target semantic information is "What are the colors of the protagonist's clothes at the beginning of the movie? What are the colors of the protagonist's clothes at the end of the movie?" Then, based on the target semantic information, it queries the global video understanding information corresponding to the video to be processed, and the target matching information obtained is "red" and "blue". Therefore, the generated response content is "red and blue".

[0327] In this embodiment, after the video to be processed is completed, the global video understanding information corresponding to the video to be processed is queried based on the target semantic information of the query content to obtain the target matching information of the target semantic information in the global video understanding information. Then, based on the target matching information, the response content corresponding to the query content is generated. In this way, by comprehensively considering the target semantic information of the query content and the target matching information of the target semantic information in the global video understanding information, the generated response content can be more accurate, thereby improving the accuracy of response content generation.

[0328] In an exemplary embodiment, the video processing method provided by this application further includes the step of displaying target information corresponding to the generated memory content on a video analysis page, specifically including the following: during the processing of the video to be processed, displaying target information corresponding to the generated memory content on a video analysis page; the target information includes at least one of summary information and key event information.

[0329] Among them, the generated memory content can refer to the memory content corresponding to the previous sub-video of the current sub-video.

[0330] Among them, the summary information corresponding to the generated memory content refers to the summary information corresponding to the generated memory content, such as "the video shows the appearance, function demonstration and actual use effect of a new product".

[0331] Among them, the key event information corresponding to the generated memory content refers to the event information of the key events corresponding to the generated memory content, specifically key event nodes, such as important events, actions, nodes, and turning points that occur in the video, such as "the first 10 seconds of the video show the product appearance, the middle 20 seconds demonstrate the function, and the last 10 seconds summarize the advantages".

[0332] For example, during the processing of the video to be processed, the terminal performs summary information generation processing based on the generated memory content to obtain summary information corresponding to the generated memory content; or, based on the generated memory content, performs key event information recognition processing to obtain key event information corresponding to the generated memory content. Then, the summary information and key event information corresponding to the generated memory content are used as target information corresponding to the generated memory content, and displayed on the video analysis page. For further explanation, see [link to example]. Figure 15 After the video to be processed, A, is displayed on the video analysis page, the terminal can obtain the target information E corresponding to the generated memory content of the video to be processed, and display the target information E corresponding to the generated memory content on the video analysis page.

[0333] In this embodiment, during the processing of the video to be processed, the target information corresponding to the generated memory content is displayed on the video analysis page, which is beneficial for timely understanding of the target information corresponding to the generated memory content.

[0334] In one exemplary embodiment, such as Figure 16 As shown, another video processing method is provided, illustrated by its application to a computer device, which can be a terminal or a server. It can be understood that this method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. (See references.) Figure 16 The video processing method includes the following steps S1601 to S1617. Wherein:

[0335] Step S1601: Obtain training data; the training data includes sample videos, sample question information corresponding to the sample videos, and sample answer information corresponding to the sample question information.

[0336] Step S1602: Using the video processing model to be trained, global video understanding information of the sample video under different processing sessions is obtained based on multiple ordered sample sub-videos corresponding to the sample video; each processing session is used to represent the process of obtaining a kind of global video understanding information of the sample video.

[0337] Step S1603: Based on the global video understanding information of the sample video under different processing sessions, obtain the predicted answer information of the sample question information under different processing sessions.

[0338] Step S1604: Based on the predicted answer information and sample answer information under different processing sessions, obtain the reward information corresponding to each processing session of the sample video; the reward information is used to characterize the similarity between the predicted answer information and the sample answer information.

[0339] Step S1605: Based on the reward information corresponding to each processing session of the sample video, obtain the generation loss of each memory content generation step in each processing session of the sample video; the memory content generation step is used to characterize the step of generating memory content corresponding to the sample sub-video.

[0340] Step S1606: Based on the generation loss of each memory content generation step in each processing session of the sample video, obtain the model loss of the video processing model to be trained.

[0341] Step S1607: Adjust the model parameters of the video processing model to be trained based on the model loss until the convergence condition is met, and obtain the pre-trained video processing model.

[0342] Step S1608: Obtain the video to be processed; according to the preset video segmentation method, segment the video to be processed to obtain multiple ordered sub-videos corresponding to the video to be processed.

[0343] The preset video segmentation method includes at least one of the following: segmentation based on preset duration, segmentation based on preset frame count, segmentation based on scene switching, segmentation based on key events, and segmentation based on semantic content consistency; the video data length corresponding to each sub-video is less than or equal to the input data length of the pre-trained video processing model.

[0344] Step S1609: Based on the ordered processing order of the sub-videos, obtain the current sub-video from the sub-videos.

[0345] Step S1610: Input the current sub-video into the pre-trained video processing model; use the pre-trained video processing model to extract visual content from the current sub-video to obtain the visual content of the current sub-video.

[0346] Step S1611: Perform feature extraction processing on the visual content of the current sub-video to obtain the multimodal feature vector corresponding to the current sub-video; based on the multimodal feature vector, obtain the information features of the current sub-video.

[0347] Step S1612: Obtain the memory content corresponding to the previous sub-video of the current sub-video, and obtain the memory content corresponding to the preceding sub-video of the current sub-video.

[0348] Step S1613: Input the memory content corresponding to the forward sub-video and the information features of the current sub-video into the pre-trained video processing model.

[0349] Step S1614: Using a pre-trained video processing model, the memory content corresponding to the forward sub-video and the information features of the current sub-video are spliced ​​together to obtain spliced ​​information features; based on the spliced ​​information features, fused information features are obtained.

[0350] Step S1615: Using a pre-trained video processing model, memory content generation processing is performed based on fused information features to obtain updated memory content corresponding to the forward sub-video; based on the updated memory content, the memory content corresponding to the current sub-video is obtained.

[0351] The length of the updated memory content is the same as the length of the memory content corresponding to the forward sub-video.

[0352] Step S1616: Return to the processing order based on the ordered sub-videos, retrieve the current sub-video from the sub-videos, until the processing completion condition is met, and obtain the memory content of the target sub-video corresponding to the processing completion condition.

[0353] Specifically, based on the ordered processing order of sub-videos, the next sub-video of the current sub-video is set as the updated current sub-video for processing until the memory content corresponding to the last sub-video of the video to be processed is obtained; based on the memory content corresponding to the last sub-video, the memory content of the target sub-video is obtained.

[0354] Step S1617: Based on the memory content of the target sub-video, obtain the global video understanding information corresponding to the video to be processed.

[0355] In the aforementioned video processing method, when understanding the video to be processed, the entire video is first divided into multiple ordered sub-videos. Then, for each current sub-video, the memory content corresponding to the preceding sub-video is combined with the information features of the current sub-video to generate the corresponding memory content. This process is iteratively executed until the processing completion condition is met. This method fully considers the temporal dependency between the current sub-video and the preceding sub-videos. When generating the memory content of the current sub-video, the memory content corresponding to the preceding sub-video and the information features of the current sub-video are integrated, which can effectively alleviate the problems of detail loss and error accumulation in traditional hierarchical summarization, thereby improving the accuracy of the generated memory content. At the same time, using the memory content corresponding to the preceding sub-video of the current sub-video to constrain the understanding of the current sub-video can eliminate the ambiguity of understanding a single sub-video, while avoiding contextual fragmentation. This is conducive to restoring the temporal logic of the entire video, making the memory content generated segment by segment more accurate and reliable, thereby improving the accuracy of the memory content of the final target sub-video. This can effectively reduce video understanding bias, thereby improving the accuracy of determining global video understanding information and thus improving the overall video understanding accuracy.

[0356] To more clearly illustrate the video processing method provided in the embodiments of this application, the following describes the video processing method in detail with a specific embodiment. In an exemplary embodiment, such as Figure 3As shown, this application also provides a long video understanding method based on reinforcement learning memory agents (i.e., an infinitely long video understanding method based on reinforcement learning memory agents). This method aims to solve the problems of high computational complexity, limited context window, and sharp performance decline with increasing video length faced by traditional multimodal large models when processing long videos. Key technical points are as follows: 1. Streaming processing mechanism: Dividing video streams of arbitrary length into small video blocks of fixed duration. The model does not need to load the entire video at once, but rather "watches" it segment by segment like a human, thus breaking the model's own context window limitation. 2. Dynamic memory and overwriting: Introducing a fixed-length "memory" module. Whenever the model processes a new video block, it combines the content of the new block with the memory from the previous moment to generate a completely new, refined, and summarized memory, and uses it to completely overwrite the old memory. This mechanism ensures that the model's single-step computation and memory usage remain constant when processing videos of arbitrary length. 3. Reinforcement learning-driven memory management: Modeling the memory update process as a reinforcement learning problem. The memory agent learns how to perform efficient memory compression and information retention through "trial and error." If a memory update helps to accurately answer questions about the entire video or complete a summary task, the agent receives a positive reward; otherwise, it does not. In this way, the model automatically learns to identify and retain information crucial for understanding the overall plot and identifying key objects, while ignoring redundant or irrelevant footage. 4. Task Decoupling: The entire process is broken down into two stages: first, the "context processing" stage, where the model iteratively processes all video blocks and updates its memory; then, the "task answering" stage, where the model uses the final global memory to answer user questions, generate summaries, or perform other downstream tasks. Through this design, this application can process infinitely long videos with linear computational complexity (proportional to the total video length) while maintaining high-quality understanding capabilities, achieving near-lossless performance extrapolation.

[0357] refer to Figure 3 The long video understanding system based on reinforcement learning memory agents includes the following modules:

[0358] 1. Video Segmentation Module: Receives the input video stream and segments it into a series of continuous video blocks C1, C2, C3...C1, C2, C3, ..., C4, each with a preset duration of 10 seconds. N .

[0359] 2. Multimodal Feature Extractor: For each video block C k A pre-trained multimodal model (Qwen2.5-VL-32B) is used to extract its visual content and transform it into a compact multimodal feature vector.

[0360] 3. Memory Agent Module: This is the core of this application.

[0361] Input: At step k, it receives two inputs: the current video block C. k The characteristics, and the memory M from the previous step k-1 The initial memory M0 is empty.

[0362] Processing: Internally, this module is a large multimodal model based on Transformer. It will use C... k Features and M k-1 The content is concatenated as input, and then a new memory M is generated. k .

[0363] Output: New memory M k This generation process is called the "overwrite" operation, M k Length and M k-1 It's exactly the same, but the content has been updated.

[0364] 4. Task Execution Module: After processing all N video blocks, the final memory M... N The data is sent to this module. This module, based on the user's specific task (such as answering questions or generating summaries), uses memory M... N To generate the final result.

[0365] The above embodiments can achieve the following technical effects:

[0366] 1. Efficiency and Scalability: It achieves linear time complexity (O(N)) processing of infinitely long videos, greatly reducing the consumption of computing resources and making real-time or near real-time analysis of ultra-long videos possible.

[0367] 2. Superior Performance and Extrapolation Capability: Through end-to-end reinforcement learning training, the model can intelligently manage long-term dependencies, effectively avoiding information loss and the "performance cliff" problem in traditional methods. When the simulated video length reaches 3.5M tokens, the traditional extended context model (Qwen2.5-VL-32B) cannot handle it due to memory exhaustion; even at a shorter length of 32K tokens, its accuracy has dropped significantly. The solution of this invention can successfully meet these challenges and maintain stable high performance under ultra-long inputs. Specific comparisons are shown in Table 1.

[0368] Table 1 Performance (i.e. accuracy) Comparison Table

[0369]

[0370] 3. Versatility and Flexibility: The framework proposed in this invention is not dependent on a specific model architecture and can be combined with any advanced multimodal large model. The design of the memory module and overwrite strategy is simple, easy to implement and deploy.

[0371] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0372] Based on the same inventive concept, this application also provides a video processing apparatus for implementing the video processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more video processing apparatus embodiments provided below can be found in the limitations of the video processing method described above, and will not be repeated here.

[0373] In one exemplary embodiment, such as Figure 17 As shown, a video processing apparatus 1700 is provided, including: a segmentation processing module 1701, a feature extraction module 1702, a fusion processing module 1703, a content generation module 1704, and an information determination module 1705, wherein:

[0374] The segmentation processing module 1701 is used to acquire the video to be processed, segment the video to be processed, and obtain multiple ordered sub-videos corresponding to the video to be processed.

[0375] The feature extraction module 1702 is used to obtain the current sub-video from the sub-videos based on the ordered processing order of the sub-videos, perform feature extraction processing on the current sub-videos, and obtain the information features of the current sub-videos.

[0376] The fusion processing module 1703 is used to obtain the memory content corresponding to the previous sub-video of the current sub-video, and to fuse the memory content corresponding to the previous sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video.

[0377] The content generation module 1704 is used to perform memory content generation processing based on the fused information features to obtain the memory content corresponding to the current sub-video, return to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and obtain the memory content of the target sub-video corresponding to the processing completion condition.

[0378] The information determination module 1705 is used to obtain global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video.

[0379] In an exemplary embodiment, the segmentation processing module 1701 is further configured to segment the video to be processed according to a preset video segmentation method to obtain multiple ordered sub-videos corresponding to the video to be processed; the preset video segmentation method includes at least one of a segmentation method based on a preset duration, a segmentation method based on a preset number of frames, a segmentation method based on scene switching, a segmentation method based on key events, and a segmentation method based on semantic content consistency; the length of the video data corresponding to each sub-video is less than or equal to the length of the input data of the pre-trained video processing model.

[0380] In an exemplary embodiment, the feature extraction module 1702 is further configured to input the current sub-video into a pre-trained video processing model; perform visual content extraction processing on the current sub-video through the pre-trained video processing model to obtain the visual content of the current sub-video; perform feature extraction processing on the visual content of the current sub-video to obtain the multimodal feature vector corresponding to the current sub-video; and obtain the information features of the current sub-video based on the multimodal feature vector.

[0381] In an exemplary embodiment, the feature extraction module 1702 is further configured to perform frame segmentation processing on the current sub-video using a pre-trained video processing model to obtain video frames of the current sub-video; extract key video frames from the video frames of the current sub-video; perform visual content extraction processing on the key video frames to obtain the visual content of the key video frames; and perform fusion processing on the visual content of each key video frame to obtain the visual content of the current sub-video.

[0382] In an exemplary embodiment, the feature extraction module 1702 is further configured to perform feature encoding processing on the visual content of the current sub-video to obtain the visual features of the current sub-video; perform feature mapping processing on the visual features of the current sub-video to obtain the mapped features; and obtain the multimodal feature vector corresponding to the current sub-video based on the mapped features.

[0383] In an exemplary embodiment, the fusion processing module 1703 is further configured to obtain the memory content corresponding to the previous sub-video of the current sub-video, and obtain the memory content corresponding to the preceding sub-video of the current sub-video.

[0384] The fusion processing module 1703 is also used to input the memory content corresponding to the forward sub-video and the information features of the current sub-video into a pre-trained video processing model; through the pre-trained video processing model, the memory content corresponding to the forward sub-video and the information features of the current sub-video are spliced ​​together to obtain spliced ​​information features; based on the spliced ​​information features, the fused information features are obtained.

[0385] In an exemplary embodiment, the content generation module 1704 is further configured to perform memory content generation processing based on fused information features using a pre-trained video processing model to obtain updated memory content corresponding to the memory content of the preceding sub-video; the content length of the updated memory content is the same as the content length of the memory content corresponding to the preceding sub-video; and based on the updated memory content, obtain the memory content corresponding to the current sub-video.

[0386] The content generation module 1704 is also used to process the next sub-video of the current sub-video as the updated current sub-video based on the ordered processing order of the sub-videos, until the memory content corresponding to the last sub-video of the video to be processed is obtained; and the memory content of the target sub-video is obtained based on the memory content corresponding to the last sub-video.

[0387] In an exemplary embodiment, the memory content corresponding to the current sub-video is obtained through a pre-trained video processing model; the video processing device 1700 further includes a model training module for acquiring training data; the training data includes sample videos, sample question information corresponding to the sample videos, and sample answer information corresponding to the sample question information; through the video processing model to be trained, based on multiple ordered sample sub-videos corresponding to the sample videos, global video understanding information of the sample videos under different processing sessions is obtained; each processing session is used to characterize the process of obtaining a kind of global video understanding information of the sample videos; based on the global video understanding information of the sample videos under different processing sessions, predicted answer information of the sample question information under different processing sessions is obtained; according to the sample question... The predicted answer information and sample answer information under different processing sessions are used to obtain the reward information corresponding to each processing session of the sample video. The reward information is used to characterize the similarity between the predicted answer information and the sample answer information. Based on the reward information corresponding to each processing session of the sample video, the generation loss of each memory content generation step in each processing session of the sample video is obtained. The memory content generation step is used to characterize the step of generating the memory content corresponding to the sample sub-video. Based on the generation loss of each memory content generation step in each processing session of the sample video, the model loss of the video processing model to be trained is obtained. The model parameters of the video processing model to be trained are adjusted based on the model loss until the convergence condition is met, and the pre-trained video processing model is obtained.

[0388] In an exemplary embodiment, the model training module is further configured to: obtain the current sample sub-video from the sample sub-videos based on the ordered processing order of the sample sub-videos; input the current sample sub-video into the video processing model to be trained; perform feature extraction processing on the current sample sub-video through the video processing model to be trained to obtain the information features of the current sample sub-video; obtain the memory content corresponding to the preceding sample sub-video of the current sample sub-video; fuse the memory content corresponding to the preceding sample sub-video and the information features of the current sample sub-video to obtain the sample fusion information features; perform corresponding memory content generation processing based on the sample fusion information features to obtain the memory content corresponding to the current sample sub-video; return to the step of obtaining the current sample sub-video from the sample sub-videos based on the ordered processing order of the sample sub-videos, until the memory content corresponding to the last sample sub-video of the sample video is obtained; different processing sessions correspond to different memory content generation processes; obtain the global video understanding information of the sample video in a single processing session based on the memory content corresponding to the last sample sub-video; and obtain the global video understanding information of the sample video in different processing sessions based on the global video understanding information of the sample video in a single processing session.

[0389] In an exemplary embodiment, the model training module is further configured to obtain semantic information of the sample question information; query the global video understanding information of the sample video under different processing sessions respectively to obtain matching information of the semantic information in the global video understanding information under different processing sessions; and generate predicted answer information of the sample question information under different processing sessions based on the matching information of the semantic information in the global video understanding information under different processing sessions.

[0390] In an exemplary embodiment, the model training module is further configured to, when the sample question information belongs to an objective question answering task, obtain the consistency comparison results between the predicted answer information and the sample answer information under different processing sessions; and, based on the consistency comparison results between the predicted answer information and the sample answer information under different processing sessions, determine the reward information corresponding to each processing session of the sample video.

[0391] In an exemplary embodiment, the model training module is further configured to, when the sample question information belongs to a generative task, obtain the semantic similarity between the predicted answer information and the sample answer information under different processing sessions; and determine the reward information corresponding to each processing session of the sample video based on the semantic similarity between the predicted answer information and the sample answer information under different processing sessions.

[0392] In an exemplary embodiment, the model training module is further configured to obtain the number of lexical units of the memory content generated in each memory content generation step in each processing session of the sample video; perform fusion processing on the number of lexical units to obtain a fused number of lexical units; perform fusion processing on the generation loss of each memory content generation step in each processing session of the sample video to obtain a fused generation loss; and obtain the model loss of the video processing model to be trained based on the ratio between the fused generation loss and the fused number of lexical units.

[0393] In an exemplary embodiment, the model training module is further configured to: obtain the reward information advantage value of each memory content generation step in each processing session of the sample video based on the reward information corresponding to each processing session of the sample video; the reward information advantage value of the memory content generation step is used to characterize the quality of the memory content generated by the memory content generation step; obtain the weight of each memory content generation step in each processing session of the sample video, and update the reward information advantage value of each memory content generation step in each processing session of the sample video using the weight to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video; obtain the divergence of the current model parameters of the video processing model to be trained relative to the initial model parameters of the video processing model to be trained, and obtain the generation loss of each memory content generation step in each processing session of the sample video based on the target reward information advantage value and divergence of each memory content generation step in each processing session of the sample video.

[0394] In an exemplary embodiment, the model training module is further configured to obtain the average reward information corresponding to each processing session of the sample video; obtain the difference between the reward information corresponding to each processing session of the sample video and the average reward information; and obtain the reward information advantage value of each memory content generation step in each processing session of the sample video based on the difference between the reward information corresponding to each processing session of the sample video and the average reward information.

[0395] In an exemplary embodiment, the model training module is further configured to: determine a first generation probability of the memory content corresponding to each memory content generation step in each processing session, based on the sample question information and the memory content corresponding to the historical memory content generation step of each memory content generation step in each processing session, under the current model parameters of the video processing model to be trained; determine a second generation probability of the memory content corresponding to each memory content generation step in each processing session, based on the sample question information and the memory content corresponding to the historical memory content generation step of each memory content generation step in each processing session, under the historical model parameters of the video processing model to be trained; and obtain the weight of each memory content generation step in each processing session of the sample video based on the ratio between the first generation probability and the second generation probability.

[0396] In an exemplary embodiment, the model training module is further configured to adjust the weights using preset coefficients to obtain the adjusted weights for each memory content generation step in each processing session of the sample video; to perform a first update process and a second update process on the reward information advantage value of each memory content generation step in each processing session of the sample video using the weights and the adjusted weights respectively, to obtain a first reward information advantage value and a second reward information advantage value for each memory content generation step in each processing session of the sample video; and to obtain a target reward information advantage value for each memory content generation step in each processing session of the sample video based on the first reward information advantage value and the second reward information advantage value.

[0397] In one exemplary embodiment, the video processing apparatus 1700 further includes an information generation module, configured to obtain candidate information corresponding to a summary information generation task for the video to be processed from global video understanding information corresponding to the video to be processed; and to perform fusion processing on the candidate information to obtain summary information corresponding to the summary information generation task.

[0398] In one exemplary embodiment, such as Figure 18 As shown, another video processing device 1800 is provided, including: a first display module 1801 and a second display module 1802, wherein:

[0399] The first display module 1801 is used to display the video to be processed on the video analysis page.

[0400] The second display module 1802 is used to display the global video understanding information corresponding to the video to be processed on the video analysis page in response to a global video understanding event for the video to be processed.

[0401] In one exemplary embodiment, such as Figure 19 As shown, another video processing device 1900 is provided, including: a video display module 1901, an event response module 1902, and a content display module 1903, wherein:

[0402] The video display module 1901 is used to display the video to be processed on the video analysis page;

[0403] The event response module 1902 is used to respond to the query content input event for the video to be processed during the processing of the video to be processed, or before the processing of the video to be processed, or after the processing of the video to be processed is completed, and to display the query content corresponding to the query content input event on the video analysis page.

[0404] The content display module 1903 is used to display the corresponding response content on the video analysis page after the video to be processed is completed.

[0405] In an exemplary embodiment, the video processing apparatus 1900 further includes a content generation module, configured to, after the video to be processed is processed, obtain target semantic information of the query content; query the global video understanding information corresponding to the video to be processed to obtain target matching information of the target semantic information in the global video understanding information; and generate response content corresponding to the query content based on the target matching information.

[0406] In an exemplary embodiment, the video processing apparatus 1900 further includes an information display module for displaying target information corresponding to the generated memory content on a video analysis page during the processing of the video to be processed; the target information includes at least one of summary information and key event information.

[0407] Each module in the aforementioned video processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0408] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 20 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as the memory content corresponding to the previous sub-video (e.g., the previous sub-video). The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0409] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 21As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a video processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0410] Those skilled in the art will understand that Figure 20 or Figure 21 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0411] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0412] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0413] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0414] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0415] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0416] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0417] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: The video to be processed is obtained, and the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed. Based on the ordered processing order of the sub-videos, the current sub-video is obtained from the sub-videos, and feature extraction processing is performed on the current sub-video to obtain the information features of the current sub-video; Obtain the memory content corresponding to the preceding sub-video of the current sub-video, and fuse the memory content corresponding to the preceding sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to characterize the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video; Based on the fused information features, memory content generation processing is performed to obtain the memory content corresponding to the current sub-video. Then, the process returns to the step of obtaining the current sub-video from the sub-video based on the ordered sub-video processing order, until the processing completion condition is met, and the memory content of the target sub-video corresponding to the processing completion condition is obtained. Based on the memory content of the target sub-video, global video understanding information corresponding to the video to be processed is obtained.

2. The method according to claim 1, characterized in that, The step of segmenting the video to be processed to obtain multiple ordered sub-videos corresponding to the video to be processed includes: According to the preset video segmentation method, the video to be processed is segmented to obtain multiple ordered sub-videos corresponding to the video to be processed; The preset video segmentation method includes at least one of the following: segmentation based on preset duration, segmentation based on preset frame count, segmentation based on scene switching, segmentation based on key events, and segmentation based on semantic content consistency; the video data length corresponding to each sub-video is less than or equal to the input data length of the pre-trained video processing model.

3. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the current sub-video to obtain the information features of the current sub-video includes: Input the current sub-video into a pre-trained video processing model; The visual content of the current sub-video is extracted using the pre-trained video processing model. The visual content of the current sub-video is subjected to feature extraction processing to obtain the multimodal feature vector corresponding to the current sub-video; Based on the multimodal feature vector, the information features of the current sub-video are obtained.

4. The method according to claim 3, characterized in that, The step of extracting visual content from the current sub-video using the pre-trained video processing model to obtain the visual content of the current sub-video includes: The pre-trained video processing model is used to perform frame segmentation on the current sub-video to obtain the video frames of the current sub-video. Extract key video frames from the video frames of the current sub-video; Visual content extraction processing is performed on the key video frames to obtain the visual content of the key video frames; The visual content of each key video frame is fused to obtain the visual content of the current sub-video.

5. The method according to claim 3, characterized in that, The step of performing feature extraction processing on the visual content of the current sub-video to obtain the multimodal feature vector corresponding to the current sub-video includes: The visual content of the current sub-video is subjected to feature encoding processing to obtain the visual features of the current sub-video; The visual features of the current sub-video are subjected to feature mapping processing to obtain the mapped features; Based on the mapped features, the multimodal feature vector corresponding to the current sub-video is obtained.

6. The method according to claim 1, characterized in that, The step of obtaining the memory content corresponding to the preceding sub-video of the current sub-video includes: Obtain the memory content corresponding to the previous sub-video of the current sub-video, and obtain the memory content corresponding to the preceding sub-video of the current sub-video; The step of fusing the memory content corresponding to the forward sub-video and the information features of the current sub-video to obtain fused information features includes: The memory content corresponding to the forward sub-video and the information features of the current sub-video are input into a pre-trained video processing model. The pre-trained video processing model is used to concatenate the memory content corresponding to the forward sub-video and the information features of the current sub-video to obtain concatenated information features. Based on the splicing information features, the fused information features are obtained.

7. The method according to claim 6, characterized in that, The process of generating memory content based on the fused information features to obtain the memory content corresponding to the current sub-video includes: Using the pre-trained video processing model, memory content generation processing is performed based on the fused information features to obtain updated memory content corresponding to the forward sub-video; the content length of the updated memory content is the same as the content length of the memory content corresponding to the forward sub-video. Based on the updated memory content, the memory content corresponding to the current sub-video is obtained; The step of returning to the processing order based on the ordered sub-videos and obtaining the current sub-video from the sub-videos, until the processing completion condition is met, and obtaining the memory content of the target sub-video corresponding to the processing completion condition, includes: Based on the ordered processing order of the sub-videos, the next sub-video of the current sub-video is set as the updated current sub-video for processing until the memory content corresponding to the last sub-video of the video to be processed is obtained. Based on the memory content corresponding to the last sub-video, the memory content of the target sub-video is obtained.

8. The method according to claim 1, characterized in that, The memory content corresponding to the current sub-video is obtained by processing a pre-trained video processing model, which is trained in the following manner: Acquire training data; the training data includes sample videos, sample question information corresponding to the sample videos, and sample answer information corresponding to the sample question information. Using the video processing model to be trained, global video understanding information of the sample video under different processing sessions is obtained based on multiple ordered sample sub-videos corresponding to the sample video; each processing session is used to represent the process of obtaining a kind of global video understanding information of the sample video. Based on the global video understanding information of the sample video under different processing sessions, the predicted answer information of the sample question information under different processing sessions is obtained; Based on the predicted answer information under different processing sessions and the sample answer information, the reward information corresponding to each processing session of the sample video is obtained; The reward information is used to characterize the similarity between the predicted answer information and the sample answer information; Based on the reward information corresponding to each processing session of the sample video, the generation loss of each memory content generation step in each processing session of the sample video is obtained. The memory content generation step is used to characterize the memory content corresponding to the generated sample sub-video; The model loss of the video processing model to be trained is obtained based on the generation loss of each memory content generation step in each processing session of the sample video. The model parameters of the video processing model to be trained are adjusted based on the model loss until the convergence condition is met, thus obtaining the pre-trained video processing model.

9. The method according to claim 8, characterized in that, The process involves using a video processing model to be trained, based on multiple ordered sub-videos corresponding to the sample video, to obtain global video understanding information of the sample video under different processing sessions, including: Based on the ordered processing order of the sample sub-videos, the current sample sub-video is obtained from the sample sub-videos and input into the video processing model to be trained. The current sample sub-video is processed by the video processing model to be trained to extract features, thereby obtaining the information features of the current sample sub-video. Obtain the memory content corresponding to the preceding sample sub-video of the current sample sub-video, and fuse the memory content corresponding to the preceding sample sub-video and the information features of the current sample sub-video to obtain the sample fusion information features; Based on the sample fusion information features, corresponding memory content generation processing is performed to obtain the memory content corresponding to the current sample sub-video. Then, the process returns to the step of obtaining the current sample sub-video from the sample sub-videos based on the ordered processing order of the sample sub-videos, until the memory content corresponding to the last sample sub-video of the sample video is obtained. Different processing sessions correspond to different memory content generation processes. Based on the memory content corresponding to the last sample sub-video, the global video understanding information of the sample video in a single processing session is obtained; Based on the global video understanding information of the sample video in a single processing session, the global video understanding information of the sample video in different processing sessions is obtained.

10. The method according to claim 8, characterized in that, The generation loss of the video processing model to be trained is obtained by taking the generation loss of each memory content generation step in each processing session based on the sample video, including: Obtain the number of lexical units of the memory content generated in each memory content generation step in each processing session of the sample video; The number of lexical units is fused to obtain the fused number of lexical units. The generation loss of each memory content generation step in each processing session of the sample video is fused to obtain the fused generation loss. The model loss of the video processing model to be trained is obtained based on the ratio between the fusion generation loss and the number of fusion tokens.

11. The method according to claim 8, characterized in that, The generation loss for each memory content generation step in each processing session of the sample video is obtained based on the reward information corresponding to each processing session of the sample video, including: Based on the reward information corresponding to each processing session of the sample video, the reward information advantage value of each memory content generation step in each processing session of the sample video is obtained; the reward information advantage value of the memory content generation step is used to characterize the quality of the memory content generated by the memory content generation step. The weight of each memory content generation step in each processing session of the sample video is obtained, and the reward information advantage value of each memory content generation step in each processing session of the sample video is updated using the weight to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video. Obtain the divergence of the current model parameters of the video processing model to be trained relative to the initial model parameters of the video processing model to be trained. Based on the target reward information advantage value of each memory content generation step in each processing session of the sample video and the divergence, obtain the generation loss of each memory content generation step in each processing session of the sample video.

12. The method according to claim 11, characterized in that, The reward information advantage value for each memory content generation step in each processing session of the sample video is obtained based on the reward information corresponding to each processing session of the sample video, including: Obtain the average reward information corresponding to each processing session of the sample video; The difference between the reward information for each processing session of the sample video and the average reward information is obtained respectively; Based on the difference between the reward information corresponding to each processing session of the sample video and the average reward information, the reward information advantage value of each memory content generation step in each processing session of the sample video is obtained.

13. The method according to claim 11, characterized in that, The step of obtaining the weight of each memory content generation step in each processing session of the sample video includes: Under the current model parameters of the video processing model to be trained, based on the sample problem information and the memory content corresponding to the historical memory content generation step of each memory content generation step in each processing session, the first generation probability of the memory content corresponding to each memory content generation step in each processing session is determined. Under the historical model parameters of the video processing model to be trained, based on the sample problem information and the memory content corresponding to the historical memory content generation step of each memory content generation step in each processing session, the second generation probability of the memory content corresponding to each memory content generation step in each processing session is determined. The weight of each memory content generation step in each processing session of the sample video is obtained based on the ratio between the first generation probability and the second generation probability.

14. The method according to claim 11, characterized in that, The step of updating the reward information advantage value of each memory content generation step in each processing session of the sample video using the weights to obtain the target reward information advantage value of each memory content generation step in each processing session of the sample video includes: The weights are adjusted using preset coefficients to obtain the adjusted weights for each memory content generation step in each processing session of the sample video; Using the weights and the adjusted weights respectively, the reward information advantage value of each memory content generation step in each processing session of the sample video is subjected to a first update process and a second update process to obtain the first reward information advantage value and the second reward information advantage value of each memory content generation step in each processing session of the sample video. Based on the first reward information advantage value and the second reward information advantage value, the target reward information advantage value is obtained for each memory content generation step in each processing session of the sample video.

15. A video processing method, characterized in that, The method includes: The video to be processed is displayed on the video analysis page; During the processing of the video to be processed, or before the processing of the video to be processed, or after the processing of the video to be processed is completed, in response to the query content input event for the video to be processed, the query content corresponding to the query content input event is displayed on the video analysis page; After the video to be processed is completed, the response content corresponding to the query content is displayed on the video analysis page; the response content is obtained by querying the global video understanding information corresponding to the video to be processed.

16. The method according to claim 15, characterized in that, Before displaying the response content corresponding to the query content on the video analysis page, the method further includes: After the video to be processed is completed, the target semantic information of the query content is obtained; Query the global video understanding information corresponding to the video to be processed to obtain the target matching information of the target semantic information in the global video understanding information; Based on the target matching information, a response is generated corresponding to the query content.

17. The method according to claim 15, characterized in that, The method further includes: During the processing of the video to be processed, the target information corresponding to the generated memory content is displayed on the video analysis page; the target information includes at least one of summary information and key event information.

18. A video processing apparatus, characterized in that, The device includes: The segmentation processing module is used to acquire the video to be processed, segment the video to be processed, and obtain multiple ordered sub-videos corresponding to the video to be processed. The feature extraction module is used to obtain the current sub-video from the sub-videos based on the ordered processing order of the sub-videos, perform feature extraction processing on the current sub-video, and obtain the information features of the current sub-video; The fusion processing module is used to obtain the memory content corresponding to the previous sub-video of the current sub-video, and to fuse the memory content corresponding to the previous sub-video and the information features of the current sub-video to obtain fused information features; wherein, the memory content corresponding to the sub-video is used to represent the video understanding information corresponding to the sub-video and the historical sub-videos of the sub-video; The content generation module is used to perform memory content generation processing based on the fused information features to obtain the memory content corresponding to the current sub-video, return to the processing order based on the ordered sub-videos, and obtain the current sub-video from the sub-videos until the processing completion condition is met, and obtain the memory content of the target sub-video corresponding to the processing completion condition. The information determination module is used to obtain global video understanding information corresponding to the video to be processed based on the memory content of the target sub-video.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 17.

Images