Video content understanding method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a video content understanding method, electronic equipment and a computer readable storage medium. Relates to the field of artificial intelligence, and comprises the following steps: clustering is carried out according to semantic contents of a plurality of events included in a target video, a plurality of tree nodes of an initial layer are generated, and video frames included in the plurality of events have a time sequence; determining a decision action for each tree node of the initial layer based on the subtitles of events included in each tree node of the initial layer and a user intention for querying the content in the target video; generating new tree nodes layer by layer in a mode of generating each tree node of the next layer according to the decision action of each tree node of the initial layer until the corresponding decision action in the new tree nodes does not have the exploration action, and obtaining a tree structure of the target video; and performing video understanding on each layer of tree nodes receiving the action according to the corresponding decision action in the tree structure to obtain an answer result. The technical problem that the video content understanding ability is insufficient in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video content understanding method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Long-form video understanding is a highly valuable yet challenging area. Unlike static images or short videos, videos lasting several hours typically contain thousands of frames, exhibiting extremely high information density and complex spatiotemporal dynamics. Effectively understanding long-form video content is crucial for applications such as automatic generation of sports highlights, film and television content summarization, and anomaly detection in surveillance videos. As video length increases, models receive massive amounts of information irrelevant to user queries, overwhelming them during processing and making it difficult to accurately locate key frames that answer questions, ultimately affecting the accuracy of understanding. Therefore, effectively mitigating the interference of massive amounts of redundant information has become a critical technical bottleneck that urgently needs to be addressed in this field.

[0003] Current technologies often perform visual clustering on frames or segments of videos, which disrupts the original temporal structure and leads to the loss of contextual information. This disruption of temporal information makes it difficult for models to understand the causal relationships between events, thus affecting their ability to answer complex questions. The hierarchical structures used in existing technologies struggle to dynamically and accurately identify the true core keyframes within complex layers, based on the user's query intent. Their keyframe selection process is not intelligent or flexible enough, resulting in a potentially suboptimal set of selected frames, thus limiting the upper limit of the model's understanding capabilities and leading to insufficient understanding of long videos. Summary of the Invention

[0004] This application provides a video content understanding method, an electronic device, and a computer-readable storage medium to alleviate or solve the technical problem of insufficient video content understanding capabilities in related technologies.

[0005] In a first aspect, embodiments of this application provide a video content understanding method, including: Clustering is performed on multiple events included in the target video according to their semantic content to generate multiple tree nodes in the initial layer. Each tree node in the initial layer includes events clustered into the same cluster, and the video frames included in the multiple events are in temporal order. Based on the captions of events included in each tree node of the initial layer, and the user intent to query the content of the target video, the decision action for each tree node of the initial layer is determined. The decision action includes a delete action, an accept action, or an explore action; the delete action means deleting the corresponding tree node, the accept action means retaining the corresponding tree node, and the explore action means determining a new decision action for the corresponding tree node based on the events in the corresponding tree node and the user intent. The tree nodes of the next layer are generated layer by layer according to the decision actions of each tree node in the initial layer, until the decision action corresponding to the new tree node does not contain the exploration action, thus obtaining the tree structure of the target video; Based on the tree nodes in the tree structure that correspond to the decision action as the acceptance action, video understanding is performed to obtain the answer result of the user's intent.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Based on the video content understanding method described in the first aspect above, this application has at least the following beneficial effects or advantages: By iteratively segmenting, generating subtitles, and clustering long videos based on dynamic events, a hierarchical tree structure is constructed. This event-based rather than discrete-frame-based segmentation method effectively preserves temporal coherence at the video-to-event granularity. By organizing video content through a coarse-to-fine tree structure, it fundamentally solves the problem of interference from massive amounts of redundant information, while maintaining the key temporal relationships necessary for accurate understanding. The user-query-driven exploration mechanism not only accurately locates keyframes but also incentivizes the model to spontaneously generate inference chains, thereby significantly improving decision-making accuracy and deep reasoning capabilities in long video understanding tasks.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0011] Figure 1 A flowchart illustrating a video content understanding method according to an embodiment of this application is shown; Figure 2 A training diagram of the video content understanding method according to an embodiment of this application is shown; Figure 3 A flowchart illustrating the video content understanding method according to an embodiment of this application is shown; Figure 4 A schematic block diagram of a video content understanding device according to an embodiment of this application is shown; Figure 5 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0012] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0014] The embodiments of this application aim to address key bottlenecks in existing long video understanding technologies, particularly the challenges faced by multimodal large language models (MM-LLMs) when processing videos lasting several hours.

[0015] This application aims to address key bottlenecks in existing long video understanding technologies, particularly the challenges faced by multimodal large language models (MM-LLMs) when processing videos lasting several hours.

[0016] Existing technologies are mainly divided into two paths: end-to-end structure and hierarchical structure.

[0017] End-to-end architectures attempt to directly input video content into a multimodal large model for processing. A typical implementation involves first uniformly extracting frames from a long video, simplifying it into a flattened frame sequence, and then using this sequence as input to the model for understanding and reasoning. Representative models include visual versions of the "Thousand Questions on General Meaning" and depth-based models. The main drawback is that as video length increases, irrelevant and redundant information grows exponentially, severely interfering with the model's judgment and leading to a significant performance degradation.

[0018] To address the information overload problem in end-to-end solutions, a hierarchical structure has been proposed. The core idea of ​​this approach is to structure the video before inputting information into a large model, thereby reducing its complexity. In such solutions, existing technologies provide video tree models, whose implementation steps are mainly as follows: (a) visual clustering and caption generation: firstly, visual clustering is performed on frames or segments in the video, and similar content is grouped; (b) frame caption generation: text descriptions are generated for the clustered video segments; (c) relevance scoring and tree construction: based on the user's question, the relevance score of the caption content for each segment is calculated, and a tree structure is dynamically constructed to represent the content of the entire video; (d) keyframe extraction: the model explores the constructed tree based on the score, extracting the keyframes most relevant to the question for final reasoning and answering.

[0019] While hierarchical methods such as Video Trees alleviate the problem of information redundancy to some extent, they still have two key technical drawbacks. First, they disrupt temporal continuity: when performing clustering or segmentation, these methods often disrupt the original temporal structure of the video, leading to the loss of contextual information. This disruption of temporal information makes it difficult for the model to understand the causal relationships between events, thus affecting its ability to answer complex questions. Second, they struggle to accurately identify keyframes: although existing hierarchical methods construct a structure, they still struggle to dynamically and accurately identify the true core keyframes within complex layers. Their keyframe selection process is not intelligent or flexible enough, resulting in a potentially suboptimal set of selected frames, thus limiting the upper limit of the model's understanding capabilities.

[0020] Existing long-video understanding methods, whether end-to-end or hierarchical extraction models, have inherent flaws. End-to-end models, through methods like uniform sampling, introduce massive amounts of irrelevant information, causing the model to be overwhelmed by irrelevant frames and leading to incorrect judgments. While existing hierarchical keyframe extraction methods attempt to reduce information, they often disrupt the original temporal structure of the video, resulting in the loss of crucial temporal information and difficulty in accurately selecting true keyframes. This approach sacrifices the integrity and comprehensibility of the video narrative while reducing data complexity. The long-video understanding framework proposed in this application constructs a hierarchical tree structure by iteratively segmenting, generating subtitles, and clustering long videos based on dynamic events. This event-based rather than discrete-frame-based segmentation effectively preserves temporal continuity at the video-to-event granularity and organizes video content through a coarse-to-fine tree structure, fundamentally solving the problem of interference from massive amounts of redundant information while maintaining the crucial temporal relationships necessary for accurate understanding.

[0021] Traditional keyframe extraction methods lack the ability to dynamically adjust based on specific user queries and the complex structure of videos. Most rely on static or pre-defined logic, failing to intelligently navigate through hierarchical data to efficiently locate the video segments most relevant to the question, resulting in suboptimal keyframe selection and poor question-answering performance.

[0022] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A flowchart illustrating a video content understanding method according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101 to S104.

[0024] Step S101: Cluster the multiple events included in the target video according to their semantic content to generate multiple tree nodes in the initial layer. Each tree node in the initial layer includes events clustered into the same cluster, and the video frames included in the multiple events exist in temporal order. Step S102: Based on the captions of the events included in each tree node of the initial layer, and the user intent to query the content in the target video, determine the decision action for each tree node of the initial layer; wherein, the decision action includes a delete action, an accept action, or an explore action; the delete action means deleting the corresponding tree node, the accept action means retaining the corresponding tree node, and the explore action means using the events and user intent in the corresponding tree node to determine the new decision action for the corresponding tree node. Step S103: Generate new tree nodes layer by layer according to the decision actions of each tree node in the initial layer to generate the tree nodes of the next layer, until there are no exploration actions corresponding to the decision actions in the new tree nodes, and obtain the tree structure of the target video. Step S104: Perform video understanding based on the tree nodes at each level in the tree structure that correspond to the decision action as the accept action, and obtain the answer result of the user's intent.

[0025] In the embodiments provided in this application, multiple events contained in the target video are clustered according to the semantic content of the events. Through deep analysis and matching of event semantics, events with similar semantics are grouped into the same cluster, thereby generating multiple tree nodes in the initial layer. Each initial layer tree node contains events clustered into the same cluster. It should be noted that, since the semantic content of multiple events is clustered, compared with the related technology of directly clustering video frames, the embodiments of this application maintain the temporal order within the events, avoiding the problem of temporal disorder leading to causal reversal in the video content.

[0026] Based on the caption information of the events contained in each tree node of the initial layer, and the user's query intent regarding the target video content, a decision action is made for each tree node of the initial layer. Decision actions are divided into three categories: deletion, acceptance, and exploration. A deletion action means that the tree node is irrelevant to the user's intent and will be removed; an acceptance action indicates that the tree node contains valid information related to the user's intent and will be retained until the final answer is generated; an exploration action indicates that further in-depth analysis is needed, combining the events in the tree node with the user's intent to continue generating the next layer of tree nodes.

[0027] The process of generating the next layer of tree nodes based on the decision actions of the initial layer tree nodes iteratively generates new tree nodes layer by layer. For nodes marked as exploration actions, the process of determining new decision actions based on events and user intent is repeated, continuously refining the node content and generating more granular child nodes under that tree node, allowing the tree structure to grow deeper. Nodes that accept actions are retained, while nodes that delete actions no longer participate in subsequent layer construction. This continues until no more exploration actions exist in the newly generated tree nodes, forming a complete tree structure of the target video. Based on the information contained in all tree nodes in the tree structure where the decision actions are accept actions, comprehensive video understanding and analysis are performed to obtain a response that addresses the user's intent. By using semantic content-based event clustering and hierarchical tree node construction, the temporal relationships of video frames can be effectively preserved, ensuring the contextual coherence of event development and avoiding misunderstandings caused by information fragmentation. This lays the foundation for accurately grasping the logic of video content. Combining user intent with tree node decision-making actions enables targeted and dynamic content filtering of target videos. Deletion actions filter out redundant content irrelevant to user intent, acceptance actions lock in valid information, and exploration actions ensure in-depth mining of potential relevant information, improving the targeting and efficiency of video understanding. The answers obtained from video understanding based on acceptance action nodes closely match user query needs, improving the accuracy and effectiveness of the answers and providing users with a better video content interaction experience. This approach demonstrates good adaptability to long-form video applications.

[0028] According to some embodiments provided in this application, before performing step S101: clustering according to the semantic content of multiple events included in the target video to generate multiple tree nodes of the initial layer, the method may further include the following processing: Perform grayscale processing on each video frame included in the target video to determine the brightness distribution characteristics corresponding to each video frame; The similarity between video frames is determined based on the brightness distribution characteristics of each video frame. The target video is segmented based on the similarity between each video frame, resulting in multiple events.

[0029] In some embodiments provided in this application, each video frame in the target video is processed in grayscale, converting color frames into grayscale frames. The corresponding brightness distribution characteristics are determined by calculating the normalized grayscale histogram of each grayscale frame, as the histogram effectively reflects the brightness distribution of the frame. Based on the brightness distribution characteristics, the similarity between each video frame is calculated; the smaller the distance, the closer the brightness distribution of the two frames, and the higher the similarity. Based on the similarity between video frames, frames with lower similarity (i.e., larger distance and more obvious differences) are selected as segmentation points, dividing the target video into multiple time-continuous and content-coherent events. This is because high similarity and small distance between adjacent frames mean little content change, and they can be classified as the same event, while low similarity and large distance indicate that the scene may be changing, making them suitable as segmentation boundaries. Video segmentation based on similarity calculation of brightness distribution features can capture the natural transitions between scenes in the video, ensuring the homogeneity of the content within each segmented event and the temporal continuity between events, thus avoiding the problems of fragmented events or mixed information caused by manually fixed-duration segmentation.

[0030] According to some embodiments provided in this application, determining the similarity between video frames based on the brightness distribution characteristics corresponding to each video frame may include the following steps: Based on the brightness distribution characteristics of each video frame, the distance between any two adjacent video frames in the target video is determined. The distance represents the image difference between any two adjacent video frames. In any two adjacent video frames, determine the adjacent video frames whose distance meets a predetermined condition, and use them as the dividing point for multiple events; The target video is segmented according to the segmentation points to obtain multiple events.

[0031] In the embodiments provided in this application, the segmentation point is determined by calculating the distance between adjacent frames based on the brightness distribution characteristics. This is used to capture the natural transition boundaries of scenes in the video, ensuring high image consistency of video frames within each segmented event and clear temporal relationships within the event. This avoids content fragmentation or mixing problems caused by fixed-duration segmentation, provides well-structured and reliable basic data, and effectively improves the accuracy and efficiency of the entire video understanding process. For example, the above segmentation method is a scene segmentation based on dynamically changing content, aiming to divide a long video stream into a series of coherent and temporally continuous event segments, including the following steps: a. Frame Feature Extraction. First, uniformly sample the input long video to obtain a sequence containing N frames. For each frame The system will calculate its normalized grayscale histogram. This histogram is obtained through the formula. The calculation shows that, among which and The width and height of the frame are given by i and j. i represents the pixel coordinates in the width direction of the video frame, and j represents the pixel coordinates in the height direction of the video frame. By traversing i and j, all pixels in the frame can be covered, and the number of pixels with different grayscale values ​​can be counted. For the Kronecker function, The histogram represents the grayscale value, ranging from 0 to 255. This histogram can effectively represent the brightness distribution characteristics of the frame.

[0032] b. Similarity calculation and segmentation point determination: The system then calculates the similarity between two adjacent frames ( and The content differences between them are measured using Bhattacharyya distance. To quantify it, its calculation formula is: The larger this distance value, the more drastic the content change between adjacent frames. The system will select a predetermined number... The point with the largest distance value is used as the scene switching point or segmentation point.

[0033] c. Event sequence generation: Based on these segmentation points, the original frame sequence is divided into... An independent, temporally consecutive event (or scene). This adaptive segmentation based on content changes, rather than fixed-duration cuts, ensures content homogeneity within each event and temporal coherence between events.

[0034] According to some embodiments provided in this application, the purpose of user-oriented caption generation is to generate concise and user-intent-related text descriptions for each segmented event. Each event segment... Along with the original questions raised by users These are input together into a pre-trained Vision-Language Model (VLM). The VLM will generate a text describing the event, i.e., a video annotation. This serves as the caption for the event. The generation process takes user feedback into account. Therefore, the subtitle content naturally focuses on information related to the question, improving the targeting of subsequent processing and making the video tree structure dynamic. Using VLM to generate event subtitles related to user intent, and filtering out irrelevant information by associating them with user questions, enhances the effectiveness of the subtitles. Dynamically focused subtitles provide a high-quality text foundation for subsequent event clustering, tree node decision-making, and other processing, enhancing the targeting and flexibility of the video tree structure, thereby improving the accuracy and efficiency of the entire video understanding process in responding to user intent. According to some embodiments provided in this application, in step S101: clustering according to the semantic content of multiple events included in the target video to generate multiple tree nodes of the initial layer may include the following steps: Based on the video frames and user intent included in multiple events, determine the corresponding captions for each of the multiple events; Based on the captions of multiple events, obtain the semantic vectors corresponding to each event; Clustering is performed based on the semantic vectors corresponding to multiple events to obtain multiple tree nodes in the initial layer.

[0035] In the embodiments provided in this application, video frames contained in multiple events and user intent are combined to determine corresponding subtitles for each event, so that the subtitles can reflect the semantic content of the event and be associated with the user intent. The subtitles of each event are converted into corresponding semantic vectors. Based on the semantic vectors, multiple events are clustered, and events with similar semantic vectors are grouped into the same cluster. Each cluster constitutes a tree node in the initial layer, thereby obtaining multiple tree nodes in the initial layer. By combining video frames and user intent to determine subtitles, the subtitle information becomes more targeted, laying a solid foundation for subsequent processing. Generating initial layer tree nodes based on semantic vector clustering effectively aggregates semantically similar events, avoiding the direct clustering of video frames found in related technologies. This prevents disruption of the temporal order between video frames and improves the logical coherence of the target video's content understanding. For example, semantic-based event clustering and tree node formation group semantically similar events together to form a tree-like node structure.

[0036] a. Video annotation vectorization: Using a text embedding model to vectorize the captions for each event. Convert to a high-dimensional numerical vector .

[0037] b. Density-based clustering: A density-based clustering algorithm (DBSCAN) is used to cluster the caption vector sets of all events. Density-based clustering algorithms can group events with similar semantic content (i.e., vectors that are close in distance in the vector space) into the same cluster.

[0038] c. Tree node construction: Each generated cluster constitutes a tree node. This node represents a set of events that may not appear consecutively in the target video but have similar content. Generating the initial layer of the tree nodes completes the construction of the first layer of the target video's tree structure. The generation process of these tree nodes is iterative. For any node in the tree, if deeper exploration is needed (i.e., the decision action is an exploration action), all video segments contained in that tree node can be merged into a new, shorter sub-video. This new sub-video is then processed in the same way as the initial layer of the target video's tree nodes, repeating the complete segmentation-subtitle generation-clustering process. This generates finer-grained child nodes under that tree node, allowing the tree to grow to deeper levels.

[0039] After constructing the dynamic video tree, it is necessary to further find the tree nodes containing key information about the answer to the question. The embodiments provided in this application design a tree-structured group relative policy optimization (T-GRPO) algorithm, which is a reinforcement learning method specifically designed for tree structure optimization, used to train a policy model to intelligently guide the exploration process.

[0040] According to some embodiments provided in this application, in step S102: clustering according to the similarity between multiple events included in the target video to generate multiple tree nodes of the initial layer may include the following steps: The user intent, along with the captions, durations, and depth information of the events included in each tree node of the initial layer, are input into the target policy model for processing to determine the decision action for each tree node in the initial layer; the depth information is used to represent the hierarchical position of the corresponding tree node in the tree structure. The target strategy model iterates based on a composite reward system that includes node-level rewards and tree-level rewards. Node-level rewards represent the quality of a single decision made by a single tree node, while tree-level rewards represent the quality of a single tree node's global decision made in response to the answer.

[0041] In the embodiments provided in this application, the user's intent, along with the captions, durations, and depth information representing the hierarchical position of each event contained in the initial layer of each tree node, are input into the target strategy model for comprehensive processing to determine the decision action for each tree node in the initial layer. The aforementioned target strategy model is iteratively optimized based on a composite reward system that includes node-level and tree-level rewards. Node-level rewards measure the quality of a single-point decision made by a single tree node, while tree-level rewards evaluate the overall decision quality of a single tree node on the final answer result, enabling the model to make more accurate decisions. This composite reward-based iterative target strategy model considers both the decision quality of individual nodes and their impact on the overall answer result, making the decision more global and effectively filtering out valuable tree nodes, thus improving the accuracy and effectiveness of the entire processing.

[0042] For example, the target policy model described above evaluates each tree node in the tree structure and determines the next action. For any node in the tree... The input to the strategy model consists of three parts: captions for all events within the tree node, the user's original question, and so on. (i.e., the aforementioned user intent), and the depth information of the tree node within the tree. This ensures that the model's decisions comprehensively consider the content of the tree node, the user's intent, and the node's position within the tree structure. The target decision model outputs one of three possible actions for each tree node: accept, indicating that the node contains sufficient key information to answer the question and is part of the final answer, requiring no further exploration; explore, indicating that the node may be relevant to the question, but the information is insufficient, requiring iteration to generate child nodes for deeper exploration; or delete, indicating that the tree node's content is irrelevant to the question and should be discarded, its branch will be pruned, and further exploration will cease.

[0043] According to some embodiments provided in this application, node-level rewards include format rewards, length rewards, and action rewards. Format rewards are used to restrict the format of the decision actions output by the target policy model, length rewards are used to enhance the decision-making content of the target policy model, and action rewards are used to assign reward values ​​to different decision actions. The method may also include the following steps: The first reward is obtained based on the sum of the length reward and the action reward; The second reward is obtained by multiplying the first reward by the tree-level reward; A composite reward is obtained based on the first and second rewards.

[0044] In the embodiments provided in this application, a composite reward function comprising node-level and tree-level rewards is designed to enable the target policy model to learn to make optimal decisions. The node-level reward evaluates the quality of decisions made at individual nodes and is further subdivided into three sub-items: format reward, length reward, and action reward. The format reward ensures that the output format is standardized, the length reward encourages the model to generate more detailed thought processes, and the action reward assigns different immediate reward values ​​to three different actions. The tree-level reward evaluates whether the final answer obtained from the entire exploration path (i.e., the entire tree structure) is correct. The tree-level reward forces the target policy model to consider the global impact of its current node decision on the final result, thereby enabling it to possess global thinking capabilities.

[0045] The length reward and action reward are added together to obtain the first reward. This first reward is then multiplied by the tree-level reward to obtain the second reward. Finally, the first and second rewards are combined to obtain the composite reward. The target policy model iteratively optimizes based on this composite reward to more accurately determine the decision action for each tree node. This composite reward calculation method organically combines local node-level rewards with global tree-level rewards, allowing the model to consider both the rationality of individual decisions and their impact on the overall response during iteration. This provides strong support for optimizing the tree structure and accurately responding to user intent.

[0046] For example, for compound reward calculation, the final total reward can be represented by the following formula: .

[0047] in, Indicates a compound reward. Indicates a formatted reward. Indicates length bonus. Indicates a reward for the action. This indicates a tree-level reward.

[0048] According to some embodiments provided in this application, the aforementioned depth information can represent the current exploration depth within the tree structure and can be used to adjust the target strategy model, thereby controlling the model's tendency to continue exploring or terminate exploration. A substance called Tree Growth Auxin is introduced. The hyperparameter of the method is defined as the ratio of the average reward for accepting and deleting actions to the reward for continuing actions. The method may also include: Determine the acceptance rewards for each tree node whose corresponding decision action is "accept", the deletion rewards for each tree node whose corresponding decision action is "delete", and the exploration rewards for each tree node whose corresponding decision action is "explore". Acceptance rewards, deletion rewards, and exploration rewards are node-level rewards. The average reward is obtained by averaging the accepted and deleted rewards. The predetermined hyperparameters set in the target strategy model are adjusted according to the ratio between the average reward and the exploration reward. These predetermined hyperparameters are used to control the hierarchical depth of the target strategy model's generation tree structure.

[0049] In the embodiments provided in this application, the acceptance reward, deletion reward, and exploration reward corresponding to each tree node for the decision action of accepting, deleting, and exploring are determined respectively. These three rewards are all node-level rewards. The average reward is obtained by calculating the average of the acceptance reward and the deletion reward. According to the ratio of the average reward to the exploration reward, the predetermined hyperparameter used to control the depth of the tree structure hierarchy in the target strategy model is adjusted. The hyperparameter is dynamically adjusted by changing the ratio, thereby controlling the depth of tree structure generation. This ensures sufficient exploration while avoiding inefficiency caused by over-exploration, thus improving the efficiency of the target strategy model's decision-making.

[0050] For example, by adjusting This allows control over the degree to which the target policy model tends to terminate the exploration early (accept / delete) or continue exploring further, thus achieving a dynamic balance between the accuracy and efficiency of the exploration. After a complete rollout, the system calculates the advantage of each decision. Then, it utilizes a loss function specifically designed for tree structures. This is used to update the network parameters of the policy model. The loss function ensures the stability of the training process through techniques such as clipping.

[0051] To facilitate understanding, an example of the training process is provided, such as... Figure 2 As shown, for the same target video, a target strategy model is used for processing. Due to the randomness of the model output, the tree structure of the q outputs can be illustrated as follows. to The reference model can be considered as an untrained policy model. KL divergence is used to control the difference between the policy model and the reference model, thus limiting the training direction of the policy model from becoming excessively biased. This is achieved by calculating... to The node-level and tree-level rewards can yield a composite reward for a tree structure with q outputs, illustrated as follows: to .based on to The relative dominance value of each tree structure can be calculated, as illustrated below. to This can reflect the advantages of different generated tree structures, which can be used to evaluate the loss function. Iterative strategy model.

[0052] In step S104: Video understanding is performed based on the tree nodes in the tree structure corresponding to the decision action of acceptance, to obtain the answer result of the user's intent. After the exploration process guided by the group-based strategy optimization of the tree structure is completed, the caption content in all nodes marked as acceptance by the strategy model is collected. These selected key information sets, along with the user's original question, are input into the large language model for the last time. The large model integrates all key information to generate a logically coherent and accurate final answer.

[0053] Compared to existing top-tier hierarchical video processing technologies, such as VideoTree, this application's embodiments, while maintaining the advantage of information structuring, overcome their key shortcomings through a unique technical solution, offering the following significant advantages: By constructing a hierarchical tree using segmentation and clustering methods based on dynamic events rather than discrete frames, this effectively organizes video content, reduces information redundancy, and maximizes the preservation of the video's original temporal continuity, overcoming the fundamental flaw of existing technologies that suffer from contextual information loss due to disruption of temporal structure. Furthermore, the tree-based intra-group policy optimization algorithm, through its composite reward mechanism encompassing both tree-level and node-level metrics, intelligently and query-drivenly explores the video tree, achieving precise dynamic positioning of keyframes and solving the problem of insufficient intelligence and accuracy in keyframe selection in existing technologies. This reinforcement learning process also incentivizes the model to spontaneously generate inference chains, greatly enhancing the model's deep reasoning capabilities when dealing with complex problems—a capability not found in existing technologies.

[0054] Based on the above embodiments and optional embodiments, this application also provides an optional implementation method that transforms an unstructured long video into a structured, hierarchical tree representation according to its content and user questions. This process not only greatly reduces the complexity of information processing, but also, through special design, preserves the original chronological order of the video to the greatest extent possible. The entire process can be referred to... Figure 3 As shown, it includes the following three core steps: step S1 scene segmentation based on dynamic content changes, step S2 subtitle generation based on user questions, and step S3 event clustering and tree node formation based on semantics.

[0055] Step S1 is scene segmentation based on dynamically changing content. The goal is to segment a long video stream into a series of coherent, temporally continuous event segments, including the following steps: a. Frame Feature Extraction. First, uniformly sample the input long video to obtain a sequence containing N frames. For each frame The system will calculate its normalized grayscale histogram. This histogram is obtained through the formula. The calculation shows that, among which and The width and height of the frame are given by i and j. i represents the pixel coordinates in the width direction of the video frame, and j represents the pixel coordinates in the height direction of the video frame. By traversing i and j, all pixels in the frame can be covered, and the number of pixels with different grayscale values ​​can be counted. For the Kronecker function, The histogram represents the grayscale value, ranging from 0 to 255. This histogram can effectively represent the brightness distribution characteristics of the frame.

[0056] b. Similarity calculation and segmentation point determination: The system then calculates the similarity between two adjacent frames ( and The content differences between them are measured using Bhattacharyya distance. To quantify it, its calculation formula is: The larger this distance value, the more drastic the content change between adjacent frames. The system will select a predetermined number... The point with the largest distance value is used as the scene switching point or segmentation point.

[0057] c. Event sequence generation: Based on these segmentation points, the original frame sequence is divided into... An independent, temporally consecutive event (or scene). This adaptive segmentation based on content changes, rather than fixed-duration cuts, ensures content homogeneity within each event and temporal coherence between events.

[0058] By iteratively segmenting long videos based on dynamic events, generating subtitles related to user questions, and then performing semantic clustering, unstructured long videos are transformed into a hierarchical tree data structure that is coarse-to-fine while maintaining key temporal information. This solves the contradiction between information redundancy and temporal information loss in existing technologies when processing long videos.

[0059] Step S2 involves generating captions addressing user questions. The goal is to generate concise and user-intent-related text descriptions for each segmented event. This involves generating captions for each event segment. Along with the original questions raised by users These are input together into a pre-trained Vision-Language Model (VLM). The VLM will generate a text describing the event, i.e., a video annotation. This serves as the caption for the event. The generation process takes user feedback into account. Therefore, the subtitle content naturally focuses on information related to the question, improving the targeting of subsequent processing and making the video tree structure dynamic. Using VLM to generate event subtitles related to user intent, and filtering out irrelevant information by associating them with user questions, enhances the effectiveness of the subtitles. Dynamically focused subtitles provide a high-quality text foundation for subsequent event clustering, tree node decision-making, and other processing, enhancing the targeting and flexibility of the video tree structure, thereby improving the accuracy and efficiency of the entire video understanding process in responding to user intent. Step S3 involves semantic event clustering and tree node formation. Semantically similar events are grouped together to form a tree-like node structure.

[0060] a. Video annotation vectorization: Use a text embedding model to convert the caption_m of each event into a high-dimensional numeric vector v_m.

[0061] b. Density-based clustering: A density-based clustering algorithm (DBSCAN) is used to cluster the caption vector sets of all events. Density-based clustering algorithms can group events with similar semantic content (i.e., vectors that are close in distance in the vector space) into the same cluster.

[0062] c. Tree Node Construction: Each generated cluster constitutes a tree node N_p. This node represents a group of events that may not appear consecutively in the target video but have similar content. Generating the initial layer of tree nodes completes the construction of the first layer of the target video's tree structure. The above tree node generation process is iterative. For any node in the tree, if deeper exploration is needed, i.e., the decision action is an exploration action, all video segments contained in that tree node can be merged and treated as a new, shorter sub-video. This new sub-video is then processed in the same way as the initial layer of tree nodes generated for the target video, repeating the complete process of segmentation-subtitle generation-clustering. This generates finer-grained child nodes under that tree node, allowing the tree to grow to deeper levels.

[0063] After constructing the dynamic video tree, it is necessary to further find the tree nodes containing key information about the answer to the question. The embodiments provided in this application design a tree-structured group relative policy optimization (T-GRPO) algorithm, which is a reinforcement learning method specifically designed for tree structure optimization, used to train a policy model to intelligently guide the exploration process.

[0064] The aforementioned objective strategy model evaluates each node in the tree structure and determines the next action. For any node in the tree... The input to the strategy model consists of three parts: captions for all events within the tree node, the user's original question, and so on. (i.e., the aforementioned user intent), and the depth information of the tree node within the tree. This ensures that the model's decisions comprehensively consider the content of the tree node, the user's intent, and the node's position within the tree structure. The target decision model outputs one of three possible actions for each tree node: accept, indicating that the node contains sufficient key information to answer the question and is part of the final answer, requiring no further exploration; explore, indicating that the node may be relevant to the question, but the information is insufficient, requiring iteration to generate child nodes for deeper exploration; or delete, indicating that the tree node's content is irrelevant to the question and should be discarded, its branch will be pruned, and further exploration will cease.

[0065] The tree-based group relative policy optimization algorithm features a reward function and training mechanism tailored to the tree structure. Regarding the reward function design, to enable the policy model to learn to make optimal decisions, a composite reward function is designed, incorporating both node-level and tree-level rewards. The final total reward can be represented by the following formula: .

[0066] in, Indicates a compound reward. Indicates a formatted reward. Indicates length bonus. Indicates a reward for the action. This indicates a tree-level reward.

[0067] This optional implementation introduces a substance called tree growth hormone ( The hyperparameter of is defined as the ratio of the average reward for accepting and deleting actions to the reward for continuing actions. This is achieved by adjusting... This allows control over how much the model tends to terminate exploration early (accept / delete) or continue exploring further, thus achieving a dynamic balance between accuracy and efficiency. After a complete exploration, the system calculates the advantage of each decision. Then, it utilizes a loss function specifically designed for tree structures. This is used to update the network parameters of the policy model. The loss function ensures the stability of the training process through techniques such as pruning.

[0068] The tree-structured group relative policy optimization algorithm dynamically makes decisions on accepting, continuing, or deleting tree nodes by training a policy model. It includes a composite reward function at the node and tree levels, as well as a tree auxin mechanism that can dynamically adjust the exploration depth, thereby achieving efficient, accurate, and query-oriented localization of keyframes.

[0069] Guiding models to generate reasoning chains through reinforcement learning strategies can incentivize them to spontaneously generate "thought-decision" chains when making decisions. This emergent reasoning ability is not merely a simple decision output, but an explanation of the decision-making process, greatly enhancing the model's deep reasoning and analytical capabilities in complex long-form video question-answering tasks.

[0070] Corresponding to the application scenarios and methods provided in the embodiments of this application, such as Figure 4 As shown in the illustration, this application also provides a video content understanding device, including: The node generation module 401 is used to cluster multiple events included in the target video according to their semantic content, and generate multiple tree nodes in the initial layer. Each tree node in the initial layer includes events clustered into the same cluster, and the video frames included in the multiple events exist in temporal order. The decision action determination module 402 is used to determine the decision action for each tree node in the initial layer based on the captions of the events included in each tree node of the initial layer and the user intent to query the content in the target video. Among them, decision actions include deletion actions, acceptance actions, or exploration actions; the deletion action means deleting the corresponding tree node, the acceptance action means retaining the corresponding tree node, and the exploration action means determining the new decision action of the corresponding tree node based on the events and user intent in the corresponding tree node. The tree structure generation module 403 is used to generate new tree nodes layer by layer according to the decision actions of each tree node in the initial layer, until there are no exploration actions corresponding to the decision actions in the new tree nodes, and obtain the tree structure of the target video. The result generation module 404 is used to perform video understanding based on the tree nodes at each level of the tree structure where the corresponding decision action is the accept action, and to obtain the answer result of the user's intent.

[0071] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0072] Figure 5 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the method described in the above embodiments. The number of memories 501 and processors 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and exchanging data.

[0073] In practical implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0074] Optionally, in a specific implementation, if the memory 501, processor 502 and communication interface 503 are integrated on a single chip, the memory 501, processor 502 and communication interface 503 can communicate with each other through an internal interface.

[0075] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0076] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0077] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0078] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0079] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0080] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0081] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0084] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0085] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0086] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0088] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video content understanding method, characterized in that, include: Clustering is performed on multiple events included in the target video according to their semantic content to generate multiple tree nodes in the initial layer. Each tree node in the initial layer includes events clustered into the same cluster, and the video frames included in the multiple events are in temporal order. Based on the captions of events included in each tree node of the initial layer, and the user intent to query the content of the target video, the decision action for each tree node of the initial layer is determined. The decision action includes a delete action, an accept action, or an explore action; the delete action means deleting the corresponding tree node, the accept action means retaining the corresponding tree node, and the explore action means determining a new decision action for the corresponding tree node based on the events in the corresponding tree node and the user intent. The tree nodes of the next layer are generated layer by layer according to the decision actions of each tree node in the initial layer, until the decision action corresponding to the new tree node does not contain the exploration action, thus obtaining the tree structure of the target video; Based on the tree nodes in the tree structure that correspond to the decision action as the acceptance action, video understanding is performed to obtain the answer result of the user's intent.

2. The method according to claim 1, characterized in that, Before clustering based on the semantic content of multiple events included in the target video to generate multiple tree nodes for the initial layer, the method further includes: Grayscale processing is performed on each video frame included in the target video to determine the brightness distribution characteristics corresponding to each video frame; The similarity between the video frames is determined based on the brightness distribution characteristics corresponding to each video frame. The target video is segmented based on the similarity between the video frames to obtain the multiple events.

3. The method according to claim 2, characterized in that, Determining the similarity between video frames based on the brightness distribution characteristics corresponding to each video frame includes: Based on the brightness distribution characteristics corresponding to each video frame, the distance between any two adjacent video frames in the target video is determined, and the distance represents the image difference between any two adjacent video frames. In any two adjacent video frames, determine the adjacent video frames whose distance satisfies the predetermined condition, and use them as the dividing points of the multiple events; The target video is segmented according to the segmentation points to obtain the multiple events.

4. The method according to claim 1, characterized in that, The step of clustering based on the semantic content of multiple events included in the target video to generate multiple tree nodes in the initial layer includes: Based on the video frames included in the multiple events and the user intent, determine the subtitles corresponding to the multiple events respectively; Based on the captions of the multiple events, obtain the semantic vectors corresponding to each of the multiple events; Clustering is performed based on the semantic vectors corresponding to the multiple events to obtain multiple tree nodes of the initial layer.

5. The method according to claim 1, characterized in that, The step of performing video understanding based on the tree nodes in the tree structure corresponding to the decision action as the acceptance action, to obtain the answer result of the user intent, includes: Based on the subtitles included in each tree node of the corresponding decision action as the receiving action, target information is generated; The answer result is obtained by analyzing the target information and the user intent.

6. The method according to any one of claims 1 to 5, characterized in that, The step of clustering based on the similarity between multiple events included in the target video to generate multiple tree nodes in the initial layer includes: The user intent and the captions, durations, and depth information of the events included in each tree node of the initial layer are input into the target strategy model for processing to determine the decision action for each tree node of the initial layer; the depth information is used to represent the hierarchical position of the corresponding tree node in the tree structure. The target strategy model iterates based on a composite reward system that includes node-level rewards and tree-level rewards. The node-level rewards represent the quality of a single-point decision made by a single tree node, and the tree-level rewards represent the quality of the global decision made by the single tree node in relation to the answer result.

7. The method according to claim 6, characterized in that, The node-level rewards include format rewards, length rewards, and action rewards. The format rewards are used to restrict the format of the decision actions output by the target policy model. The length rewards are used to enhance the decision-making content of the target policy model. The action rewards are used to assign reward values ​​to different decision actions. The method further includes: The first reward is obtained based on the sum of the length reward and the action reward; The second reward is obtained by multiplying the first reward by the tree-level reward; The composite reward is obtained based on the first reward and the second reward.

8. The method according to claim 6, characterized in that, The method further includes: The acceptance reward for each tree node corresponding to the decision action of accepting, the deletion reward for each tree node corresponding to the decision action of deleting, and the exploration reward for each tree node corresponding to the decision action of exploring are determined; the acceptance reward, the deletion reward, and the exploration reward belong to the node-level rewards. The average reward is obtained by averaging the accepted reward and the deleted reward. The predetermined hyperparameters set in the target strategy model are adjusted according to the ratio between the average reward and the exploration reward. The predetermined hyperparameters are used to control the hierarchical depth of the tree structure generated by the target strategy model.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images