A video semantic information extraction enhancement and optimization method for complex event situation understanding

Abstract

This invention relates to a method for extracting, enhancing, and optimizing video semantic information for complex event situational understanding. The method includes: Step 1. Semantic information extraction and enhancement based on hierarchical division; Step 2. Targeted supplementation of event elements based on key nodes. The superior technical effect of the method of this invention is that, compared with existing methods that directly select and focus on local key information, the method of this invention first performs targetless video semantic enhancement on long videos through three levels: complete video level, video segment level, and keyframe level, and then carries out targeted supplementation around key nodes and missing event elements. While focusing on key content, it can reduce the loss of local details and auxiliary clues in the early processing stage.

Description

Technical Field

[0001] This invention relates to the fields of multimodal data fusion, video understanding, knowledge graph construction, event extraction and information completion, and specifically to a method for enhancing and optimizing video semantic information extraction for complex event situation understanding. Background Technology

[0002] In complex event analysis tasks, the acquisition of event information typically relies on the combined support of multiple modalities, including text, images, and videos. Among these, video modality is a key data source for complex event understanding because it can directly convey event processes, subject interactions, and dynamic scene changes. However, unlike text modality, which can directly express semantics in discrete symbolic form, information in videos exists implicitly as continuous temporal visual signals, making it difficult to directly and fully utilize. Typically, video understanding tasks such as detection, question answering, summarization, and description are needed to extract and organize the semantics of video content, further forming structured or unstructured semantic representations before they can be used for event analysis and understanding. Therefore, existing video understanding methods usually utilize multimodal models or visual language models, employing methods such as video caption generation, video question answering, video retrieval, or visual language reasoning to extract and express the semantics of scenes, characters, actions, and event processes in videos.

[0003] To address the computational overhead and temporal redundancy issues associated with long videos, existing video understanding methods generally employ various compression strategies to reduce computational complexity. Depending on the stage at which compression occurs, related methods can be broadly classified into two categories: The first category is input-side compression methods, which involve compression before the video enters the model. This is achieved through uniform sampling (see reference 1: Lin B, Zhu B, Ye Y, et al. Video-LLaVA: Learning united visual representation by alignment before projection[C] / / Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2024.), sparse sampling (see reference 2: Lei J, Li L, Zhou L, et al. Less isMore: ClipBERT for Video-and-Language Learning via Sparse Sampling[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society, 2021: 7331-7341.), and adaptive keyframe selection (see reference 3: Tang X, et al. Adaptive Keyframe). Methods such as Sampling for Long Video Understanding[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2025. can reduce the number of input frames to reduce the computational burden on subsequent tasks (see reference 4: Miech A, Laptev I, Sivic J).Learnable pooling with contextgating for video classification [C] / / Proceedings of the International Conference on Learning Representations. San Juan: ICLR, 2017), weight-based feature aggregation (see reference 5: Fu Y, Wang Y, Wang Letal., CompFeat: Comprehensive feature aggregation for video question answering [C] / / Proceedings of the AAAIConference on Artificial Intelligence. Palo Alto: AAAI Press, 2021, 35(2):1303-1311.), and token-level compression (see reference 6 iang J, Sangineto E, Zhong Y, et al. STORM: Token efficient understanding for multimodal LLMs [C]Proceedings of the long video IEEE / CVFInternational Conference on Computer Vision Workshops. LoAlamitos: IEEE Computer Society, Methods such as those mentioned above (e.g., 2025) compress the intermediate representation size and reduce redundant features. Although these methods act on the input side and feature side respectively, they essentially belong to the technical route of compressing long video information at different stages and to different degrees. These methods can improve video understanding efficiency to some extent, but they are also prone to losing key segments in the event evolution during compression, weakening multi-stage temporal relationships, and diminishing the expression of important information such as local scene details, subject attributes, and state changes. They are particularly detrimental to the complete modeling of the continuous evolution process of multiple sub-events in complex event videos.

[0004] In addition to the feature compression methods mentioned above, there is also a class of video understanding methods in the existing technology that focus on the selection of key information. These methods typically no longer attempt to retain all video content indiscriminately, but instead start from task relevance, using attention allocation and question-guided feature selection (see reference 7: Wang W, Bao J, Xu D, et al. Longvideo question answering: A matching-guided attention model[J]. PatternRecognition, 2020, 107: 107480.), adaptive frame-level attention[3], fragment-level content localization (see reference 8: Du H, Zhang S, Xie B, et al. Uncovering what, why and how: A comprehensive benchmark for causation understanding of video anomaly[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos:IEEE Computer Society, 2024: 18793-18803.) and memory retrieval (see reference 9 (Qian R, DongX, Zhang P, et al. Streaming Long Video Understanding with Large Language Models[C] / / Advances in Neural Information Processing)). Mechanisms such as Systems. Red Hook: Curran Associates, 2024. prioritize and focus on key frames, key segments, or local features in videos, allowing the model to focus on the local content most relevant to the current task, thereby improving understanding efficiency and task relevance. While these methods mitigate the information dilution problem caused by indiscriminate compression to some extent, their scope is often limited by task cues, relevance scoring, or local selection strategies, easily overlooking auxiliary cues that are equally important for understanding subsequent events but currently less significant. Furthermore, existing methods often lack a unified multi-granularity organization and continuous supplementation mechanism for the relationships between multiple sub-events in complex events, the interactions between different subjects, and the complete coverage of event elements.

[0005] Building upon this foundation, some studies have further combined textual captions, images, or external knowledge information to supplement and interpret video semantics, thereby enhancing overall event comprehension. With the development of graph structure modeling, knowledge graphs, and large language models, existing methods have gradually expanded from early feature splicing and representation alignment to a fusion approach that incorporates entity relationship modeling, event structure representation, and knowledge-driven reasoning. Some methods construct entity relation graphs within videos or cross-frame dynamic graphs to uniformly model the entities, relationships, and temporal evolution processes in videos, thereby enhancing the ability to model object associations and locate key content in long video understanding (see Reference 10: Chu M, Li Y, Chua T S. Understanding longvideos via LLM-powered entity relation graphs[EB / OL]. arXiv:2501.1592025[2026-04-22].). Other methods further combine entity-event-scene knowledge graphs to map multimodal semantics from videos, images, and texts into a unified knowledge structure, and utilize knowledge-driven matching and reasoning mechanisms to improve the interpretation and analysis capabilities of complex events (see Reference 11: (Li W, Wang C, Cui X, et al. COSINT-Agent: A knowledge-driven multimodal agent for Chinese open source intelligence[EB / OL]. arXiv:2503.03215, 2025 [2026-04-22].); In addition, there are methods that supplement the background knowledge, event connections and potential relationships in multi-source information by introducing external event knowledge graphs to enhance cross-source reasoning ability in complex event scenarios (see reference 12: Yang ZR, Wang YL, Shi ZY, et al. EventRAG: Enhancing LLM generation with event knowledge graphs[C] / / Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2025.).However, most existing technologies are still limited to information aggregation, shallow alignment, or single-round reasoning. They are still insufficient in identifying missing information between different modalities, screening candidate evidence, and providing targeted completion for specific event elements. As a result, the event information in the fusion results often appears weakly structured, fragmented, and missing in terms of elements.

[0006] Therefore, while existing technologies have made some progress in long video compression, key information selection, and knowledge-driven reasoning, they still struggle to simultaneously ensure the continuity of the overall event process, the full expression of local details, the effective organization of cross-modal semantic relationships, and the complete coverage of key event information in video understanding tasks targeting complex events. This results in problems such as coarse expression, loose structure, and missing elements in complex event information, making it difficult to meet the needs of subsequent refined event analysis and situational understanding. Current video understanding technologies for complex event analysis, although capable of extracting some semantic information from videos, still have significant shortcomings in serving complex event understanding and situational analysis.

[0007] First, while existing video understanding methods based on compression strategies can improve processing efficiency to some extent, they essentially compress the original video information in different forms. This can easily lead to the loss of key segments in the development of events, weakening of multi-stage temporal relationships, and insufficient expression of local scene details. As a result, important information in the video that reflects the evolution of events, the behavior of people, the state of the environment, and changes in the outcome cannot be fully preserved. Ultimately, only a relatively coarse-grained semantic representation of the video can be obtained, which is not conducive to the continuous process modeling and detailed understanding of complex events.

[0008] Secondly, while existing video understanding methods focused on key information selection no longer indiscriminately compress all video content, they utilize mechanisms such as attention allocation, question-guided feature selection, frame-level attention, segment-level localization, and memory retrieval to prioritize and focus on the most relevant local content for the current task, thus alleviating the information dilution problem to some extent. However, the scope of these methods is often limited by task prompts, relevance scoring, or local candidate selection strategies, often highlighting only the most significant and directly relevant local content while easily overlooking auxiliary cues that are equally important but less significant for understanding subsequent complex events. Furthermore, existing methods typically lack a unified multi-granularity organization and continuous supplementation capability for the continuous connections between multiple sub-events in complex events, the interactions between different subjects, and the overall coverage of key event information, making it difficult to balance local focus with overall integrity.

[0009] Finally, while existing multimodal methods combining graph structure modeling, knowledge graphs, and external knowledge enhancement have gradually evolved from early feature splicing and representation alignment to structured organization and auxiliary interpretation of semantic information in videos, images, and texts through entity relationship graphs, event knowledge graphs, and knowledge-driven reasoning mechanisms, achieving some progress in object association modeling, event relationship expression, and cross-source reasoning, most methods remain at the stage of information aggregation, shallow alignment, or single-round reasoning. Existing methods are generally insufficient for identifying missing information between different modalities, screening candidate evidence, tracing cross-granularity semantic clues, and targeted completion of specific event elements. This often results in fragmented event information expression, unclear structure, and incomplete element coverage in the fusion results, making it difficult to form a truly continuous, complete, and interpretable structured expression for complex events. Summary of the Invention

[0010] To address the aforementioned issues, this invention proposes a video semantic enhancement and optimization method for complex event situation understanding. Firstly, addressing the problems of key information loss and coarse-grained semantic expression in long video processing, this method extracts and enhances information at three levels: complete video, video segments, and keyframes, through dynamic segmentation, keyframe extraction, and hierarchical semantic fusion. This extracts semantic content related to the event process, local scenes, and static details from the video modality, effectively supplementing video information. Secondly, by combining event graph structure, event element modeling, and missing information detection mechanisms, missing event element nodes or potential relationships are used as guides to selectively extract and accurately complete important information related to the core elements of the event from multimodal candidate sources, further supplementing and optimizing the video information extraction results. Through these two progressive processing rounds, this method enhances the video's ability to effectively express information, improves the relevance of the added information to event situation understanding, and strengthens the final fusion result's ability to represent event elements in a complete and structured manner.

[0011] The video semantic information extraction, enhancement, and optimization method for complex event situation understanding specifically includes:

[0012] Step 1. Enhanced semantic information extraction based on hierarchical partitioning:

[0013] To address the issue that long video data often requires sampling and compression due to the limited number of input tokens when inputting into multimodal or visual-language models, leading to the loss of temporal information and local details, this paper proposes a method to supplement and enrich the original event information by hierarchically dividing the video and performing multi-granular semantic extraction at the segment and frame levels. Here, the token is the basic unit for information encoding and computation in the multimodal model. The content of different modalities, such as text, image blocks, and video frames, needs to be uniformly converted into token representations and then used as model input for processing.

[0014] Step 2. Targeted supplementation of event elements based on key nodes:

[0015] To address the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement, knowledge-driven methods are used to complete important information related to the core elements of the event in a goal-driven manner. Through the above two rounds of progressive processing, the structured enhancement and complete expression of complex event situation information are achieved.

[0016] Furthermore, regarding the issue in step 1 that long video data, when input into a multimodal model or vision-language model, requires sampling and compression due to the limited number of input tokens, leading to the loss of temporal information and local details, this step supplements and enriches the original event information by hierarchically dividing the video and performing multi-granular semantic extraction at the segment and frame levels. Specifically, this includes:

[0017] Step 1.1 Obtain input multimodal data and construct a unified semantic input:

[0018] The process involves acquiring raw multimodal data corresponding to the event to be analyzed, including video data and text data. Video data serves as the core object for semantic information extraction and enhancement, while text data serves as the information source for subsequent semantic fusion. To facilitate unified analysis, the video modal data is first preprocessed using existing video understanding and visual language models to derive a general description of the video content, which then serves as the text semantic representation for further processing. ;

[0019] Step 1.2 Segment the video data using content change detection:

[0020] Extract consecutive video frames from the input video. Dynamically divide the long video into multiple semantically consistent and temporally continuous video segments based on content changes between adjacent frames. Let the i-th segment be... Frame and the The frames are denoted as follows: and Its inter-frame variation score Represented as:

[0021] ,

[0022] in, Indicates the number of visual feature types used. Indicates the first The weights corresponding to visual features, This indicates that the video frame is being processed. Mapping function for visual feature extraction, This represents the difference measure function between corresponding features, calculated using norm distance, cosine distance, statistical difference measure, or other similarity / difference calculation methods. Visual features include: color space-based visual features, edge features, optical flow features, features extracted by deep learning models based on CNN and VIT, and cross-modal semantic features extracted by visual language models or other pre-trained multimodal models. Different features can be used individually according to actual application needs, or they can be combined in a weighted manner to improve the ability of content change detection to perceive scene transitions, subject changes, action changes, or semantic transitions. At that time, the judgment of the first There are content change boundaries near the frame, where, A preset threshold is set based on the segmentation accuracy requirements of the target task and the system's computing resources. A smaller threshold results in more detailed segmentation but increases computational overhead, while a larger threshold reduces the number of segments but ignores some fine-grained changes. Based on the detected boundary points, the original long video is segmented... Divided into A series of segments ;in The value range is 3.0-9.0;

[0023] Step 1.3 Extract keyframes from each video segment:

[0024] For each video segment obtained in step 2, keyframes are further extracted from each video segment to supplement the static scene details that are ignored in the complete video-level or segment-level description, including people, locations, environment and local behavioral features. The keyframe extraction methods include time-sampling-based keyframe extraction, visual feature clustering-based keyframe extraction, and deep learning model-based keyframe extraction.

[0025] Step 1.4 Generate fragment-level and keyframe-level description information respectively:

[0026] Semantic description information is generated for video segments and keyframes respectively. Segment-level description is used to preserve local time process and event stage information, while keyframe-level description is used to enhance the expression of static details and local semantics. Segment-level description and keyframe description can be obtained by multimodal models such as video understanding models or vision-language models.

[0027] Each video segment is input into the video understanding model for local semantic reasoning, resulting in a corresponding segment-level description:

[0028] ,

[0029] in, Indicates the first The segment-level description results corresponding to each segment, in this way, while maintaining the temporal correlation of events, separate the multi-event content that is easily mixed up under the uniform compression of the whole video, so as to alleviate the semantic ambiguity caused by long sequence feature compression;

[0030] Input the keyframes into the multimodal model to generate corresponding keyframe-level descriptions:

[0031] ,

[0032] in, Indicates the first The first segment One keyframe This represents the corresponding keyframe description text, which is used to supplement static visual information that is ignored in fragment-level semantics, including scene layout, subject attributes, environmental features, and local detail cues.

[0033] Let the complete video-level description obtained from preprocessing in step 1.1 be... , No. Each fragment level description is , No. The first segment Each keyframe is described as Then, the semantic set of the video after semantic information extraction and enhancement is represented as:

[0034] ,

[0035] in, Indicates the number of video segments. Indicates the first The number of keyframes in a video clip;

[0036] Step 1.5 Construct fragment-level local semantic units and align them with frame-level descriptions:

[0037] To achieve fine-grained fusion of keyframe information and video segment information, it is necessary to first divide the segment-level description information into local semantic units, let the first unit be... Fragment-level description It is divided into several local semantic units, represented as:

[0038] ,

[0039] in, Indicates the first The number of local semantic units in a video segment-level information. Indicates the first The first fragment-level description Each local semantic unit corresponds to a sentence, phrase, or local event semantic fragment in the fragment description.

[0040] To naturally integrate keyframe details into fragment descriptions, a vector encoding function is used to represent fragment-level local semantic units and keyframe descriptions using vectors:

[0041] ,

[0042] ,

[0043] in, This represents the sentence vector encoding function. Indicates the first The first segment Vector representation of local semantic units, Indicates the first The first segment The vector representations of keyframe descriptions are used to calculate the semantic similarity between keyframe descriptions and local semantic units:

[0044] ,

[0045] in, Keyframe description The corresponding fragment local semantic unit index is then used to construct the alignment structure of the fragment semantic backbone and keyframe details:

[0046] Describe each keyframe based on its similarity score. Corresponding to the most relevant local semantic unit in the fragment description, let the first... In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: Then we have:

[0047] ,

[0048] Subsequently, the alignment relationship between the semantic backbone of the fragment and the details of the keyframe is constructed, that is, each keyframe description is assigned to the local semantic unit of the fragment most relevant to its semantics, providing a basis for subsequent fusion generation;

[0049] Step 1.6 Fusion of keyframe description and fragment-level description:

[0050] Based on the semantic similarity results obtained in step 1.5, describe each keyframe. The alignment relationship is mapped to the most relevant local semantic unit in the fragment description and used as auxiliary information. This alignment is then input into the large language model along with the corresponding fragment-level description for fusion, resulting in an enhanced fragment description.

[0051] Let the first In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: This indicates a keyframe description. With fragment-level description The first in The most semantically relevant local semantic units are identified by organizing fragment-level descriptions, keyframe descriptions, and alignment relationships between keyframes and local semantic units into fused cue information. This information is then input into a large language model for fusion generation, yielding the [number]th [unit]. Enhanced description information corresponding to each fragment:

[0052] ,

[0053] in, Indicates the fusion generation function, Indicates the first The enhanced description result after fusing keyframe details of each segment allows keyframe information related to different themes or objects in the same video segment to be added to the most relevant semantic positions in the segment description, avoiding disordered splicing of different details during the fusion process and improving the local semantic consistency and accuracy of detail expression in the segment description.

[0054] Step 1.7 Enhance the fusion of segment descriptions and complete video descriptions:

[0055] After enhancing all the segments, the enhanced segment description sequence is obtained:

[0056] ,

[0057] in, Indicates the first Enhanced description results after fusing keyframe details from individual segments Indicates the number of video segments. Represents the set of all enhanced fragment description sequences;

[0058] This step, as the second-level fusion process, continues the semantic alignment logic from step 1.5 and the fusion generation function from step 1.6 to fuse the enhanced segment description and the complete video description. Specifically, in step 1.6, keyframe descriptions are used as supplementary information, and segment-level descriptions are fused... Unlike the semantic backbone, this step uses the complete video-level description as the overall semantic backbone and the descriptions of each enhanced segment as supplementary information to the corresponding local event content in the complete video-level description. Specifically, the complete video-level description is divided into local semantic units to obtain the set of local semantic units corresponding to the complete video-level description. Using the same vector encoding and semantic similarity calculation method as in step 1.5, the semantic similarity between each enhanced segment description and each local semantic unit in the complete video-level description is calculated. Each enhanced segment description is then mapped to the local semantic unit in the complete video-level description that is most semantically relevant to it, thereby constructing the alignment relationship between the complete video semantic backbone and the supplementary information of the enhanced segments.

[0059] Let the first Enhanced fragment description The corresponding local semantic unit index in the complete video-level description is Then it means the first The enhanced fragment description and the first in the full video-level description The local semantic units are the most semantically relevant. The complete video-level description, the enhanced segment description sequence, and the alignment relationship between the enhanced segment description and the complete video-level local semantic units are organized together into fused cue information. This information is then input into the large language model using the fusion generation function from step 1.6 to generate the final enhanced video text description information.

[0060] ,

[0061] in, This refers to the fusion generation function used in step 1.6. This represents the final enhanced video text description information.

[0062] Furthermore, step 2 involves targeted supplementation of event elements based on key node-driven processes:

[0063] To address the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement, knowledge-driven methods are used to perform goal-oriented completion of important information related to the core elements of the event. Through the above two rounds of progressive processing, a structured enhancement and complete expression of complex event situation information is achieved, specifically including the following steps:

[0064] Step 2.1 Input text captions and video information extracted in the first round of enhancement:

[0065] The input text and the enhanced video description from the first round are used together as input for the targeted supplementation of event elements in this round. Let the input text be denoted as... The enhanced video description obtained from the first round of semantic information extraction and enhancement based on hierarchical division is denoted as... Then the semantic set of the input in this round is represented as:

[0066] ,

[0067] in, Provides textual descriptions of the event's background, participants, and outcomes. It provides visual semantic information enhanced at the video, segment, and keyframe levels, which together constitute the basic input for this round of event modeling and targeted supplementation.

[0068] Step 2.2 Extract entity, relation, and event element classifications, and model sub-events:

[0069] For the input semantic set Entity recognition, relation extraction, and event element classification are performed to transform the original natural language description into structured "entity-relationship-entity" semantic units. Sub-event semantics are organized around events, with the first event as the central element. Each sub-event is denoted as:

[0070] ,

[0071] in, Indicates the first The event entity itself, i.e., the WHAT element; This represents the set of event elements, and here we use a structure of five types of event elements other than the event WHAT.

[0072] Step 2.3 Constructing the event graph and multi-level evidence index:

[0073] The entity-relationship extraction results of the text captions and enhanced video descriptions are merged, and a unified event graph is constructed under the event center constraint:

[0074] ,

[0075] in, Represents a set of entity nodes. This represents a set of edges representing relationships between entities. If entities extracted from different modalities within the same data set have the same identifier, they are merged into the same node to reduce semantic redundancy caused by duplicate entities across modalities and to enhance the explicit associations between different modalities around the same event. Relationships connect head entities and tail entities in the form of edges, which are used to characterize the semantic connections between event elements. The relationship text is retained as an edge attribute and is no longer constructed as a separate node.

[0076] To support subsequent missing element detection and candidate evidence localization, a multi-level evidence index is constructed for entities and relationships:

[0077] ,

[0078] in, Indicates the specific source identifier, Represents the event entity The set of locations appearing in different sources of evidence. The evidence source hierarchy is represented, with different levels corresponding to information sources of different granularities. This establishes an "entity-source" mapping relationship, enabling each semantic unit in the graph to be organized in a unified manner and traced back to its corresponding modality and local evidence range. The multi-level evidence index provides a foundation for subsequent semantic missing location, candidate evidence screening, and targeted information supplementation.

[0079] Step 2.4 Semantic missing event detection based on graph structure:

[0080] For each event entity Feature detection is performed in its local neighborhood subgraph. Let the event features obtained in its neighborhood subgraph be:

[0081] ,

[0082] in, Let be the neighborhood hop count. If an entity node corresponding to the target element type exists within the neighborhood hop count range, the target element is considered to have been expressed in the local semantic structure of the current event. During the missing element detection phase, priority is given to judging based on element nodes with direct or nearest-neighbor semantic relationships within the local neighborhood of the event entity to avoid interference from other sub-events or weakly related information in judging the coverage of the current event element. Otherwise, it is considered that the target event element information is missing in the local semantic structure corresponding to the current event. Thus, the set of missing elements of the event entity is defined as:

[0083] ,

[0084] in, This represents the set of feature entities that have been detected in the local neighborhood subgraph of the event entity, when When, then the sub-event is considered The relevant information is complete; conversely, when... At that time, it was considered The semantic information of the event is still incomplete and needs to be further mined and supplemented with multimodal evidence;

[0085] Step 2.5. Joint ranking of sub-event importance and evidence:

[0086] For each event entity with missing elements Based on the source records corresponding to the event entity and related entity nodes in its local neighborhood, a set of candidate evidence is constructed:

[0087] ,

[0088] in, A candidate evidence source is represented, with each candidate evidence source corresponding to the source position of the relevant entity node in the keyframe description, video clip description, complete video-level description, text caption, or fused description. A candidate evidence source does not indicate that it already contains the missing event element, but rather represents the source position that has a semantic relationship with the current event entity and carries information about its missing element, used for further filtering and backtracking. An evidence score is calculated for each candidate evidence source.

[0089] ,

[0090] in, The level of this evidence. For hierarchical weight functions, Represents the current event entity The number of times related entity nodes that have a direct semantic connection with the candidate evidence source appear in the candidate evidence source is used to assign weights to the association strength, taking into account the different semantic similarities between different sources and the current event.

[0091] After obtaining the scores for each source of evidence, the overall strength of evidence score for the event entity is further calculated:

[0092] ,

[0093] in, This represents the set of candidate evidence sources related to the event entity. The overall evidence strength score comprehensively reflects the semantic support of the event entity in different modalities and granularities of evidence sources. It is used to rank the event entities to be mined, so as to prioritize the event entities with more sufficient evidence support and higher potential for completion.

[0094] Step 2.6 Identify the objects to be explored and candidate sources of evidence:

[0095] This step is used to identify event entities that need further supplementation from the event graph and its corresponding multi-level evidence index, and to screen evidence sources that can support subsequent local visual information mining from the candidate evidence sources obtained in step 2.5. During the construction of the event graph and multi-level evidence index in step 2.3, the entity nodes extracted from text descriptions, complete video-level descriptions, enhanced segment descriptions, or keyframe descriptions all record their corresponding source type and source location information. The source types include keyframe-level sources, video segment-level sources, complete video-level description sources, text description sources, and fused description sources. For entity nodes from keyframe-level descriptions, their source location information includes at least the time position of the corresponding keyframe in the original video; for entity nodes from video segment-level descriptions, their source location information includes at least the start and end times of the corresponding video segment in the original video; for entity nodes from complete video-level descriptions, text descriptions, or fused descriptions, their source information is used to provide event semantic support and candidate screening criteria.

[0096] For event entities with missing elements Step 2.5 has already constructed a candidate evidence set based on the entity node source records related to the event entity. This step further retains keyframe-level sources and video segment-level sources with original video local temporal localization capabilities from the candidate evidence set as the candidate evidence source set for subsequent in-depth mining:

[0097] ,

[0098] in, This refers to the event entity obtained in step 2.5. The relevant element nodes and their candidate evidence sets, The first element related to the event entity and its missing elements. One candidate source of evidence, This indicates the source type of the candidate evidence source; each candidate evidence source... The information should include at least the type of evidence source, the corresponding entity node or relation edge, the type of missing related element, the evidence score, and the source location information. Indicates keyframe evidence, The video clip evidence, the complete video-level description source, the text caption source, and the fused description source serve as coarse-grained semantic sources for event scoring and candidate selection, but do not directly participate in subsequent in-depth mining of local areas of the original video;

[0099] Among them, keyframe-level evidence sources can locate specific time positions in the original video, while video clip-level evidence sources can locate continuous time ranges in the original video. When the semantic correlation between candidate evidence sources and the current event entity and its missing elements is the same or similar, keyframe-level evidence sources have more accurate time positioning capabilities and are given priority as subsequent local in-depth mining objects; video clip-level evidence sources are used to support further mining of continuous event processes, subject interaction behaviors, or state change information.

[0100] For each video's corresponding information, all event entities are ranked according to the event's strength of evidence score. Assume the video information contains... After sorting the event entities according to their event scores from highest to lowest, select the top... A significant proportion of high-value events are entering the in-depth analysis phase:

[0101] ,

[0102] in, This indicates the order of events ranked from highest to lowest based on the strength of evidence scores. Each event entity This indicates the screening ratio for entering the in-depth analysis stage. After identifying the events requiring in-depth analysis, the corresponding candidate inference data fragments or frames are further located, and the highest-scoring data fragments or frames are selected from the candidate evidence set. One piece of evidence was chosen as the final object of in-depth investigation.

[0103] Step 2.7 Determine the targeted data mining strategy based on the type of candidate evidence source:

[0104] The video clips and keyframes obtained in steps 1.1 to 1.7 are used to initially extract local semantic information from the original video and provide candidate evidence sources with temporal localization capabilities to supplement subsequent event elements. The video clip-level descriptions and keyframe-level descriptions obtained in the first round of processing still have problems with insufficient expression of local action processes, changes in subject attributes, environmental details, or event outcome information. In this step, for the selected candidate evidence sources, we trace back to their corresponding time positions or time ranges in the original video and perform dense sampling in the corresponding local areas using a sampling frequency higher than that in the first round of semantic extraction to construct a local visual input frame sequence for directional reasoning:

[0105] For candidate evidence sources that are keyframe-level evidence, let the time position of the keyframe in the original video be denoted as . Set a local sampling time window centered on this time position. And adopt local encrypted sampling interval The original video frames within the time window are resampled to obtain the keyframe neighborhood visual input frame sequence:

[0106] ,

[0107] in, This represents the local encrypted sampling interval within the keyframe neighborhood time window. The local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. That is, the original visual information is re-acquired at a higher sampling frequency in the neighborhood range before and after the keyframe. By using the above-mentioned neighborhood encrypted sampling method centered on the keyframe time position, a more continuous and detailed visual context within the segment can be obtained on the basis of the first round of segment-level semantic expression. This is used to supplement the event development process, subject interaction relationship, scene state change and event result information that were not fully extracted due to the limited initial sampling density or coarse semantic generalization granularity.

[0108] For evidence sourced from video clips, let the corresponding clip's time range be... Then, uniform sampling is performed within the corresponding segment range to obtain a segment-level visual evidence frame sequence:

[0109] For candidate evidence sources that are video clips, let the time range corresponding to the video clip in the original video be... And employ fragment local encryption sampling interval The original video frames within the specified time range are resampled to obtain a segment-level visual input frame sequence:

[0110] ,

[0111] in, This indicates the local encrypted sampling interval within the video segment range. The local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. That is, the original visual information is re-acquired at a higher sampling frequency within the candidate video segment range. By performing local encrypted sampling on the target video segment as described above, a more continuous and detailed visual context within the segment can be obtained on the basis of the first round of segment-level semantic expression. This is used to supplement the event development process, subject interaction relationship, scene state change and event result information that were not fully extracted due to the limited initial sampling density or coarse semantic generalization granularity.

[0112] The visual input frame sequence obtained from the above sampling is input into the visual language model. Targeted reasoning is performed around the set of missing elements of the current event entity to obtain supplementary information such as location, time, related subjects, environmental information, or impact of the result. The newly added semantics are written back into the event graph to achieve continuous updating of the event structure and semantic completeness.

[0113] Further, in step 1.3, the keyframe extraction method based on uniform sampling (time-driven) extracts a keyframe sequence by uniformly sampling each video segment according to a preset sampling frequency. Let the first... The duration of each video clip is The sampling interval is The keyframe sequence extracted from this segment is represented as:

[0114] ,

[0115] in, Indicates the first Each video clip at the sampling time The corresponding video frame, To meet The largest integer, using uniform sampling, can achieve simple operation and low computational overhead, and is suitable for fast processing of large-scale video data. Previously, long videos have been dynamically segmented. The time span within each video segment is relatively short and the range of semantic changes is relatively limited. When the segment granularity is small, as long as the sampling frequency is set reasonably, uniform sampling can capture the main visual content and local details within the segment.

[0116] The clustering-based keyframe extraction method (content-driven) first extracts the visual features of each video frame within a video segment. Then, it clusters the video frames within the same segment based on feature similarity. From each cluster, it selects the center frame, the frame closest to the cluster center, or other representative frames as keyframes. The keyframe extraction method used for clustering is K-means, DBSCAN, or other feature similarity-based clustering methods. Let the i-th... There are a total of [number] video clips The frame, corresponding to the feature representation is:

[0117] ,

[0118] in, Indicates the first In the video clip, the first The visual features corresponding to the frames are used to divide the frame features within the segment into... The i-th cluster, let denoted as i-th. Each cluster is Its cluster center is The set of keyframes extracted from this segment can then be represented as:

[0119] ,

[0120] in, Indicates the first A set of keyframes corresponding to a video segment. Indicates the first In the video clip, the first The corresponding video frame. Indicates the first The index of the representative frame closest to the cluster center in each cluster. The advantage of using clustering is that the extracted keyframes are more semantically representative, can better cover different scene states or semantic change patterns within the video clip, avoid redundant frames from appearing repeatedly, and can improve the effective information density when generating subsequent visual language descriptions.

[0121] The keyframe extraction method based on deep learning models (model-driven) inputs candidate video frames or candidate frame sequences from a video segment into a pre-trained or trained keyframe selection model. The keyframe selection model outputs an importance score or keyframe selection result for each candidate video frame based on the visual content features of the candidate video frames, the temporal variation features between adjacent frames, and / or cross-modal features related to event semantics. The keyframe selection model can be implemented using an end-to-end trained deep neural network model, or it can be implemented by combining a pre-trained visual feature extraction model with a frame importance scoring module. Its network structure includes convolutional neural networks, 3D convolutional neural networks, recurrent neural networks, temporal convolutional networks, visual Transformer models, video understanding models, visual language models, or other deep learning models capable of evaluating the importance of video frames.

[0122] Let the first The first video clip Frame is Its corresponding importance score is:

[0123] ,

[0124] in, This represents the keyframe scoring model. Indicates the first The context information corresponding to each video segment includes the temporal information of adjacent frames within the video segment, segment-level visual features, and / or event semantic cues.

[0125] Based on the importance score of each candidate video frame, from the first... From a given video segment, a predetermined number of candidate video frames with the highest scores are selected, or candidate video frames with importance scores higher than a predetermined threshold are selected, to form the set of keyframes corresponding to that video segment.

[0126] ,

[0127] in, Indicates the first A set of keyframes extracted from a video clip using a deep learning model. This represents the threshold for keyframe importance scoring.

[0128] Furthermore, in step 1.3, to reduce the repeated selection of candidate video frames with similar time positions and repetitive visual content, a minimum time interval constraint or similarity deduplication constraint is set during the keyframe screening process. A keyframe extraction method based on a deep learning model is adopted, and adaptive selection is made according to the importance of different candidate video frames in the video segment to the expression of the event. This method can not only focus on frames with significant changes in the picture content, but also select more representative key frames by combining changes in subject behavior, changes in scene state, temporal context and event semantic relevance, thereby improving the coverage of complex event semantic information by key frames and the effective information density of subsequent visual language description generation.

[0129] Furthermore, in step 2.2, the structure of the five types of event elements includes:

[0130] The set of event elements can then be further represented as:

[0131] ,

[0132] in, Indicates the first Sub-events in event element types The corresponding elements and content;

[0133] By using the above method, local semantic information that was originally scattered in the text captions and enhanced video descriptions is reorganized into a unified representation around the same sub-event, providing a structural basis for subsequent missing information detection, candidate evidence screening and targeted supplementation.

[0134] The superior technical effects of the video semantic information extraction, enhancement, and optimization method for complex event situation understanding described in this invention are as follows:

[0135] 1. Compared with existing methods that directly select and focus on key local information, the method described in this invention first...

[0136] Long videos undergo untargeted semantic enhancement at three levels: full video level, video segment level, and keyframe level. Then, targeted supplementation is carried out around key nodes and missing event elements. Therefore, while focusing on key content, it can reduce the loss of local details and auxiliary clues in the early processing stage.

[0137] 2. Existing methods for selecting key information typically rely directly on task objectives, filtering key frames from the video.

[0138] While reasoning based on key segments or local features can improve targeting, it can easily overlook information that is equally important for understanding complex events but is currently less significant. The method described in this invention first performs three-level semantic extraction and fusion on long videos, enhancing process information at the level of complete videos, stage information at the level of video segments, and static details at the level of key frames. Then, based on this, it conducts target-driven targeted supplementation around key nodes and missing event elements, which can focus on key information while retaining and supplementing more local detail clues.

[0139] 3. This involves organizing and... (The sentence is incomplete and requires more context to translate accurately.)

[0140] Compared to other reasoning methods, the method described in this invention achieves explicit modeling of event elements centered on the event itself, while simultaneously integrating multi-level semantic information from video with external textual information and clarifying the processing path for subsequent information supplementation. Existing graph structure methods typically focus on the structured organization and relational reasoning of existing semantic results, lacking explicit modeling of event elements and struggling to uniformly organize key information such as location, time, related subjects, and results around a specific event. The method described in this invention, by constructing a structured representation of event-sub-event-element, enables multimodal information to be uniformly expressed around a specific event, thereby achieving event-centered structured organization. Furthermore, existing methods often use video or single-modal information as the primary modeling object, limiting the utilization of external information.

[0141] 4. When constructing the graph structure, the method described in this invention integrates complete video-level, video segment-level, and keyframe-level data.

[0142] In addition to semantic information, external textual modal data is introduced to uniformly map information from different sources into the event structure, thereby achieving cross-modal semantic association and fusion, and jointly guiding the further mining of key content in the video. Furthermore, existing graph structure methods often lack explicit guidance for the subsequent information supplementation process, relying on implicit reasoning for questions such as "which event elements to supplement," "which nodes to prioritize," and "from which sources to mine." The method described in this invention, based on event element modeling and multi-level evidence organization, can clearly identify the event elements to be supplemented, key nodes, and corresponding information sources, thereby forming a targeted mining path for event elements and providing a clear basis for subsequent supplementation. Attached Figure Description

[0143] Figure 1 This is a schematic diagram of a video semantic information extraction, enhancement, and optimization method for complex event situation understanding in an embodiment of the present invention;

[0144] Figure 2This is a schematic diagram of the semantic information extraction and enhancement process based on hierarchical division in an embodiment of the present invention;

[0145] Figure 3 This is a flowchart illustrating the targeted supplementation of event elements based on key node traction in an embodiment of the present invention. Detailed Implementation

[0146] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0147] Example

[0148] The video semantic information extraction, enhancement, and optimization method for complex event situation understanding described in this invention, such as... Figure 1 As shown, it includes:

[0149] Step 1. Enhanced semantic information extraction based on hierarchical partitioning:

[0150] To address the issue that long video data often requires sampling and compression due to the limited number of input tokens when inputting into multimodal or visual-language models, leading to the loss of temporal information and local details, this paper proposes a method to supplement and enrich the original event information by hierarchically dividing the video and performing multi-granular semantic extraction at the segment and frame levels. Here, the token is the basic unit for information encoding and computation in the multimodal model. Different modal contents such as text, image blocks, and video frames need to be uniformly converted into token representations and then used as model input for processing.

[0151] Step 2. Targeted supplementation of event elements based on key nodes:

[0152] To address the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement, knowledge-driven methods are used to complete important information related to the core elements of the event in a goal-driven manner. Through the above two rounds of progressive processing, the structured enhancement and complete expression of complex event situation information are achieved.

[0153] In some specific embodiments of the method described in this invention, such as Figure 2 As shown, step 1 addresses the issue that long video data, when input into multimodal models or vision-language models, typically requires sampling and compression due to the limited number of input tokens, leading to the loss of temporal information and local details. This is addressed by hierarchically dividing the video and performing multi-granular semantic extraction at the segment and frame levels to supplement and enrich the original event information. Specifically, this includes:

[0154] Step 1.1 Acquire input multimodal data and construct a unified semantic input. Obtain the original multimodal data corresponding to the event to be analyzed, including video data and text data. Among them, video data serves as the core object for semantic information extraction and enhancement, while text data serves as the information source for subsequent semantic fusion. To facilitate subsequent unified analysis, the video modal data is preprocessed using existing video understanding models and visual language models to infer a summary description of the video content, which serves as the text semantic representation for further processing. ;

[0155] Step 1.2 Segment the video data using content change detection:

[0156] Extract consecutive video frames from the input video. Dynamically divide the long video into multiple semantically consistent and temporally continuous video segments based on content changes between adjacent frames. Let the i-th segment be... Frame and the The frames are denoted as follows: and Its inter-frame variation score Represented as:

[0157] ,

[0158] in, Indicates the number of visual feature types used. Indicates the first The weights corresponding to visual features, This indicates that the video frame is being processed. Mapping function for visual feature extraction, This represents the difference measurement function between corresponding features, calculated using norm distance, cosine distance, statistical difference measure, or other similarity / difference calculation methods. Visual features include: visual features based on color space, edge features, optical flow features, features extracted by deep learning models based on CNN and VIT, and cross-modal semantic features extracted by visual language models or other pre-trained multimodal models. Different features can be used individually according to actual application needs, or they can be combined in a weighted manner to improve the ability of content change detection to perceive scene transitions, subject changes, action changes, or semantic transitions. At that time, the judgment of the first There are content change boundaries near the frame. A preset threshold is set based on the segmentation accuracy requirements of the target task and the system's computing resources. A smaller threshold results in more detailed segmentation but increases computational overhead, while a larger threshold reduces the number of segments but ignores some fine-grained variations. Based on the detected boundary points, the original long video is segmented... Divided into A series of segments ;in The value range is 3.0-9.0;

[0159] Step 1.3 Extract keyframes from each video segment:

[0160] For each video segment obtained in step 2, keyframes are further extracted from each video segment to supplement static scene details that are easily overlooked in the complete video-level or segment-level description, including people, locations, environment and local behavioral features. The keyframe extraction methods include time-sampling-based keyframe extraction, visual feature clustering-based keyframe extraction, and deep learning model-based keyframe extraction. The keyframe extraction strategies can be replaced, extended or combined according to specific application requirements.

[0161] Step 1.4 Generate fragment-level and keyframe-level description information respectively:

[0162] Semantic description information is generated for video segments and keyframes respectively. Segment-level description is used to preserve local time process and event stage information, while keyframe-level description is used to enhance the expression of static details and local semantics. Segment-level description and keyframe description can be obtained by multimodal models such as video understanding models or vision-language models.

[0163] Each video segment is input into the video understanding model for local semantic reasoning, resulting in a corresponding segment-level description:

[0164] ,

[0165] in, Indicates the first The segment-level description results corresponding to each segment, in this way, while maintaining the temporal correlation of events, separate the multi-event content that is easily mixed up under the uniform compression of the whole video, so as to alleviate the semantic ambiguity caused by long sequence feature compression;

[0166] Input the keyframes into the multimodal model to generate corresponding keyframe-level descriptions:

[0167] ,

[0168] in, Indicates the first The first segment One keyframe This represents the corresponding keyframe description text. The keyframe description text is used to supplement static visual information that is easily overlooked in fragment-level semantics, including: scene layout, subject attributes, environmental features, and local detail cues.

[0169] Let the complete video-level description obtained from preprocessing in step 1.1 be... , No. Each fragment level description is , No. The first segment Each keyframe is described as Then, the semantic set of the video after semantic information extraction and enhancement is represented as:

[0170] ,

[0171] in, Indicates the number of video segments. Indicates the first The number of keyframes in a video clip;

[0172] Step 1.5 Construct fragment-level local semantic units and align them with frame-level descriptions:

[0173] To achieve fine-grained fusion of keyframe information and video segment information, it is necessary to first divide the segment-level description information into local semantic units, let the first unit be... Fragment-level description It is divided into several local semantic units, represented as:

[0174] ,

[0175] in, Indicates the first The number of local semantic units in a video segment-level information. Indicates the first The first fragment-level description Each local semantic unit corresponds to a sentence, phrase, or local event semantic fragment in the fragment description.

[0176] To naturally integrate keyframe details into fragment descriptions, a vector encoding function is used to represent fragment-level local semantic units and keyframe descriptions using vectors:

[0177] ,

[0178] ,

[0179] in, This represents the sentence vector encoding function. Indicates the first The first segment Vector representation of local semantic units, Indicates the first The first segment The vector representations of keyframe descriptions are used to calculate the semantic similarity between keyframe descriptions and local semantic units:

[0180] ,

[0181] in, Keyframe description The corresponding fragment local semantic unit index is then used to construct the alignment structure of the fragment semantic backbone and keyframe details:

[0182] Based on the similarity score, each keyframe description is mapped to the most relevant local semantic unit in the fragment description. Let the first keyframe be the most relevant local semantic unit in the fragment description. In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: Then we have:

[0183] ,

[0184] Subsequently, the alignment relationship between the semantic backbone of the fragment and the details of the keyframe is constructed, that is, each keyframe description is assigned to the local semantic unit of the fragment most relevant to its semantics, providing a basis for subsequent fusion generation;

[0185] Step 1.6 Fusion of frame description and fragment description:

[0186] Based on the semantic similarity results obtained in step 1.5, each keyframe description is mapped to the most relevant local semantic unit in the fragment description. This alignment relationship is then used as auxiliary information and fused with the corresponding fragment-level description into the large language model to obtain the enhanced fragment description result.

[0187] Let the first In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: This indicates a keyframe description. With fragment-level description The first in The most semantically relevant local semantic units are identified by organizing fragment-level descriptions, keyframe descriptions, and alignment relationships between keyframes and local semantic units into fused cue information. This information is then input into a large language model for fusion generation, yielding the [number]th [unit]. Enhanced description information corresponding to each fragment:

[0188] ,

[0189] in, This represents the first-stage fusion generation function. Indicates the first The enhanced description result after fusing keyframe details of each segment allows keyframe information related to different themes or objects in the same video segment to be added to the most relevant semantic positions in the segment description, avoiding disordered splicing of different details during the fusion process and improving the local semantic consistency and accuracy of detail expression in the segment description.

[0190] Step 1.7 Enhance the fusion of segment descriptions and complete video descriptions:

[0191] After enhancing all the segments, the enhanced segment description sequence is obtained:

[0192] ,

[0193] in, Indicates the first Enhanced description results after fusing keyframe details from individual segments Indicates the number of video segments. Represents the set of all enhanced fragment description sequences;

[0194] This step, as the second-level fusion process, continues the semantic alignment logic from step 1.5 and the fusion generation function from step 1.6 to fuse the enhanced segment description and the complete video description. Specifically, in step 1.6, keyframe descriptions are used as supplementary information, and segment-level descriptions are fused... Unlike the semantic backbone, this step uses the complete video-level description as the overall semantic backbone and the descriptions of each enhanced segment as supplementary information to the corresponding local event content in the complete video-level description. Specifically, the complete video-level description is divided into local semantic units to obtain the set of local semantic units corresponding to the complete video-level description. Using the same vector encoding and semantic similarity calculation method as in step 1.5, the semantic similarity between each enhanced segment description and each local semantic unit in the complete video-level description is calculated. Each enhanced segment description is then mapped to the local semantic unit in the complete video-level description that is most semantically relevant to it, thereby constructing the alignment relationship between the complete video semantic backbone and the supplementary information of the enhanced segments.

[0195] Let the first Enhanced fragment description The corresponding local semantic unit index in the complete video-level description is Then it means the first The enhanced fragment description and the first in the full video-level description The local semantic units are the most semantically relevant. The complete video-level description, the enhanced segment description sequence, and the alignment relationship between the enhanced segment description and the complete video-level local semantic units are organized together into fused cue information. This information is then input into the large language model using the fusion generation function from step 1.6 to generate the final enhanced video text description information.

[0196] ,

[0197] in, This refers to the fusion generation function used in step 1.6. This represents the final generated enhanced video text description information;

[0198] Through the above-mentioned layer-by-layer fusion process, without changing the fusion generation mechanism, keyframe-level local detail information can be first integrated into the corresponding segment-level description, and then the enhanced segment-level semantic information can be integrated into the complete video-level description. This allows the final generated video text description to simultaneously retain the ability to express local details, segment-level process information, and overall event narrative information at the complete video level.

[0199] In some specific embodiments of the method described in this invention, combined with Figure 3 The diagram illustrates targeted supplementation of event elements based on key node guidance. Step 2, targeted supplementation of event elements based on key node guidance, addresses the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement. It utilizes knowledge guidance to perform target-driven supplementation of important information related to the core elements of the event. Through the above two progressive processes, it achieves structured enhancement and complete expression of complex event situation information, specifically including the following steps:

[0200] Step 2.1 Input text captions and video information extracted in the first round of enhancement:

[0201] The input text and the enhanced video description from the first round are used together as input for the targeted supplementation of event elements in this round. Let the input text be denoted as... The enhanced video description obtained from the first round of semantic information extraction and enhancement based on hierarchical division is denoted as... Then the semantic set of the input in this round is represented as:

[0202] ,

[0203] in, Provides textual descriptions of the event's background, participants, and outcomes. It provides visual semantic information enhanced at the video, segment, and keyframe levels, which together constitute the basic input for this round of event modeling and targeted supplementation.

[0204] Step 2.2 Extract entity, relation, and event element classifications, and model sub-events:

[0205] For the input semantic set Entity recognition, relation extraction, and event element classification are performed to transform the original natural language description into structured "entity-relationship-entity" semantic units. Sub-event semantics are organized around events, with the first event as the central element. Each sub-event is denoted as:

[0206] ,

[0207] in, Indicates the first The event entity itself, i.e., the WHAT element; This represents the set of event elements. In this step, five types of event element structures were used, excluding the event WHAT.

[0208] Step 2.3 Constructing the event graph and multi-level evidence index:

[0209] The entity-relationship extraction results of the text captions and enhanced video descriptions are merged, and a unified event graph is constructed under the event center constraint:

[0210] ,

[0211] in, Represents a set of entity nodes. This represents a set of edges representing relationships between entities. If entities extracted from different modalities within the same data set have the same identifier, they are merged into the same node to reduce semantic redundancy caused by duplicate entities across modalities and to enhance the explicit associations between different modalities around the same event. Relationships connect head entities and tail entities in the form of edges, which are used to characterize the semantic connections between event elements. The relationship text is retained as an edge attribute and is no longer constructed as a separate node.

[0212] To support subsequent missing element detection and candidate evidence localization, a multi-level evidence index is constructed for entities and relationships:

[0213] ,

[0214] in, Indicates the specific source identifier, Represents the event entity The set of locations appearing in different sources of evidence. The evidence source hierarchy is represented, with different levels corresponding to information sources of different granularities. This establishes an "entity-source" mapping relationship, enabling each semantic unit in the graph to be organized in a unified manner and traced back to its corresponding modality and local evidence range. The multi-level evidence index provides a foundation for subsequent semantic missing location, candidate evidence screening, and targeted information supplementation.

[0215] Step 2.4 Semantic missing event detection based on graph structure:

[0216] For each event entity Feature detection is performed in its local neighborhood subgraph. Let the event features obtained in its neighborhood subgraph be:

[0217] ,

[0218] in, Let be the neighborhood hop count. If an entity node corresponding to the target element type exists within the neighborhood hop count range, the target element is considered to have been expressed in the local semantic structure of the current event. During the missing element detection phase, priority is given to judging based on element nodes with direct or nearest-neighbor semantic relationships within the local neighborhood of the event entity to avoid interference from other sub-events or weakly related information in judging the coverage of the current event element. Otherwise, it is considered that the target event element information is missing in the local semantic structure corresponding to the current event. Thus, the set of missing elements of the event entity is defined as:

[0219] ,

[0220] in, This represents the set of feature entities that have been detected in the local neighborhood subgraph of the event entity, when When, then the sub-event is considered The relevant information is complete; conversely, when... At that time, it was considered The semantic information of the event is still incomplete and needs to be further mined and supplemented with multimodal evidence;

[0221] Step 2.5. Joint ranking of sub-event importance and evidence:

[0222] For each missing event For the missing elements of a node, construct its candidate evidence set:

[0223] ,

[0224] in, Each source of evidence corresponds to a specific frame, fragment, or text description location. An evidence score is calculated for each source of evidence.

[0225] ,

[0226] in, The level of this evidence. For hierarchical weight functions, for In evidence The frequency of occurrences within a source is considered, and the semantic similarity between different sources and the current event is taken into account to assign weights to the association strength. Specifically, the location where the event entity directly appears and its position in the event graph are considered. Evidence that establishes a direct semantic connection within a skip neighborhood is given a higher weight than missing elements of the same type that appear only in the same local area but have not yet formed a direct connection; the latter is given a higher weight than missing elements of the same type that appear only in a larger area, thus making the evidence score reflect the local semantic clues that are more closely related to the current event.

[0227] After obtaining the scores for each source of evidence, the overall strength of evidence score for the event entity is further calculated:

[0228] ,

[0229] in, It represents the set of evidence sources related to the event entity. The overall evidence strength score comprehensively reflects the semantic support of the event in different modalities and granularities of evidence. It is used to sort the events to be mined, so as to prioritize the events with denser evidence and higher potential for completion.

[0230] Step 2.6 Identify the objects to be explored and candidate sources of evidence:

[0231] This step is used to locate potentially information-dense areas in multimodal data and to supplement missing event elements in a targeted manner using visual reasoning. Considering that subsequent in-depth mining requires clear temporal localization, only fine-grained evidence sources with local localization capabilities are retained as the actual mining targets, namely keyframe-level evidence and video clip-level evidence:

[0232] ,

[0233] in, This refers to the event entity obtained in step 2.5. The relevant element nodes and their candidate evidence sets, The first element related to the event entity and its missing elements. One candidate source of evidence, This indicates the source type of the candidate evidence source; each candidate evidence source... The information should include at least the type of evidence source, the corresponding entity node or relation edge, the type of missing related element, the evidence score, and the source location information. Indicates keyframe evidence, The video clip evidence, the complete video-level description source, the text caption source, and the fused description source serve as coarse-grained semantic sources for event scoring and candidate selection, but do not directly participate in subsequent in-depth mining of local areas of the original video;

[0234] Among them, keyframe-level evidence sources can be located to a specific time position in the original video, while video clip-level evidence sources can be located to a continuous time range in the original video. Therefore, when the semantic correlation between candidate evidence sources and the current event entity and its missing elements is the same or similar, keyframe-level evidence sources are given priority as subsequent local in-depth mining objects due to their more accurate time positioning capabilities; video clip-level evidence sources are used to support further mining of continuous event processes, subject interaction behaviors, or state change information.

[0235] For each video's corresponding information, all event entities are ranked according to the event's strength of evidence score. Assume the video information contains... After sorting the event entities according to their event scores from highest to lowest, select the top... A significant proportion of high-value events are entering the in-depth analysis phase:

[0236] ,

[0237] in, This indicates the order of events ranked from highest to lowest based on the strength of evidence scores. Each event entity This indicates the screening ratio for entering the in-depth analysis stage. After identifying the events requiring in-depth analysis, the corresponding candidate inference data fragments or frames are further located, and the highest-scoring data fragments or frames are selected from the candidate evidence set. One piece of evidence was chosen as the final object of in-depth investigation.

[0238] Step 2.7 Determine the targeted mining strategy based on the type of evidence source:

[0239] Based on the video clips and keyframes obtained in steps 1.1 to 1.7, local semantic information is initially extracted from the original video, providing candidate evidence sources with temporal localization capabilities to supplement subsequent event elements. The video clip-level and keyframe-level descriptions obtained in the first round of processing still suffer from insufficient expression of local action processes, changes in subject attributes, environmental details, or event outcome information. This step, for the selected candidate evidence sources, traces back to their corresponding time position or time range in the original video, and performs intensive sampling within the corresponding local region using a sampling frequency higher than that of the first round of semantic extraction, to construct a local visual input frame sequence for directional reasoning.

[0240] For candidate evidence sources that are keyframe-level evidence, let the time position of the keyframe in the original video be denoted as . Set a local sampling time window centered on this time position. And adopt local encrypted sampling interval The original video frames within the time window are resampled to obtain the keyframe neighborhood visual input frame sequence:

[0241] ,

[0242] in, This represents the local encrypted sampling interval within the keyframe's neighborhood time window. This local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. In other words, it re-acquires the original visual information within the neighborhood before and after the keyframe at a higher sampling frequency. Through this neighborhood encrypted sampling method centered on the keyframe's time position, a more continuous and detailed visual context within the segment can be obtained on top of the first round of segment-level semantic expression. This is used to supplement information on event development, subject interaction relationships, scene state changes, and event outcomes that were not fully extracted due to limited initial sampling density or coarse semantic generalization granularity. For cases where the evidence source is a video segment, the corresponding segment's time range is set as... Then, uniform sampling is performed within the corresponding segment range to obtain a segment-level visual evidence frame sequence:

[0243] For candidate evidence sources that are video clips, let the time range corresponding to the video clip in the original video be... And employ fragment local encryption sampling interval The original video frames within the specified time range are resampled to obtain a segment-level visual input frame sequence:

[0244] ,

[0245] in, This indicates the local encrypted sampling interval within the video segment range. The local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. That is, the original visual information is re-acquired at a higher sampling frequency within the candidate video segment range. By performing local encrypted sampling on the target video segment as described above, a more continuous and detailed visual context within the segment can be obtained on the basis of the first round of segment-level semantic expression. This is used to supplement the event development process, subject interaction relationship, scene state change and event result information that were not fully extracted due to the limited initial sampling density or coarse semantic generalization granularity.

[0246] The visual input frame sequence obtained from the above sampling is input into the visual language model. Targeted reasoning is performed around the set of missing elements of the current event entity to obtain supplementary information such as location, time, related subjects, environmental information, or impact of the result. The newly added semantics are written back into the event graph to achieve continuous updating of the event structure and semantic completeness.

[0247] In some specific embodiments of the method described in this invention, in step 1.3, based on the uniform sampling keyframe extraction method (time-driven), each video segment is uniformly sampled according to a preset sampling frequency to extract the keyframe sequence. Let the first... The duration of each video clip is The sampling interval is The keyframe sequence extracted from this segment is represented as follows:

[0248] ,

[0249] in, Indicates the first Each video clip at the sampling time The corresponding video frame, To meet The largest integer, using uniform sampling, can achieve simple operation and low computational overhead, making it suitable for fast processing of large-scale video data. Given that the long video has been dynamically segmented in the previous step, the time span within each video segment is relatively short and the range of semantic changes is relatively limited. When the segment granularity is small, as long as the sampling frequency is set reasonably, uniform sampling can fully capture the main visual content and local detail information within the segment. Compared with directly applying a higher sampling rate to the entire long video and then performing uniform processing, this invention can reduce the burden of subsequent feature processing and model inference while better preserving local effective information, thereby providing support for subsequent semantic description generation.

[0250] The clustering-based keyframe extraction method (content-driven) first extracts the visual features of each video frame within a video segment. Then, it clusters the video frames within the same segment based on feature similarity. From each cluster, it selects the center frame, the frame closest to the cluster center, or other representative frames as keyframes. The keyframe extraction method used for clustering is K-means, DBSCAN, or other feature similarity-based clustering methods. Let the i-th... There are a total of [number] video clips The frame, corresponding to the feature representation is:

[0251] ,

[0252] in, Indicates the first In the video clip, the first The visual features corresponding to the frames are used to divide the frame features within the segment into... The i-th cluster, let denoted as i-th. Each cluster is Its cluster center is The set of keyframes extracted from this segment can then be represented as:

[0253] ,

[0254] in, Indicates the first A set of keyframes corresponding to a video segment. Indicates the first In the video clip, the first frame

[0255] The corresponding video frame, Indicates the first The index of the representative frame closest to the cluster center in each cluster. The advantage of using clustering is that the extracted keyframes are more semantically representative, can better cover different scene states or semantic change patterns within the video clip, avoid redundant frames from appearing repeatedly, and can improve the effective information density when generating subsequent visual language descriptions.

[0256] The keyframe extraction method based on deep learning models (model-driven) first inputs candidate video frames or candidate frame sequences from a video segment into a pre-trained or trained keyframe selection model. The keyframe selection model then outputs an importance score or keyframe selection result for each candidate video frame based on the visual content features of the candidate video frames, the temporal variation features between adjacent frames, and / or cross-modal features related to event semantics. The keyframe selection model can be implemented using an end-to-end trained deep neural network model, or it can be implemented by combining a pre-trained visual feature extraction model with a frame importance scoring module. Its network structure includes, but is not limited to, convolutional neural networks, 3D convolutional neural networks, recurrent neural networks, temporal convolutional networks, visual Transformer models, video understanding models, visual language models, or other deep learning models capable of evaluating the importance of video frames.

[0257] Let the first The first video clip Frame is Its corresponding importance score is:

[0258] ,

[0259] in, This represents the keyframe scoring model. Indicates the first The context information corresponding to each video segment includes the temporal information of adjacent frames within the video segment, segment-level visual features, and / or event semantic cues.

[0260] Based on the importance score of each candidate video frame, from the first... From a given video segment, a predetermined number of candidate video frames with the highest scores are selected, or candidate video frames with importance scores higher than a predetermined threshold are selected, to form the set of keyframes corresponding to that video segment.

[0261] ,

[0262] in, Indicates the first A set of keyframes extracted from a video clip using a deep learning model. This represents the threshold for keyframe importance scoring.

[0263] In some specific embodiments of the method described in this invention, in step 1.3, in order to reduce the repeated selection of candidate video frames with similar time positions and repetitive visual content, a minimum time interval constraint or similarity deduplication constraint can be set during the keyframe screening process. A keyframe extraction method based on a deep learning model is adopted, which can adaptively select according to the importance of different candidate video frames in the video segment to the event expression. It can not only focus on frames with significant changes in screen content, but also select more representative key frames by combining changes in subject behavior, changes in scene state, temporal context and event semantic relevance, thereby improving the coverage of complex event semantic information by key frames and the effective information density of subsequent visual language description generation.

[0264] In some specific examples of the method described in this invention, in step 2.2, the five types of event element structure include The set of event elements can then be further represented as:

[0265] ,

[0266] in, Indicates the first Sub-events in event element types The corresponding elements and content;

[0267] By using the above method, local semantic information that was originally scattered in the text captions and enhanced video descriptions is reorganized into a unified representation around the same sub-event, providing a structural basis for subsequent missing information detection, candidate evidence screening and targeted supplementation.

[0268] This invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope. All such changes and modifications fall within the scope of the invention as defined by the appended claims.

Claims

1. A video semantic enhancement and optimization method for complex event situation understanding, comprising: Step 1. Semantic information extraction enhancement based on hierarchical partitioning: To address the problem that long video data is subject to sampling and compression due to the limited number of input tokens when inputting into multimodal models or vision-language models, resulting in the easy loss of temporal information and local details, this step supplements and enriches the original event information by hierarchically partitioning the video and performing multi-granular semantic extraction at the segment and frame levels. Step 2. Targeted supplementation of event elements based on key nodes: To address the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement, knowledge-driven methods are used to complete important information related to the core elements of the event in a goal-driven manner. Through the above two rounds of progressive processing, the structured enhancement and complete expression of complex event situation information are achieved.

2. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, In step 1, the token is the basic unit for information encoding and calculation in the multimodal model. The different modal contents of text, image blocks, and video frames need to be uniformly converted into token representations and then used as model inputs for processing.

3. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, Step 1 addresses the issue that when long video data is input into a multimodal model or vision-language model, sampling and compression are required due to the limited number of input tokens, leading to the loss of temporal information and local details. This is addressed by hierarchically dividing the video and performing multi-granular semantic extraction at the segment and frame levels to supplement and enrich the original event information. Specifically, this includes: Step 1.1 Obtain input multimodal data and construct a unified semantic input: The process involves acquiring raw multimodal data corresponding to the event to be analyzed, including video data and text data. Video data serves as the core object for semantic information extraction and enhancement, while text data serves as the information source for subsequent semantic fusion. To facilitate unified analysis, the video modal data is first preprocessed using existing video understanding and visual language models to derive a general description of the video content, which then serves as the text semantic representation for further processing. ; Step 1.2 Segment the video data using content change detection: Extract consecutive video frames from the input video. Dynamically divide the long video into multiple semantically consistent and temporally continuous video segments based on content changes between adjacent frames. Let the i-th segment be... Frame and the The frames are denoted as follows: and Its inter-frame variation score Represented as: ， in, Indicates the number of visual feature types used. Indicates the first The weights corresponding to visual features This indicates that the video frame is being processed. Mapping function for visual feature extraction, This represents the difference measure function between corresponding features, calculated using norm distance, cosine distance, statistical difference measure, or other similarity / difference calculation methods. Visual features include: color space-based visual features, edge features, optical flow features, features extracted by deep learning models based on CNN and VIT, and cross-modal semantic features extracted by visual language models or other pre-trained multimodal models. Different features can be used individually according to actual application needs, or they can be combined in a weighted manner to improve the ability of content change detection to perceive scene transitions, subject changes, action changes, or semantic transitions. At that time, the judgment of the first There are content change boundaries near the frame, where, The preset threshold; Step 1.3 Extract keyframes from each video segment: For each video segment obtained in step 2, keyframes are further extracted from each video segment to supplement the static scene details that are ignored in the complete video-level or segment-level description, including people, locations, environment and local behavioral features. The keyframe extraction methods include time-sampling-based keyframe extraction, visual feature clustering-based keyframe extraction, and deep learning model-based keyframe extraction. Step 1.4 Generate fragment-level and keyframe-level description information respectively: Semantic description information is generated for video segments and keyframes respectively. Segment-level description is used to preserve local time process and event stage information, while keyframe-level description is used to enhance the expression of static details and local semantics. Segment-level description and keyframe description can be obtained by multimodal models such as video understanding models or vision-language models. Each video segment is input into the video understanding model for local semantic reasoning, resulting in a corresponding segment-level description: ， in, Indicates the first The segment-level description results corresponding to each segment, in this way, while maintaining the temporal correlation of events, separate the multi-event content that is easily mixed up under the uniform compression of the whole video, so as to alleviate the semantic ambiguity caused by long sequence feature compression; Input the keyframes into the multimodal model to generate corresponding keyframe-level descriptions: ， in, Indicates the first The first segment in the segment One keyframe This represents the corresponding keyframe description text, which is used to supplement static visual information that is ignored in fragment-level semantics, including scene layout, subject attributes, environmental features, and local detail cues. Let the complete video-level description obtained from preprocessing in step 1.1 be... , No. Each fragment level description is , No. The first segment Each keyframe is described as Then, the semantic set of the video after semantic information extraction and enhancement is represented as: ， in, Indicates the number of video segments. Indicates the first The number of keyframes in a video clip; Step 1.5 Construct fragment-level local semantic units and align them with frame-level descriptions: To achieve fine-grained fusion of keyframe information and video segment information, it is necessary to first divide the segment-level description information into local semantic units, let the first unit be... Fragment-level description It is divided into several local semantic units, represented as: ， in, Indicates the first The number of local semantic units in a video segment-level information. Indicates the first The first fragment-level description Each local semantic unit corresponds to a sentence, phrase, or local event semantic fragment in the fragment description. To naturally integrate keyframe details into fragment descriptions, a vector encoding function is used to represent fragment-level local semantic units and keyframe descriptions using vectors: ， ， in, This represents the sentence vector encoding function. Indicates the first The first segment Vector representation of local semantic units, Indicates the first The first segment The vector representations of keyframe descriptions are used to calculate the semantic similarity between keyframe descriptions and local semantic units: ， in, Keyframe description The corresponding fragment local semantic unit index is then used to construct the alignment structure of the fragment semantic backbone and keyframe details: Describe each keyframe based on its similarity score. Corresponding to the most relevant local semantic unit in the fragment description, let the first... In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: Then we have: ， Subsequently, the alignment relationship between the semantic backbone of the fragment and the details of the keyframe is constructed, that is, each keyframe description is assigned to the local semantic unit of the fragment most relevant to its semantics, providing a basis for subsequent fusion generation; Step 1.6 Fusion of keyframe description and fragment-level description: Based on the semantic similarity results obtained in step 1.5, describe each keyframe. The alignment relationship is mapped to the most relevant local semantic unit in the fragment description and used as auxiliary information. This alignment is then input into the large language model along with the corresponding fragment-level description for fusion, resulting in an enhanced fragment description. Let the first In the video clip, the first The index of the local semantic unit corresponding to each keyframe description is: This indicates a keyframe description. With fragment-level description The first in The most semantically relevant local semantic units are identified by organizing fragment-level descriptions, keyframe descriptions, and alignment relationships between keyframes and local semantic units into fused cue information. This information is then input into a large language model for fusion generation, yielding the [number]th [unit]. Enhanced description information corresponding to each fragment: ， in, Indicates the fusion generation function, Indicates the first The enhanced description result after fusing keyframe details of each segment allows keyframe information related to different themes or objects in the same video segment to be added to the most relevant semantic positions in the segment description, avoiding disordered splicing of different details during the fusion process and improving the local semantic consistency and accuracy of detail expression in the segment description. Step 1.7 Enhance the fusion of segment descriptions and complete video descriptions: After enhancing all the segments, the enhanced segment description sequence is obtained: , in, Indicates the first Enhanced description results after fusing keyframe details from individual segments Indicates the number of video segments. Represents the set of all enhanced fragment description sequences; This step, as the second-level fusion process, continues the semantic alignment logic from step 1.5 and the fusion generation function from step 1.6 to fuse the enhanced segment description and the complete video description. Specifically, in step 1.6, keyframe descriptions are used as supplementary information, and segment-level descriptions are fused... Unlike the semantic backbone, this step uses the complete video-level description as the overall semantic backbone and the descriptions of each enhanced segment as supplementary information to the corresponding local event content in the complete video-level description. Specifically, the complete video-level description is divided into local semantic units to obtain the set of local semantic units corresponding to the complete video-level description. Using the same vector encoding and semantic similarity calculation method as in step 1.5, the semantic similarity between each enhanced segment description and each local semantic unit in the complete video-level description is calculated. Each enhanced segment description is then mapped to the local semantic unit in the complete video-level description that is most semantically relevant to it, thereby constructing the alignment relationship between the complete video semantic backbone and the supplementary information of the enhanced segments. Let the first Enhanced fragment description The corresponding local semantic unit index in the complete video-level description is Then it means the first The enhanced fragment description and the first in the full video-level description The local semantic units are the most semantically relevant. The complete video-level description, the enhanced segment description sequence, and the alignment relationship between the enhanced segment description and the complete video-level local semantic units are organized together into fused cue information. This information is then input into the large language model using the fusion generation function from step 1.6 to generate the final enhanced video text description information. ， in, This refers to the fusion generation function used in step 1.

6. This represents the final enhanced video text description information.

4. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, In step 1.2, the The preset threshold value is set based on the target task's requirements for segmentation accuracy and the system's computing resources. When the threshold is small, the segmentation results in more detailed segments but increase computational overhead; when the threshold is large, the number of segments decreases but some fine-grained changes are ignored. Based on the detected boundary points, the original long video is... Divided into A series of segments ;in The preset threshold value range is 3.0-9.

0.

5. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, In step 1.3, in order to reduce the repeated selection of candidate video frames that are close in time and have repetitive visual content, a minimum time interval constraint or similarity deduplication constraint is set in the keyframe screening process. A keyframe extraction method based on a deep learning model is adopted, and adaptive selection is made according to the importance of different candidate video frames in the video segment to the expression of the event.

6. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, In step 1.3, the keyframe extraction method based on uniform sampling performs uniform sampling on each video segment according to a preset sampling frequency to extract the keyframe sequence. Let the first keyframe be... The duration of each video clip is If the sampling interval is Δt, then the keyframe sequence extracted from this segment is represented as: ， in, Indicates the first Each video clip at the sampling time The corresponding video frame, To meet The largest integer, using uniform sampling, can achieve simple operation and low computational overhead, and is suitable for fast processing of large-scale video data. Previously, long videos have been dynamically segmented. The time span within each video segment is relatively short and the range of semantic changes is relatively limited. When the segment granularity is small, as long as the sampling frequency is set reasonably, uniform sampling can capture the main visual content and local details within the segment. Cluster-based keyframe extraction first extracts the visual features of each video frame within a video segment. Then, it clusters the video frames within the same segment based on feature similarity. From each cluster, it selects the center frame, the frame closest to the cluster center, or other representative frames as keyframes. The keyframe extraction method used for clustering is K-means, DBSCAN, or other feature similarity-based clustering methods. Let the i-th... There are a total of [number] video clips The frame, corresponding to the feature representation is: ， in, Indicates the first In the video clip, the first The visual features corresponding to the frames are used to divide the frame features within the segment into... The i-th cluster, let denoted as i-th. The clusters are as follows Its cluster center is The set of keyframes extracted from this segment can then be represented as: ， in, Indicates the first A set of keyframes corresponding to a video segment. Indicates the first In the video clip, the first The corresponding video frame. Indicates the first The index of the representative frame closest to the cluster center in each cluster. The advantage of using clustering is that the extracted keyframes are more semantically representative, can better cover different scene states or semantic change patterns within the video clip, avoid redundant frames from appearing repeatedly, and can improve the effective information density when generating subsequent visual language descriptions. Keyframe extraction based on deep learning models involves inputting candidate video frames or candidate frame sequences from a video segment into a pre-trained or trained keyframe selection model. The keyframe selection model then outputs an importance score or keyframe selection result for each candidate video frame based on the visual content features of the candidate video frames, the temporal variation features between adjacent frames, and / or cross-modal features related to event semantics. The keyframe selection model can be implemented using an end-to-end trained deep neural network model or by combining a pre-trained visual feature extraction model with a frame importance scoring module. Its network structure includes convolutional neural networks, 3D convolutional neural networks, recurrent neural networks, temporal convolutional networks, visual Transformer models, video understanding models, visual language models, or other deep learning models capable of evaluating the importance of video frames. Let the first The first video clip Frame is Its corresponding importance score is: ， in, This represents the keyframe scoring model. Indicates the first The context information corresponding to each video segment includes the temporal information of adjacent frames within the video segment, segment-level visual features, and / or event semantic cues. Based on the importance score of each candidate video frame, from the first... From a given video segment, a predetermined number of candidate video frames with the highest scores are selected, or candidate video frames with importance scores higher than a predetermined threshold are selected, to form the set of keyframes corresponding to that video segment. ， in, Indicates the first A set of keyframes extracted from a video clip using a deep learning model. This represents the threshold for keyframe importance scoring.

7. The video semantic enhancement and optimization method for complex event situation understanding as described in claim 1, characterized in that, Step 2 describes the targeted supplementation of event elements based on key node-driven processes: To address the issues of missing event elements and insufficient cross-modal correlations after the first round of extraction and enhancement, knowledge-driven methods are used to perform goal-oriented completion of important information related to the core elements of the event. Through the above two rounds of progressive processing, a structured enhancement and complete expression of complex event situation information is achieved, specifically including the following steps: Step 2.1 Input text captions and video information extracted in the first round of enhancement: The input text and the enhanced video description from the first round are used together as input for the targeted supplementation of event elements in this round. Let the input text be denoted as... The enhanced video description obtained from the first round of semantic information extraction and enhancement based on hierarchical division is denoted as... Then the semantic set of the input in this round is represented as: ， in, Provides textual descriptions of the event's background, participants, and outcomes. It provides visual semantic information enhanced at the video, segment, and keyframe levels, which together constitute the basic input for this round of event modeling and targeted supplementation. Step 2.2 Extract entity, relation, and event element classifications, and model sub-events: For the input semantic set Entity recognition, relation extraction, and event element classification are performed to transform the original natural language description into structured "entity-relationship-entity" semantic units. Sub-event semantics are organized around events, with the first event as the central element. Each sub-event is denoted as: ， in, Indicates the first The event entity itself, i.e., the WHAT element; This represents the set of event elements, and here we use a structure of five types of event elements other than the event WHAT. Step 2.3 Constructing the event graph and multi-level evidence index: The entity-relationship extraction results of the text captions and enhanced video descriptions are merged, and a unified event graph is constructed under the event center constraint: ， in, Represents a set of entity nodes. This represents the set of edges that represent relationships between entities. If entities extracted from different modalities within the same set of data have the same identifier, they are merged into the same node to reduce semantic redundancy caused by repeated entities across modalities and to enhance the explicit associations between different modalities around the same event. Relationships connect head entities and tail entities in the form of edges, which are used to represent the semantic connections between event elements. The relationship text is retained as an edge attribute and is no longer constructed as a node separately. To support subsequent missing element detection and candidate evidence localization, a multi-level evidence index is constructed for entities and relationships: ， in, Indicates the specific source identifier, Represents the event entity The set of locations appearing in different sources of evidence. The evidence source hierarchy is represented, with different levels corresponding to information sources of different granularities. This establishes an "entity-source" mapping relationship, enabling each semantic unit in the graph to be organized uniformly and traced back to its corresponding modality and local evidence range. The multi-level evidence index provides a foundation for subsequent semantic missing location, candidate evidence screening, and targeted information supplementation. Step 2.4 Semantic missing event detection based on graph structure: For each event entity Feature detection is performed in its local neighborhood subgraph. Let the event features obtained in its neighborhood subgraph be: ， in, Let be the neighborhood hop count. If an entity node corresponding to the target element type exists within the neighborhood hop count range, the target element is considered to have been expressed in the local semantic structure of the current event. During the missing element detection phase, priority is given to judging based on element nodes with direct or nearest-neighbor semantic relationships within the local neighborhood of the event entity to avoid interference from other sub-events or weakly related information in judging the coverage of the current event element. Otherwise, it is considered that the target event element information is missing in the local semantic structure corresponding to the current event. Thus, the set of missing elements of the event entity is defined as: ， in, This represents the set of feature entities that have been detected in the local neighborhood subgraph of the event entity, when When, then the sub-event is considered The relevant information is complete; conversely, when... At that time, it was considered The semantic information of the event is still incomplete and needs to be further mined and supplemented with multimodal evidence; Step 2.

5. Joint ranking of sub-event importance and evidence: For each event entity with missing elements Based on the source records corresponding to the event entity and related entity nodes in its local neighborhood, a set of candidate evidence is constructed: ， in, Each candidate evidence source represents the source position of the relevant entity node in the keyframe description, video clip description, complete video-level description, text caption, or fused description. A candidate evidence source does not necessarily mean that it already contains the missing event element; rather, it represents the source position that is semantically related to the current event entity and carries information about its missing element. This information is used for further filtering and backtracking. An evidence score is calculated for each candidate evidence source. ， in, The level of this evidence, For hierarchical weight functions, Represents the current event entity The number of times related entity nodes that have a direct semantic connection with the candidate evidence source appear in the candidate evidence source is used to assign weights to the association strength, taking into account the different semantic similarities between different sources and the current event. After obtaining the scores for each source of evidence, the overall strength of evidence score for the event entity is further calculated: ， in, This represents the set of candidate evidence sources related to the event entity. The overall evidence strength score comprehensively reflects the semantic support of the event entity in different modalities and granularities of evidence sources. It is used to rank the event entities to be mined, so as to prioritize the event entities with more sufficient evidence support and higher potential for completion. Step 2.6 Identify the objects to be explored and candidate sources of evidence: This step is used to identify event entities that need further supplementation from the event graph and its corresponding multi-level evidence index, and to screen evidence sources that can support subsequent local visual information mining from the candidate evidence sources obtained in step 2.

5. During the construction of the event graph and multi-level evidence index in step 2.3, the entity nodes extracted from text descriptions, complete video-level descriptions, enhanced segment descriptions, or keyframe descriptions all record their corresponding source type and source location information. The source types include keyframe-level sources, video segment-level sources, complete video-level description sources, text description sources, and fused description sources. For entity nodes from keyframe-level descriptions, their source location information includes at least the time position of the corresponding keyframe in the original video; for entity nodes from video segment-level descriptions, their source location information includes at least the start and end times of the corresponding video segment in the original video; for entity nodes from complete video-level descriptions, text descriptions, or fused descriptions, their source information is used to provide event semantic support and candidate screening criteria. For event entities with missing elements Step 2.5 has already constructed a candidate evidence set based on the entity node source records related to the event entity. This step further retains keyframe-level sources and video segment-level sources with original video local temporal localization capabilities from the candidate evidence set as the candidate evidence source set for subsequent in-depth mining: ， in, This refers to the event entity obtained in step 2.

5. The relevant element nodes and their candidate evidence sets, The first element related to the event entity and its missing elements. One candidate source of evidence, This indicates the source type of the candidate evidence source; each candidate evidence source... The information should include at least the type of evidence source, the corresponding entity node or relation edge, the type of missing related element, the evidence score, and the source location information. Indicates keyframe evidence, The video clip evidence, the complete video-level description source, the text caption source, and the fused description source serve as coarse-grained semantic sources for event scoring and candidate selection, but do not directly participate in subsequent in-depth mining of local areas of the original video; Among them, keyframe-level evidence sources can locate specific time positions in the original video, while video clip-level evidence sources can locate continuous time ranges in the original video. When the semantic correlation between candidate evidence sources and the current event entity and its missing elements is the same or similar, keyframe-level evidence sources have more accurate time positioning capabilities and are given priority as subsequent local in-depth mining objects; video clip-level evidence sources are used to support further mining of continuous event processes, subject interaction behaviors, or state change information. For each video's corresponding information, all event entities are ranked according to the event's strength of evidence score. Assume the video information contains... After sorting the event entities according to their event scores from highest to lowest, select the top... A significant proportion of high-value events are entering the in-depth analysis phase: ， in, This indicates the order of events ranked from highest to lowest based on the strength of evidence scores. Each event entity This indicates the screening ratio for entering the in-depth analysis stage. After identifying the events requiring in-depth analysis, the corresponding candidate inference data fragments or frames are further located, and the highest-scoring data fragments or frames are selected from the candidate evidence set. One piece of evidence was chosen as the final object of in-depth investigation. Step 2.7 Determine the targeted data mining strategy based on the type of candidate evidence source: The video clips and keyframes obtained in steps 1.1 to 1.7 are used to initially extract local semantic information from the original video and provide candidate evidence sources with temporal localization capabilities to supplement subsequent event elements. The video clip-level descriptions and keyframe-level descriptions obtained in the first round of processing still have problems with insufficient expression of local action processes, changes in subject attributes, environmental details, or event outcome information. In this step, for the selected candidate evidence sources, we trace back to their corresponding time positions or time ranges in the original video and perform dense sampling in the corresponding local areas using a sampling frequency higher than that in the first round of semantic extraction to construct a local visual input frame sequence for directional reasoning: For candidate evidence sources that are keyframe-level evidence, let the time position of the keyframe in the original video be denoted as . Set a local sampling time window centered on this time position. And adopt local encrypted sampling interval The original video frames within the time window are resampled to obtain the keyframe neighborhood visual input frame sequence: ， in, This represents the local encrypted sampling interval within the keyframe neighborhood time window. The local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. That is, the original visual information is re-acquired at a higher sampling frequency in the neighborhood range before and after the keyframe. By using the above-mentioned neighborhood encrypted sampling method centered on the keyframe time position, a more continuous and detailed visual context within the segment can be obtained on the basis of the first round of segment-level semantic expression. This is used to supplement the event development process, subject interaction relationship, scene state change and event result information that were not fully extracted due to the limited initial sampling density or coarse semantic generalization granularity. For evidence sourced from video clips, let the corresponding clip's time range be... Then, uniform sampling is performed within the corresponding segment range to obtain a segment-level visual evidence frame sequence: For candidate evidence sources that are video clips, let the time range corresponding to the video clip in the original video be... And employ fragment local encryption sampling interval The original video frames within the specified time range are resampled to obtain a segment-level visual input frame sequence: ， in, This indicates the local encrypted sampling interval within the video segment range. The local encrypted sampling interval is smaller than the sampling interval used in the first round of semantic extraction of the corresponding video segment. That is, the original visual information is re-acquired at a higher sampling frequency within the candidate video segment range. By performing local encrypted sampling on the target video segment as described above, a more continuous and detailed visual context within the segment can be obtained on the basis of the first round of segment-level semantic expression. This is used to supplement the event development process, subject interaction relationship, scene state change and event result information that were not fully extracted due to the limited initial sampling density or coarse semantic generalization granularity. The visual input frame sequence obtained from the above sampling is input into the visual language model. Targeted reasoning is performed around the set of missing elements of the current event entity to obtain supplementary information such as location, time, related subjects, environmental information, or impact of the result. The newly added semantics are written back into the event graph to achieve continuous updating of the event structure and semantic completeness.

8. The video semantic enhancement and optimization method for complex event situational understanding as described in claim 1, characterized in that, In step 2.2, the structure of the five types of event elements includes: The set of event elements can then be further represented as: ， in, Indicates the first Sub-events in event element types The corresponding elements and content; By using the above method, local semantic information that was originally scattered in the text captions and enhanced video descriptions is reorganized into a unified representation around the same sub-event, providing a structural basis for subsequent missing information detection, candidate evidence screening and targeted supplementation.

Images