A video event graph generation method and system based on a multi-modal large model

Abstract

The application discloses a video event graph generation method and system based on a multi-modal large model, relates to the technical field of artificial intelligence, and comprises the following steps: performing multi-modal data analysis on video data to obtain a time-stamped image frame sequence and a time-stamped text stream; aligning the image frame sequence and the text stream to the time axis of real-time data to be processed according to respective time stamps to form a time-synchronized multi-modal data pool; through multi-modal data analysis and time alignment, event trigger detection and argument extraction are performed in combination with a multi-modal large model, and problems, such as non-matching of information modalities, delayed event trigger detection, single evidence system and rough time alignment in traditional methods in video event understanding, are effectively solved. Therefore, accurate identification and high-quality structured representation of events in video data can be realized, and fine-grained and high-reliability intelligence support is provided for complex scenes such as video analysis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for generating video event graphs based on a multimodal large model. Background Technology

[0002] Driven by the digital wave, video data has experienced explosive growth, and the dynamic event information it contains has become a key element for intelligent decision-making and knowledge graph construction. Existing technologies have reached a bottleneck in building high-precision, traceable video event graphs, especially in fields with extremely high requirements for timeliness and accuracy, such as military command, security monitoring, and traffic safety. Traditional methods can no longer support the needs of in-depth event analysis in complex scenarios.

[0003] Traditional video event understanding methods suffer from several problems when processing video data, including information modality mismatch leading to loss of details, event trigger detection delays and missed detections, weak and singular evidence systems, and coarse and inaccurate graphs due to mismatched temporal alignment granularity. These issues fail to meet the needs for fine-grained intelligence and highly reliable analysis. Summary of the Invention

[0004] The purpose of this application is to provide a video event graph generation method and system based on a multimodal large model, which can effectively reduce the loss of fine-grained information, reduce the event false negative rate, enhance the reliability of evidence, and achieve refined time series analysis.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a video event graph generation method based on a multimodal large model. The method includes: acquiring video data to be processed; performing multimodal data parsing on the video data to obtain a sequence of image frames with timestamps and a text stream with timestamps; wherein the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from an audio stream; aligning the image frame sequence and the text stream to the time axis of the real-time data to be processed according to their respective timestamps to form a time-synchronized multimodal data pool; acquiring image frames and corresponding text fragments from the multimodal data pool using a sliding window or keyframe method, and inputting the image frames and text fragments into a first multimodal large model for cross-modal event trigger detection, outputting a detection result, wherein the detection result indicates that a preset type of event has occurred in the video data. The event, the detection result includes event type, trigger description, confidence level, and trigger point timestamp of the event occurrence; in response to the detection result, a time window is determined centered on the trigger point timestamp, and based on a preset event schema, the corresponding event elements are extracted from video clips and text within the time window in the form of dynamically generated question-and-answer pairs using a second multimodal large model, and the specific image frame ID or text word-level timestamp referenced by the model when generating each event element is recorded as the source of evidence; cross-modal consistency verification is performed on the extracted candidate event elements, and the detection results that pass the verification, the extracted event elements, and the source of evidence are integrated to construct a structured event instance; the constructed multiple event instances are sorted in chronological order, and an event graph containing event nodes and relationship edges is generated based on the logical relationships between events.

[0006] Optionally, the multimodal data parsing includes: The FFmpeg toolset is used to extract keyframes from video data and generate millisecond-level timestamps for each frame. The existing subtitle text in the video data is parsed using an SRT file parser; For the audio stream in the video data, the Whisper-large-v3 model is used for ASR recognition, and transcribed text with word-level timestamps is generated.

[0007] Optionally, the first multimodal large model is a GLM-4.5V model; The process of outputting the detection results includes: The GLM-4.5V model is guided to perform cross-modal inference by structured prompts, and a confidence threshold is set for the detection results output by the model to filter out detection results with a confidence level higher than the confidence threshold, and non-maximum suppression is performed to remove duplicates of the same event type detected consecutively.

[0008] Optionally, the second multimodal large model is the GLM-4.5V model; The dynamically generated question-and-answer pairs are transformed based on the argument structure defined in the event schema; The process of using a second multimodal large model to extract corresponding event elements from video clips and text within the time window in the form of dynamically generated question-and-answer pairs, and recording the specific image frame ID or text word-level timestamp referenced by the model when generating each event element as a source of evidence, includes: The question-and-answer pair, along with the image frames and text within the time window, are input into the GLM-4.5V model. This guides the GLM-4.5V model to specify the image frame ID or text timestamp on which the answer is based as the source of evidence while answering the question.

[0009] Optionally, before or during the construction of the structured event instance, the method further includes: The detection results and extracted event elements are input into the third multimodal large model, and semantic event names and descriptions are automatically generated through instructions and integrated into the event instance.

[0010] Optionally, the cross-modal consistency check includes: Calculate the semantic similarity of event elements extracted from different modalities using a pre-trained language model, and determine whether it exceeds a preset threshold; and / or, The event schema is used to perform rule-based validation on the data type, format, and completeness of event elements, and noisy data that does not meet the constraints is removed.

[0011] Optionally, the step of sorting the constructed multiple event instances in chronological order and analyzing the logical relationships between events to generate an event graph containing event nodes and relationship edges includes: By using rule-based logical relationship analysis or logical relationship analysis based on pre-trained language models, the causal or sequential relationships between events are determined, and the event graph is generated in the form of a directed graph, where nodes are event instances and edges are logical relationships between events with relation labels.

[0012] Optionally, the event instance includes the source of evidence corresponding to each event element, and the source of evidence is an image frame ID or a text timestamp, forming a traceable chain of evidence.

[0013] Optionally, the event instances are represented in JSON-LD format and embedded with schema:Event semantic web vocabulary to support integration with external knowledge graphs.

[0014] Secondly, this application provides a video event graph generation system based on a multimodal large model, used to implement the video event graph generation method based on a multimodal large model as described in any one of the first aspects above, including: The video data acquisition module is used to acquire the video data to be processed. A multimodal data parsing module is used to perform multimodal data parsing on the video data to obtain a time-stamped image frame sequence and a time-stamped text stream; wherein, the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from the audio stream; The multimodal timestamp alignment module is used to calibrate and align the image frame sequence and the text stream to the time axis of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool; The event result detection module is used to acquire image frames and corresponding text fragments from the multimodal data pool in the form of sliding window or keyframe, and input the image frames and text fragments into the first multimodal large model to perform cross-modal event trigger detection, and output the detection result. The detection result indicates that a preset type of event has occurred in the video data. The detection result includes event type, trigger description, confidence level and the trigger point timestamp of the event. The event element extraction module is used to respond to the detection result, determine a time window centered on the trigger point timestamp, and extract corresponding event elements from video clips and text within the time window in the form of dynamically generated question-and-answer pairs based on a preset event schema and using a second multimodal large model. The module also records the specific image frame ID or text word-level timestamp referenced by the model when generating each event element as a source of evidence. The event instance construction module is used to perform cross-modal consistency verification on the extracted candidate event elements, and integrate the verification results, the extracted event elements and the evidence sources to construct a structured event instance. The temporal structure representation module is used to sort multiple constructed event instances in chronological order and generate an event graph containing event nodes and relation edges based on the logical relationships between events.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method and system for generating video event graphs based on a multimodal large model. The method involves: acquiring video data to be processed; performing multimodal data parsing on the video data to obtain a sequence of image frames with timestamps and a text stream with timestamps; aligning the image frame sequence and text stream to the timeline of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool; resolving modal mismatch and temporal coarseness issues through multimodal data parsing and timestamp alignment; and then, acquiring image frames and corresponding text fragments from the multimodal data pool using a sliding window or keyframe method, inputting the image frames and text fragments into a first multimodal large model for cross-modal event trigger detection, and outputting the detection results. The detection results indicate that a preset type of event has occurred in the video data. The results include event type, trigger description, confidence level, and trigger point timestamp. In response to the detection results, a time window is determined centered on the trigger point timestamp. Based on a pre-defined event schema, a second multimodal large model is used to extract corresponding event elements from video clips and text within the time window in the form of dynamically generated question-and-answer pairs. The model records the specific image frame ID or text word-level timestamp referenced when generating each event element as a source of evidence. Cross-modal consistency verification is performed on the extracted candidate event elements. The detection results that pass the verification, the extracted event elements, and the source of evidence are integrated to construct a structured event instance. The constructed event instances are sorted chronologically, and the logical relationships between events are analyzed to generate an event graph containing event nodes and relationship edges. This application effectively solves the problems of information modality mismatch, delayed event trigger detection, single evidence system, and coarse time alignment in video event understanding by combining multimodal data parsing and time alignment with a multimodal large model for event trigger detection and argument extraction. It has the advantages of effectively reducing the loss of fine-grained information, lowering the event false negative rate, enhancing the reliability of evidence, and enabling refined temporal analysis. Therefore, it can achieve accurate identification and high-quality structured representation of events in video data, providing fine-grained and highly reliable intelligence support for complex scenarios such as video analysis. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the video event graph generation method based on a multimodal large model provided in Embodiment 1 of this application. Figure 2A visual schematic diagram of the "traffic accident-traffic congestion" event map generated in Embodiment 1 of this application; Figure 3 This is a schematic diagram of the functional modules of the video event graph generation system based on a multimodal large model provided in Embodiment 3 of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment takes a video analysis scenario as an example to generate an event graph from a video of an enemy armored force attack. The scope of protection of this application is not limited to this embodiment.

[0020] In an exemplary embodiment, a video event graph generation method based on a multimodal large model is provided. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is illustrated using a server as an example. Figure 1 As shown, the video event graph generation method based on a multimodal large model provided in Embodiment 1 of this application includes the following steps: S110. Obtain the video data to be processed.

[0021] Specifically, it acquires video data containing visual images and audio streams. The video data may include embedded subtitles, narration, and other text-related information. The data format supports mainstream video encoding formats and meets the basic requirements of multimodal parsing.

[0022] For example, the video data can be traffic monitoring video data containing visual images, audio content, and text content, and can be in MP4 encoding format, supporting FFmpeg parsing.

[0023] The visuals could show a multi-vehicle rear-end collision involving multiple cars and a truck on a highway, with vehicle debris scattered and vehicles behind braking suddenly; the audio stream could show continuous impact sounds, braking sounds, and a driver's cries for help: "Call the police! There's a three-vehicle rear-end collision at kilometer 35 southbound!"; and the text content could be a real-time warning message generated by the traffic management center: "An abnormal parking event has been detected on the G50 expressway southbound at kilometer 35, suspected to be a multi-vehicle accident."

[0024] S120. Perform multimodal data parsing on the video data to obtain a time-stamped image frame sequence and a time-stamped text stream; wherein, the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from the audio stream by an automatic speech recognition model.

[0025] In an exemplary embodiment, multimodal data parsing includes: Keyframes were extracted from the video data using the FFmpeg toolset, and millisecond-level timestamps were generated for each frame. Existing subtitle text in the video data was parsed using an SRT file parser. For the audio stream (narration or voice-over) in the video data, the Whisper-large-v3 model was used for ASR recognition, and transcribed text with word-level timestamps was generated.

[0026] Specifically, using the FFmpeg toolset, keyframes in the video data are extracted through a select filter (e.g., select='eq (pict_type,I)'). A millisecond-level timestamp is generated for each frame (e.g., at 00:02:35.120, a keyframe clearly showing the moment of the vehicle collision and the flying debris is extracted, and a millisecond-level timestamp is generated for this frame and a frame ID is assigned: 00:02:35.120). This efficiently captures visual changes and reduces the number of redundant frames. For existing subtitle text in the video data, SRT (SubRip) is used. The subtitle file parser directly reads the subtitle text content and precise start and end timestamps (e.g., the SRT parser extracts the traffic management center's warning text "An abnormal parking event was detected on the G50 expressway southbound at K35, suspected to be a multi-vehicle accident," with the corresponding timestamp being 00:02:36.000-->00:02:40.000). For narration, voice-over, or other audio content in the video data, OpenAI's Whisper-large-v3 model is used for automatic speech recognition (ASR) to generate transcribed text with word-level timestamps (e.g., using the Whisper-large-v3 model to perform ASR recognition on the audio stream, the transcription is "Quickly report! A three-vehicle rear-end collision occurred at K35 southbound!", and a word-level timestamp is generated for each word, with "three-vehicle rear-end collision" corresponding to the timestamp 00:02:35.500). At the same time, the audio parsing process is integrated with ASR recognition, using the transcribed text as a supplementary text stream to the audio modality.

[0027] S130. The image frame sequence and text stream are aligned to the time axis of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool.

[0028] Specifically, the image frame sequence and text stream parsed in step S120 are aligned to the time axis of the real-time data to be processed according to their respective timestamps to form a time-synchronized multimodal data pool. For any time point t in the multimodal data pool, the context window is set to [t-Δ, t+Δ] (Δ can be configured according to the scenario requirements, such as 2 seconds). All image frames and text fragments within this window are aggregated as the multimodal context at that time point to ensure that synchronized multimodal information can be obtained at any time in subsequent processing.

[0029] For example, the extracted keyframes, subtitle text, and ASR transcribed text are aligned to the same video timeline to construct a multimodal data pool; the context window is set to Δ=2 seconds. For the time point 00:01:15.200, its context window [00:01:13.200, 00:01:17.200] aggregates the keyframes at the moment of collision, the "three-car rear-end collision" and coordinate-related content in the ASR transcribed text.

[0030] S140. Obtain image frames and corresponding text fragments from the multimodal data pool using a sliding window or keyframe method, and input the image frames and text fragments into the first multimodal large model to perform cross-modal event trigger detection. Output the detection results, which indicate that a preset type of event has occurred in the video data. The detection results include the event type, trigger description, confidence level, and the trigger point timestamp of the event.

[0031] In an exemplary embodiment, the first multimodal large model is the GLM-4.5V model; The process of outputting the detection results includes: The GLM-4.5V model is guided to perform cross-modal inference by using structured cue words, and a confidence threshold is set for the detection results output by the model to filter out detection results with a confidence level higher than the confidence threshold, and non-maximum suppression is used to remove duplicates of the same event type detected consecutively.

[0032] For example, the vehicle collision keyframe and the corresponding ASR transcribed text within the context window of 00:02:35.120 are obtained from the data pool and input into the first multimodal large model (GLM-4.5V model). The structured prompts are input into the GLM-4.5V model: "You are a traffic safety incident analysis expert. Please analyze the given image and associated text to determine whether the following events have occurred: ['traffic accident', 'illegally parked vehicle', 'driving against traffic', 'pedestrian trespassing', 'congestion']. If they have occurred, please output them in JSON format: {'event_type': '...', 'trigger_phrase': '...', 'confidence': 0.95, 'timestamp': '...'}."; The GLM-4.5V model outputs the detection result: the event type is "traffic accident", the trigger description is "three-vehicle rear-end collision", the confidence level is 0.92, and the timestamp is 00:02:35.120; with a confidence threshold of 0.7, this result is higher than the threshold, and there are no consecutive detection results of the same event type, so deduplication is not required, and it is determined to be a valid event trigger signal detection result.

[0033] Among them, the GLM-4.5V's excellent ability to understand long texts and complex instructions ensures its analysis accuracy when dealing with complex traffic scene descriptions and lengthy ASR texts.

[0034] S150. In response to the detection results, a time window is determined with the trigger point timestamp as the center. Based on the preset event schema, the second multimodal large model is used to extract the corresponding event elements from the video clips and text within the time window in the form of dynamically generated question-and-answer pairs. The specific image frame ID or text word-level timestamp referenced by the model when generating each event element is recorded as the source of evidence.

[0035] In an exemplary embodiment, the second multimodal large model is the GLM-4.5V model; Dynamically generated question-and-answer pairs are transformed based on the argument structure defined in the event schema; Using a second multimodal large model, corresponding event elements are extracted from video clips and text within a time window in the form of dynamically generated question-and-answer pairs. The specific image frame ID or text word-level timestamp referenced by the model when generating each event element is recorded as a source of evidence, including: Input the question-and-answer pair along with the image frames and text within the time window into the GLM-4.5V model. This will guide the GLM-4.5V model to specify the image frame ID or text timestamp as the source of evidence when answering the question.

[0036] Specifically, the event schema is defined in YAML or JSON format, containing information such as event type, required arguments, optional arguments, argument type, and argument description. Structured question-and-answer pairs are dynamically generated based on the event schema, transforming the open visual question-and-answer task into a targeted information filling task. The question-and-answer pairs, along with key image frames and synchronized text fragments within the time window, are input into the GLM-4.5V model to obtain accurate event elements in a question-and-answer format. The specific image frame ID or text timestamp referenced by the model when generating each event element is recorded as the source of evidence for that event element, forming a traceable chain of evidence.

[0037] For example, in response to the detection results, the keyframes, caption text, and ASR transcription text within the time window [00:02:30.000, 00:02:40.000] are locked, centered on the trigger point timestamp 00:02:35.120; the schema (JSON format) of the traffic safety scenario "traffic accident" event is invoked, which defines the required arguments as: accident location (road description), number of vehicles involved (integer), accident type (rear-end collision / side collision / frontal collision, etc.), and whether anyone is injured (yes / no); a structured question-and-answer pair is generated based on the schema, and the question-and-answer pair and the multimodal data within the time window are input into the GLM-4.5V model to extract event elements and record the source of evidence: Question 1: "Based on the provided images and text, what is the exact location of this traffic accident? Please provide the road name and mileage." Answer: "K35 on the G50 Expressway, southbound to northbound." Evidence source: ASR text (00:02:35.500) and caption text (00:02:36.000). Question 2: "How many vehicles are shown in the image or text? Please extract the exact number." Answer: "4 vehicles (3 cars and 1 truck)". Evidence source: Visual keyframes (frame ID: 00:02:35.120, 00:02:35.800). Question 3: "What is the specific type of traffic accident that can be identified from the image?" Answer: "Multiple rear-end collision". Evidence source: Visual keyframe (frame ID: 00:02:35.120).

[0038] S160. Perform cross-modal consistency verification on the extracted candidate event elements, including: Calculate the semantic similarity of event elements extracted from different modalities using a pre-trained language model, and determine whether it exceeds a preset threshold; and / or, The event schema is used to perform rule-based validation on the data type, format, and completeness of event elements, and to remove noisy data that does not meet the constraints.

[0039] Specifically, pre-trained language models (such as Sentence-BERT) are used to calculate semantic embedding vectors of event elements extracted from different modalities (visual, caption, ASR transcribed text). Semantic consistency is judged by calculating cosine similarity. When the cosine similarity is higher than a preset threshold (such as 0.8), it is determined that the semantics point to the same entity. For key entities such as location coordinates and equipment numbers, precise string matching or regular expressions are used for verification. The data type, format, and completeness of event elements are validated by rules using a pre-defined event schema. For example, it checks whether the number of vehicles involved is an integer, whether the accident location contains the necessary road information, and whether all required arguments are filled. Event elements that do not meet the constraints are removed through a rule engine or lightweight ontology reasoning.

[0040] For example, using Sentence-BERT to calculate semantic similarity, the cosine similarity between "K35 km southbound" in the ASR text and the location information shown on the road sign in the visual keyframe is 0.91 (threshold 0.8), and the cosine similarity between "G50 Expressway K35 section southbound" in the subtitle text and the ASR location is 0.94, both of which are determined to be semantically consistent. There is a difference in the number of vehicles involved between "three-vehicle rear-end collision" (ASR text) and "4 vehicles involved" detected by visual inspection. Through verification, it was found that 4 vehicles are indeed visible in the visual keyframe. The ASR driver's statement in the emergency situation is an approximate number. Based on the visual evidence, the number of vehicles involved is corrected to 4.

[0041] Schema constraint validation: All required arguments have been filled, the accident location format conforms to the "road name + direction + mileage" specification, the number of vehicles involved is an integer of 4, and the accident type is "rear-end collision", which is a preset type enumeration value. Validation passed.

[0042] S170. Input the detection results and extracted event elements into the third multimodal large model, automatically generate semantic event names and event descriptions through instructions, and integrate them into the event instance.

[0043] Specifically, the verified detection results, extracted event elements, and corresponding contextual information are input into the third multimodal large model GLM-4.5V. The model is instructed to generate semantic and generalized event names and detailed event descriptions. The event names should concisely reflect the core elements of the event, and the event descriptions should fully include key information such as the event's occurrence time, subject, action, object, and environment.

[0044] For example, the detection results, validated event elements, and contextual information within the time window are input into the GLM-4.5V model. The input command is: "Generate a semantically complete and highly generalized event name and detailed description based on the following information: Event type is traffic accident, location is G50 expressway southbound at kilometer 35, involving 4 vehicles (3 cars and 1 truck), accident type is chain-reaction rear-end collision, trigger time is 00:02:35.120"; Model generation: Incident title: "Four-vehicle rear-end collision occurred at kilometer 35 on the G50 expressway southbound to northbound"; Event Description: "At 00:02:35.120, a multi-vehicle rear-end collision occurred at kilometer 35 on the G50 expressway heading north, involving four vehicles (three cars and one truck). Video footage shows debris flying at the moment of impact, and vehicles behind braking suddenly. Audio recordings show a driver calling for help, 'Call the police! There's a three-car rear-end collision at kilometer 35 heading north!' The traffic management center issued a warning at 00:02:36.000 confirming the abnormal parking situation on this section of the road."

[0045] S180. Integrate the detection results, extracted event elements, and sources of evidence to construct a structured event instance.

[0046] Optionally, the event instance contains the source of evidence for each event element, which can be an image frame ID or a text timestamp, forming a traceable chain of evidence.

[0047] Optionally, event instances are represented in JSON-LD format and embedded with schema:Event semantic web vocabulary to support integration with external knowledge graphs.

[0048] For example, by integrating the above detection results, extracted event elements, and evidence sources, a structured event instance in JSON-LD format is constructed, embedding the schema:Event semantic web terminology to fully preserve the evidence source information. A specific example is as follows: json { "@context":"https: / / schema.org", "@type":"Event", "name":"A four-vehicle rear-end collision occurred at kilometer 35 on the G50 expressway, southbound." "eventType":"Traffic accident", "startTime":"00:01:15.200", "triggerSignal":{ "triggerPhrase": "Three-car rear-end collision", "confidence":0.92, "timestamp":"00:01:15.200" }, "arguments":[ { "argName":"Accident location", "argValue":"G50 Expressway, southbound to northbound, at kilometer 35", "evidenceSource": "ASR text (00:02:35.500)" }, { "argName":"Number of vehicles involved", "argValue":"4 vehicles (3 cars, 1 truck)", "evidenceSource": "Visual keyframes (00:02:35.120, 00:02:35.800)" }, { "argName":"Accident Type", "argValue":"Multi-vehicle rear-end collision", "evidenceSource": "Visual keyframe (00:02:35.120)" } ], "evidenceFrameIDs": ["00:02:35.120", "00:02:35.800"], "description": "At 00:02:35.120, a multi-vehicle rear-end collision occurred at kilometer 35 on the G50 expressway heading north, involving four vehicles (three cars and one truck). Video footage shows debris flying at the moment of impact, and vehicles behind braking suddenly. Audio recordings show a driver calling for help, 'Call the police! There's a three-car rear-end collision at kilometer 35 heading north!' The traffic management center issued a warning at 00:02:36.000 confirming abnormal parking on this section of the road." Simultaneously, for the "rear traffic congestion" event at 00:02:40.000 in the video, a corresponding structured event instance is constructed following the steps described above.

[0049] S190. Sort the constructed event instances in chronological order, analyze the logical relationships between the events, and generate an event graph containing event nodes and relationship edges.

[0050] In an exemplary embodiment, the constructed multiple event instances are sorted chronologically, and the logical relationships between the events are analyzed to generate an event graph containing event nodes and relationship edges, including: By using rule-based logical relationship analysis or logical relationship analysis based on pre-trained language models, causal or sequential relationships between events are determined, and an event graph is generated in the form of a directed graph, where nodes are event instances and edges are logical relationships between events with relation labels.

[0051] Specifically, the logical relationships between events are analyzed using either rule-based or pre-trained language model-based methods. Rule-based analysis includes using temporal proximity and domain lexicons (such as a causal lexicon for traffic safety events) for judgment, while analysis based on pre-trained language models includes using semantic reasoning for judgment. The determined logical relationships between events include at least causal and sequential relationships. Relationship labels (such as causes, followedBy) and time interval information can be added to the relationship edges between events. Using event instances as nodes and the logical relationships between events as labeled directed edges, a visual event graph is generated to intuitively display the occurrence sequence and logical connections of events in the video.

[0052] For example, the "traffic accident" event instance (00:02:35.120) and the "rear traffic congestion" event instance (00:02:40.500) are sorted by timestamp; By combining the traffic safety incident causal lexicon (including the "traffic accident-traffic congestion" association pair) and the semantic reasoning of the pre-trained language model, it is determined that there is a causal relationship between the two events (the chain-rear-end collision is the direct cause of the congestion of the vehicles behind) and a sequential relationship (the congestion occurred 5.38 seconds after the accident).

[0053] like Figure 2 As shown, Figure 2 This is a visualization diagram of the "traffic accident-traffic congestion" event map generated in Embodiment 1 of this application. The yellow node represents Event 1: "A four-vehicle rear-end collision occurred at K35 on the G50 Expressway from south to north". The red node represents Event 2: "The traffic accident caused severe congestion of vehicles behind". The directed edges are labeled with causal relationships, sequential relationships and time intervals.

[0054] Using event 1 and event 2 instances as nodes, and labeled directed edges to represent logical relationships, generate a directed graph-structured event graph: Event 1: A four-vehicle rear-end collision occurred at kilometer 35 on the G50 expressway from south to north. The node is connected to Event 2: The traffic accident caused severe traffic congestion behind the vehicle. Event graph generation is completed by pointing the directed edge labeled "causes" / "followedBy" with an interval of 5.38s.

[0055] Example 2: The processing results of Example 1 were verified by evaluating accuracy, efficiency, and practicality. The results are as follows: Accuracy metrics: 100% accuracy in event type identification (correctly identifies "traffic accident" and "traffic congestion" event types), 100% completeness in event element extraction (all required arguments are correctly extracted), 100% pass rate in cross-modal validation (no semantic conflicts, format errors, or missing information), and 100% accuracy in event relationship identification (correctly identifies causal, sequential, and time interval relationships). Efficiency metrics: Single event processing time is 0.8 seconds (including data parsing, model inference, verification, and structuring), which is 65% more efficient than the traditional multimodal fusion method (average time 2.3 seconds); the redundancy rate of key frame extraction is reduced by 40% (only 15 key frames are retained, while traditional fixed frame rate sampling requires 25 frames). Practicality metrics: The generated event graph intuitively displays the logical chain of traffic events from "traffic accident to traffic congestion". The evidence chain preserved by the event instances supports manual backtracking and verification (clicking the frame ID can view the original keyframe, and clicking the text timestamp can play the corresponding audio segment), fully meeting the core requirements of intelligent traffic monitoring for interpretability and traceability.

[0056] Example 3, based on the same inventive concept, such as Figure 3 As shown in the embodiments of this application, a video event graph generation system based on a multimodal large model is also provided for implementing the video event graph generation method based on a multimodal large model described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the video event graph generation system based on a multimodal large model provided below can be found in the limitations of the video event graph generation method based on a multimodal large model described above, and will not be repeated here.

[0057] This video event graph generation system based on a multimodal large model specifically includes: The video data acquisition module 301 is used to acquire video data to be processed.

[0058] The multimodal data parsing module 302 is used to perform multimodal data parsing on video data to obtain a time-stamped image frame sequence and a time-stamped text stream; wherein, the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from the audio stream.

[0059] The multimodal timestamp alignment module 303 is used to calibrate and align the image frame sequence and text stream to the time axis of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool.

[0060] The event result detection module 304 is used to acquire image frames and corresponding text fragments from the multimodal data pool in the form of sliding window or keyframe, input the image frames and text fragments into the first multimodal large model, perform cross-modal event trigger detection, and output the detection result. The detection result indicates that a preset type of event has occurred in the video data. The detection result includes the event type, trigger description, confidence level and the trigger point timestamp of the event.

[0061] The event element extraction module 305 is used to respond to the detection results, determine the time window centered on the trigger point timestamp, and extract the corresponding event elements from the video clips and text within the time window in the form of dynamically generated question-and-answer pairs based on the preset event schema and the second multimodal large model. It also records the specific image frame ID or text word-level timestamp referenced by the model when generating each event element as the source of evidence.

[0062] The event instance construction module 306 is used to perform cross-modal consistency verification on the extracted candidate event elements, and integrate the verification results, the extracted event elements and the evidence sources to construct a structured event instance.

[0063] The temporal structure representation module 307 is used to sort the constructed multiple event instances in chronological order and generate an event graph containing event nodes and relation edges based on the logical relationships between events.

[0064] After the system in this embodiment is running, it can automatically complete the entire process from video data input to event graph output. The processing result is consistent with that in Embodiment 1. It also supports batch processing of multiple videos, and a single server can process multiple video streams simultaneously, meeting the efficiency requirements of practical applications.

[0065] It should be noted that the multimodal large models, pre-trained language models, toolsets, etc. used in the embodiments of this application can be replaced with other models / tools with the same or similar functions. For example, GLM-4.5V can be replaced with other multimodal large models that support cross-modal reasoning, Whisper-large-v3 can be replaced with other highly robust ASR models, and Sentence-BERT can be replaced with other pre-trained language models that support semantic embedding. Such replacements all fall within the protection scope of this application.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating video event graphs based on a multimodal large model, characterized in that, include: Acquire the video data to be processed; The video data is subjected to multimodal data parsing to obtain a time-stamped image frame sequence and a time-stamped text stream; wherein, the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from the audio stream; The image frame sequence and the text stream are aligned and calibrated to the timeline of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool; Image frames and corresponding text fragments are acquired from the multimodal data pool using a sliding window or keyframe method. The image frames and text fragments are then input into a first multimodal large model to perform cross-modal event trigger detection. The detection results are output, and the detection results indicate that a preset type of event has occurred in the video data. The detection results include the event type, trigger description, confidence level, and the trigger point timestamp of the event. In response to the detection result, a time window is determined with the trigger point timestamp as the center. Based on the preset event schema, the second multimodal large model is used to extract the corresponding event elements from the video clips and text within the time window in the form of dynamically generated question-and-answer pairs. The specific image frame ID or text word-level timestamp referenced by the model when generating each event element is recorded as the source of evidence. Cross-modal consistency verification is performed on the extracted candidate event elements, and the verification results, extracted event elements and evidence sources are integrated to construct a structured event instance; The cross-modal consistency check includes: Calculate the semantic similarity of event elements extracted from different modalities using a pre-trained language model, and determine whether it exceeds a preset threshold; and / or, The event schema is used to perform rule-based validation on the data type, format, and completeness of event elements, and to remove noisy data that does not meet the constraints. The constructed event instances are sorted in chronological order, and an event graph containing event nodes and relationship edges is generated based on the logical relationships between the events.

2. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The multimodal data parsing includes: The FFmpeg toolset is used to extract keyframes from video data and generate millisecond-level timestamps for each frame. The subtitle text in the video data is parsed using an SRT file parser; For the audio stream in the video data, the Whisper-large-v3 model is used for ASR recognition, and transcribed text with word-level timestamps is generated.

3. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The first multimodal large model is the GLM-4.5V model; The process of outputting the detection results includes: The GLM-4.5V model is guided to perform cross-modal inference by structured prompts, and a confidence threshold is set for the detection results output by the model to filter out detection results with a confidence level higher than the confidence threshold, and non-maximum suppression is performed to remove duplicates of the same event type detected consecutively.

4. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The second multimodal large model is the GLM-4.5V model; The dynamically generated question-and-answer pairs are transformed based on the argument structure defined in the event schema; The process of using a second multimodal large model to extract corresponding event elements from video clips and text within the time window in the form of dynamically generated question-and-answer pairs, and recording the specific image frame ID or text word-level timestamp referenced by the model when generating each event element as a source of evidence, includes: The question-and-answer pair, along with the image frames and text within the time window, are input into the GLM-4.5V model. This guides the GLM-4.5V model to specify the image frame ID or text timestamp on which the answer is based as the source of evidence while answering the question.

5. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, Before or during the construction of the structured event instance, the method further includes: The detection results and extracted event elements are input into the third multimodal large model, and semantic event names and descriptions are automatically generated through instructions and integrated into the event instance.

6. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The process of sorting the constructed multiple event instances in chronological order, analyzing the logical relationships between events, and generating an event graph containing event nodes and relationship edges includes: By using rule-based logical relationship analysis or logical relationship analysis based on pre-trained language models, the causal or sequential relationships between events are determined, and the event graph is generated in the form of a directed graph, where nodes are event instances and edges are logical relationships between events with relation labels.

7. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The event instance contains the evidence source corresponding to each event element. The evidence source is an image frame ID or a text timestamp, forming a traceable evidence chain.

8. The video event graph generation method based on a multimodal large model according to claim 1, characterized in that, The event instances are represented in JSON-LD format and embedded with schema:Event semantic web vocabulary to support integration with external knowledge graphs.

9. A video event graph generation system based on a multimodal large model, characterized in that, A method for generating video event graphs based on a multimodal large model as described in any one of claims 1-8 includes: The video data acquisition module is used to acquire the video data to be processed. A multimodal data parsing module is used to perform multimodal data parsing on the video data to obtain a time-stamped image frame sequence and a time-stamped text stream; wherein, the text stream includes text extracted from embedded subtitles in the video and / or text transcribed from the audio stream; The multimodal timestamp alignment module is used to calibrate and align the image frame sequence and the text stream to the time axis of the real-time data to be processed according to their respective timestamps, forming a time-synchronized multimodal data pool; The event result detection module is used to acquire image frames and corresponding text fragments from the multimodal data pool in the form of sliding window or keyframe, and input the image frames and text fragments into the first multimodal large model to perform cross-modal event trigger detection, and output the detection result. The detection result indicates that a preset type of event has occurred in the video data. The detection result includes event type, trigger description, confidence level and the trigger point timestamp of the event. The event element extraction module is used to respond to the detection result, determine a time window centered on the trigger point timestamp, and extract corresponding event elements from video clips and text within the time window in the form of dynamically generated question-and-answer pairs based on a preset event schema and using a second multimodal large model. The module also records the specific image frame ID or text word-level timestamp referenced by the model when generating each event element as a source of evidence. The event instance construction module is used to perform cross-modal consistency verification on the extracted candidate event elements, and integrate the verification results, extracted event elements, and evidence sources to construct a structured event instance. The cross-modal consistency verification includes: using a pre-trained language model to calculate the semantic similarity of event elements extracted from different modalities and determining whether it is higher than a preset threshold; and / or using the event schema to perform rule verification on the data type, format, and completeness of event elements, and removing noisy data that does not meet the constraints. The temporal structure representation module is used to sort multiple constructed event instances in chronological order and generate an event graph containing event nodes and relation edges based on the logical relationships between events.

Images