A time series structured data generation method and system based on a multi-modal large language model
Patent Information
- Application Number
- CN202611076197.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-10-27
Smart Images
Figure CN122986262A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of large model technology, and more specifically, to a method and system for generating temporal structured data based on a multimodal large language model. Background Technology
[0002] The maturity of drone technology has led to the widespread application of drones equipped with high-definition cameras in urban traffic, security patrols, and emergency rescue scenarios. Drones generate a large amount of video streams during missions, accompanied by flight parameter data, including altitude, attitude angles (pitch, roll, yaw), GPS coordinates, speed, and timestamps. Currently, this type of video data with flight parameters is mainly processed through manual annotation or traditional visual algorithms, which has significant limitations. Manual processing requires professionals to review the video frame by frame, manually recording target objects, scene information, and event behaviors, converting them into structured data. Simultaneously, video timestamps must be manually aligned with flight parameter data. This method is extremely inefficient, costly, and slow to respond quickly when dealing with high frame rates and long video streams. Inconsistent understandings and standards among different annotators result in inconsistent data quality.
[0003] Traditional computer vision solutions employ object detection algorithms such as YOLO and Faster R-CNN to identify objects in video frames, combine this with OCR to extract text information, and then use a rule engine to correlate it with flight parameters. While this approach improves efficiency, it has significant limitations: it can only recognize predefined categories and lacks semantic understanding capabilities for complex open-domain scenes; it can only process single-frame images, lacking temporal modeling and failing to capture changes, trajectories, and event evolutions of targets over time; the spatiotemporal alignment of flight parameters and videos typically uses simple timestamp matching, neglecting viewpoint shifts caused by drone attitude changes and GPS errors; and the generated data is a flat table, lacking hierarchical organization and making it difficult to support complex queries. Another approach is a hybrid architecture that combines CNNs to extract visual features with expert rules for data structuring. This approach is effective in specific scenarios, but knowledge acquisition faces bottlenecks—expert rules require extensive domain knowledge and are difficult to cover all scenarios. When scenarios change, the feature extraction network and rules need to be redesigned, resulting in high migration costs.
[0004] In summary, existing solutions have shortcomings in semantic understanding, temporal modeling, spatiotemporal alignment accuracy, data hierarchicalization, and system scalability, making it difficult to meet the needs of large-scale, high-efficiency, and intelligent processing. Summary of the Invention
[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide a method for generating temporal structured data based on a multimodal large language model, which realizes accurate spatiotemporal fusion of video content and flight parameter data, and improves semantic understanding ability and temporal data consistency in open scenarios.
[0006] The objective of this application is achieved through the following technical solution: Firstly, this application proposes a method for generating temporal structured data based on a multimodal large language model, including: S1. Receive UAV video stream with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract the image frame sequence, flight parameter data stream and corresponding metadata. S2. Based on the timestamp, perform linear interpolation to align the image frame sequence and flight parameter data stream, and establish a spatial mapping from image pixel coordinates to geographic coordinates based on the UAV attitude angle, camera intrinsic parameters and collinearity equations, and encapsulate them to form original structured data units. S3. Input the image frame into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompt words, and output single-frame structured semantic information; S4. Define a time window, aggregate the structured semantic information of a single frame, construct a four-level time-series data structure, and establish a spatiotemporal index; S5. In scenarios with continuous video stream input, maintain a time-series data queue based on a sliding time window, perform incremental updates by calculating the differences between the old and new windows, and maintain a global object ID mapping. S6. Perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated time series data, and output the final time series structured data.
[0007] In one possible implementation, the step of linearly interpolating and aligning the image frame sequence with the flight parameter data stream based on timestamps includes: Let the set of video frame timestamps be... The original timestamp set of flight parameter data is For any video frame timestamp The corresponding flight parameter values are obtained by performing linear interpolation calculations on adjacent flight parameter data points. : ,in and for The timestamps of two adjacent flight parameters.
[0008] In one possible implementation, the step of establishing a spatial mapping from image pixel coordinates to geographic coordinates based on the UAV attitude angle, camera intrinsic parameters, and collinearity equation includes: pass Establish a mapping relationship, where For the camera intrinsic parameter matrix, The rotation and translation matrix is composed of the pitch angle, roll angle, and yaw angle of the UAV. Image pixel coordinates, For world geographical coordinates, is the scale factor.
[0009] In one possible implementation, step S3 includes: Image frames are input into the visual encoder of a multimodal large language model to generate high-dimensional visual feature vectors; Guide a multimodal large language model to perform at least one of the following tasks: object detection and recognition, scene understanding and classification, attribute extraction, and relational reasoning, using structured prompts; The natural language results output by the multimodal large language model are parsed and transformed into structured single-frame semantic information.
[0010] In one possible implementation, the four-level time-series data structure includes: The scene layer is used to describe the global environmental context and weather and lighting conditions. The object layer is used to record the unique identifier of the target object and its movement trajectory over time. The attribute layer is used to record the fine-grained characteristics and temporal evolution of objects. The relationship layer is used to record the interaction relationships and event sequences between objects.
[0011] In one possible implementation, the step of establishing the spatiotemporal index in step S4 includes: Establish a time series index for a four-level time series data structure to support conditional queries based on time ranges; A spatial index is created for the four-level time-series data structure to support queries based on geographic range; Establish a global object identifier index for the target object in the object layer, supporting object tracking across time windows.
[0012] In one possible implementation, the step of incrementally updating by calculating the difference features between the old and new windows in S5 includes: Maintain a fixed-length sliding time window queue When a new window Upon arrival, remove the oldest window that entered the queue. ; Calculate the difference features between the new window and its neighboring windows. ,pass Update the time-series data and maintain the global object ID mapping table through time-series consistency checks.
[0013] In one possible implementation, step S6 includes: Check the monotonicity and continuity of all timestamps in the dataset, and mark any detected timestamp anomalies; Based on pre-defined domain knowledge rules, it verifies whether the changes in the target's spatial coordinates conform to the motion speed constraints, whether the changes in the target's attributes conform to common sense rules, and whether the relationships between objects satisfy transitivity and symmetry. To address the loss of semantic information due to image quality degradation or model recognition failure, weighted interpolation based on temporal distance or invoking a multimodal large language model to perform inference and completion based on the context of adjacent frames are adopted. To address data gaps caused by dropped video frames, motion trajectories are calculated based on flight parameters or spatiotemporal predictions are made based on visual content to generate supplementary data.
[0014] In one possible implementation, the method further includes: The final time-series structured data output is serialized and stored using JSON Schema, and the data includes timestamp and geographic coordinate fields.
[0015] Secondly, this application proposes a temporal structured data generation system based on a multimodal large language model, comprising: The video preprocessing module is used to receive UAV video streams with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract image frame sequences, flight parameter data streams and corresponding metadata. The original structured data construction module is used to perform linear interpolation alignment of image frame sequences and flight parameter data streams based on timestamps, and to establish a spatial mapping from image pixel coordinates to geographic coordinates based on UAV attitude angles, camera intrinsic parameters and collinearity equations, and encapsulate them into original structured data units. The visual recognition module of the multimodal large language model is used to input image frames into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompts, and output single-frame structured semantic information; The temporal structure extraction module is used to define time windows, aggregate structured semantic information of single frames, construct a four-level temporal data structure, and establish a spatiotemporal index. The streaming incremental update module is used to maintain a time-series data queue based on a sliding time window in scenarios with continuous video stream input. It performs incremental updates by calculating the differences between the old and new windows and maintains a global object ID mapping. The timing consistency verification and inference module is used to perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated timing data, and output the final timing structured data.
[0016] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.
[0017] This application discloses a method and system for generating temporal structured data based on a multimodal large language model. First, it receives a UAV video stream with flight parameters, decodes and splits it into frames, extracting image frame sequences, flight parameter data streams, and metadata. Then, it performs linear interpolation alignment and establishes a spatial mapping, encapsulating the original structured data units. The image frames are input into the multimodal large language model, which guides target detection, scene understanding, and relational reasoning through structured prompts, outputting single-frame semantic information. A time window is defined to aggregate semantic information, constructing a four-level temporal data structure and establishing a spatiotemporal index. Incremental updates are performed through a sliding time window, and object ID mapping is maintained. Finally, timestamp verification, logical consistency checks, and missing value inference and completion are performed, outputting high-quality temporal structured data. This achieves accurate spatiotemporal fusion of video content and flight parameter data, improving semantic understanding capabilities and temporal data consistency in open scenarios. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 The diagram shows a flowchart of a temporal structured data generation method based on a multimodal large language model proposed in an embodiment of this application. Detailed Implementation
[0020] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0021] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] This invention aims to solve the following problems: achieving accurate spatiotemporal alignment and fusion of video content and flight parameter data; utilizing a multimodal large language model to understand complex open-domain scenarios; constructing a multi-level, temporally ordered data structure to track target state evolution; incremental updates and consistency maintenance in streaming data scenarios; and establishing a general processing paradigm to enable the system to quickly adapt to different application scenarios. This application proposes a method for generating temporally ordered structured data based on a multimodal large language model.
[0023] Please refer to Figure 1 , Figure 1 This paper illustrates a flowchart of a temporal structured data generation method based on a multimodal large language model, as proposed in an embodiment of this application, including: S1. Receive UAV video stream with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract the image frame sequence, flight parameter data stream and corresponding metadata.
[0024] First, the input video stream is decoded and restored using an H.265 / HEVC decoding engine, extracting metadata information such as encoding parameters, frame rate, resolution, and color space. Then, the video is decomposed into a discrete sequence of image frames according to a preset sampling strategy, assigning each frame a unique frame number and recording its time offset relative to the start of the video. Simultaneously, the drone's flight control logs or real-time data stream are parsed to extract flight parameters such as timestamps, latitude and longitude, altitude, pitch angle, roll angle, yaw angle, ground speed, airspeed, and battery level. Furthermore, an EXIF parsing library is used to extract shooting parameters and proprietary metadata embedded by the drone manufacturer from the images, obtaining the camera intrinsic parameter matrix and distortion coefficients. After these processes, a standardized sequence of image frames, a synchronized flight parameter data stream, and complete metadata information are finally output.
[0025] S2. Based on the timestamp, perform linear interpolation to align the image frame sequence with the flight parameter data stream, and establish a spatial mapping from image pixel coordinates to geographic coordinates based on the UAV attitude angle, camera intrinsic parameters and collinearity equations, and encapsulate them to form original structured data units.
[0026] First, a linear interpolation algorithm is used to resample the sparse flight parameter data stream according to the timestamps of the video frames, ensuring that each frame corresponds to the accurate flight parameter state. Then, based on the UAV's attitude angle and the camera's intrinsic parameter matrix, a spatial mapping relationship between image pixel coordinates and real geographic coordinates is established through collinearity equations to correct deviations caused by UAV movement, viewpoint changes, and lens distortion. Finally, each frame, the synchronously interpolated flight parameter data, and the spatial mapping parameters are encapsulated into a complete original structured data unit.
[0027] The steps for linearly interpolating and aligning the image frame sequence with the flight parameter data stream based on timestamps include: Let the set of video frame timestamps be... , indicating the first to the last extracted after decoding The timestamp sequence corresponding to the frame image, and the set of timestamps for the original flight parameter data are as follows: For any video frame timestamp The corresponding flight parameter values are obtained by performing linear interpolation calculations on adjacent flight parameter data points. : ,in and for Two adjacent flight parameter timestamps. The raw flight parameter data timestamp set represents the timestamp sequence of flight parameters recorded by the UAV flight control system. The sampling frequency is usually lower than the video frame rate, hence m <n。
[0028] For any video frame timestamp First, locate the timestamps of its two adjacent flight parameters. and (satisfy ≤ ≤ Then, the flight parameter values corresponding to the video frame are calculated using a linear interpolation formula. : ,in and for Two adjacent flight parameter timestamps, among which and These are the timestamps in the raw flight parameter data. and The corresponding flight parameter values, This is the time difference between the video frame timestamp and the timestamp of the adjacent preceding flight parameter. This represents the time interval between two adjacent flight parameter timestamps. Through this interpolation calculation, the asynchronously acquired flight parameter data is resampled to be completely synchronized with the video frame timestamps, achieving precise alignment of multi-source heterogeneous data in the time dimension.
[0029] The steps for establishing a spatial mapping from image pixel coordinates to geographic coordinates based on UAV attitude angles, camera intrinsic parameters, and collinearity equations include: pass Establish a mapping relationship, where For the camera intrinsic parameter matrix, The rotation and translation matrix is composed of the pitch angle, roll angle, and yaw angle of the UAV. Image pixel coordinates, For world geographical coordinates, is the scale factor.
[0030] Calculate the rotation matrix based on the pitch angle, roll angle, and yaw angle from the UAV flight data. (Describe the rotational relationship between the UAV's body coordinate system and the world geographic coordinate system), and determine the translation vector by combining the UAV's GPS coordinates. (Describing the position of the camera's optical center in the world coordinate system), together forming the rotation and translation matrix. Introducing the camera intrinsic parameter matrix (Includes parameters such as focal length and principal point coordinates, obtained by parsing drone manufacturer-specific metadata such as DJI XMP data) and distortion coefficients. (Used to correct lens distortion); Image pixel coordinates are established using collinearity equations. with world geographical coordinates Mapping relationship: ,in This is the scale factor, used to unify the scale of homogeneous coordinates; It is a homogeneous vector of pixel coordinates.
[0031] This mapping allows any pixel in an image to be projected back into a 3D geographic space, achieving the conversion from pixel coordinates to geographic coordinates. Subsequently, the basic attribute encapsulation module encapsulates each frame of image, time-aligned flight parameter data, and spatial mapping results into raw structured data units, ultimately outputting a standardized data sequence that is time-synchronized and spatially mapped.
[0032] S3. Input the image frame into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompt words, and output single-frame structured semantic information; The preprocessed single-frame image is input into the selected open-source multimodal large language model. First, a high-dimensional visual feature vector is generated through a visual encoder. Then, structured prompts containing task instructions and format constraints guide the model to perform at least one visual semantic task among target detection and recognition, scene understanding and classification, and object relationship reasoning. Finally, the natural language results output by the model are parsed and transformed into unified single-frame structured semantic information.
[0033] Step S3 includes: Image frames are input into the visual encoder of a multimodal large language model to generate high-dimensional visual feature vectors; Guide a multimodal large language model to perform at least one of the following tasks: object detection and recognition, scene understanding and classification, attribute extraction, and relational reasoning, using structured prompts; The natural language results output by the multimodal large language model are parsed and transformed into structured single-frame semantic information.
[0034] The system first selects a suitable open-source multimodal large language model from the Qwen-VL series, LLaVA series, InternVL series, or Yi-VL series based on the application scenario and loads it. Then, image frames are fed into the model's visual encoder, which extracts deep semantic features of the image through convolution or Transformer architecture, generating high-dimensional visual feature vectors. ,in The number of feature channels, and These are the height and width of the feature map, respectively. This vector encodes multi-layered visual information in the image, including shape, texture, color, and spatial layout.
[0035] The system design includes structured prompts for task instructions, output format constraints, and examples. These high-dimensional visual feature vectors, along with the prompts, are input into the model's language decoder. The model leverages its multimodal alignment capabilities and intrinsic knowledge to perform at least one visual semantic understanding task: object detection and recognition, scene understanding and classification, attribute extraction, and relational reasoning.
[0036] The system receives natural language text output by the model and extracts structured fields from it using a pre-defined parsing template or a rule-based information extraction algorithm. These fields include scene categories and descriptions, unique identifiers for each target, bounding box coordinates, attribute key-value pairs, and relation triples. Finally, these fields are organized according to a predefined JSON Schema to generate a standardized single-frame semantic information data structure.
[0037] S4. Define a time window, aggregate the structured semantic information of a single frame, construct a four-level time-series data structure, and establish a spatiotemporal index.
[0038] Define a dynamically adjustable time window The structured semantic information of a single frame falling within the window range is aggregated to form a continuous data segment with time as the axis.
[0039] The four-level time-series data structure includes: The scene layer is used to describe the global environmental context and weather and lighting conditions. The object layer is used to record the unique identifier of the target object and its movement trajectory over time. The attribute layer is used to record the fine-grained characteristics and temporal evolution of objects. The relationship layer is used to record the interaction relationships and event sequences between objects.
[0040] The four-level time-series data structure includes: The scene layer is used to describe the global environment context and dynamic conditions, including information such as scene category, environmental features, and traffic status. It generates scene descriptions of all single frames within the window and takes the dominant semantics to reflect the overall environmental features during the window.
[0041] The object layer is used to record the unique identifiers of all target objects within the window and their motion trajectories over time. The semantic association reasoning module analyzes the visual features and spatial positions of objects in adjacent frames, establishes cross-frame object correspondence, assigns a globally unique ID to each object, and concatenates its position coordinates, speed, direction and other state information in the order of timestamps to form a continuous temporal trajectory.
[0042] The attribute layer is used to record the fine-grained features and temporal evolution of objects. For each target in the object layer, its static and dynamic attributes are extracted and stored. By tracking the changes in attribute values within the window, a temporal sequence of attribute changes over time is constructed, supporting fine-grained tracking of object states.
[0043] The relationship layer is used to record the interaction relationships and event sequences between objects. By analyzing the trajectory of the object layer and the features of the attribute layer, spatial relationships, logical relationships and events are identified. Relationships and events are sorted according to the occurrence time to form an interaction sequence within the window, supporting the semantic understanding of group behavior.
[0044] The steps in step S4 for establishing the spatiotemporal index include: Establish a time series index for a four-level time series data structure to support conditional queries based on time ranges; A spatial index is created for the four-level time-series data structure to support queries based on geographic range; Establish a global object identifier index for the target object in the object layer, supporting object tracking across time windows.
[0045] A time series index is built for the four-level time series data structure to support time range-based conditional queries. A B+ tree structure is used to build an index on the timestamp field of the data at each level.
[0046] A spatial index is built for the four-level time-series data structure to support geographic range-based queries. GeoHash encoding is used to divide the geographic coordinate field in each layer of data into spatial grids and build a hash index to achieve a fast response for geofence queries such as "querying all objects within 50 meters of a certain coordinate point".
[0047] A global object identifier index is established for the target objects in the object layer to support object tracking across time windows. A hash table structure is used to map the global ID of an object to its time window, storage address and trajectory segment, ensuring that objects appearing in a new time window can be quickly associated with historical trajectories through their IDs, thus achieving full lifecycle tracking.
[0048] S5. In scenarios with continuous video stream input, maintain a time-series data queue based on a sliding time window, perform incremental updates by calculating the differences between the old and new windows, and maintain a global object ID mapping. The system first maintains a sliding time window queue of fixed length. Each window The output corresponds to four levels of time-series structured data within a time window. As new video frames arrive consecutively, the system continuously aggregates them to form a new window according to a preset time window size. And add it to the tail of the queue; if the queue length exceeds the threshold Then, the oldest window that entered at the head of the queue is removed synchronously. This ensures that the system processes only historical data of a limited length, enabling real-time response to an unlimited video stream and controllable memory usage.
[0049] The incremental update steps in S5, which involve calculating the differences between the old and new windows, include: Maintain a fixed-length sliding time window queue When a new window Upon arrival, remove the oldest window that entered the queue. ; Calculate the difference features between the new window and its neighboring windows. ,pass Update the time-series data and maintain the global object ID mapping table through time-series consistency checks.
[0050] The incremental feature calculation module does not recalculate all existing data, but only extracts the new feature data. Difference information compared to adjacent windows , This includes newly appearing objects, disappearing objects, extended segments of the trajectory of existing objects, attribute changes, and newly added relational events. Then, the system will... and differences Perform a union operation to generate an updated dataset. This significantly reduces computational and storage overhead. During this process, the historical data fusion module simultaneously performs time-series consistency checks to ensure that incremental updates do not disrupt the logical consistency of existing data. It also maintains a global object ID mapping table, assigning a unique and unchanging identifier to each physical entity appearing within the sliding window, ensuring that objects can be accurately tracked even when they span multiple windows.
[0051] The data persistence module writes updated data to the time-series database, employing a tiered storage strategy. Based on the data's timeliness and access frequency, the system stores data from different time ranges in storage media with different performance levels: recently generated, frequently accessed data is stored in in-memory databases such as Redis to support millisecond-level queries; older, lukewarm data is migrated to relational or columnar time-series databases such as PostgreSQL or ClickHouse; and cold data exceeding the business retention period is archived to low-cost object storage systems such as MinIO or HDFS. Through this tiered storage strategy, the system can ensure real-time query performance while also considering the economics of long-term data storage.
[0052] S6. Perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated time series data, and output the final time series structured data.
[0053] First, the monotonicity and continuity of all timestamps in the dataset are verified, and duplicate timestamps or abrupt changes are marked. Then, a logical consistency check is performed based on domain knowledge rules—verifying whether changes in object spatial coordinates conform to velocity constraints, whether attribute changes follow common sense, and whether relationships between objects satisfy transitivity and symmetry. If violations are found, an anomaly signal is output through the consistency constraint function. For semantic gaps caused by video blurring or occlusion, weighted interpolation based on temporal distance or contextual reasoning using a multimodal large language model is used for completion. For data gaps caused by dropped video frames, reasonable completion data is generated through flight parameter motion trajectory estimation or visual spatiotemporal prediction. Finally, high-quality time-series structured data with continuous timestamps, logical consistency, and complete semantics is output.
[0054] The steps in S6 include: Check the monotonicity and continuity of all timestamps in the dataset, and mark any detected timestamp anomalies; Based on pre-defined domain knowledge rules, it verifies whether the changes in the target's spatial coordinates conform to the motion speed constraints, whether the changes in the target's attributes conform to common sense rules, and whether the relationships between objects satisfy transitivity and symmetry. To address the loss of semantic information due to image quality degradation or model recognition failure, weighted interpolation based on temporal distance or invoking a multimodal large language model to perform inference and completion based on the context of adjacent frames are adopted. To address data gaps caused by dropped video frames, motion trajectories are calculated based on flight parameters or spatiotemporal predictions are made based on visual content to generate supplementary data.
[0055] First, the monotonicity and continuity of all timestamps in the dataset are checked, and any detected timestamp anomalies are marked. The timestamp verification module iterates through all timestamps in the time-series dataset. To ensure that the constraints are met: if the system uses a constant sampling interval, then it is required that... ,in For constant sampling intervals, if non-uniform sampling is used, then it is required that... Strictly monotonically increasing. For detected duplicate timestamps, timestamp rollbacks, or abnormal jumps, the system marks these points as abnormal and triggers subsequent repair processes.
[0056] Secondly, based on pre-defined domain knowledge rules, it verifies whether changes in the target's spatial coordinates conform to motion speed constraints, whether changes in target attributes conform to common sense rules, and whether relationships between objects satisfy transitivity and symmetry. The logical consistency check module defines consistency constraint functions. When constraints are violated, an abnormal signal is output. Specifically, it includes three aspects: (1) Spatial consistency: Check whether the spatial coordinate changes of the object conform to the laws of physical motion, that is ,in (1) Maximum reasonable speed preset according to object category; (2) Attribute consistency: verify whether the attribute changes of the object conform to common sense, such as the static attributes of the vehicle such as color and model will not change suddenly in a very short time; (3) Relationship consistency: ensure that the spatial or logical relationship between objects remains transitive and symmetrical.
[0057] Furthermore, to address the semantic information loss caused by image quality degradation or model recognition failure, weighted interpolation based on temporal distance or inference completion based on the context of adjacent frames using a multimodal large language model is employed. The missing value inference module first identifies missing or low-confidence records in semantic fields (such as target category, attribute, and location) in the data. For continuous numerical missing values (such as coordinates and velocity), linear interpolation or Gaussian process regression is used for filling. The weight The calculation is based on the time distance between the target time and known data points, with closer distances carrying greater weight. For missing discrete semantic information, the system utilizes the contextual reasoning capabilities of a multimodal large language model to organize the structured semantic information of adjacent frames before and after the missing time into prompt words, guiding the model to infer the most likely completion value based on logical coherence.
[0058] Finally, to address data gaps caused by dropped video frames, motion trajectories are calculated based on flight parameters or spatiotemporal predictions are performed based on visual content to generate complete data. The temporal completion module specifically handles the complete loss of time-series data caused by video transmission packet loss, encoding errors, or brief occlusions. For such data gaps, the system uses aligned flight parameters to calculate motion trajectories, such as extrapolating the target's geographical location using a uniform velocity or uniform acceleration model. Simultaneously, combining existing visual semantic information before and after the gap, a reasonable completion sequence is generated using a physical model-based approach or a lightweight spatiotemporal prediction network. The completed data is then marked with special tags and confidence scores to ensure that downstream applications can distinguish between original observations and inferred values. After the above integrity restoration, the system outputs high-quality, consistency-verified final temporal structured data.
[0059] The method further includes: The final time-series structured data output is serialized and stored using JSON Schema, and the data includes timestamp and geographic coordinate fields.
[0060] The final time-series structured data output from step S6 is serialized and stored using JSON Schema. This data format includes timestamp and geographic coordinate fields in each level and each record node, thereby supporting efficient queries based on time range and efficient queries based on spatial range, facilitating seamless integration with existing big data ecosystem components such as MongoDB, Elasticsearch, and Spark.
[0061] In this embodiment, the drone cruises and takes pictures at a height of 100 meters above the main urban road along a preset route. The video encoding format is H.265, the resolution is 4K (3840×2160), the frame rate is 30fps, and the flight parameter log is recorded at the same time.
[0062] Step S1: Start the FFmpeg decoding engine to decode the video stream and extract metadata such as encoding parameters, frame rate, resolution, and color space. Frames are split according to a sampling strategy of extracting 1 frame every 5 frames, resulting in an image sequence of 6 frames per second. A unique frame number (e.g., frame_001200) is assigned to each image, and its time offset relative to the start time of the video (e.g., 00:03:20.000) is recorded. Simultaneously, the flight parameter log file is parsed to extract parameters such as the UTC timestamp, GPS coordinates (longitude 116.3974°, latitude 39.9093°, altitude 100.5m), attitude angles (pitch -5°, roll 2°, yaw 45°), and ground speed (15m / s) for each recorded point. The camera intrinsic matrix (focal length 24mm, principal point coordinates (1920, 1080)) and distortion coefficients (k1=0.12, k2=-0.05, p1=0.001, p2=-0.002) are extracted from the image XMP metadata using the DJI SDK. After processing, the system outputs a standardized image frame sequence and synchronized flight parameter data units.
[0063] Timestamp alignment is performed. Taking frame_001200 as an example, its timestamp t=200.000s, and the timestamps of adjacent flight parameter recording points are τ_j=199.800s and τ_{j+1}=200.200s, respectively, corresponding to GPS coordinates (116.3974,39.9093,100.2) and (116.3975,39.9094,100.8), respectively. The GPS coordinates of the flight parameters corresponding to this frame are calculated as (116.39745,39.90935,100.5) through linear interpolation. Similarly, interpolation is performed on the attitude angle and velocity parameters. Then, spatial coordinate mapping is performed: the rotation matrix is calculated based on the UAV pitch angle of -5°, roll angle of 2°, and yaw angle of 45°, and the mapping from pixel coordinates to geographic coordinates is established by combining the camera intrinsic parameter matrix. For the image center pixel (1920, 1080), its corresponding ground geographic coordinates are calculated as (116.3975, 39.9093, 0) through inverse projection. Finally, each frame of image and its associated flight parameter data and spatial mapping parameters are encapsulated into raw structured data units.
[0064] The image frame_001200.jpg is fed into the open-source multimodal large language model Qwen-VL-Chat. Structured prompts are designed: "Please analyze this urban traffic monitoring image, identify all vehicles, pedestrians, and traffic signs, describe the scene type, and analyze the positional relationships between vehicles." After the model's visual encoder extracts image features, the language decoder outputs multi-dimensional semantic information: In terms of object detection, it identifies 3 cars (bounding boxes and confidence levels of [1200, 800, 1400, 950] with confidence of 0.95, [1500, 820, 1700, 960] with confidence of 0.93, and [800, 780, 1000, 920] with confidence of 0.92), 2 pedestrians (confidence levels 0.88–0.90), and 1 traffic light. (Confidence level 0.96); The scene understanding output scene category is "urban main road intersection", described as "two-way four-lane urban road, with traffic lights at the intersection, and vehicles are waiting to pass"; attribute extraction indicates that the first white sedan (vehicle A) is stationary, the second black SUV (vehicle B) is stationary, and the pedestrian is located in the zebra crossing area; relational reasoning concludes that vehicle A is about 5 meters in front of vehicle B, all vehicles are waiting behind the traffic light stop line, and the pedestrian is located on the sidewalk to the right of the vehicles. The natural language output is transformed into structured single-frame semantic information through template parsing.
[0065] A time window size of 5 seconds (corresponding to 30 frames) is set, and the semantic information of each frame within this window is aggregated. Taking a white sedan (ID: obj_001) as an example, its temporal trajectory within the window is tracked. It is found that the vehicle remains stationary for 5 seconds, thus the event is determined to be "waiting at a traffic light". A four-level data structure is constructed: the scene layer records the scene category (city intersection), weather (sunny day), and lighting (daytime) of the window; the object layer records the unique ID, category, and time-stamped trajectory sequence of all identified objects; the attribute layer records the color, model, status, and other attributes of each object and their changes over time (e.g., vehicle A is always white, and its status changes from stationary to moving); the relationship layer records the spatial relationships between objects (e.g., "obj_001 is 5 meters ahead of obj_002") and event sequences (e.g., "obj_001 is stopped at a traffic light"). A spatiotemporal index is also established: the time index uses a B+ tree structure to support range queries, the spatial index uses GeoHash encoding to support geographical range queries, and a globally unique identifier is assigned to each object to support cross-window tracking. The final output is a time-series structured data structure.
[0066] As the drone continues its cruise, new video frames arrive continuously. A sliding time window queue with a fixed length of 10 windows (corresponding to 50 seconds of historical data) is maintained. When the 11th window arrives, the oldest window is automatically removed. For the red car (ID obj_004) appearing in the new window, the system assigns a new global ID and creates a trajectory record. For the existing object obj_001, its state changes (e.g., from "stopped" to "moving") continue to be tracked in the new window, and its trajectory record is updated using an incremental trajectory fusion algorithm, adding the state at the end of the original trajectory (speed 10m / s). The updated data is written to a time-series database, using a hierarchical storage strategy: hot data is kept in Redis memory, warm data is written to PostgreSQL, and cold data is archived to MinIO object storage.
[0067] Consistency checks are performed on the incrementally updated time-series data. Timestamp verification confirms that all timestamps are strictly monotonically increasing, with no jumps or repetitions. Logical consistency checks reveal that obj_001's state is "stopped" at t=205.0s and "moving" at t=205.167s, with its speed changing from 0 to 10 m / s. This change conforms to the "green light" event logic, thus passing the consistency check. Assuming that at t=210.0s, image blurring causes a loss of color attribute recognition for obj_001, the system uses color information from preceding and following frames (both "white") to fill in the missing information through linear interpolation. Assuming that at t=215.0s, occlusion causes a loss of position information for obj_001, the system uses flight parameter data to calculate the UAV's position change and, combined with obj_001's historical trajectory (moving north at a constant speed of 10 m / s), infers its current position as a 10-meter extension of its historical position along the heading direction multiplied by the time difference. After the above verification and reasoning completion, the system outputs the final time-series structured data that has passed the consistency check. This data is stored serialized in JSON Schema, and each data node contains a timestamp field and a spatial coordinate field, which can support efficient time-series and spatial queries such as "querying all vehicles that have passed through a certain area in the past 5 minutes".
[0068] Compared with the prior art, the embodiments of this application have the following beneficial effects: First, by using linear interpolation time resampling and spatial coordinate mapping based on collinear equations, the problem of accurate matching of heterogeneous data sources in time and space dimensions is solved, providing spatiotemporally consistent input for subsequent semantic understanding.
[0069] Secondly, by leveraging the zero-shot and few-shot learning capabilities of multimodal large language models, object detection, scene understanding, and relational reasoning can be completed without predefined categories and a large amount of labeled data, significantly improving the generalization ability in complex scenarios.
[0070] Third, a four-layer data structure of scene layer, object layer, attribute layer and relation layer is proposed, which can fully describe the evolution of target state, attribute changes and interaction events, and support complex semantic queries oriented towards time sequence and space.
[0071] Fourth, by adopting a sliding time window and incremental update mechanism for differential features, only the changes between the old and new windows are calculated, which greatly reduces the computation and storage overhead and adapts to the continuous processing needs of infinitely long video streams.
[0072] Fifth, by using timestamp monotonicity verification, logical consistency checks (spatial, attribute, relational), and multi-strategy missing value inference and completion, data defects caused by video quality or model recognition failures are effectively repaired, ensuring the continuity and logical rationality of the output data.
[0073] Sixth, this solution is not only applicable to drone video streams, but can also be migrated to video sources with spatiotemporal metadata such as fixed monitoring, vehicle recorders, and robot vision. At the same time, its data model and incremental verification mechanism can be extended to other fields such as industrial equipment monitoring and medical image time series analysis.
[0074] The following presents a possible implementation of a temporal structured data generation system based on a multimodal large language model. This system executes the various steps and corresponding technical effects of the temporal structured data generation method shown in the above embodiments and possible implementations. The system includes: The video preprocessing module is used to receive UAV video streams with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract image frame sequences, flight parameter data streams and corresponding metadata. The original structured data construction module is used to perform linear interpolation alignment of image frame sequences and flight parameter data streams based on timestamps, and to establish a spatial mapping from image pixel coordinates to geographic coordinates based on UAV attitude angles, camera intrinsic parameters and collinearity equations, and encapsulate them into original structured data units. The visual recognition module of the multimodal large language model is used to input image frames into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompts, and output single-frame structured semantic information; The temporal structure extraction module is used to define time windows, aggregate structured semantic information of single frames, construct a four-level temporal data structure, and establish a spatiotemporal index. The streaming incremental update module is used to maintain a time-series data queue based on a sliding time window in scenarios with continuous video stream input. It performs incremental updates by calculating the differences between the old and new windows and maintains a global object ID mapping. The timing consistency verification and inference module is used to perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated timing data, and output the final timing structured data.
[0075] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for generating temporal structured data based on a multimodal large language model, characterized in that, include: S1. Receive UAV video stream with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract the image frame sequence, flight parameter data stream and corresponding metadata. S2. Based on the timestamp, perform linear interpolation to align the image frame sequence and flight parameter data stream, and establish a spatial mapping from image pixel coordinates to geographic coordinates based on the UAV attitude angle, camera intrinsic parameters and collinearity equations, and encapsulate them to form original structured data units. S3. Input the image frame into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompt words, and output single-frame structured semantic information; S4. Define a time window, aggregate the structured semantic information of a single frame, construct a four-level time-series data structure, and establish a spatiotemporal index; S5. In scenarios with continuous video stream input, maintain a time-series data queue based on a sliding time window, perform incremental updates by calculating the differences between the old and new windows, and maintain a global object ID mapping. S6. Perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated time series data, and output the final time series structured data.
2. The method for generating time-series structured data as described in claim 1, characterized in that, The steps for linearly interpolating and aligning the image frame sequence with the flight parameter data stream based on timestamps include: Let the set of video frame timestamps be... The original timestamp set of flight parameter data is For any video frame timestamp The corresponding flight parameter values are obtained by performing linear interpolation calculations on adjacent flight parameter data points. : ,in and for The timestamps of two adjacent flight parameters.
3. The method for generating time-series structured data as described in claim 1, characterized in that, The steps for establishing a spatial mapping from image pixel coordinates to geographic coordinates based on UAV attitude angles, camera intrinsic parameters, and collinearity equations include: pass Establish a mapping relationship, where For the camera intrinsic parameter matrix, The rotation and translation matrix is composed of the pitch angle, roll angle, and yaw angle of the UAV. Image pixel coordinates, For world geographical coordinates, is the scale factor.
4. The method for generating time-series structured data as described in claim 1, characterized in that, Step S3 includes: Image frames are input into the visual encoder of a multimodal large language model to generate high-dimensional visual feature vectors; Guide a multimodal large language model to perform at least one of the following tasks: object detection and recognition, scene understanding and classification, attribute extraction, and relational reasoning, using structured prompts; The natural language results output by the multimodal large language model are parsed and transformed into structured single-frame semantic information.
5. The method for generating time-series structured data as described in claim 1, characterized in that, The four-level time-series data structure includes: The scene layer is used to describe the global environmental context and weather and lighting conditions. The object layer is used to record the unique identifier of the target object and its movement trajectory over time. The attribute layer is used to record the fine-grained characteristics and temporal evolution of objects. The relationship layer is used to record the interaction relationships and event sequences between objects.
6. The method for generating time-series structured data as described in claim 1, characterized in that, The steps in step S4 for establishing the spatiotemporal index include: Establish a time series index for a four-level time series data structure to support conditional queries based on time ranges; A spatial index is created for the four-level time-series data structure to support queries based on geographic range; Establish a global object identifier index for the target object in the object layer, supporting object tracking across time windows.
7. The method for generating time-series structured data as described in claim 1, characterized in that, The incremental update steps in S5, which involve calculating the differences between the old and new windows, include: Maintain a fixed-length sliding time window queue When a new window Upon arrival, remove the oldest window that entered the queue. ; Calculate the difference features between the new window and its neighboring windows. ,pass Update the time-series data and maintain the global object ID mapping table through time-series consistency checks.
8. The method for generating time-series structured data as described in claim 1, characterized in that, The steps in S6 include: Check the monotonicity and continuity of all timestamps in the dataset, and mark any detected timestamp anomalies; Based on pre-defined domain knowledge rules, it verifies whether the changes in the target's spatial coordinates conform to the motion speed constraints, whether the changes in the target's attributes conform to common sense rules, and whether the relationships between objects satisfy transitivity and symmetry. To address the loss of semantic information due to image quality degradation or model recognition failure, weighted interpolation based on temporal distance or invoking a multimodal large language model to perform inference and completion based on the context of adjacent frames are adopted. To address data gaps caused by dropped video frames, motion trajectories are calculated based on flight parameters or spatiotemporal predictions are made based on visual content to generate supplementary data.
9. The method for generating time-series structured data as described in claim 1, characterized in that, The method further includes: The final time-series structured data output is serialized and stored using JSON Schema, and the data includes timestamp and geographic coordinate fields.
10. A temporal structured data generation system based on a multimodal large language model, characterized in that, include: The video preprocessing module is used to receive UAV video streams with flight parameters, decode the video stream, split it into frames and parse the flight parameters, and extract image frame sequences, flight parameter data streams and corresponding metadata. The original structured data construction module is used to perform linear interpolation alignment of image frame sequences and flight parameter data streams based on timestamps, and to establish a spatial mapping from image pixel coordinates to geographic coordinates based on UAV attitude angles, camera intrinsic parameters and collinearity equations, and encapsulate them into original structured data units. The visual recognition module of the multimodal large language model is used to input image frames into the multimodal large language model, guide the model to perform object detection, scene understanding and relation reasoning through structured prompts, and output single-frame structured semantic information; The temporal structure extraction module is used to define time windows, aggregate structured semantic information of single frames, construct a four-level temporal data structure, and establish a spatiotemporal index. The streaming incremental update module is used to maintain a time-series data queue based on a sliding time window in scenarios with continuous video stream input. It performs incremental updates by calculating the differences between the old and new windows and maintains a global object ID mapping. The timing consistency verification and inference module is used to perform timestamp monotonicity verification, logical consistency check, and missing value inference and completion on the updated timing data, and output the final timing structured data.