Query analysis method, device and equipment fusing multi-modal spatio-temporal data

Abstract

The invention discloses a query analysis method, device and equipment fusing multi-modal spatio-temporal data. The method comprises the steps of receiving a query request; based on a large language model, space-time elements in the query request are extracted, and the space-time elements are converted into corresponding query task sequences; executing the query task sequence, and obtaining a multi-modal query result based on the dynamic spatio-temporal knowledge graph and the original database; and returning a multi-modal query result to the large language model, and generating a multi-modal response report. By means of the mode, non-technical users can conduct deep exploration on complex multi-source heterogeneous data through daily languages by introducing the natural language understanding and analysis capacity of the large language model, and the user query threshold is greatly lowered; by constructing the dynamic space-time knowledge graph, barriers among different modal data are broken, so that linkage fusion analysis can be carried out among the different modal data, and a basis is provided for space-time concept analysis of a large language model.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and big data processing technology, and in particular to a query and parsing method, apparatus and equipment for integrating multimodal spatiotemporal data. Background Technology

[0002] In fields such as smart cities, intelligent transportation, and public safety, massive amounts of multimodal spatiotemporal data are generated during operation. This multimodal spatiotemporal data has the following characteristics: Multi-modal: The data sources are diverse, including video streams from urban surveillance cameras, sensor data streams from Internet of Things (IoT) devices (such as environmental sensors and smart meters), vehicle GPS location data, and social media check-in data.

[0003] Spatio-temporal attributes: All data incorporates two key dimensions: time and space. For example, a video might be taken at a specific time at an intersection, or an air quality reading might come from a specific monitoring station at a specific moment.

[0004] The existing technology still has the following drawbacks when processing the above-mentioned multimodal spatiotemporal data: 1) The "Spatiotemporal Blind Spot" of Large Language Models (LLMs): Traditional LLMs (such as the GPT series) primarily derive their knowledge from text corpora, lacking an intrinsic understanding of continuous and dynamic spatiotemporal concepts in the physical world. This prevents them from directly processing and reasoning about queries that require spatiotemporal concepts, such as "the density of people in a certain area between 3 pm and 4 pm"; 2) Limitations of traditional spatiotemporal databases: While Geographic Information Systems (GIS) and spatiotemporal databases can efficiently query structured spatiotemporal data (points, lines, and polygons), they struggle to handle unstructured data such as video content and sensor text descriptions. Furthermore, their query methods rely on Structured Query Language (SQL), posing a significant barrier to entry for non-professional users. 3) Data silo problem: Data from different modalities (video, IoT, and GPS) are often stored in independent systems, making cross-modal fusion analysis difficult. For example, to answer the question "find all illegally parked vehicles that caused traffic congestion yesterday afternoon and retrieve relevant video evidence", it is necessary to manually link multiple data systems, which is inefficient and difficult. Summary of the Invention

[0005] This application provides a query parsing method, apparatus, and device that integrates multimodal spatiotemporal data to solve the technical problems in the prior art where large language models cannot understand spatiotemporal concepts, making it difficult to perform cross-modal fusion analysis and resulting in high query thresholds.

[0006] To address the aforementioned technical problems, this application adopts the following technical solution: providing a query and parsing method for fusing multimodal spatiotemporal data, comprising: Receive query requests; Based on the large language model, the spatiotemporal elements in the query request are extracted and converted into a corresponding query task sequence; The query task sequence is executed to obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database; The multimodal query results are returned to the large language model to generate a multimodal response report.

[0007] In one optional embodiment of this application, the dynamic spatiotemporal knowledge graph is constructed in the following manner: Real-time acquisition of multimodal spatiotemporal data; The multimodal spatiotemporal data is characterized as spatiotemporal events containing spatiotemporal labels; Based on the aforementioned spatiotemporal events and pre-acquired static geographic information, the dynamic spatiotemporal knowledge graph is constructed.

[0008] In one optional embodiment of this application, the multimodal spatiotemporal data includes at least video data, sensor data, and trajectory data; The step of characterizing the multimodal spatiotemporal data as spatiotemporal events with spatiotemporal labels includes: The video data is subjected to target detection based on the target detection model, and behavior recognition is performed based on the dual-path spatiotemporal network to obtain the event label corresponding to the video data. The sensor data is received based on the MQTT protocol, and the sensor data is parsed to obtain the event tag corresponding to the sensor data; The trajectory data is spatiotemporally segmented based on a clustering algorithm to obtain the event label corresponding to the trajectory data; Each event tag is assigned a corresponding spatiotemporal tag to obtain each spatiotemporal event.

[0009] In an optional embodiment of this application, the step of extracting spatiotemporal elements from the query request based on a large language model and converting the spatiotemporal elements into a corresponding query task sequence includes: Based on a large language model, the query intent in the query request is identified; In response to the query intent being a fact query, the spatiotemporal elements in the query request are extracted, and the spatiotemporal elements are converted into a list of query tasks for the dynamic spatiotemporal knowledge graph; In response to the query intent being a multimodal query, the spatiotemporal elements in the query request are extracted, and the spatiotemporal elements are converted into a list of query tasks for the dynamic spatiotemporal knowledge graph and the original database.

[0010] In one optional embodiment of this application, the execution of the query task sequence to obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database includes: The query task sequence is executed, the structured query results are obtained based on the dynamic spatiotemporal knowledge graph, and the original data is obtained from the original database based on the structured query results, and then combined to obtain the multimodal query results.

[0011] In one optional embodiment of this application, returning the multimodal query result to the large language model and generating a multimodal response report includes: The multimodal query results are returned to the large language model, and the original data and the structured query results are fused using natural language generation technology to generate a multimodal response report.

[0012] In an optional embodiment of this application, after receiving the query request, the method further includes: Based on the query domain of the query request, the large language model pre-trained in the corresponding domain is dynamically selected.

[0013] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a query and parsing device for fusing multimodal spatiotemporal data, comprising: The request receiving module is used to receive query requests; The task decomposition module is used to extract spatiotemporal elements from the query request based on the large language model, and convert the spatiotemporal elements into a corresponding query task sequence; The execution fusion module is used to execute the query task sequence and obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database; The response generation module is used to return the multimodal query results to the large language model and generate a multimodal response report.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, including a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the above-mentioned query and parsing method for fusing multimodal spatiotemporal data.

[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a storage medium on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the above-mentioned query and parsing method for fusing multimodal spatiotemporal data.

[0016] The beneficial effects of this application are as follows: Unlike existing technologies, this application discloses a query parsing method, apparatus, and device that integrates multimodal spatiotemporal data. This method uses a large language model to understand and analyze query requests, extract spatiotemporal elements, and convert these elements into a query task sequence. The query task sequence is then executed on a constructed dynamic spatiotemporal knowledge graph and the original database to obtain multimodal query results. Finally, a multimodal response report is generated using the large language model. By introducing the natural language understanding and analysis capabilities of the large language model, non-technical users can also deeply explore complex, multi-source, heterogeneous data using everyday language, greatly reducing the user's query threshold. Furthermore, by constructing a dynamic spatiotemporal knowledge graph, the barriers between different modalities of data are broken down, enabling interconnected and fused analysis of different modalities, and providing a foundation for the spatiotemporal concept analysis of the large language model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an embodiment of the query and parsing method for fusing multimodal spatiotemporal data provided in this application; Figure 2 This is a schematic diagram of the interactive response process of an embodiment of the query and parsing method for fusing multimodal spatiotemporal data provided in this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the query and parsing device for fusing multimodal spatiotemporal data provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the storage medium provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] This application provides a query and parsing method for fusing multimodal spatiotemporal data, see reference. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the query and parsing method for fusing multimodal spatiotemporal data provided in this application. The query and parsing method for fusing multimodal spatiotemporal data includes: S10: Receive query request.

[0022] Multimodal spatio-temporal data is multi-source heterogeneous data with multimodal and spatio-temporal attributes. Data sources may include video streams generated by urban surveillance cameras, sensor data streams from Internet of Things (IoT) devices (such as environmental sensors and smart meters), vehicle GPS positioning data, and social media check-in data, etc. All data embeds the two key dimensions of time and space.

[0023] In this application, an interactive interface can be designed to provide users with a query entry point. Users can send query requests by entering non-technical natural language dialogue. The content of the query can be any multi-source heterogeneous data with temporal and spatial attributes, that is, multimodal spatiotemporal data in any scenario.

[0024] This application, after receiving the query request, also includes: Based on the query domain of the query request, a large language model pre-trained in the corresponding domain is dynamically selected.

[0025] In this application, upon receiving a user's query request, the system can perform vertical domain-specific identification of the query request's domain and dynamically select the corresponding large language model to understand and analyze the query request based on the domain identification results. For example, if the query request belongs to the transportation domain, a large language model pre-trained based on transportation data is selected; if the query request belongs to the public safety domain, a large language model pre-trained based on safety incident data is selected; and if the query request belongs to the environmental monitoring domain, a large language model pre-trained based on environmental data is selected.

[0026] Large language models corresponding to each domain can be pre-trained using a base model combined with domain fine-tuning within that domain. Specifically, for the selection of the base model, a powerful general-purpose large language model, such as Qwen or GLM, can be chosen; no specific limitation is made here. For domain fine-tuning, instruction datasets corresponding to each domain can be constructed by collecting relevant data from each domain. These instruction datasets contain a large number of paired samples of "natural language query - decomposed subtask sequence," and the base model can be fine-tuned using each instruction dataset. For training strategies, parameter efficient fine-tuning (PEFT) methods, such as LoRA, can be used to reduce training costs while maintaining the model's generality, achieving efficient training.

[0027] For example, if a user enters a query request through the interactive interface, such as "find which intersection's traffic congestion was caused by a traffic accident during last Friday's evening rush hour, and provide a video of the accident scene," after receiving the query request, the system performs vertical domain specificity identification on the domain to which the query request belongs. If the domain identification result is "traffic domain," then the query request is input into the large language model corresponding to the traffic domain.

[0028] Furthermore, large language models applicable to query requests of varying complexity can be trained. The appropriate large language model can be dynamically selected based on the complexity of the query request. Additionally, other classifications of query requests can be performed to train corresponding large language models. Moreover, the large language model in this application is a pluggable, dynamic structure, allowing access to large language models from different vendors or of different scales.

[0029] S20: Based on the large language model, extract the spatiotemporal elements in the query request and convert the spatiotemporal elements into the corresponding query task sequence.

[0030] In this application, based on a large language model, spatiotemporal elements are extracted from the query request, and these spatiotemporal elements are converted into a corresponding query task sequence, including: Based on a large language model, identify the query intent in the query request; In response to the query intent being a fact query, the spatiotemporal elements in the query request are extracted and converted into a list of query tasks for the dynamic spatiotemporal knowledge graph. In response to the query intent being a multimodal query, the spatiotemporal elements in the query request are extracted and converted into a list of query tasks targeting the dynamic spatiotemporal knowledge graph and the original database.

[0031] In this application, after inputting the query request into a large language model in the corresponding domain, the query intent in the natural language query request is identified through the natural language understanding (NLU) capability of the large language model. Query intent can be divided into factual queries and multimodal queries according to their complexity. For factual queries, such as "What is the design speed of Binhai Road?", only factual natural language data needs to be provided to the user as a response, which is a low-complexity query request. For multimodal queries, such as "Show me the accident video that caused the congestion on Binhai Road", the response also needs to provide the user with relevant original video data, that is, provide a multimodal content response, which is a high-complexity query request.

[0032] For query intent recognition, in addition to the two categories of query intents categorized by complexity—factual queries and multimodal queries—as mentioned above, other aspects of intent recognition can be extended. For example, identifying query intents such as causal analysis, trend prediction, or path planning can be done, and corresponding algorithms can be added and invoked for these intents. There are no specific limitations on the algorithms corresponding to the query intents here; they can be flexibly selected according to needs. For example, if the query intent is trend prediction, a time-series prediction model (such as LSTM, Transformer, or other deep learning models) can be invoked to make predictions after relevant data is retrieved. Similarly, if the query intent is path planning, a path planning model (such as A*, RRT, or other classic algorithms) can be invoked to plan paths after relevant data is retrieved. Furthermore, a single query request may contain multiple query intents, such as both prediction and planning intents. The appropriate model can be invoked based on the intent recognition results. Intent recognition can improve the accuracy of subsequent spatiotemporal element extraction and further enhance the accuracy of the response.

[0033] After identifying the intent of the query request, key spatiotemporal elements (such as time, space, object attributes, and behavioral events) are extracted from the query request based on a large language model. For example, if the query request is "find the red car that was speeding near Yanta Road yesterday afternoon", the extractable spatiotemporal elements include the time "yesterday afternoon", the space "Yanta Road", the behavioral event "speeding", and the object attribute "red car".

[0034] After extracting the spatiotemporal elements from the query request, the large language model uses techniques such as Chain-of-Thought (CoT) to transform these elements into a sequence of query tasks. Furthermore, the query task sequence targets different query objects depending on the query intent. Chain-of-Thought is a prompting technique that guides artificial intelligence models (especially large language models) to solve problems by demonstrating a progressive reasoning process.

[0035] When the query intent is a fact query, the query task list targets the dynamic spatiotemporal knowledge graph as the query object; when the query intent is a multimodal query, the query task list can target both the dynamic spatiotemporal knowledge graph and the original database as the query object, so as to obtain the original data pointers of the multimodal content.

[0036] For example, if the query request is "find the red car that was speeding near Yanta Road yesterday afternoon", then after extracting the spatiotemporal elements, the query task sequence can be converted into "1. Query the knowledge graph for 'red cars' that match the attributes; 2. Filter for cars that have 'speeding' behavior within the specified spatiotemporal range; 3. Retrieve the associated raw data".

[0037] Unlike existing technologies that rely on structured query languages ​​(such as SQL) for querying traditional spatiotemporal databases, posing a high barrier to entry for non-technical users, the query parsing method for integrating multimodal spatiotemporal data provided in this application, through intent recognition and spatiotemporal element extraction based on a large language model, transforms users' natural language query requests into a sequence of executable query tasks. This allows non-technical users to conduct in-depth exploration of complex data (such as urban data) using everyday language, significantly lowering the barrier to data analysis and enhancing the naturalness and intelligence of the interaction.

[0038] S30: Execute the query task sequence to obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database.

[0039] In this application, the dynamic spatiotemporal knowledge graph is constructed in the following ways: S31: Real-time acquisition of multimodal spatiotemporal data.

[0040] In this application, the multimodal spatiotemporal data includes at least video data, sensor data, and trajectory data.

[0041] This application acquires heterogeneous raw data in real time from various data sources, including video surveillance, IoT sensors, and GPS devices. This data contains temporal and spatial attributes, i.e., multimodal spatiotemporal data. The acquired multimodal spatiotemporal data includes at least video data, sensor data, and trajectory data. In addition to these urban data, the multimodal spatiotemporal data acquired in this application can also be multi-source heterogeneous data containing temporal and spatial attributes from other scenarios. The type of multimodal spatiotemporal data is not limited here.

[0042] S32: Characterize multimodal spatiotemporal data as spatiotemporal events with spatiotemporal labels.

[0043] In this application, multimodal spatiotemporal data is characterized as spatiotemporal events with spatiotemporal labels, including: S321: Target detection is performed on video data based on the target detection model, and behavior recognition is performed based on the dual-path spatiotemporal network to obtain the event labels corresponding to the video data.

[0044] For multimodal spatiotemporal data obtained from different data sources, the data is converted into event labels using their respective parsing and processing methods. For example, the air quality data obtained by the sensor at a certain location is converted into event labels {type: "air_quality", value: 85, location: [lat, lng]}, which includes three event labels: type, value, and location.

[0045] The process of converting multimodal spatiotemporal data into event tags is essentially the process of atomizing unstructured and semi-structured data. Unstructured data refers to data without a fixed format or predefined model, such as video streams (e.g., cam_003.mp4), images, and audio. Semi-structured data refers to data that is partially structured but contains free text or variable fields, such as sensor JSON data and GPS track points. Atomization refers to decomposing data into its smallest, indivisible, and business-meaning unit. Here, it involves decomposing unstructured and semi-structured data into structured event tags. Atomization is the cornerstone of building efficient, reliable, and flexible data systems.

[0046] For real-time video streams, the video content needs to be parsed before the unstructured video data can be converted into event tags. Parsing the video content can be done by using an object detection model to detect objects such as vehicles and pedestrians, and then combining this with a two-path spatiotemporal (SlowFast) network for behavior recognition, identifying behavioral events such as illegal parking and accidents, and obtaining corresponding structured event tags for these events. The object detection model can be a YOLO series model, such as YOLOv8; the behavior recognition model, besides the two-path spatiotemporal network, can also be replaced by behavior recognition models such as Temporal Segment Networks (TSN), without specific limitations here.

[0047] S322: Receives sensor data based on the MQTT protocol, parses the sensor data, and obtains the event tags corresponding to the sensor data.

[0048] For sensor data, the sensor data is received in real time based on the MQTT protocol. The sensor data is semi-structured data, such as {"device_id": "S001", "PM2.5": 75, "timestamp": "2024-05-15T18:00"}, which can be directly parsed into structured event tags.

[0049] S323: Based on the clustering algorithm, the trajectory data is spatiotemporally segmented to obtain the event labels corresponding to the trajectory data.

[0050] For GPS trajectory data, the trajectory data can be segmented into spatiotemporal segments using a clustering algorithm (DBSCAN). For each segment of the trajectory data, stop points or abnormal movement events can be identified, and the semi-structured trajectory data can be converted into structured event labels.

[0051] S324: Attach the corresponding spatiotemporal label to each event label to obtain each spatiotemporal event.

[0052] For each set of event labels obtained in steps S321 to S323 above, a unified spatiotemporal label is attached to each set of event labels, thereby representing multimodal spatiotemporal data as spatiotemporal events containing spatiotemporal labels. The spatiotemporal label is the standardized spatiotemporal metadata attached to all multimodal data. The spatiotemporal metadata is the metadata of the spatiotemporal attributes that the multimodal spatiotemporal data has at the time of acquisition, including location, start time and end time, i.e., location: [lat, lng], start_time: "timestamp1", end_time: "timestamp2".

[0053] For example, a spatiotemporal event of illegal parking can be characterized as: {event_type: "illegal_parking",object_id: "SGB1234A", location: [1.283, 103.851], start_time: "timestamp1",end_time: "timestamp2", source_data: "video_clip_url"}.

[0054] In spatiotemporal events, core fields are fixed, such as the `event_type` field, which is a spatiotemporal attribute metadata field. Other fields are flexibly set according to the spatiotemporal event attributes to ensure compatibility. The `event_type` field uses an enumeration type to predefine common event types (such as `traffic_congestion`, `illegal_parking`, etc.), and supports dynamically extending new types (such as `air_quality_alert`, etc.) through configuration files.

[0055] The above method of representing multimodal spatiotemporal data as spatiotemporal events with spatiotemporal labels allows for the development of event label conversion functionality corresponding to new data types when new data types are introduced. The core query part does not require radical modification, and scalability is improved through flexible decoupling.

[0056] It should be noted that the execution order of the above steps S321 to S323 is parallel, and the above steps S321 to S324 are executed in real time.

[0057] S33: Construct a dynamic spatiotemporal knowledge graph based on spatiotemporal events and pre-acquired static geographic information.

[0058] In this application, multimodal spatiotemporal data is characterized as spatiotemporal events containing spatiotemporal labels. Then, entities (Nodes) and relationships (Edges) in a knowledge graph are extracted from the spatiotemporal events and pre-acquired static geographic information to construct a dynamic spatiotemporal knowledge graph.

[0059] Static geographic information refers to basic, static geographic information obtained through standard interfaces (such as GIS APIs) or by importing common geographic data files (such as Shape files or Geo JSON), including road networks, administrative divisions, and points of interest (POIs). Static geographic information can serve as static geographic entities in dynamic spatiotemporal knowledge graphs.

[0060] In contrast to static geographic information, dynamic geographic information is dynamically extracted from real-time data streams. For example, in video data, the geographic coordinates of an event can be calculated by combining pre-calibrated camera position parameters with the target location detected by a computer vision model. Similarly, in sensor or trajectory data, the latitude and longitude coordinates reported by the corresponding devices can be directly parsed. Dynamic geographic information can also serve as dynamic geographic entities in a dynamic spatiotemporal knowledge graph.

[0061] Physical objects (such as vehicles and cameras), geographical regions, and abstract events (such as traffic congestion) in spatiotemporal events are treated as entities in a dynamic spatiotemporal knowledge graph. The center coordinates, length, level, and speed limit of each location in static geographic information are also treated as entities in the dynamic spatiotemporal knowledge graph. Relationships between entities describe the spatiotemporal relationships between them, such as "located_at," "happened_during," and "interacts_with" relationships in spatiotemporal events, and road connections in static geographic information. These entities and relationships are dynamically updated as new data (new spatiotemporal events and new static geographic information) flows in. The dynamic spatiotemporal knowledge graph provides a structured knowledge base of the physical world's state for large language models.

[0062] It should be noted that the dynamic geographic information contained in spatiotemporal events is a specific coordinate point or area (such as [114.057, 22.543] or a circular range), describing the "location" where the event occurs; while static geographic information has rich attributes and relationships. For example, a "coastal road" entity not only has its center coordinates, but also attributes such as its length, grade, connecting intersections, and speed limits. Static geographic information describes the "location" and its context. Since static geographic information changes very infrequently and belongs to basic background data, while spatiotemporal events are high-frequency dynamic data, processing the two separately can improve data management efficiency.

[0063] Unlike existing technologies where different modalities of data (video, IoT, GPS) are typically stored in independent systems, making cross-modal fusion analysis difficult and inefficient, this application provides a query and parsing method for fusing multimodal spatiotemporal data. By constructing a dynamic spatiotemporal knowledge graph, it breaks down the barriers between different modalities of data such as video, IoT, and GPS. This enables linked analysis, such as between "video content" and "sensor readings," revealing deep-seated correlations between multimodal data, improving the depth and breadth of data analysis, while also being highly efficient and easy to query.

[0064] In this application, a query task sequence is executed to obtain multimodal query results based on a dynamic spatiotemporal knowledge graph and the original database, including: The system executes a sequence of query tasks, obtains structured query results based on a dynamic spatiotemporal knowledge graph, retrieves raw data from the original database based on the structured query results, and combines these results to obtain multimodal query results.

[0065] In this application, for a query task sequence, depending on the query object it targets, the query is selected to be performed in the dynamic spatiotemporal knowledge graph or a combined query of the dynamic spatiotemporal knowledge graph and the original database.

[0066] The raw database, also known as the underlying database, is a logical concept referring to a collection of heterogeneous systems for storing and managing various types of raw data, not a single database. Raw databases include: 1. Graph databases (NetworkExploration and Optimization 4 Java, Neo4j), used to store and manage dynamic spatiotemporal knowledge graphs; 2. Time-series databases (InfluxDB), specifically designed for efficient storage and querying of massive amounts of time-series data generated by IoT sensors; 3. Object storage (MinIO), used to store unstructured raw files such as videos and images; 4. Relational databases (PostgreSQL), used to store structured data such as system configurations, user information, and metadata; the attributes of object class entities are stored in the relational database.

[0067] Structured query results refer to data obtained from dynamic spatiotemporal knowledge graphs and raw databases that has undergone system processing and organization. This data has a clear format and structure. Specifically, it includes: spatiotemporal event information, such as traffic congestion events and traffic accident events, containing structured fields such as time, location, and event type; entity relationship data, such as the relationship between vehicles and locations, and causal relationships between events; statistical data, such as quantitative indicators such as traffic flow and average speed within a specific time period; and analytical results, such as processed data including trend analysis and correlation analysis. For example, when querying "Which intersection's traffic congestion during last Friday's evening rush hour was caused by a traffic accident", the structured query results might include: a list of congestion events: {event_type: "traffic_congestion", location: "intersection of Binhai Road and Central Avenue", time: "2023-10-20 17:30-19:00", severity: "high"} and traffic accident information: {event_type: "vehicle_collision", location: "intersection of Binhai Road and Central Avenue", time: "2023-10-20 17:45", involved_vehicles: 2}.

[0068] Raw data pointers are references or links to raw, unprocessed multimodal data sources, not the raw data itself. These pointers are used to access and display raw multimedia content. Examples include video clip URLs like "https: / / storage.example.com / videos / camera_023 / 2023-10-20_17-45.mp4?start=1745&duration=300" or surveillance screenshots like " / data / captures / cam_023_20231020_1745.jpg".

[0069] Raw data refers to unstructured or semi-structured data stored in object storage (such as video files) or time-series databases (such as raw sensor readings).

[0070] When the query intent is a fact query, the corresponding query task sequence only needs to target the dynamic spatiotemporal knowledge graph as the query object. For example, if the query request is "What is the design speed of Binhai Road?", then you only need to query the attributes of the "Binhai Road" entity in the dynamic spatiotemporal knowledge graph according to the query task sequence to obtain the query results.

[0071] When the query intent is a multimodal query, the corresponding generated query task sequence targets both the dynamic spatiotemporal knowledge graph and the original database. For example, if the query request is "Show me the video of the accident that caused the traffic congestion on Binhai Road," then according to the query task sequence, the relevant spatiotemporal events need to be found in the dynamic spatiotemporal knowledge graph first to obtain the structured query results. Then, based on the original data pointers (such as video clip URLs) contained in the spatiotemporal events, the original data associated with the spatiotemporal events is retrieved from the original database. At this point, the structured query results and the original data together constitute the multimodal query results.

[0072] In some embodiments, while dynamic spatiotemporal knowledge graphs are an excellent bridge connecting language and the physical world, in certain simplified scenarios, tasks can also be accomplished by providing a specially designed set of API tools for large language models, allowing them to directly call the query interface of time-series databases.

[0073] S40: Return the multimodal query results to the large language model and generate a multimodal response report.

[0074] In this application, the multimodal query results are returned to the large language model to generate a multimodal response report, including: The multimodal query results are returned to the large language model, and the original data and structured query results are merged based on natural language generation technology to generate a multimodal response report.

[0075] In this application, after executing the query task sequence to obtain multimodal query results, the multimodal query results are returned to the large language model. After receiving these heterogeneous multimodal query results, the large language model uses its powerful natural language generation (NLG) technology to merge the raw data and structured query results in the multimodal query results. Specifically, the structured query results are translated into a natural language summary, and the raw data (such as multimodal content such as data visualization charts, interactive maps, or video clips) are embedded in the natural language summary, thereby generating a comprehensive multimodal response report.

[0076] In one embodiment, a user inputs a natural language query request through an interactive interface: "Query which intersection's traffic congestion during last Friday evening rush hour was caused by a traffic accident, and provide a video of the accident scene." First, the query request is identified as belonging to the traffic domain. The query request is then input into the corresponding large language model for the traffic domain. Upon receiving the query request, the large language model identifies the query intent as a multimodal query of causal relationship and extracts spatiotemporal elements such as time (last Friday evening rush hour), event 1 (traffic congestion), event 2 (traffic accident), and relationship (due to). Subsequently, the large language model converts these spatiotemporal elements into a corresponding query task sequence: a. Search for congestion events and their locations within a specified time period in a dynamic spatiotemporal knowledge graph; b. Search for traffic accident events near the same locations; c. Associate spatiotemporally overlapping "congestion-accident" pairs; d. Extract the URLs of video data associated with the accidents. Subsequently, a query task sequence was executed. A dynamic spatiotemporal knowledge graph query revealed a strong spatiotemporal correlation between the traffic congestion event at the intersection of Binhai Road and Central Avenue and a vehicle_collision (traffic accident) event. The corresponding video clip URL was found in the original database, yielding a multimodal query result. Finally, the multimodal query result was returned to the large language model. Based on the result, the model generated a natural language summary: "The traffic congestion at the intersection of Binhai Road and Central Avenue is likely caused by a traffic accident…", along with a playable on-site monitoring video clip and a traffic flow chart showing the congestion period. This comprehensive multimodal response report was then returned to the user.

[0077] In scenarios such as traffic management, public safety early warning, or emergency resource dispatch, the query and parsing method that integrates multimodal spatiotemporal data provided in this application can automatically complete the end-to-end automated processing flow from user natural language query input to multimodal comprehensive response report output without human intervention. It can shorten the manual analysis work that used to take several hours or even days to minutes, significantly improving the efficiency and intelligence level of urban management.

[0078] In one embodiment, see Figure 2 , Figure 2This is a schematic diagram of the interactive response process of an embodiment of the query parsing method for integrating multimodal spatiotemporal data provided in this application. The user inputs a query request in natural language on the interactive interface. After domain recognition, the query request is input into a large language model of the corresponding domain for understanding and decomposition. Specifically, the large language model performs intent recognition on the query request, then extracts the spatiotemporal elements from the query request, converting these elements into a query task list, thus completing the query request task decomposition. On the other hand, multimodal spatiotemporal data, including video data, IoT sensor data, GPS trajectory data, and static geographic data, is acquired in real time. This multimodal spatiotemporal data is converted into spatiotemporal events and stored in a dynamic spatiotemporal knowledge graph. After obtaining the query task list, the query task list is executed, and multimodal query results are obtained from the dynamic spatiotemporal knowledge graph and the original database. The multimodal query results are returned to the large language model, generating a multimodal response report, including text summaries, data charts, and interactive maps, and a comprehensive multimodal response report is output and returned to the user.

[0079] This application provides a query and parsing device that integrates multimodal spatiotemporal data, see reference. Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the query and parsing device for fusing multimodal spatiotemporal data provided in this application. The query and parsing device for fusing multimodal spatiotemporal data includes: Request receiving module 10 is used to receive query requests; The task decomposition module 20 is used to extract spatiotemporal elements from the query request based on the large language model and convert the spatiotemporal elements into the corresponding query task sequence. The execution fusion module 30 is used to execute the query task sequence and obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database; The response generation module 40 is used to return the multimodal query results to the large language model and generate a multimodal response report.

[0080] The above-mentioned request receiving module 10, task decomposition module 20, execution fusion module 30 and response generation module 40 interact to realize the query and parsing process of fused multimodal spatiotemporal data. You can refer to the specific description of steps S10 to S40 above. The repeated parts will not be repeated here.

[0081] See Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the storage medium provided in this application.

[0082] The storage medium 400 stores program data 410, which, when executed by the processor, implements, as follows: Figure 1 The steps of the query parsing method that integrates multimodal spatiotemporal data are described.

[0083] The program data 410 is stored in a storage medium 400 and includes several instructions for causing a network device (which may be a router, personal computer, server, or other network device) or processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0084] Optionally, the storage medium 400 can be any medium that can store program data, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), disk, or optical disc.

[0085] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the computer device provided in this application.

[0086] The device 500 includes a processor 520 and a memory 510 connected to each other. The memory 510 stores a computer program. When the processor 520 executes the computer program, it implements the steps of the query and parsing method for fusing multimodal spatiotemporal data as described above.

[0087] Unlike existing technologies, this application discloses a query parsing method, apparatus, and device that integrates multimodal spatiotemporal data. This method uses a large language model to understand and analyze query requests, extract spatiotemporal elements, and convert these elements into a query task sequence. The query task sequence is then executed on a constructed dynamic spatiotemporal knowledge graph and the original database to obtain multimodal query results. Finally, a multimodal response report is generated using the large language model. By introducing the natural language understanding and analysis capabilities of the large language model, non-technical users can deeply explore complex, multi-source, heterogeneous data using everyday language, significantly lowering the query threshold for users. Furthermore, by constructing a dynamic spatiotemporal knowledge graph, the barriers between different modalities of data are broken down, enabling collaborative and fusion analysis between different modalities and providing a foundation for spatiotemporal concept analysis using the large language model.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the storage medium embodiments and computer device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0089] This application can be used in a wide range of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, distributed computing environments including any of the above systems or devices, etc.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A query and parsing method for integrating multimodal spatiotemporal data, characterized in that, include: Receive query requests; Based on the large language model, the spatiotemporal elements in the query request are extracted and converted into a corresponding query task sequence; The query task sequence is executed to obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database; The multimodal query results are returned to the large language model to generate a multimodal response report.

2. The query and parsing method for fusing multimodal spatiotemporal data according to claim 1, characterized in that, The dynamic spatiotemporal knowledge graph is constructed in the following ways: Real-time acquisition of multimodal spatiotemporal data; The multimodal spatiotemporal data is characterized as spatiotemporal events containing spatiotemporal labels; Based on the aforementioned spatiotemporal events and pre-acquired static geographic information, the dynamic spatiotemporal knowledge graph is constructed.

3. The query and parsing method for fusing multimodal spatiotemporal data according to claim 2, characterized in that, The multimodal spatiotemporal data includes at least video data, sensor data, and trajectory data; The step of characterizing the multimodal spatiotemporal data as spatiotemporal events with spatiotemporal labels includes: The video data is subjected to target detection based on the target detection model, and behavior recognition is performed based on the dual-path spatiotemporal network to obtain the event label corresponding to the video data. The sensor data is received based on the MQTT protocol, and the sensor data is parsed to obtain the event tag corresponding to the sensor data; The trajectory data is spatiotemporally segmented based on a clustering algorithm to obtain the event label corresponding to the trajectory data; Each event tag is assigned a corresponding spatiotemporal tag to obtain each spatiotemporal event.

4. The query and parsing method for fusing multimodal spatiotemporal data according to claim 1, characterized in that, The step of extracting spatiotemporal elements from the query request based on a large language model and converting these spatiotemporal elements into a corresponding query task sequence includes: Based on a large language model, the query intent in the query request is identified; In response to the query intent being a fact query, the spatiotemporal elements in the query request are extracted, and the spatiotemporal elements are converted into a list of query tasks for the dynamic spatiotemporal knowledge graph; In response to the query intent being a multimodal query, the spatiotemporal elements in the query request are extracted, and the spatiotemporal elements are converted into a list of query tasks for the dynamic spatiotemporal knowledge graph and the original database.

5. The query and parsing method for fusing multimodal spatiotemporal data according to claim 1, characterized in that, The execution of the query task sequence, based on the dynamic spatiotemporal knowledge graph and the original database, yields multimodal query results, including: The query task sequence is executed, the structured query results are obtained based on the dynamic spatiotemporal knowledge graph, and the original data is obtained from the original database based on the structured query results, and then combined to obtain the multimodal query results.

6. The query and parsing method for fusing multimodal spatiotemporal data according to claim 5, characterized in that, The step of returning the multimodal query results to the large language model and generating a multimodal response report includes: The multimodal query results are returned to the large language model, and the original data and the structured query results are fused based on natural language generation technology to generate a multimodal response report.

7. The query and parsing method for fusing multimodal spatiotemporal data according to claim 1, characterized in that, After receiving the query request, it also includes: Based on the query domain of the query request, the large language model pre-trained in the corresponding domain is dynamically selected.

8. A query and parsing device for integrating multimodal spatiotemporal data, characterized in that, include: The request receiving module is used to receive query requests; The task decomposition module is used to extract spatiotemporal elements from the query request based on the large language model, and convert the spatiotemporal elements into a corresponding query task sequence; The execution fusion module is used to execute the query task sequence and obtain multimodal query results based on the dynamic spatiotemporal knowledge graph and the original database; The response generation module is used to return the multimodal query results to the large language model and generate a multimodal response report.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the query and parsing method for fusing multimodal spatiotemporal data as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the query parsing method for fusing multimodal spatiotemporal data as described in any one of claims 1-7.

Images