An agent-based urban digital twin interaction method

Abstract

The present application relates to a kind of city digital twin interaction methods based on agent, comprising: constructing space-time resource knowledge graph;Text semantic embedding is carried out to each node in space-time resource knowledge graph;Natural language instruction of user is disassembled into subtask by agent, the text semantic embedding vector of each subtask is calculated, and semantic matching retrieval is carried out according to text semantic embedding vector, obtains the node most relevant in space-time resource knowledge graph with each subtask;The action generation context of each subtask is built, and the action generation context is input into agent, and action language message is generated;Finally, the front end of digital twin city system receives the action language message sequence of each subtask that the back end of agent outputs, then according to the order of subtask, the action language message of each subtask is parsed and executed in turn.The method realizes low threshold, efficient intention-driven interaction.

Description

Technical Field

[0001] This invention relates to the field of digital twin technology, and in particular to a city digital twin interaction method based on intelligent agents. Background Technology

[0002] Currently, digital twin city systems have become a key support for promoting the digital transformation of cities. A digital twin city system can be understood as constructing an identical, real-time synchronized, and computable digital copy of the real city in virtual space. Through technologies such as the Internet of Things, big data, and artificial intelligence, it enables the city to be "traceable in the past, perceptible in the present, and predictable in the future," providing decision support for planning, construction, governance, and services.

[0003] Existing digital twin city systems primarily rely on menu-driven or control-driven graphical user interfaces for user interaction. Users need to click preset buttons, navigate through function menus, or operate specific tools to trigger system responses. In contrast, the front-end of the digital twin city system pre-configures interactive controls such as function menus, buttons, and sliders. Users trigger built-in fixed actions or call preset interfaces by clicking, checking, or dragging, thereby enabling operations such as layer display control, parameter adjustment, spatial positioning, and attribute querying. This type of solution typically relies on a static binding relationship of "control—function—interface," with the system executing corresponding operation flows according to the user's triggering order of controls.

[0004] However, this static interaction method has two major drawbacks: first, it has a high barrier to entry, requiring users to master technical terminology and interface layout; second, the system's functions are rigid, making it difficult to respond to unpredictable requests that require dynamic calculation or reasoning, lacking intelligence and flexibility. These drawbacks hinder the flexible analysis and user-friendly interaction of digital twin city systems. Therefore, further improvements to existing technologies are needed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a city digital twin interaction method based on intelligent agents, which can realize low-threshold and high-efficiency intention-driven interaction in light of the above-mentioned prior art.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a city digital twin interaction method based on intelligent agents, characterized by including the following steps: Step 1: Construct a spatiotemporal resource knowledge graph with nodes and relational edges; the nodes include interactive target nodes and spatiotemporal resource nodes, and the relational edges are the relational edges established between interactive target nodes and their associated spatiotemporal resource nodes; Step 2: Perform text semantic embedding on each node in the spatiotemporal resource knowledge graph to obtain the text semantic embedding vector of each node; Step 3: Obtain the user's natural language instructions, and decompose the user's natural language instructions into sub-tasks through the intelligent agent to obtain a sub-task sequence. The sub-task sequence includes multiple sub-tasks arranged in sequence. Calculate the text semantic embedding vector of each sub-task, and perform semantic matching retrieval based on the text semantic embedding vector to obtain the nodes most relevant to each sub-task in the spatiotemporal resource knowledge graph, and establish a mapping set between the sub-task and its most relevant nodes. Step 4: Based on the mapping set between subtasks and their most relevant nodes, construct the action generation context for each subtask, and input the action generation context of each subtask into the agent in the form of structured prompts. The backend of the agent then generates the action language message for each subtask. Step 5: Establish communication between the front end of the digital twin city system and the back end of the intelligent agent. After receiving the action language message sequence of each subtask output by the intelligent agent's back end, the front end of the digital twin city system parses and executes the action language messages of each subtask in the order of the subtasks.

[0007] Preferably, the specific method for establishing the interactive target node in step 1 is as follows: Extract interactive candidate items from the front-end page resources of the digital twin city system, generate a globally unique interactive target identifier for each candidate item, extract the semantic fields of each candidate item, and bind a resource identifier to each candidate item; Each candidate is constructed as an interaction target node, and each interaction target node includes an interaction target identifier, a semantic field, and a bound resource identifier.

[0008] Preferably, the specific method for establishing the spatiotemporal resource nodes in step 1 is as follows: obtain a set of spatiotemporal resource objects from the backend entity database of the digital twin city system, and establish each spatiotemporal resource object in the set of spatiotemporal resource objects as a spatiotemporal resource node, with each spatiotemporal resource node including at least a resource identifier.

[0009] Preferably, the specific method for establishing the relation edges in step 1 is as follows: All spatiotemporal resource nodes are grouped into a spatiotemporal resource node set. In the spatiotemporal resource node set, spatiotemporal resource nodes that match the resource identifiers bound to the current interactive target node are retrieved to obtain the set of associated spatiotemporal resource nodes of the current interactive target node. Establish relationship edges between each interactive target node and each spatiotemporal resource node in its corresponding set of associated spatiotemporal resource nodes.

[0010] Preferably, in step 2, the Sentence-BERT model is used to encode the semantic fields of each node to obtain the text semantic embedding vector of each node.

[0011] Preferably, in step 3, the Sentence-BERT model is used to encode the description text of each subtask to obtain the text semantic embedding vector of each subtask.

[0012] Preferably, in step 3, the first step is calculated according to the following formula. Sub-tasks Most relevant node , ∈{1, 2, ..., J}, where J is the total number of subtasks in the subtask sequence; ; in, In order to enable The node that reaches its maximum value. For the first Sub-tasks Text semantic embedding vectors, It is a set of nodes in a spatiotemporal resource knowledge graph. ∈ , For a set of nodes Middle node The text semantic embedding vector; This is the vector cosine similarity function.

[0013] Preferably, in step 4, for the first Sub-tasks In other words, if with The most relevant node is the interaction target node, then the current subtask The action generation context includes at least the description text of the current subtask, the interaction target node most relevant to the current subtask, the set of associated spatiotemporal resource nodes of the interaction target node most relevant to the current subtask, and a preset action language message specification; the action language message specification includes at least the following fields: a first field for characterizing the action type, a second field for identifying the interaction target to be operated, a third field for identifying the spatiotemporal resource object to which it is pointed, and a fourth field for carrying the action parameter set corresponding to the action type; If with If the most relevant node is a non-interactive target node, then the current subtask is recorded as an unexecutable subtask.

[0014] Preferably, the specific process of parsing and executing the action language messages of each subtask in the front-end of the digital twin city system in step 5 is as follows: After parsing the action language message of a subtask, the front end of the digital twin city system obtains at least the first field, the second field, the third field, and the fourth field. The front end of the digital twin city system matches the corresponding executor based on the first field in the preset mapping relationship between action type and executor. The mapping relationship between action type and executor is used to assign different action types to different types of executors. The parsed second, third and fourth fields are passed as execution input parameters to the matched executor, and the executor then executes the corresponding action.

[0015] Compared with existing technologies, the advantages of this invention are as follows: By constructing a spatiotemporal resource knowledge graph, an explicit association between "operable objects and resource objects" is achieved; subsequently, an intelligent agent decomposes the user's natural language commands into subtasks, and performs semantic matching retrieval between the subtasks and nodes in the spatiotemporal resource knowledge graph to determine executable interaction targets and their resources; based on subtasks, interaction targets, bound resources, and parameter constraints, action language messages are generated, and finally parsed and executed using the front end of the digital twin city system. Thus, this method can automatically convert the user's natural language commands into executable front-end operation sequences for the digital twin city system, achieving low-threshold, high-efficiency intent-driven interaction, reducing multi-level menu switching and repeated clicks, shortening operation paths, reducing interaction costs, and improving efficiency; furthermore, it lowers the barrier to entry, allowing users to complete common operations and combinations without needing to be familiar with technical terminology and interface layout, improving usability and universality for non-professionals. Attached Figure Description

[0016] Figure 1 This is a flowchart of a city digital twin interaction method based on intelligent agents in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0018] like Figure 1 As shown, the city digital twin interaction method based on intelligent agents in this embodiment includes the following steps: Step 1: Construct a spatiotemporal resource knowledge graph with nodes and relational edges; nodes include interactive target nodes and spatiotemporal resource nodes, and relational edges are the relational edges established between interactive target nodes and their associated spatiotemporal resource nodes; Step 2: Perform text semantic embedding on each node in the spatiotemporal resource knowledge graph to obtain the text semantic embedding vector of each node; Step 3: Obtain the user's natural language instructions, and decompose the user's natural language instructions into sub-tasks through the intelligent agent to obtain a sub-task sequence. The sub-task sequence includes multiple sub-tasks arranged in sequence. Calculate the text semantic embedding vector of each sub-task, and perform semantic matching retrieval based on the text semantic embedding vector to obtain the nodes most relevant to each sub-task in the spatiotemporal resource knowledge graph, and establish a mapping set between the sub-task and its most relevant nodes. Step 4: Based on the mapping set between subtasks and their most relevant nodes, construct the action generation context for each subtask, and input the action generation context of each subtask into the agent in the form of structured prompts. The backend of the agent then generates the action language message for each subtask. Step 5: Establish communication between the front end of the digital twin city system and the back end of the intelligent agent. After receiving the action language message sequence of each subtask output by the intelligent agent's back end, the front end of the digital twin city system parses and executes the action language messages of each subtask in the order of the subtasks.

[0019] The specific method for establishing the interactive target node in step 1 is as follows: extract candidate items with interactive significance from the front-end page resources of the digital twin city system, generate a globally unique interactive target identifier for each candidate item, extract the semantic field of each candidate item, and bind a resource identifier to each candidate item; construct each candidate item into an interactive target node, and each interactive target node includes an interactive target identifier, a semantic field, and a bound resource identifier.

[0020] In this embodiment, the specific method for selecting candidates is as follows: Step 1-1: Obtain the front-end page code of the digital twin city system. The front-end page code of the digital twin city system includes at least the DOM tree structure, node tags, and node data attributes; construct the page DOM tree and traverse the page nodes through the HTML parsing module. Step 1-2: Based on node labels and node data attributes, identify the set of interactive control nodes. The interactive controls in this set of interactive control nodes include at least buttons, drop-down lists, and sliders. Steps 1-3: Extract each interactive control node from the identified set of interactive control nodes as a candidate interactive target and construct a set of interactive target candidates. Each candidate in the set of interactive target candidates contains at least the original attribute information inherited by its corresponding control node. The original attribute information is used for subsequent extraction of semantic description and bound resource identifier.

[0021] In this embodiment, a globally unique interactive target identifier is generated for each candidate using the rule of "prefix + date string + random string". In this embodiment, the prefix is ​​"T-" and the random string is a 4-digit random number string. For example, one of the interactive target identifiers is "T-20240520-1001".

[0022] In this embodiment, the semantic field of each candidate includes at least the name of the interactive target, the type of the interactive control, and a text description. Additionally, each interactive target node also includes parameter information for the interactive control. When the interactive control is a slider, the parameter information is maximum, minimum, or error; when the interactive control has no parameters, the parameter information is empty.

[0023] The specific method for establishing spatiotemporal resource nodes in step 1 is as follows: obtain a set of spatiotemporal resource objects from the backend entity database of the digital twin city system, and establish each spatiotemporal resource object in the set of spatiotemporal resource objects as a spatiotemporal resource node. Each spatiotemporal resource node includes at least a resource identifier. In this embodiment, the set of spatiotemporal resource objects needs to be standardized into a set of spatiotemporal resource nodes.

[0024] The specific method for establishing the relationship edges in step 1 is as follows: All spatiotemporal resource nodes are grouped into a spatiotemporal resource node set. Within this set, spatiotemporal resource nodes that match the resource identifiers bound to the current interactive target node are retrieved, resulting in the associated spatiotemporal resource node set for the current interactive target node. Relationship edges are then established between each interactive target node and each spatiotemporal resource node in its corresponding associated spatiotemporal resource node set. In this embodiment, matching is performed based on resource identifiers. If the strings corresponding to the resource identifiers are the same, they match; otherwise, they do not match.

[0025] In step 2, the Sentence-BERT model is used to encode the semantic field of each node, resulting in a text semantic embedding vector for each node. The semantic field of each node is the same text formed by concatenating the interaction target name and text description semantic field of each candidate. The Sentence-BERT model is an existing technology, meaning that the final text semantic embedding vector can be obtained by encoding the text using the Sentence-BERT model.

[0026] In step 3, the intelligent agent performs task analysis on the natural language instruction text to generate a sequence of subtasks. This sequence of subtasks also includes the subtask description text after inference. Similarly, in step 3, the Sentence-BERT model is used to encode the description text of each subtask to obtain the text semantic embedding vector of each subtask.

[0027] In addition, the first step in step 3 is calculated according to the following formula. Sub-tasks Most relevant node , ∈{1, 2, ..., J}, where J is the total number of subtasks in the subtask sequence; ; in, In order to enable The node that reaches its maximum value. For the first Sub-tasks Text semantic embedding vectors, It is a set of nodes in a spatiotemporal resource knowledge graph. ∈ , For a set of nodes Middle node The text semantic embedding vector; This is the vector cosine similarity function.

[0028] The above nodes For the interactive target node or spatiotemporal resource node, in step 4, for the first... Sub-tasks In other words, if with Most relevant node If the target node for interaction is the current subtask The action generation context includes at least the description text of the current subtask, the interaction target node most relevant to the current subtask, the set of associated spatiotemporal resource nodes of the interaction target node most relevant to the current subtask, and the preset action language message specification.

[0029] The aforementioned preset action language message specification is a JSON structure constraint that conforms to the system action specification, used to limit the field composition and value range of the action language message. The action language message specification includes at least the following fields: a first field for representing the action type, a second field for identifying the interaction target to be operated, a third field for identifying the spatiotemporal resource object to which it is pointed, and a fourth field for carrying the action parameter set corresponding to the action type.

[0030] If with Most relevant node If the target node is not interactive, it is determined that the current subtask cannot be mapped to an interactive target node that can be triggered by the front end of the digital twin city system. The generation of action language messages for the current subtask is terminated, the current subtask is recorded as an unexecutable subtask, and then the next subtask processing flow is executed.

[0031] The agent in step 4 is a large language model, and the action language messages output by the large language model can satisfy the above field constraints in the JSON structure of the action language messages.

[0032] In step 5 above, the WebSocket protocol is used to establish communication between the front end of the digital twin city system and the back end of the intelligent agent, ensuring a closed loop of real-time action push, execution and feedback, while adding an exception handling mechanism to ensure stability.

[0033] The specific method for establishing communication is as follows: The front end of the digital twin city system initiates a WebSocket connection: After the front end digital twin page loads, it actively initiates a WebSocket connection request to the back end of the intelligent agent; The backend establishes a session and returns a handshake response: After receiving the connection request from the frontend of the digital twin city system, the backend of the agent generates a unique session identifier string, creates a session context (storing action sequences and execution status), and returns a handshake response to the frontend of the digital twin city system. After the connection is established, the system enters the message listening state: After receiving the handshake response, the front end of the digital twin city system stores a unique session identifier string, enters the message listening loop, and registers an exception callback.

[0034] The specific process of parsing and executing the action language messages of each subtask in the front-end of the digital twin city system in step 5 is as follows: After parsing the action language message of a subtask, the front end of the digital twin city system obtains at least the first field, the second field, the third field, and the fourth field. The front end of the digital twin city system matches the corresponding executor based on the first field in the preset mapping relationship between action type and executor. The mapping relationship between action type and executor is used to assign different action types to different types of executors. The parsed second, third and fourth fields are passed as execution input parameters to the matched executor, and the executor then executes the corresponding action.

[0035] In this embodiment, the mapping relationship between action type and actuator is pre-set in the digital twin city system. For example: layer switch → layer management actuator; parameter adjustment → parameter control actuator; spatial positioning → map positioning actuator.

[0036] This application parses action language messages through the front end of a digital twin city system and dispatches executors to complete the interactive operations of the digital twin city system. This method can effectively bridge the semantic gap between user intent and the functions of the digital twin city system, significantly improve interaction efficiency and system intelligence, promote the transformation of the digital twin city system from "passive response" to "active understanding" interaction paradigm, and realize the ability to directly control the interaction of the digital twin city system using natural language.

[0037] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for urban digital twin interaction based on intelligent agents, characterized in that... Includes the following steps: Step 1: Construct a spatiotemporal resource knowledge graph with nodes and relational edges; the nodes include interactive target nodes and spatiotemporal resource nodes, and the relational edges are the relational edges established between interactive target nodes and their associated spatiotemporal resource nodes; The specific method for establishing the interactive target node is as follows: Extract interactive candidate items from the front-end page resources of the digital twin city system, generate a globally unique interactive target identifier for each candidate item, extract the semantic fields of each candidate item, and bind a resource identifier to each candidate item; Each candidate is constructed as an interaction target node, and each interaction target node includes an interaction target identifier, a semantic field, and a bound resource identifier; Step 2: Perform text semantic embedding on each node in the spatiotemporal resource knowledge graph to obtain the text semantic embedding vector of each node; Step 3: Obtain the user's natural language instructions, and use the intelligent agent to break down the user's natural language instructions into sub-tasks to obtain a sub-task sequence, which includes multiple sub-tasks arranged in a sequential order. Calculate the text semantic embedding vector for each subtask, and perform semantic matching retrieval based on the text semantic embedding vector to obtain the nodes most relevant to each subtask in the spatiotemporal resource knowledge graph, and establish a mapping set between the subtask and its most relevant nodes. Step 4: Based on the mapping set between subtasks and their most relevant nodes, construct the action generation context for each subtask, and input the action generation context of each subtask into the agent in the form of structured prompts. The backend of the agent then generates the action language message for each subtask. Step 5: Establish communication between the front end of the digital twin city system and the back end of the intelligent agent. After receiving the action language message sequence of each subtask output by the intelligent agent's back end, the front end of the digital twin city system parses and executes the action language messages of each subtask in the order of the subtasks.

2. The urban digital twin interaction method according to claim 1, characterized in that: The specific method for establishing spatiotemporal resource nodes in step 1 is as follows: obtain a set of spatiotemporal resource objects from the backend entity database of the digital twin city system, and establish each spatiotemporal resource object in the set of spatiotemporal resource objects as a spatiotemporal resource node. Each spatiotemporal resource node includes at least a resource identifier.

3. The urban digital twin interaction method according to claim 2, characterized in that: The specific method for establishing the relation edges in step 1 is as follows: All spatiotemporal resource nodes are grouped into a spatiotemporal resource node set. In the spatiotemporal resource node set, spatiotemporal resource nodes that match the resource identifiers bound to the current interactive target node are retrieved to obtain the set of associated spatiotemporal resource nodes of the current interactive target node. Establish relationship edges between each interactive target node and each spatiotemporal resource node in its corresponding set of associated spatiotemporal resource nodes.

4. The urban digital twin interaction method according to any one of claims 1 to 3, characterized in that: In step 2, the Sentence-BERT model is used to encode the semantic fields of each node to obtain the text semantic embedding vector of each node.

5. The urban digital twin interaction method according to claim 4, characterized in that: In step 3, the Sentence-BERT model is used to encode the description text of each subtask to obtain the text semantic embedding vector of each subtask.

6. The urban digital twin interaction method according to claim 5, characterized in that: In step 3, the number is calculated according to the following formula. Sub-tasks Most relevant node , ∈{1, 2, ..., J}, where J is the total number of subtasks in the subtask sequence; ； in, In order to enable The node that reaches its maximum value. For the first Sub-tasks Text semantic embedding vectors, It is a set of nodes in a spatiotemporal resource knowledge graph. ∈ , For a set of nodes Middle node The text semantic embedding vector; Let be the vector cosine similarity function.

7. The urban digital twin interaction method according to claim 3, characterized in that: In step 4, for the first Sub-tasks In other words, if with The most relevant node is the interaction target node, then the current subtask The action generation context includes at least the description text of the current subtask, the interaction target node most relevant to the current subtask, the set of associated spatiotemporal resource nodes of the interaction target node most relevant to the current subtask, and a preset action language message specification; the action language message specification includes at least the following fields: a first field for characterizing the action type, a second field for identifying the interaction target to be operated, a third field for identifying the spatiotemporal resource object to which it is pointed, and a fourth field for carrying the action parameter set corresponding to the action type; If with If the most relevant node is a non-interactive target node, then the current subtask is recorded as an unexecutable subtask.

8. The urban digital twin interaction method according to claim 7, characterized in that: The specific process of parsing and executing the action language messages of each subtask in the front-end of the digital twin city system in step 5 is as follows: After parsing the action language message of a subtask, the front end of the digital twin city system obtains at least the first field, the second field, the third field, and the fourth field. The front end of the digital twin city system matches the corresponding executor based on the first field in the preset mapping relationship between action type and executor. The mapping relationship between action type and executor is used to assign different action types to different types of executors. The parsed second, third and fourth fields are passed as execution input parameters to the matched executor, and the executor then executes the corresponding action.

Images