Language model prompt construction and agent task planning method based on semantic graph

Abstract

The invention relates to the technical field of language model and agent perception control, provides a language prompt (Prompt) construction and task planning method based on a semantic scene graph, and solves the problems that in an existing system, a graph structure cannot adapt to a language model, and task planning lacks a perception closed loop. The method comprises the following steps: acquiring environmental data through a multi-modal sensor and constructing a semantic scene graph; automatically generating a language prompt based on the semantic graph structure, wherein the language prompt is used for guiding a language model to perform natural language task planning; wherein the language prompt adopts a multi-level template generation mechanism, room-object-attribute information is embedded into a structured template, hierarchical semantic blocks of an environment section, a target section and an operation section are formed, and the expression ability and generalization of a language model are improved; and a planning result output by the language model is implemented by the execution module. According to the method, a closed-loop path from a perceptual graph to language reasoning is established, and the method is suitable for a semantic task execution system in complex scenes such as unmanned aerial vehicle inspection, search and rescue and detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically a method for constructing language model prompts and planning intelligent agent tasks based on semantic graphs. Background Technology

[0002] In recent years, with the breakthroughs achieved by pre-trained large language models (LLM) in the fields of natural language understanding and generation, how to introduce their powerful semantic reasoning capabilities into intelligent agent systems has become a hot topic in cutting-edge research such as robot task execution and intelligent interaction.

[0003] In complex and dynamic environments, intelligent agents (such as drones and service robots) need to efficiently understand and rationally decompose human natural language task instructions to complete tasks such as scene perception, path planning, and action control. Although large language models perform excellently in understanding natural language expressions, their lack of explicit modeling capabilities for environmental entities and limited ability to resolve contextual relationships still lead to problems in embodied intelligent task planning, such as unclear reasoning chains, semantic drift, and poor execution feasibility.

[0004] In current research, a common approach is to construct scene graphs or semantic maps using visual perception techniques and then use these as the basis for task reasoning. However, existing methods often overlook the input adaptation problem between the semantic graph and the language model, resulting in prompts that fail to fully express the task-related semantic structure.

[0005] Furthermore, existing prompts are typically presented in a single-level, linear text structure, lacking encoding and efficient organization of semantic hierarchical structures, which affects the inference efficiency of language models and the controllability of output strategies.

[0006] Therefore, how to construct semantic scene graphs from dynamic perception environments and transform them into multi-level structured language prompts that can be efficiently understood by language models, thereby driving task reasoning and execution, has become one of the key challenges in current language model-driven intelligent agent task planning.

[0007] To address the aforementioned issues, a semantic graph-based language model prompting construction and agent task planning method is proposed. This method effectively integrates spatial semantic structure and natural language expression, achieving a closed-loop process from perception mapping to language understanding and task execution. It has broad application prospects and theoretical value in embodied intelligence systems. Summary of the Invention

[0008] The purpose of this invention is to provide a method for constructing language model prompts and planning tasks for intelligent agents based on semantic graphs. This method constructs a three-dimensional scene graph with multi-layered semantic structure and transforms it into structured natural language prompts to guide a pre-trained language model to perform multi-round semantic reasoning and policy generation. This enables an embodied intelligent agent to perceive, understand, and execute closed-loop control of complex task environments, thereby improving the accuracy, adaptability, and interpretability of task planning.

[0009] To achieve the above objectives, the present invention adopts the following technical solution, and the overall method flow is as follows: Figure 1 As shown:

[0010] S1 This invention first acquires image data and spatial point cloud information of the target environment through multimodal sensors (including RGB cameras, depth cameras, LiDAR, etc.) to obtain raw multimodal data with semantic and geometric features.

[0011] S2 inputs image data into a pre-trained semantic segmentation model, performs pixel-level semantic annotation on the image, extracts key semantic label information, including object category, spatial location, attribute status (such as on / off, color, material, weight, etc.), and constructs a two-dimensional semantic label set.

[0012] S3 uses spatial registration methods such as geometric mapping, projection transformation, and camera intrinsic parameter inverse calculation to align the semantic information extracted from the image with the spatial coordinates of the point cloud, thereby achieving a consistent match between semantic content and spatial structure and establishing a complete spatial semantic relationship.

[0013] S4 constructs a 3D semantic scene graph structure based on the semantic space alignment results, such as Figure 2 As shown, the scene graph represents semantic units such as rooms, objects, and attributes in the form of nodes, and expresses spatial adjacency, physical connection, and semantic dependency in the form of edges, thus forming a structured representation with semantic hierarchy and spatial topology.

[0014] Based on the completed 3D semantic graph, S5 constructs natural language prompts, as follows: Figure 3 As shown, the specific steps include the following:

[0015] (1) The nodes in the semantic graph are divided into semantic levels, including Room-Level, Object-Level and Attribute-Level, which represent description units with different granularities and semantic depths, respectively.

[0016] (2) Preset structured language templates for each level, such as "Room A contains..." for the room level, "An object is located in..." for the object level, and "...which is open / closed / heavy" for the attribute level, etc.

[0017] (3) Embed the specific content of the semantic graph node into the corresponding hierarchical template to generate semantic fragments under the standard syntax, ensuring that the language output has consistent expression logic and contextual continuity;

[0018] (4) Based on the current task objective or user input instructions, perform task relevance calculation on each node in the semantic graph, and select the subgraph region most relevant to the task for template filling by referring to its semantic category, spatial proximity, operation possibility and other indicators.

[0019] (5) Semantic fragments are assembled in the hierarchical order of "room → object → attribute" to form a complete natural language prompt text. The text includes the following three parts: environment description: used to summarize the spatial structure and distribution relationship of each room and key object in the current scene; target intent: used to describe the object, target action or interaction intent of the current task; operation instruction: used to guide the language model to generate control behavior suggestions with operability, sequence and parametric properties.

[0020] S6 feeds the above language prompts as input text into a pre-trained large language model, which performs semantic understanding, intent reasoning, and task generation based on the prompt content.

[0021] After understanding the prompts, the S7 language model performs multiple rounds of semantic reasoning, outputting task decision text including target location inference, path planning schemes, subtask partitioning results, and action sequence planning, such as... Figure 4 As shown, this demonstrates the reasoning and integration capabilities of the language model between perception and decision-making.

[0022] S8 inputs the natural language task strategy generated by the language model into the semantic parsing module, which performs language structure analysis and semantic entity extraction to extract information such as task objectives, action types, control order, and parameter configurations. The semantic parsing module can be implemented based on technologies such as natural language dependency parsing, named entity recognition (NER), and event extraction.

[0023] S9 constructs structured low-level control instructions based on the parsing results. The control instructions include path coordinates, movement speed, task sequence, operation parameters, etc., and performs instruction format conversion and platform protocol adaptation to ensure that the instructions are compatible with the control interface of intelligent agent platforms (such as drones and mobile robots).

[0024] The S10 sends low-level control commands to the target embodied intelligence platform via a communication module, completing a closed-loop control process from natural language input to action execution, supporting single-round execution and multi-round feedback interaction. The system can dynamically adjust the prompt construction or re-infer based on the task execution results to enhance the flexibility and robustness of task completion.

[0025] In specific application scenarios, the method of this invention can be applied to tasks such as indoor search and rescue, warehouse material handling, and inspection of complex building spaces. For example, when a human issues the instruction "Please find the open red drawer in the kitchen," the system can parse the instruction intent, activate relevant nodes in the semantic graph, generate prompt input, and complete the reasonable decomposition and path control of the instruction, ensuring the accuracy and executability of the task.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] (1) A language prompt construction method oriented towards semantic graph structure is proposed. Based on multi-layer semantic template and node matching mechanism, the language input has a clear structure and semantic logic, which significantly enhances the language model's understanding of the scene and its ability to respond to instructions.

[0028] (2) Introduce task-related filtering and fragment compression mechanisms to effectively compress the prompt length while ensuring semantic integrity, improve model inference efficiency, and reduce information redundancy and generation bias caused by excessively long context.

[0029] (3) Supports multi-turn semantic reasoning mechanism, the language model can iteratively generate better strategies based on dialogue history or feedback content, and enhances adaptability, robustness and interactivity in complex dynamic tasks;

[0030] (4) Construct a complete closed-loop path of "semantic graph construction → language prompting → model reasoning → strategy parsing → action execution", with good system integration and end-to-end deployment capabilities;

[0031] (5) The method has good scalability and is suitable for single or multi-agent systems. It is widely used in fields such as intelligent inspection, spatial navigation, semantic search, and interactive guidance. It has high practical value and application prospects in complex dynamic environments. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0033] Figure 2 This is a schematic diagram of the structural composition of the semantic scene graph construction module in this invention.

[0034] Figure 3 This is a flowchart of the language prompt construction process based on semantic hierarchy in this invention.

[0035] Figure 4 This is a closed-loop control structure diagram of the multi-turn language model reasoning and task execution mapping in this invention. Detailed Implementation

[0036] To make the technical solution of the present invention clearer and more complete, the present invention will be described in detail below with reference to specific embodiments and accompanying drawings. These embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0037] This invention proposes a semantic graph-based language model prompting construction and intelligent agent task planning method, which is applicable to various embodied intelligent platforms (such as drones, ground mobile robots, etc.) to complete high-level task instructions in complex dynamic environments.

[0038] like Figure 1 As shown, the overall system flow of the present invention includes the following steps:

[0039] The first step is the acquisition and processing of sensor data:

[0040] (1) Use multimodal sensors (RGB camera, depth camera, LiDAR, etc.) to collect images, point clouds and other perception data in the target scene to obtain visual and geometric information of the environment.

[0041] (2) Perform semantic segmentation on the image data. Use a pre-trained Transformer-based segmentation model (such as SegFormer, Mask2Former, etc.) to extract the semantic label corresponding to each pixel, such as object category, feasible area, wall, etc.

[0042] (3) Use camera calibration parameters to geometrically register the semantic label information in the image with the point cloud data, and use libraries such as Open3D or PCL to register the point cloud and generate a three-dimensional point cloud with semantic labels.

[0043] (4) Based on the registered semantic point cloud, clustering and spatial partitioning algorithms (such as DBSCAN and Octomap) are used to construct a three-dimensional semantic map of the scene and build the initial structure of the semantic scene map.

[0044] The second step is semantic scene graph construction and structured modeling:

[0045] (1) Map the objects, room boundaries, switches and other elements identified in the semantic point cloud to nodes in the semantic graph and mark their attributes (position, size, status, etc.).

[0046] (2) Establish the connection of edges in the graph based on spatial adjacency and semantic logical relationship, such as "object A in room B" and "object C next to object D".

[0047] (3) Divide the graph structure into three levels: Room-Level, Object-Level, and Attribute-Level, forming a scene semantic graph with a clear semantic hierarchy, such as... Figure 2 As shown.

[0048] (4) Store the structural information of the scene graph. A graph database such as Neo4j can be used to facilitate subsequent querying, updating and prompt generation.

[0049] Step 3: Constructing the prompt message:

[0050] (1) Design corresponding language template fragments for each layer of nodes, for example: Room layer: “Room A contains…”; Object layer: “There is an object B located in…”; Attribute layer: “The object is…and its state is…”.

[0051] (2) Select subgraph regions related to the target based on task requirements, and rank the importance of nodes through semantic relevance scoring mechanisms (such as keyword matching, task context matching, etc.). For example, use TF-IDF or BERTembedding cosine similarity to calculate the matching degree between task keywords and graph node descriptions.

[0052] (3) For nodes with higher scores, fill in the corresponding language templates to generate structured semantic fragments.

[0053] (4) Organize template fragments in the order of “room → object → attribute” and splice them together to generate the final language prompt input, which includes three parts: Environment Description; Goal Statement; and Action Instruction.

[0054] like Figure 3 As shown, the final generated message is in the form of: "Room A contains Object B, which is closed. Please find an open object near it."

[0055] Step 4: Large language model drives task inference:

[0056] (1) Input structured natural language prompts into large language models (such as GPT-4, LLaMA2, etc.) and perform multi-round semantic reasoning in combination with context.

[0057] (2) The output includes: task target recognition (e.g., "turn on the nearest switch"); subtask decomposition (e.g., "find the switch", "approach the switch", "execute the operation"); action sequence planning (e.g., "GoTo(roomX)→Detect(objectY)→Manipulate(objectY)").

[0058] (3) The obtained text sequence is extracted by the semantic parsing module to extract action instructions and parameters, including action category, target object, position coordinates, etc.

[0059] (4) Transform the instructions into the underlying executable format of the embodied intelligence platform, such as waypoint flight instructions for UAVs, movement and operation instructions for ground robots, etc. For example, 'GoTo(RoomA)' is converted into UAV waypoint commands [x=2.3,y=1.5,z=1.2].

[0060] (5) The final instruction is sent to the intelligent agent platform through the communication interface (such as ROS, MAVLink, etc.) to realize the closed-loop control process from scene perception to natural language decision-making to action execution.

[0061] Step 5: Module Supplementation and Control Logic Details:

[0062] (1) The control module includes a Prompt Manager, a Model Interface Manager (LLMInterface), a Semantic Parser, and a Task Scheduler.

[0063] (2) The prompt manager is used to maintain various prompt templates, dynamically combine prompt texts, and adapt them to task intent.

[0064] (3) The model interface manager is responsible for communicating with various large language models and can automatically select the cloud or local inference engine.

[0065] (4) The semantic parser converts the text into a structured intermediate representation and performs secondary retrieval and comparison of the scene semantic graph through a graph matching mechanism.

[0066] (5) The task scheduler supports multi-task switching and execution monitoring, and adjusts the execution path in real time based on environmental feedback.

[0067] Step 6, Application Example Explanation:

[0068] For example, if a user issues the command "Find an accessible open area and land", the system will execute the following process: The perception module acquires images and point clouds; constructs a semantic map to identify open areas (such as open grasslands); generates a prompt using a template: "There is an open field in Room A..."; the large language model determines that the area is accessible for landing and generates a task path; the control module issues the command "NavigateTo(Field)→Descend→Land".

[0069] This demonstrates the effectiveness of the present invention in open instruction understanding and semantic mapping tasks in complex environments.

Claims

1. A method for constructing language prompts and planning tasks based on semantic scene graphs, characterized in that, Includes the following steps: S1 semantic scene graph construction: Includes the following sub-steps: (1) Multimodal data acquisition: Image information and spatial point cloud data are acquired through multimodal sensors. Image information is used to identify semantic labels, and point cloud information is used to represent spatial structure. (2) Image semantic processing: Semantic segmentation is performed on image data to extract object category, spatial location, and attribute information; (3) Semantic space alignment: The semantic information extracted from the image is spatially registered with the point cloud data, and the semantic labels are aligned with the spatial structure through methods such as projection matching, geometric transformation or camera parameter calculation. (4) Semantic graph structure generation: Based on the spatial alignment results, semantic tags such as "room", "object", and "attribute" are constructed as nodes, and edge connections are established based on spatial adjacency or semantic dependency to generate a three-dimensional semantic scene graph. S2 language prompt message construction: The process of converting semantic graphs into natural language prompts includes the following sub-steps: (1) Semantic layer hierarchical division: The nodes in the semantic scene graph are divided according to semantic levels, including Room-Level, Object-Level, and Attribute-Level; each level represents a different semantic granularity and spatial range, which is used to guide the downstream template embedding. (2) Multi-level template matching: For different levels, template fragments are set up, such as "Room A contains…" for the room level, "An object is located…" for the object level, and "…which is open / closed / heavy" for the attribute level. Each node generates a local semantic fragment by calling a language template that matches its semantic category; (3) Task relevance screening: Based on the current task objective, semantic graph nodes are filtered, priority scores are calculated based on the semantic or spatial relevance between nodes and the target task, and highly relevant subgraphs are selected for template filling. (4) Generation of structured semantic fragments: The filtered node information is filled into the language templates of the corresponding levels to generate several structured semantic fragments, each fragment maintaining internal syntactic consistency and semantic coherence; (5) Semantic hierarchy assembly: Following the semantic hierarchy of "room → object → attribute", the semantic fragments are spliced ​​and assembled into a complete language input text; the assembled result includes three main segments: Environment description section: Used to summarize the scene structure and the distribution of major entities; Objective / Intent Section: Clearly state the objectives and intent of the current task; Operation instruction segment: Guides the language model to output the expected operation plan or control behavior; S3 Language Model Inference: The generated structured language prompts are input into the pre-trained language model to guide the language model in scene understanding, multi-round semantic reasoning and task strategy planning, and output high-level task decision results related to the semantic scene graph. S4 Task Strategy Output: The language model outputs structured strategy content, including task target location, path planning, subtask decomposition, or action instruction sequence, based on input prompts, to guide the execution process of downstream tasks. S5 task execution mapping: The natural language task strategy output by the language model is parsed and structurally transformed, and mapped into low-level control instructions that can be executed by an embodied intelligent agent (such as a drone or robot), thereby realizing closed-loop control from natural language instructions to physical action execution.

2. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 1, characterized in that, The Prompt construction includes the following steps: S1 performs semantic hierarchical classification of nodes in the semantic scene graph, dividing them into room layer, object layer and attribute layer according to semantic granularity. S2 is based on a multi-level template generation mechanism, which embeds structural information of each level into the corresponding language template to form a structured prompt text with semantic hierarchical relationship; S3 selects relevant subgraph regions in the semantic graph based on the task objective, extracts task-related nodes, fills them into the language template, and finally constructs natural language prompt information containing environmental description segments, target intent segments, and operation instruction segments. The prompt information described in S4 is used as input for language model reasoning, thereby improving the language model's ability to perceive complex semantic structures and its reasoning performance.

3. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 1, characterized in that, The semantic graph generation includes the following steps: S1 collects data about the target environment through a multimodal sensor, including image information and depth point cloud information; S2 uses an image semantic segmentation model to process image data and extract object categories, spatial locations, and attribute information. S3 spatially registers and aligns the semantic information extracted from the image with the point cloud data, and combines the spatial geometric structure and semantic labels to construct a three-dimensional semantic scene map; The semantic scene graph described in S4 uses nodes to represent semantic units such as rooms, objects, and attributes in the environment, and edges to represent their spatial relationships and semantic connections, in order to support the subsequent Prompt construction and task reasoning process.

4. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 1, characterized in that, The language model possesses multi-turn task reasoning capabilities, including the following steps: The S1 language model analyzes the task objective and environmental state based on input prompts and generates initial task decision suggestions. S2 uses the previous round's output as context to embed subsequent prompts based on feedback instructions or contextual dialogue content, and constructs a new round of input prompt information; The S3 language model combines the output of the previous wheel with the input of the new wheel to perform iterative reasoning, gradually refining the task execution strategy until a complete and executable task decomposition result or path planning scheme is generated. The multi-turn task reasoning capability described in S4 enhances the language model's ability to understand complex task structures and its dynamic programming capabilities, thereby improving the agent's decision-making stability in uncertain environments.

5. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 1, characterized in that, The task strategy output includes the following types: S1 Target Location: Specifies the specific coordinates or spatial region that the agent needs to go to, used to locate the target location of the task; S2 Navigation Path: Based on the semantic graph spatial structure and obstacle information, it outputs path planning results that meet the task objectives; S3 Subtask Objective: Decompose the overall task hierarchically and output several independently executable subtask units; S4 Operation Instruction Sequence: Generates a structured sequence of action instructions for each subtask objective, including control actions, execution order, and parameter configuration; Any of the above outputs of S5 can be used individually or in combination for the agent to execute strategies, improving the versatility and adaptability of the planning system.

6. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 2, characterized in that, The Prompt construction process includes a compression mechanism to handle information redundancy when the number of semantic graph nodes exceeds a set threshold, including the following steps: S1 calculates semantic relevance scores for each node in the semantic graph based on the current task objective, taking into account indicators such as spatial distance, task-related labels, and semantic level. S2 selects highly relevant nodes according to their scores to form subgraph regions, and uses these regions to generate an environment summary. S3 fills language templates with filtered subgraph information, constructs concise but key-information-retaining Prompt statements, reduces input redundancy, and improves the inference efficiency of language models. The S4 compression mechanism ensures that the length of the input prompt is within the model's processing range while maintaining the semantic integrity required for task decision-making.

7. The method for constructing language prompts and planning tasks based on semantic scene graphs according to claim 1, characterized in that, The task execution interface is used to convert language output into action execution, and specifically includes the following steps: S1 performs syntactic parsing and semantic understanding on the natural language task strategies generated by the language model, extracting control information such as operation goals, action types, and execution parameters; Based on the parsing results, S2 maps natural language expressions into structured control command formats, including executable parameters such as path coordinates, speed control, and task sequence. S3 converts structured commands into a set of action instructions supported by the target intelligent agent (such as drones or robots), and generates low-level control signals adapted to its hardware interface. S4 sends the control signal to the execution intelligent agent platform through the communication interface, realizing the closed-loop conversion from language to action and completing the task execution.

Images