Method, system and equipment for planning visual language structured space and medium

Abstract

The invention relates to the technical field of visual language intelligence, and discloses a visual language structured space planning method, system and device and a medium, which are applied to a body-equipped intelligent agent, and the method comprises the following steps: obtaining an image data stream of an indoor environment and a current position of the body-equipped intelligent agent in real time; constructing a visual scene hierarchical graph corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchical graph represents a spatial hierarchical structure and a mutual relation of objects in the indoor environment; mapping a natural language instruction of a user to a layer corresponding to the visual scene hierarchical graph, and generating a standardized language corresponding to the natural language instruction; and determining an optimal action path according to the standardized language and the current position, and executing the optimal action path until a target answer corresponding to the natural language instruction is obtained. The method has the beneficial effects that complex tasks can be better dealt with, and the solving capability and the interpretability are improved.

Description

Technical Field

[0001] The present application relates to the field of visual language intelligent technology, and in particular to a planning method, system, device and medium for visual language structured space. Background Art

[0002] The core goal of the EmbodiedQA task is to enable intelligent agents to understand and execute human instructions in real-world environments, make dynamic decisions, and interact with the environment. For example, the agent needs to find objects, organize objects, or answer questions about the environment based on instructions.

[0003] In recent years, embodied agents based on LLMs have developed rapidly. The reasoning ability of LLMs enables agents to process complex instructions and generate action plans. However, these agents still face challenges such as insufficient environment understanding, hallucination problems, and interpretability in embodied question answering tasks. Summary of the invention

[0004] The present application provides a planning method, system, device and medium for a visual language structured space, which solves the technical problems of insufficient environmental understanding, hallucination problems and interpretability that are still faced in embodied question-answering tasks, and achieves the technical effect of better coping with complex tasks and improving problem-solving capabilities and interpretability.

[0005] In order to achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, an embodiment of the present application provides a planning method for a visual language structured space, which is applied to an embodied intelligent body, and the method includes: Acquire in real time an image data stream of the indoor environment and the current position of the embodied intelligent body; Constructing a visual scene hierarchy diagram corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy diagram represents the spatial hierarchy and mutual relationship of objects in the indoor environment; Mapping the user's natural language instructions to the layer corresponding to the visual scene hierarchy diagram, and generating a standardized language corresponding to the natural language instructions; An optimal action path is determined based on the standardized language and the current position, and the optimal action path is executed until a target answer corresponding to the natural language instruction is obtained.

[0006] The present embodiment provides a visual language structured space planning method, which realizes real-time perception and positioning of the indoor environment by acquiring the image data stream of the indoor environment and the current position of the embodied intelligent body in real time. Then, a visual scene hierarchy diagram of the indoor environment is constructed based on the image data stream, and the hierarchy diagram represents the spatial structure and mutual relationship of objects in the indoor environment. It is possible to highly abstract and organize environmental information from a visual level, which helps the embodied intelligent body understand the spatial layout of the environment, the relative position of objects, and the relationship between them. Then the user's natural language instructions are mapped to the layer corresponding to the visual scene hierarchy diagram, and a standardized language is generated. The natural language instructions can be converted into a standardized expression that can be understood by the machine and associated with the spatial hierarchy diagram. This process makes human-computer interaction more natural, and the user can instruct the embodied intelligent body to perform tasks through simple language, while avoiding language ambiguity and misunderstanding. Finally, the optimal action path is determined based on the standardized language and the current position, and the path is executed until the target answer is obtained. It ensures that the embodied intelligent body can complete the task efficiently and accurately, avoids redundant or invalid actions in path planning, and improves the operation efficiency and the success rate of task execution.

[0007] Optionally, constructing the visual scene hierarchy diagram corresponding to the indoor environment based on the image data stream includes: determining objects of different levels in the indoor environment and spatial relationships and hierarchical relationships between objects according to the image data stream; Determine the object as a node, and the spatial relationship and the hierarchical relationship as edges; The visual scene hierarchy graph of different layers is constructed according to the nodes and the edges.

[0008] This embodiment uses image data streams to identify objects and their spatial relationships, providing an accurate environmental perception basis for subsequent processing; secondly, by converting objects and their relationships into graph models, this information can be efficiently stored and processed; finally, a visual scene hierarchy graph is constructed to enhance the comprehensive understanding of complex scenes through layered representation. This significantly improves the accuracy and efficiency of indoor environment modeling, providing strong support for applications such as intelligent interaction and navigation.

[0009] Optionally, the visual scene hierarchy diagram includes a floor layer, a room layer, a non-moving object layer and a moving object layer; wherein, The floor layer is the highest layer of the visual scene hierarchy diagram, and the floor layer includes a plurality of room layers; The room layers are the rooms in the indoor environment, and each of the room layers includes the non-moving object layer and the moving object layer; The non-movable object layer is the non-movable objects in the room; The moving object layer is movable objects in the room.

[0010] Optionally, mapping the user's natural language instruction to a layer corresponding to the visual scene hierarchy diagram to generate a standardized language corresponding to the natural language instruction includes: Obtaining a natural language instruction from a user and identifying keywords of the natural language instruction; Determining, based on the keywords, an inquiry object and an inquiry relationship corresponding to the natural language instruction; The query object and the query relationship are mapped to a layer corresponding to the visual scene hierarchy diagram, and a standardized language corresponding to the natural language instruction is generated.

[0011] This embodiment achieves preliminary recognition of user intentions and information extraction by obtaining the user's natural language instructions and identifying the keywords therein. It can automatically identify key information in user input, providing a basis for subsequent processing. Then, by analyzing the keywords, the inquiry object and inquiry relationship corresponding to the natural language instruction are determined, and the user's needs are converted into specific layers and structures. It is possible to establish a connection between the natural language instruction and the specific objects or relationships in the visual scene, so as to clarify the specific query content of the user. The inquiry object and inquiry relationship are then mapped to the visual scene hierarchy diagram, and a standardized language is generated. Through the precise mapping of the layer structure of the visual scene, natural language can be converted into a standardized expression that can be understood and processed by the computer system, which enhances the fusion of natural language and visual data, improves the intelligence and accuracy of the interaction, and supports embodied intelligent agents to perform accurate queries and operations in complex visual environments.

[0012] Optionally, during the process of executing the optimal action path, the method further includes: Determining whether the current position meets the visual field requirement according to a preset visual perception condition; When the field of view requirement is not met, the field of view angle is adjusted until the field of view requirement is met.

[0013] This embodiment realizes intelligent perception and evaluation of the current field of view by judging whether the current position meets the field of view requirements according to preset visual perception conditions. It can automatically evaluate whether the field of view meets the requirements according to preset conditions, so as to judge whether the field of view needs to be adjusted. This enhances the adaptive ability of the embodied intelligent body, enabling it to flexibly respond to different visual perception requirements. When the current field of view does not meet the requirements, the field of view angle is automatically adjusted to ensure that the field of view conditions are met. The field of view angle can be dynamically adjusted to ensure the adaptability and integrity of the observation area. It avoids the tediousness of users manually adjusting the viewing angle, and improves the user experience.

[0014] Optionally, the determining whether the current position meets the field of view requirement is performed by judging through a multimodal large model; wherein the multimodal large model includes at least one of ChatGPT and LLM.

[0015] Optionally, the method further comprises: In the case where the natural language instruction includes an object attribute indicating a target object, a pre-judgment perception result is determined according to the object attribute using an LLMs model; wherein the pre-judgment perception result is used to characterize long-range perception or short-range perception; Based on the predicted perception results, an action plan is generated to guide the embodied intelligent agent to approach the target object.

[0016] This embodiment uses the LLMs model to predict the perception results according to the object attributes in the natural language instructions, and can predict the perception results in advance through the object attribute information in the natural language, whether it is long-range perception or close-range perception. This predictive perception can help the embodied intelligent agent more efficiently identify the position, characteristics or state of the target object, so as to make intelligent decisions for the next action and reduce the uncertainty in real-time perception. An action plan is generated based on the predicted perception results to guide the embodied intelligent agent to approach the target object. It is possible to formulate a suitable action path based on the previous perception prediction, so as to ensure that the embodied intelligent agent can approach the target object more accurately. This action plan not only optimizes the action process, but also enables the embodied intelligent agent to better cope with different environments and task requirements, and improve the efficiency and accuracy of task completion.

[0017] In a second aspect, an embodiment of the present application provides a planning system for a visual language structured space, which is applied to an embodied intelligent body, and the system includes: A data acquisition unit, used to acquire an image data stream of the indoor environment and a current position of the embodied intelligent body in real time; A graph construction unit, configured to construct a visual scene hierarchy graph corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy graph represents the spatial hierarchical structure and mutual relationship of objects in the indoor environment; An instruction mapping unit, used to map the user's natural language instruction to a layer corresponding to the visual scene hierarchy diagram, and generate a standardized language corresponding to the natural language instruction; A path determination unit is used to determine an optimal action path according to the standardized language and the current position, and execute the optimal action path until a target answer corresponding to the natural language instruction is obtained.

[0018] In a third aspect, an embodiment of the present application provides a computer device, including: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned visual language structured space planning method by executing the computer instructions.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the above-mentioned visual language structured space planning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a method for planning a visual language structured space provided in an embodiment of the present application; Figure 2 A flowchart of step S3 provided in an embodiment of the present application; Figure 3 A flowchart of step S5 provided in an embodiment of the present application; Figure 4 In the process of executing the optimal action path provided in the embodiment of the present application, the method also includes a flowchart of specific steps; Figure 5 A flowchart of another visual language structured space planning method provided in an embodiment of the present application; Figure 6 A block diagram of a planning system for a visual language structured space provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0023] The core goal of the EmbodiedQA task is to enable the intelligent agent to understand and execute human instructions in a real environment, interact with the environment through perception and action, and answer complex questions raised by humans. The significance of this task is not only that the intelligent agent can answer traditional text questions, but also that the intelligent agent needs to be able to make dynamic decisions in the actual environment and make reasonable responses based on real-time perception information. For example, the intelligent agent may need to find specific objects in the room, organize objects, or answer questions about the layout of the room according to instructions.

[0024] As technology develops, intelligent agents need to have the following capabilities when performing embodied question-answering tasks: Natural language understanding and generation: The ability to understand and generate natural language, especially to understand complex, multi-step task instructions.

[0025] Environmental perception and interaction: perceive the environment in real time through visual, auditory and other means of perception, and interact with objects in the environment.

[0026] Mission planning and decision-making: Make reasonable plans and decisions based on current perception results and mission objectives.

[0027] Self-correction and learning: Make adjustments and corrections based on feedback during task execution.

[0028] In recent years, research on entity agents based on LLMs has developed rapidly. The powerful reasoning ability of LLMs enables agents to understand instructions, infer inferences, and generate complex action plans at a higher level. LLMs can process complex sentence structures through natural language and reason about multi-round dialogues, and are one of the core technologies for achieving embodied question-answering tasks.

[0029] However, current LLMs technology still faces many challenges when applied in the field of intelligent agents: (1) Lack of sufficient understanding of the environment Current LLM-based task planning methods tend to generate a task plan at one time, lacking dynamic, real-time perception of the environment. The agent may start to execute the task according to this generated plan, but this "one-time generation" approach may be inflexible when the environment changes or there is a lot of uncertainty. For example, the agent may misjudge the location of an object, fail to consider the existence of obstacles, or be unable to cope with sudden changes in the environment.

[0030] This makes task planning very difficult when facing complex and changing environments. The intelligent agent may not be able to accurately make adjustments or corrections based on environmental information, which will affect the effectiveness of task execution.

[0031] (2) Hallucination Problem of LLMs “Hallucination” refers to the fact that LLMs may generate unrealistic or false information when generating text. In agent task planning, LLMs’ hallucinations may manifest as agents executing unrealistic or unachievable action plans. For example, LLMs may ignore certain physical obstacles or misunderstand the state of objects in the action routes generated based on instructions, resulting in deviations in the final task execution. Because the generation mechanism of LLMs is not always based on accurate environmental feedback, they are prone to generate speculative information that does not match the actual situation.

[0032] This problem is particularly serious in dynamic environments, where the agent may not fully verify the real-time status of the environment, resulting in task failure or poor results.

[0033] (3) Lack of explainability and difficulty in error diagnosis The “black box” nature of LLMs means that their reasoning process is often unexplainable to users and developers. When an agent fails to perform a complex task, it is difficult to determine the specific location or cause of the error. For example, the agent may fail to identify the location of a target object, but because the reasoning path of LLMs cannot be clearly traced, developers cannot directly understand what caused the error. This greatly increases the difficulty of diagnosing and fixing the problem.

[0034] In addition, LLMs may make mistakes during task execution due to inaccurate reasoning or insufficient environmental perception. Such errors often lack sufficient explainability, making the problem location and repair process more complicated.

[0035] In order to solve the above technical problems, according to an embodiment of the present application, a planning method embodiment of a visual language structured space is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] In this embodiment, a planning method for a visual language structured space is provided. Figure 1 A flowchart of a visual language structured space planning method provided in an embodiment of the present application, such as Figure 1 As shown, applied to an embodied agent, the process includes the following steps: Step S1, real-time acquisition of the image data stream of the indoor environment and the current position of the embodied intelligent body.

[0037] Specifically, visual sensors such as cameras are used to capture image data streams of indoor environments in real time. These data streams can provide the visual information required by the embodied intelligent body for subsequent environmental understanding and navigation. The current position of the embodied intelligent body is determined by indoor positioning technology. Current positioning technologies can be roughly divided into neighboring information method, scene analysis method and geometric feature method. These methods usually require the use of auxiliary nodes, such as RFID tags or infrared and ultrasonic sensors, to calculate the position of the node to be measured relative to the auxiliary node through different ranging methods, and then compare it with the data collected in advance in the database to determine the current position.

[0038] First, auxiliary nodes with fixed positions are set up in the indoor environment. The positions of these nodes are known. Some location information is directly stored in the nodes, such as radio frequency identification (RFID) tags, and some is stored in the database of the computer terminal, such as infrared, ultrasonic, etc.

[0039] Then the measurement uses the transmitting and receiving devices to measure the distance of the embodied intelligent body to determine the relative position. Using a certain method to measure the distance usually requires a pair of transmitting and receiving devices, which can be roughly divided into two types according to the location of the transmitter and the receiver: one is that the transmitter is located at the node being measured and the receiver is located at the auxiliary node, such as infrared, ultrasonic and radio frequency identification (RFID); the other is that the transmitter is located at the auxiliary node and the receiver is located at the node being measured, such as WiFi, ultra-wideband (UWB), ZigBee. Based on the ranging results, the embodied intelligent body can use positioning algorithms such as neighboring information method, scene analysis method or geometric feature method to determine its relative position.

[0040] Step S3, constructing a visual scene hierarchy diagram corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy diagram represents the spatial hierarchy and mutual relationship of objects in the indoor environment.

[0041] Specifically, use pre-trained deep learning models such as CNN to detect and recognize objects in image data streams to determine the category and location of objects. Then build a hierarchical structure. Floor layer: determine the floor layout of the indoor environment and use each room area as a node. Room layer: based on the floor layer, refine it to specific rooms, and each room is a node. Non-moving object layer: in the room layer, identify non-movable objects such as furniture, walls, etc. as nodes. Moving object layer: identify movable moving objects such as cups, books, etc. as nodes. Then determine the spatial relationship and hierarchical relationship. Spatial relationship: analyze the spatial relationship between objects, such as "above...", "next to...", etc. Hierarchical relationship: determine the hierarchical relationship between objects, such as a moving object is above a non-moving object, or a non-moving object is inside the room. Finally, use objects as nodes, spatial relationships and hierarchical relationships as edges to form a topological graph, that is, a visual scene hierarchical graph.

[0042] Step S5, mapping the user's natural language instructions to the layer corresponding to the visual scene hierarchy diagram, and generating a standardized language corresponding to the natural language instructions.

[0043] Specifically, natural language processing techniques, such as named entity recognition (NER) and relation extraction, are used to identify objects in instructions and the relationships between them. The user's instruction intent is determined, which helps to select the appropriate layer and generate the correct query. Based on the identified entities and relationships, the corresponding layers of these elements in the visual scene hierarchy diagram are determined. For example, if the entity is a movable moving object, it should be mapped to the moving object layer V4. A standardized language is defined that can be understood and executed. For example, a natural language instruction can be converted into (V3→V1), indicating that the query is about the position of an object in the non-moving object layer V3 relative to the room layer V1.

[0044] Step S7, determining the optimal action path according to the standardized language and the current position, and executing the optimal action path until the target answer corresponding to the natural language instruction is obtained.

[0045] Specifically, key information is extracted from the standardized language, such as the target object (book) and location (on the coffee table). The location information of the target object is retrieved in the visual scene hierarchy diagram to determine the layer and specific coordinates of the target object. Through path planning algorithms such as A*, Dijkstra or RRT. Based on the current position and target position of the embodied agent, an optimal path is generated. The embodied agent moves along the planned path, which may include navigation, obstacle avoidance and other behaviors. During the execution process, the position and environmental changes of the embodied agent are monitored in real time, and the path is adjusted if necessary. After reaching the target position, the target object is identified using a visual sensor until the target answer corresponding to the instruction is obtained and fed back using the large language model LLM.

[0046] In the process of obtaining and feeding back the target answer corresponding to the instruction using the large language model LLM, the existence of the target object (e.g., book) is confirmed through image recognition technology. The features of the identified target object are matched with the features described in the natural language instruction to ensure that the identified object is the object of the inquiry. Confirm whether the location of the target object is consistent with the location specified in the natural language instruction (e.g., whether the book is on the coffee table). Verify the spatial relationship between the target object and the surrounding environment to ensure that it conforms to the description in the natural language instruction. Use the large language model LLM for semantic analysis to ensure that the extracted information semantically matches the natural language instruction and feed back the target answer to the user.

[0047] The present embodiment provides a visual language structured space planning method, which realizes real-time perception and positioning of the indoor environment by acquiring the image data stream of the indoor environment and the current position of the embodied intelligent body in real time. Then, a visual scene hierarchy diagram of the indoor environment is constructed based on the image data stream, and the hierarchy diagram represents the spatial structure and mutual relationship of objects in the indoor environment. It is possible to highly abstract and organize environmental information from a visual level, which helps the embodied intelligent body understand the spatial layout of the environment, the relative position of objects, and the relationship between them. Then the user's natural language instructions are mapped to the layer corresponding to the visual scene hierarchy diagram, and a standardized language is generated. The natural language instructions can be converted into a standardized expression that can be understood by the machine and associated with the spatial hierarchy diagram. This process makes human-computer interaction more natural, and the user can instruct the embodied intelligent body to perform tasks through simple language, while avoiding language ambiguity and misunderstanding. Finally, the optimal action path is determined based on the standardized language and the current position, and the path is executed until the target answer is obtained. It ensures that the embodied intelligent body can complete the task efficiently and accurately, avoids redundant or invalid actions in path planning, and improves the operation efficiency and the success rate of task execution.

[0048] Figure 2 The flowchart of step S3 provided in the embodiment of the present application may include the following steps: Step S31, determining objects of different levels in the indoor environment and the spatial relationship and hierarchical relationship between the objects according to the image data stream.

[0049] Specifically, through deep learning models, such as convolutional neural networks (CNN), objects in the image data stream are detected and identified to determine the category and location of the objects. Stereoscopic vision or deep learning technology is used to estimate the depth information of objects in the image, and a three-dimensional model of the indoor environment is constructed. Based on the three-dimensional model of the indoor environment, objects at different levels are identified and constructed, such as the floor level, room level, non-moving object level, and moving object level. The spatial relationship and hierarchical relationship between objects are obtained through geometric analysis or machine learning algorithms.

[0050] Step S33: determine the objects as nodes, and the spatial relationships and hierarchical relationships as edges.

[0051] Specifically, each object is represented as a node in the graph, and each node contains the attributes of the object, such as category, location, size, etc. Analyze the spatial relationship between objects, such as "above", "next to", "inside", etc. Determine the relative positions between objects, such as front and back, up and down, left and right, etc. Determine the boundaries of the room and associate them with the area of ​​the floor layer. Associate non-moving objects in the room with room nodes to indicate that they belong to the same room. Associate moving objects with the non-moving objects they are in to indicate affiliation.

[0052] For example, the floor layer V1 represents the plan layout of the entire indoor environment, such as different room areas or spaces. The room layer V2 represents specific rooms or space areas, which are subdivisions of the indoor environment. The non-moving object layer V3 represents large, immovable objects in the room, such as furniture, walls, etc. The moving object layer V4 represents small, freely movable objects in the room, which are usually attached to non-moving objects.

[0053] Edge E 12 (Floor layer to room layer): Indicates the relationship between the area in the floor layer and the specific room in the room layer. 23 (Room layer to non-moving object layer): indicates the relationship between the room in the room layer and the non-moving object in the non-moving object layer. 34 (Non-moving object layer to moving object layer): Indicates the relationship between objects in the non-moving object layer and moving objects in the moving object layer. 22 (Room layer relationship): Indicates the spatial relationship between different rooms in the room layer. 33 (Relationship within the non-moving object layer): Indicates the spatial relationship between different non-moving objects within the non-moving object layer. 44 (Relationship within the moving object layer): represents the spatial relationship between different moving objects within the moving object layer.

[0054] Step S35, constructing a visual scene hierarchy graph of different layers based on the nodes and edges.

[0055] In some preferred embodiments, the visual scene hierarchy diagram includes a floor layer, a room layer, a non-moving object layer, and a moving object layer; wherein, The floor layer is the highest layer of the visual scene hierarchy graph, and the floor layer includes multiple room layers; The room layer refers to each room in the indoor environment, and each room layer contains a non-moving object layer and a moving object layer; The non-movable object layer is the immovable objects in the room; The moving object layer is the movable objects in the room.

[0056] Specifically, construct the floor layer: create a node for each room, which belongs to the floor layer. If there is a spatial relationship between rooms (such as adjacent or contained), create edges between the corresponding nodes. Construct the room layer: In each room layer, create nodes for the non-moving objects and moving object layers in the room respectively. Inside the room layer, create edges between the non-moving object layer and the moving object layer to indicate that they belong to the same room. Construct the non-moving object layer: create a node for each non-movable non-moving object in the room. If there is a spatial relationship between non-moving objects, such as side by side or stacked, create edges between the corresponding nodes. Construct the moving object layer: create a node for each movable moving object in the room. If there is a spatial relationship between moving objects or between moving objects and non-moving objects, create edges between the corresponding nodes. Integrating the floor layer, room layer, non-moving object layer, and moving object layer, we get a visual scene hierarchy graph, which reflects the spatial structure and hierarchical relationship of objects in the indoor environment, and provides a clear environment model for the embodied intelligent agent to support its navigation and operation in the indoor environment.

[0057] and Figure 1 Compared with the embodiment shown in the figure, this embodiment identifies objects and their spatial relationships through image data streams, providing an accurate environmental perception basis for subsequent processing; secondly, by converting objects and their relationships into graph models, this information can be efficiently stored and processed; finally, a visual scene hierarchy graph is constructed to enhance the comprehensive understanding of complex scenes through hierarchical representation. This significantly improves the accuracy and efficiency of indoor environment modeling, providing strong support for applications such as intelligent interaction and navigation.

[0058] Figure 3 The flowchart of step S5 provided in the embodiment of the present application may include the following steps: Step S51, obtaining the user's natural language instructions and identifying keywords of the natural language instructions.

[0059] Specifically, the embodied intelligent agent receives the user's natural language instructions through a speech recognition system or a text input interface, and uses natural language processing (NLP) techniques such as word segmentation, part-of-speech tagging, and named entity recognition (NER) to identify keywords in the natural language instructions.

[0060] Step S53: determining the inquiry object and inquiry relationship corresponding to the natural language instruction according to the keywords.

[0061] Specifically, the keywords and context are analyzed to determine the object (such as "tea table") and query relationship (such as "located at") of the user's query. That is, the user's intention is determined, such as querying location, attributes or status.

[0062] Step S55, mapping the query object and the query relationship to the layer corresponding to the visual scene hierarchy diagram, and generating a standardized language corresponding to the natural language instruction.

[0063] Specifically, according to the determined query object and query relationship, large language models LLMs are used to map them to the corresponding layers in the visual scene hierarchy. For example, if the query is about the location of an object, it needs to be mapped to the non-moving object layer V3 and the moving object layer V4. For example, the natural language instruction is "Where is the OBJ located in?", which is parsed into a pattern in the visual scene hierarchy (such as (room layer V2→floor layer V1) or (non-moving object layer V3→floor layer V1)).

[0064] The natural language instructions are converted into a standardized query language, which is usually pre-defined, into (non-moving object layer V3 → floor layer V1) or (moving object layer V4 → floor layer V1), indicating that the query is about the position of the object relative to the room or floor.

[0065] It should be noted that the large language models (LLMs) here are pre-trained by fine-tuning instructions, so that the model can convert complex natural language instructions into standardized languages ​​mapped to the layers corresponding to the visual scene hierarchy diagram, which provide the data basis for subsequent embodied intelligent body navigation and final answer, thereby improving the accuracy and efficiency of task planning and execution.

[0066] and Figure 1 Compared with the embodiment shown, this embodiment achieves preliminary recognition of user intentions and information extraction by obtaining the user's natural language instructions and identifying the keywords therein. It can automatically identify key information in user input, providing a basis for subsequent processing. Then, by analyzing the keywords, the inquiry object and inquiry relationship corresponding to the natural language instruction are determined, and the user's needs are converted into specific layers and structures. It is possible to establish a connection between the natural language instruction and the specific objects or relationships in the visual scene, so as to clarify the specific query content of the user. The inquiry object and inquiry relationship are then mapped to the visual scene hierarchy diagram, and a standardized language is generated. Through the precise mapping of the layer structure of the visual scene, natural language can be converted into a standardized expression that can be understood and processed by the computer system, which enhances the fusion of natural language and visual data, improves the intelligence and accuracy of the interaction, and supports embodied intelligent agents to perform accurate queries and operations in complex visual environments.

[0067] Figure 4 In the process of executing the optimal action path provided in the embodiment of the present application, the method also includes a flowchart of specific steps, which may include the following steps: Step S711, determining whether the current position meets the field of view requirement according to preset visual perception conditions.

[0068] In some preferred embodiments, whether the current position meets the field of view requirement is determined by judging through a multimodal large model; wherein the multimodal large model includes at least one of ChatGPT and LLM.

[0069] Step S713: If the field of view requirement is not met, adjust the field of view angle until the field of view requirement is met.

[0070] Specifically, pre-defined visual perception conditions such as clarity, field of view, object recognition, etc., these conditions will be used to evaluate whether the current position meets the field of view requirements. Use a multimodal large model (such as ChatGPT, LLM) combined with language understanding and visual information to determine whether the current position meets the preset visual perception conditions. Match the collected visual information (here refers to the image data stream) with the preset visual perception conditions to determine whether the field of view requirements are met. If the field of view requirements are not met, such as the inability to clearly see the entire room, adjust the field of view angle. Here, the embodied intelligent agent can rotate the camera, adjust the focal length of the camera, or rotate the angle of the embodied intelligent agent. After adjustment, re-collect visual information and evaluate whether the field of view requirements are met. If the field of view still does not meet the requirements, continue to adjust and evaluate until the conditions are met.

[0071] For example, visually determine whether the embodied agent has moved to the target location, such as in front of a coffee table. Use a multimodal large model to evaluate whether the coffee table is clear in the field of view and whether the information about the coffee table and its attached objects can be clearly seen. If the visual observation results do not match expectations, such as only a small corner can be seen instead of the entire coffee table, an error is detected. The embodied agent uses mobility capabilities to adjust its position, such as translation or rotation, to obtain a better field of view. After adjustment, re-evaluate whether the field of view requirements are met. Feedback the user on the current status and any required actions, such as requesting more instructions or confirmation. After moving to the final target location, begin observing the target. Observe the target object in conjunction with the initial natural language instructions, such as recognizing colors or text, and use large language models such as LLMs to provide answers.

[0072] and Figure 1 Compared with the embodiment shown in the figure, this embodiment realizes intelligent perception and evaluation of the current field of view by judging whether the current position meets the field of view requirements according to the preset visual perception conditions. It can automatically evaluate whether the field of view meets the requirements according to the preset conditions, so as to judge whether the field of view needs to be adjusted. This enhances the adaptive ability of the embodied intelligent body, enabling it to flexibly respond to different visual perception needs. When the current field of view does not meet the requirements, the field of view angle is automatically adjusted to ensure that the field of view conditions are met. The field of view angle can be dynamically adjusted to ensure the adaptability and integrity of the observation area. It avoids the tediousness of users manually adjusting the viewing angle, and improves the user experience.

[0073] Figure 5 A flowchart of another visual language structured space planning method provided in an embodiment of the present application, the process may include the following steps: Step S2, when the natural language instruction contains object attributes indicating the target object, the LLMs model is used to determine the predicted perception result according to the object attributes; wherein the predicted perception result is used to characterize the long-range perception or the short-range perception.

[0074] Step S4, based on the predicted perception results, generates an action plan to guide the embodied intelligent agent to approach the target object.

[0075] Specifically, the LLMs’ extensive common sense knowledge is used to perceive and reason about the external environment, and different action rules are formulated based on the object’s attribute characteristics. Predict whether the attribute needs to be perceived remotely or closely, and formulate action rules based on the object’s attribute characteristics. The language processing capabilities of the LLMs are used to generate specific content that needs to be perceived based on the current steps of the action plan, helping the embodied agent to effectively perceive and reason about the external environment, and generate an action plan, including the macro steps of where to go first and where to go next. This action plan focuses more on the logical order of the task and the necessary action steps, without involving specific path details. The purpose of the action plan is to guide the embodied agent to have a clear action framework when performing the task, ensuring the consistency of the task and the achievement of the goal. The optimal path is a specific navigation path calculated under the macro steps determined by the action plan. The two complement each other. The action plan guides the calculation of the optimal path, and the optimal path ensures that the action plan can be executed efficiently and accurately. During the execution of the task, it may be necessary to adjust the optimal path based on the action plan many times to cope with environmental changes and new situations, and ensure the smooth progress of the task.

[0076] and Figure 1 Compared with the embodiment shown, this embodiment uses the LLMs model to predict the perception results according to the object attributes in the natural language instructions, and can predict the perception results in advance through the object attribute information in the natural language, whether it is long-range perception or close-range perception. This predictive perception can help the embodied intelligent agent to more efficiently identify the position, characteristics or state of the target object, so as to make intelligent decisions for the next action and reduce the uncertainty in real-time perception. An action plan is generated based on the predicted perception results to guide the embodied intelligent agent to approach the target object. It is possible to formulate a suitable action path based on the previous perception prediction, so as to ensure that the embodied intelligent agent can approach the target object more accurately. This action plan not only optimizes the action process, but also enables the embodied intelligent agent to better cope with different environments and task requirements, and improve the efficiency and accuracy of task completion.

[0077] Accordingly, please refer to Figure 6A block diagram of a planning system for a visual language structured space provided in an embodiment of the present application, applied to an embodied intelligent body, the system comprising: A data acquisition unit 101 is used to acquire the image data stream of the indoor environment and the current position of the embodied intelligent body in real time; A graph construction unit 103 is used to construct a visual scene hierarchy graph corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy graph represents the spatial hierarchy structure and mutual relationship of objects in the indoor environment; The instruction mapping unit 105 is used to map the user's natural language instruction to the layer corresponding to the visual scene hierarchy diagram, and generate a standardized language corresponding to the natural language instruction; The path determination unit 107 is used to determine the optimal action path according to the standardized language and the current position, and execute the optimal action path until a target answer corresponding to the natural language instruction is obtained.

[0078] In some optional implementations, the graph construction unit 103 includes: According to the image data stream, objects at different levels in the indoor environment and the spatial and hierarchical relationships between the objects are determined; the objects are determined as nodes, and the spatial and hierarchical relationships are determined as edges; Based on nodes and edges, construct visual scene hierarchical graphs of different layers.

[0079] In some optional embodiments, the visual scene hierarchy diagram includes a floor layer, a room layer, a non-moving object layer, and a moving object layer; wherein, The floor layer is the highest layer of the visual scene hierarchy graph, and the floor layer includes multiple room layers; The room layer refers to each room in the indoor environment, and each room layer contains a non-moving object layer and a moving object layer; The non-movable object layer is the immovable objects in the room; The moving object layer is the movable objects in the room.

[0080] In some optional implementations, the instruction mapping unit 105 includes: Obtaining the user's natural language instructions and identifying keywords in the natural language instructions; Determine the inquiry object and inquiry relationship corresponding to the natural language instruction based on the keywords; Map the query objects and query relations to the layers corresponding to the visual scene hierarchy graph, and generate a standardized language corresponding to the natural language instructions.

[0081] In some optional implementations, during the process of executing the optimal action path, the system further includes: Determine whether the current position meets the visual field requirements according to the preset visual perception conditions; If the field of view requirement is not met, adjust the field of view angle until the field of view requirement is met.

[0082] In some optional implementations, whether the current position meets the field of view requirement is determined by judging through a multimodal large model; wherein the multimodal large model includes at least one of ChatGPT and LLM.

[0083] In some optional embodiments, the system further comprises: In the case where the natural language instruction includes an object attribute indicating a target object, the LLMs model is used to determine a predicted perception result according to the object attribute; wherein the predicted perception result is used to characterize long-range perception or short-range perception; Based on the predicted perception results, an action plan is generated to guide the embodied agent to approach the target object.

[0084] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0085] In this embodiment, a visual language structured space planning system is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0086] See also Figure 7 , Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0087] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0088] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0089] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0090] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0091] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0092] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0093] The systems and units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0094] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods and systems. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods and systems according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0100] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0101] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0102] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A planning method for a visual language structured space, characterized in that: Applied to an embodied intelligent body, the method comprises: acquiring in real time an image data stream of an indoor environment and a current position of the embodied intelligent body; Constructing a visual scene hierarchy diagram corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy diagram represents the spatial hierarchy and mutual relationship of objects in the indoor environment; Mapping the user's natural language instructions to the layer corresponding to the visual scene hierarchy diagram, and generating a standardized language corresponding to the natural language instructions; An optimal action path is determined based on the standardized language and the current position, and the optimal action path is executed until a target answer corresponding to the natural language instruction is obtained.

2. The method according to claim 1, characterized in that: The step of constructing a visual scene hierarchy diagram corresponding to the indoor environment based on the image data stream includes: Determining objects at different levels in the indoor environment and spatial relationships and hierarchical relationships between objects according to the image data stream; Determine the object as a node, and the spatial relationship and the hierarchical relationship as edges; The visual scene hierarchy graph of different layers is constructed according to the nodes and the edges.

3. The method according to claim 1 or 2, characterized in that: The visual scene hierarchy diagram includes a floor layer, a room layer, a non-moving object layer and a moving object layer; wherein, The floor layer is the highest layer of the visual scene hierarchy diagram, and the floor layer includes a plurality of room layers; The room layers are the rooms in the indoor environment, and each of the room layers includes the non-moving object layer and the moving object layer; The non-movable object layer is the non-movable objects in the room; The moving object layer is movable objects in the room.

4. The method according to claim 1, characterized in that The mapping of the user's natural language instruction to the layer corresponding to the visual scene hierarchy diagram to generate a standardized language corresponding to the natural language instruction includes: Obtaining a natural language instruction from a user and identifying keywords of the natural language instruction; Determining, based on the keywords, an inquiry object and an inquiry relationship corresponding to the natural language instruction; The query object and the query relationship are mapped to a layer corresponding to the visual scene hierarchy diagram, and a standardized language corresponding to the natural language instruction is generated.

5. The method according to claim 1, characterized in that In the process of executing the optimal action path, the method further includes: Determining whether the current position meets the visual field requirement according to a preset visual perception condition; When the field of view requirement is not met, the field of view angle is adjusted until the field of view requirement is met.

6. The method according to claim 5, characterized in that The determination of whether the current position meets the field of view requirement is performed by judging through a multimodal large model; wherein the multimodal large model includes at least one of ChatGPT and LLM.

7. The method according to claim 1, characterized in that The method further comprises: In the case where the natural language instruction includes an object attribute indicating a target object, a pre-judgment perception result is determined according to the object attribute using an LLMs model; wherein the pre-judgment perception result is used to characterize long-range perception or short-range perception; Based on the predicted perception results, an action plan is generated to guide the embodied intelligent agent to approach the target object.

8. A planning system for visual language structured space, characterized in that: Applied to an embodied intelligent body, the system comprises: a data acquisition unit for acquiring an image data stream of an indoor environment and a current position of the embodied intelligent body in real time; A graph construction unit, configured to construct a visual scene hierarchy graph corresponding to the indoor environment based on the image data stream; wherein the visual scene hierarchy graph represents the spatial hierarchical structure and mutual relationship of objects in the indoor environment; An instruction mapping unit, used to map the user's natural language instruction to a layer corresponding to the visual scene hierarchy diagram, and generate a standardized language corresponding to the natural language instruction; A path determination unit is used to determine an optimal action path according to the standardized language and the current position, and execute the optimal action path until a target answer corresponding to the natural language instruction is obtained.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the visual language structured space planning method described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the planning method of the visual language structured space according to any one of claims 1 to 7.

Images