Dynamic environment object searching method based on common sense and memory enhanced reasoning

Abstract

The invention discloses a dynamic environment object searching method based on common sense and memory enhanced reasoning, and belongs to the technical field of robot object searching. The method comprises the steps that a robot obtains an article search task and historical interaction data input by a current user; rGB-D images of the surrounding environment are collected in real time, and various object instances are recognized; establishing a functional three-dimensional scene graph comprising a container layer, a movable object layer and a region layer according to the detected environment and object information; a hierarchical action space comprising a macroscopic level and a microscopic level is designed, a macroscopic action planning cue word is input into LLM according to current scene information, and the optimal next-step action of the robot is obtained; and repeating the steps until the task is completed or failed. According to the method, functional three-dimensional scene graph construction, layered action planning and a dynamic environment response mechanism are organically combined, and efficient target searching in a dynamic environment can be achieved.

Description

Technical Field

[0001] This invention belongs to the field of robot object search technology, specifically, it refers to a dynamic environment object search method based on common sense and memory-enhanced reasoning. Background Technology

[0002] Existing robot object search technologies mainly include the following schemes: (1) Heuristic search method: The robot searches for objects according to preset rules and priority order, usually using breadth-first or depth-first search strategies, checking containers and areas in the environment one by one. (2) Learning-based search method: The robot is trained to identify the possible locations of objects using deep learning models, and learns the relationship between objects and the environment through historical data. (3) Language model-based search method: The robot uses large language models to understand search instructions, but is mainly limited to object localization in static environments.

[0003] However, the above technologies have the following drawbacks: (1) Lack of functional area understanding: Existing technologies cannot effectively identify and utilize the functional divisions of the environment, such as storage areas and leisure areas, resulting in the inability to effectively narrow the search range and low search efficiency. (2) Limitation of atomic operations: Existing methods rely on atomic actions and can only operate on one container at a time. When facing multi-container environments, the search speed is slow and parallel or batch processing cannot be achieved. (3) Static environment assumption: Existing technologies assume that the position of objects is relatively fixed and lack the ability to adapt to dynamic environments. They cannot handle situations such as objects being moved or repositioned. (4) Insufficient memory integration: They cannot effectively integrate historical search memory and common sense reasoning, making it difficult to quickly adapt to and relocate target objects when the environment changes. Summary of the Invention

[0004] To address the limitations of existing object search technologies, such as insufficient understanding of functional regions, poor dynamic adaptability, and low search efficiency, and to solve the problem of object search in dynamic environments, this invention proposes a dynamic environment object search method based on common sense and memory-enhanced reasoning.

[0005] The dynamic environment object search method based on common sense and memory-enhanced reasoning of the present invention performs the following steps:

[0006] Step 1: The robot acquires the current user's item search task and historical interaction data;

[0007] Step 2: The robot obtains its own position, uses its onboard sensing devices to collect RGB-D images of the surrounding environment, identifies and segments various object instances from the images, and obtains the attributes of each identified object instance, including semantic labels and 3D point cloud coordinates.

[0008] Step 3: The robot builds a functional 3D scene map containing container layers, movable object layers, and region layers based on the detected environment and object information;

[0009] First, the robot constructs a BEV occupancy map and generates a Voronoi navigation map. Then, the robot uses LLM to classify objects into two categories: containers and movable objects, and identifies containers and movable objects with support relationships. Next, LLM is used to perform region clustering based on spatial proximity and semantic relationships to obtain the containers contained in each region and the region's functional description. The region division results are optimized, including merging single-container regions to reduce the total number of regions and correcting container location classification errors.

[0010] Step 4: Based on the designed hierarchical motion space and the current F3DSG information, input the macro motion planning prompts into the LLM, and the LLM outputs the robot's best next action;

[0011] The hierarchical action space includes macro-level and micro-level actions; among which, the macro-level action instructions include: (1) explore(room_name), which means navigating to an unexplored frontier point in a specified room; (2) open(room_name, door_name), which means navigating to the location of a specified door in a specified room, performing the door opening operation, and entering the space behind the door; (3) inspect(room_name, area_name), which means navigating to each container in a specified area of ​​a specified room in the order of inspection, and performing the opening operation after navigation if the container is closed and operable; (4) done(), which means terminating the task execution when the target object is successfully located or the continuous action execution fails; the micro-level action instructions refer to determining the inspection order of each container in the specified area when performing the inspect action, and the inspection order is determined by calculating the comprehensive selection score of the container.

[0012] Step 5: After the robot completes each macroscopic action, repeat steps 2 to 4; when the target object is detected, the robot stops moving and the task is successful; if no target object is found, and the maximum number of interactions is reached or there is no next action to be performed, the task fails.

[0013] In step 4, each container to be inspected within the area is... Calculate the overall selection score Choose the container with the highest rating. As the target of the next navigation and inspection; among which, the computing container Overall selection score as follows:

[0014] ;

[0015] ;

[0016] ;

[0017] The A* algorithm is used to calculate the distance from the robot's current position to the container. Path distance to the nearest Voronoi node The path distance components are obtained after normalization. , , These represent the path distances from the robot's current position to the nearest and farthest Voronoi nodes in the navigation map, respectively; calculated using the SBERT model. Cosine similarity between container name and target object description The semantic relevance components are obtained after normalization. , , These represent the lowest and highest cosine similarity between the container name and the target name, respectively. To search for risk factors, it means that the target object does not exist except for The joint probability in other containers outside, The target object is in the container The probability of it; These are the weighted coefficients of the path distance component, the semantic relevance component, and the search risk factor, respectively. It is the collection of containers to be inspected.

[0018] The dynamic environment object search method based on common sense and memory-enhanced reasoning of the present invention is applicable to item search tasks without initial map information, as well as item search tasks with initial map information and known initial position of the target object. For an item search task with initial map information and known initial location of the target object, the robot obtains the F3DSG of a portion of the scene and the initial location of the target object from the memory data. First, it queries the F3DSG to find the initial observed location of the target object, including: (1) first locating the area where the target object is initially located. If there are more than one candidate area, the robot prioritizes the candidate area based on the A* path distance from the current location of the robot to the nearest container in each candidate area; (2) identifying the initial container of the target object and recording the name, spatial location and associated information of the container in the F3DSG. Then, the robot navigates to the area where the target object is initially located, checks the initial container, and sets the check order based on the comprehensive selection score of other containers in the area. The robot navigates to each container location one by one and checks the containers. The robot compares the currently observed movable objects with the historical information recorded in the F3DSG in real time. If the movable objects on the container change, the F3DSG is dynamically updated. When the target object is found, the task is completed. Otherwise, the above steps 2-5 are continued.

[0019] The advantages and positive effects of this invention are as follows:

[0020] (1) The dynamic environment object search method of the present invention intelligently classifies objects in the environment through a large language model, performs regional clustering by combining object name, spatial coordinates and carrying object information, infers the functional attributes of each region and generates an environmental representation with semantic understanding ability, and corrects container classification errors by merging single container regions and distance threshold judgment, optimizes region division, and establishes a functional three-dimensional scene map containing container layer, movable object layer and region layer, transforming the environment from traditional geometric space segmentation to functional region segmentation, overcoming the limitation of existing 3D scene maps that only provide coarse-grained spatial segmentation and cannot identify functional regions, enabling robots to understand the functional semantics of regions such as "rest area, storage area" like humans, thereby narrowing the search range of target objects.

[0021] (2) The dynamic environment object search method of the present invention designs a two-level action space of macro-region-level actions and micro-container-level actions, and constructs a hierarchical action planning framework; at the macro level, search instructions are generated in units of functional regions, which reduces the number of calls to the large language model and the accumulation of errors; at the micro level, the container inspection order is optimized through a comprehensive scoring mechanism. This scoring mechanism considers three dimensions at the same time: path distance, semantic relevance and search risk, which avoids the problem of low search efficiency caused by traditional methods relying on only a single indicator.

[0022] (3) The dynamic environment object search method of the present invention establishes a dynamic environment response mechanism based on memory and common sense, maintains environmental memory to record the original position information of objects, locates the initial area and initial container of the target object through this information, prioritizes checking the initial container, and uses a large language model to infer the possible relocation area of ​​the object when the target is not found in the initial area. This method is based on the observation that movable objects tend to move in specific functional areas in daily life. By combining common sense about object use and functional area correlation analysis, it can quickly infer the possible new position of the target object when the environment changes dynamically, which significantly improves the success rate of target relocation in dynamic environments.

[0023] (4) The dynamic environment object search method of the present invention integrates F3DSG construction, hierarchical motion planning and dynamic environment response mechanism, enabling the robot to integrate real-time perception, historical memory and functional common sense information, and achieve efficient target search in some observable dynamic environments. Through comparative experiments with advanced methods, the present invention has significant advantages in scene understanding accuracy, search success rate, interaction efficiency and environmental adaptability. Attached Figure Description

[0024] Figure 1 This is a flowchart of the dynamic environment object search method of the present invention for searching for items without an initial map;

[0025] Figure 2 This is a flowchart of the dynamic environment object search method of the present invention for searching for objects when there is a partial initial map and the initial position of the target object is known;

[0026] Figure 3 This is a framework diagram of the robot dynamic environment object search method constructed according to an embodiment of the present invention;

[0027] Figure 4 This is an example diagram illustrating how the present invention uses LLM to classify objects into two categories: containers and movable objects.

[0028] Figure 5 This is an example diagram illustrating how LLM is used to perform region clustering of objects in an embodiment of the present invention;

[0029] Figure 6 This is an example diagram of the macro-action planning prompts for the LLM constructed according to an embodiment of the present invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0031] The dynamic environment object search method based on common sense and memory-enhanced reasoning in this invention can handle two types of item search tasks, depending on whether initial map information is available. For item search tasks without initial map information, the execution flow of this invention is as follows: Figure 1 As shown. For tasks with partial initial map information and known initial target positions, the execution flow of the method of this invention is as follows. Figure 2 As shown in the figure, a framework diagram of the robot dynamic environment object search method implemented in this embodiment of the invention is shown below. Figure 3 As shown.

[0032] For item search tasks without initial map information, one implementation step of the dynamic environment object search method based on common sense and memory-enhanced reasoning in this invention is as follows: Figure 1 As shown, it includes four steps.

[0033] Step 1: Initialize the scene. The robot rotates once in place to obtain the initial scene information.

[0034] The robot, equipped with an RGBD camera, continuously moves and captures images in its environment. The Semantic-SAM image segmentation model is used to identify and segment instances in each frame of the RGB image, resulting in semantically masked image images. The RGBD camera simultaneously captures color images containing three channels (Red (R), Green (G), and Blue (B)) and point clouds representing depth information (the distance between the camera and the object's surface). Semantic-SAM combines semantic segmentation and general object segmentation, automatically identifying and segmenting various objects in the image, while assigning semantic labels such as "table" and "chair" to each object. Combining the segmentation mask of the RGB image with the corresponding depth information, the pixel coordinates of each object instance are converted into 3D point cloud coordinates. A feature-based point cloud registration method is used to accurately align and fuse the object point clouds from different viewpoints, constructing a complete 3D object model. Finally, structured data records are calculated for each object in the scene, including its object ID, category label (semantic label), center coordinates, and bounding box vertex coordinates.

[0035] Step 2: Using basic scene information and common sense about the functionality of items, the LLM infers the F3DSG. Step 2 of this embodiment includes the following four sub-steps.

[0036] Step 21: Navigation Map Generation. This invention constructs a Bird's-Eye-View (BEV) occupancy map, which reflects the distribution of explored areas and obstacles in the environment. The occupancy map is a rasterized map that identifies each small cell as either an obstacle, passable free space, or an unexplored area. Based on the free space areas in this BEV occupancy map, a Voronoi (Thieson polygon) navigation map is generated, which is used for robot path planning and navigation. Simultaneously, frontiers are defined as the boundaries between explored and unexplored areas; these frontiers guide the robot in active environmental exploration. Furthermore, for each object in the scene, its nearest Voronoi node is calculated to facilitate robot navigation and manipulation.

[0037] Step 22: Perform object classification and support relationship recognition. In this embodiment of the invention, all objects in the identified scene, set O, are first classified into containers using LLM. and movable objects Two basic categories, example prompt words as follows Figure 4 As shown. Then, the support relationships between objects are identified through spatial geometric relationships. The support relationship is determined based on the position of the bounding boxes. If the bounding boxes of two objects meet the condition that they overlap in the XY plane and are close to or overlap in the Z axis, they are considered to have a potential support relationship and are added to the set. In the middle. Furthermore, if a pair of object... Satisfy the formula This forms a valid container – a movable object pair – and is contained within the collection. In this context, all containers that do not satisfy this relationship are categorized into the set of isolated containers. Movable items that do not satisfy this relationship are classified into other item sets. .

[0038] Figure 4 The example uses LLM to categorize objects [yogurt, coffee table, cup, armchair, towel, desktop cabinet, banana, refrigerator, apple, oven, file, soap, washing machine, grapes] into two categories: containers and movable objects.

[0039] Step 23: Implement LLM-based region clustering and functional inference. In this embodiment of the invention, the names, spatial coordinates, and information about the movable objects they carry are input into the LLM. The LLM performs region clustering based on spatial proximity and semantic relationships. It records information about movable objects carried by the container, collection The movable object information carried by the container is set to None. This invention's system uses a structured prompt template, requiring the LLM output of the region division results, including a list of containers contained in each region, a functional description (e.g., "region for cooking"), and a reasoned explanation for the grouping. Examples of prompt words are shown below. Figure 5 As shown. LLM uses its built-in common sense knowledge to automatically infer the functional attributes of each region based on container type, spatial distribution, and semantic relationships of the objects it carries.

[0040] Figure 5 The example provides the names, spatial coordinates, and movable objects carried by all containers in Room 1. Using LLM, region clustering is performed to obtain two regions.

[0041] Step 24: Optimize Region Partitioning Results. To address two main issues in the initial region partitioning generated by LLM: an excessive number of regions leading to increased complexity in subsequent action planning, and containers with abnormal spatial distributions being incorrectly classified into inappropriate regions, this invention optimizes the LLM region partitioning results in two aspects. The first improvement is to reduce the total number of regions by merging single-container regions, thereby reducing the complexity of the search space. The second improvement is to correct location classification errors by reassigning containers with abnormal spatial distributions to more spatially reasonable regions.

[0042] The region partitioning optimization algorithm provided by this invention sets two retention conditions for each container to determine whether it should remain in the original region: (1) The first retention condition applies to containers in a multi-container region. If a container's region contains multiple containers, and the distance between this container and at least one other container in the same region is less than a set threshold, then this container is retained in the original region. This condition ensures that in a multi-container region, containers that are indeed spatially adjacent can continue to remain in the same region. (2) The second retention condition applies to containers in a single-container region. If a container forms a region on its own, and its distance from all other containers is not less than a set threshold, it means that this container should indeed form a region independently, and therefore is retained in the original region. The redistribution rule of the region partitioning optimization algorithm is: for containers that do not meet the above two retention conditions, the container is redistributed to the region containing the container closest to it. This ensures that each container can be assigned to the most spatially reasonable position. The formula of the region partitioning optimization algorithm of this invention is as follows:

[0043] ;

[0044] in This represents the first reservation condition, applicable to containers in a multi-container area. This represents the second reservation condition, applicable to containers in a single-container area. It is a container In the initial region, yes The number of containers in the area; It is a distance threshold, which can be adjusted by the scene size; It is a container and The Euclidean distance between them; It is an optimized container area; It is a collection of all regions. It is one of the areas; Operator selection includes The area of ​​the nearest neighbor.

[0045] To eliminate the influence of container processing order on the optimization results, this invention employs a two-stage optimization strategy: the first stage collects the optimization decisions for all containers, and the second stage executes the optimization decisions for all containers simultaneously. After optimization is complete, Each item is categorized into the area containing the nearest item.

[0046] Step 3: Input the information from F3DSG into the LLM analysis, allowing it to select and execute the optimal next action, continuously acquiring new environmental information during the task. This invention constructs a hierarchical action planning framework, generating search instructions at the macro level based on functional regions, and determining the container inspection order at the micro level through a comprehensive scoring mechanism. This step can be specifically divided into the following two sub-steps.

[0047] Step 31: Design a hierarchical action space. This invention designs a hierarchical action space, including macroscopic and microscopic actions.

[0048] At the macro level, the following high-level action instructions are defined: (1) explore(room_name): Navigate to an unexplored frontier point within a specified room; room_name is the name of the specified room; (2) open(room_name, door_name): Navigate to the location of the specified door in the specified room, perform the door-opening operation, and enter the space behind the door; door_name is the name of the specified door; (3) inspect(room_name, area_name): Navigate sequentially to each container within the specified area of ​​the specified room according to the inspection sequence for observation. If a container is closed and operable, perform the opening operation after navigation; area_name is the name of the specified area; (4) done(): Terminate task execution when one of the following conditions is met: successful location of the target object, or failure of continuous action execution. This macro action space significantly reduces decision complexity and improves search efficiency.

[0049] At the micro level, this invention requires determining the specific inspection order of containers within the target area during the inspection process to optimize search efficiency. To this end, the method of this invention designs a comprehensive scoring mechanism that calculates a selection score for each container to be inspected within the area, selecting the container with the highest score as the next inspection target. Specifically, for containers within the area... Its comprehensive selection score The calculation formula is:

[0050] ;

[0051] ;

[0052] ;

[0053] The A* algorithm is used to calculate the distance from the robot's current position to the container. Path distance to the nearest Voronoi node Then, normalization is performed to obtain the normalized path distance components. The closer the containers are, the higher their path distance component score. and These represent the path distances from the robot's current position to the nearest and farthest Voronoi nodes in the navigation map, respectively. The SBERT (Sentence-BERT) model is used to calculate the cosine similarity between the container name and the target object description. The semantic relevance components were obtained by normalization. Containers with higher semantic relevance score have higher semantic relevance component scores. and These represent the lowest and highest cosine similarity between the container name and the target name, respectively. (Score) It is obtained by weighting three components: path distance component, semantic relevance component, and search risk factor. To search for risk factors, it means that the target object does not exist except for The joint probability of the remaining containers outside the main container. The target object is in the container The probability of being selected. Search risk factors encourage prioritizing the examination of containers that, if skipped, might lead to search failure. These are weighted coefficients for the path distance component, semantic relevance component, and search risk factor, used to balance the impact of the three dimensions: path efficiency, semantic relevance, and search risk. The weighted coefficient values ​​can be adjusted according to actual conditions. Based on the scoring formula, the system, at each micro-action decision, selects from the remaining set of containers to be checked. Choose the container with the highest rating As the target for the next navigation and inspection.

[0054] After checking all containers in a given area, containers within that area are added to a blacklist to avoid duplicate checks. This micro-action planning mechanism effectively combines spatial efficiency, semantic reasoning, and risk assessment, significantly improving the success rate of target search in complex environments.

[0055] Step 32: Design macro-motion planning prompts for LLM.

[0056] The macro-action planning prompts designed in this invention comprise the following core components: First, defining the robot's system role and task objectives; second, listing all available macro-actions and their functional descriptions; third, specifying output format requirements, including analysis, reasoning, and instructions; fourth, providing current scene state information, using a hierarchical structure of "room-area-container-object" to describe the discovered environmental information, including area function descriptions, container lists, and lists of movable objects; fifth, providing historical action execution records and their results; sixth, listing currently executable candidate action options, including: inspectable areas with distance information such as "very close" or "near," rooms with unexplored space, and rooms containing closed doors; and finally, explicitly querying the best next action and providing execution constraints, such as parameters must be selected from a given list, and repeated failed actions may be infeasible. This invention, by constructing macro-action planning prompts, provides information about inspectable areas to a large language model, including the area's functional attributes, explored state, and container types. Based on the current F3DSG and task objective, the LLM is input with the aforementioned macro-action planning prompts. The large language model then determines, based on the current environmental state and task objective, whether there are areas that need to be inspected, or whether exploration-related instructions should be executed. Examples of prompts are as follows: Figure 6 As shown.

[0057] Step 4: After the robot completes an action, it acquires its own position in real time, dynamically perceives the surrounding environment, and repeats steps 2 and 3. When the target object is detected, it stops moving immediately, and the task is successful; if no target object is found and the maximum number of interactions is reached or there is no next action to be performed, the task fails.

[0058] After completing a macroscopic action, the system enters a loop iteration process, repeatedly executing the F3DSG construction and action planning steps. Specifically, after the robot successfully navigates, opens a door, or opens a container, it will obtain new scene observation information, including newly discovered objects and their spatial positional relationships. The system dynamically updates the F3DSG based on this incremental information, integrating the support relationship between the newly detected container and movable objects, regional functional attributes, etc., into the scene graph. Then, the updated F3DSG is input into the large language model again for macroscopic action planning, and the next optimal action is selected and executed. During the entire task execution process, real-time object detection continues to run. Once the vision system detects a target object, it immediately triggers the done() action to terminate the task, and the task is judged as successful. If no target object is found and any of the following termination conditions occur, the task is judged as failed: (1) multiple consecutive action execution failures indicate that there is no feasible next action in the current environment; (2) the preset maximum number of interactions is reached. This closed-loop iteration mechanism enables the robot to continuously integrate new information, update environmental cognition, and optimize search strategies in a partially observable dynamic environment, thereby achieving efficient target object localization.

[0059] For tasks with partial initial map information and known initial locations of target objects, a process for item search using the method of this invention is as follows: Figure 2 As shown, the method includes the following three steps. The method of this invention uses a dynamic environmental response mechanism, retrieves the initial position of the target object through environmental memory, prioritizes checking the original container and dynamically updates the F3DSG, and when the target is not found, uses a large language model to infer potential relocation regions.

[0060] Step 1: Obtain the initial position information of the target object from F3DSG.

[0061] This invention utilizes the robot's environmental memory to retrieve the initial position information of a target object from the F3DSG. Specifically, in the initial exploration phase, the robot has constructed an F3DSG of part of the scene, which records the support relationship between movable objects and containers, as well as the functional area to which the containers belong. When performing a dynamic target search task, the system first queries the F3DSG for the initial observed position information of the target object, including two levels: (1) Region level: locate the functional area where the target object is initially located. If there are multiple candidate regions, prioritize them based on the A* path distance from the robot's current position to the nearest container in each candidate region, and navigate to the nearest candidate region first; (2) Container level: identify the container where the target object is initially stored, and record the name, spatial location, and associated information of the container in the F3DSG. This environmental memory mechanism is based on common sense observation of human object usage habits—movable objects in daily life usually stay within a specific functional area, for example, a TV remote control often moves between the sofa and coffee table in a leisure area, providing prior knowledge for subsequent hierarchical relocation search.

[0062] Step 2: Check the initial area of ​​the target object and dynamically update the scene information.

[0063] After navigating to the initial area of ​​the target object, the robot first checks the initial container, then determines the inspection order according to the comprehensive selection and scoring mechanism defined above, and checks other containers. During the inspection process, the robot navigates to the container location one by one. If the container is an openable closed container, it performs an opening operation and obtains complete observation information of all objects carried on the container. The system compares the currently observed movable objects with the historical information recorded in F3DSG in real time. If changes are detected in the movable objects on the container, such as objects being removed, added, or replaced, the F3DSG is dynamically updated to reflect the current scene configuration. Throughout the inspection process, the vision system continuously performs real-time object detection. Once a target object is detected, the done() action is immediately triggered, the robot stops moving, and the task is considered successful.

[0064] Step 3: If the target object is not found after checking all the initial areas, continue to perform the operations in steps 2, 3 and 4 above, allowing LLM to infer other areas where the target object may exist, or to further explore unexplored scenes, until the task succeeds or fails.

[0065] like Figure 3As shown, in this embodiment of the invention, the robot acquires RGB-D images of the environment through a perception module, extracts semantically meaningful object instance mask images from the images, and acquires its own positioning data in real time to generate basic scene information. Subsequently, the system uses a memory module containing user finger and historical interaction data, and a common sense knowledge module containing object roles and functional attributes as context inputs, and uses a Large Language Model (LLM) as the reasoning core for hierarchical action planning. During this process, the system constructs and dynamically maintains a hierarchical three-dimensional dynamic scene graph (F3DSG) containing rooms, areas, containers, and movable objects, and optimizes the area division based on semantic distance. Finally, based on the reasoning results, a dynamic environment response is generated, driving the robot to perform hierarchical search actions including macro-area guidance and micro-container investigation, and continuously updates the positioning and perception information through a feedback mechanism until the accurate search for the target object is achieved.

[0066] The embodiments of the present invention were compared with state-of-the-art methods in the interactive indoor simulation dataset iGibson and the constructed real-world scene, verifying the significant advantages of the present invention in terms of scene understanding accuracy, search success rate, interaction efficiency and environmental adaptability.

[0067] Regarding scene understanding accuracy, the comparison was primarily made between the geometric classification algorithm (OpenIN) for container classification accuracy and between the door-based segmentation algorithm (MoMa-LLM) and the traditional geometric segmentation algorithm (Hydra) for room segmentation accuracy. See Tables 1 and 2. In the tables, Ours represents the method of this invention.

[0068] Table 1 Container Classification Accuracy

[0069]

[0070] Four metrics were used to compare the classification results: Accuracy (accuracy), Precision (precision), Recall (recall), and F1-Score (F1 score).

[0071] Table 2. Room segmentation (assessed using Veno plot node purity) and classification accuracy

[0072]

[0073] In terms of item search, this invention mainly compares several methods: simple heuristic strategies Random and Greedy, the learning / reinforcement learning method HIMOS, and various schemes based on large language models such as ESC-Interactive, UnstructuredLLM, MoMa-LLM, and their different room segmentation strategy variants, such as MoMa-LLM w / Hydra and MoMa-LLM w / R3DSG. Ablation experiments were conducted on the sub-modules of the method (Ours) of this invention. "Ours w / o area optimization" represents the ablation experiment with the region segmentation optimization algorithm removed from the method of this invention; "Ours w / randdom order" represents the ablation experiment using a random order to check containers; and "Ours w / greedy order" represents the method using a greedy algorithm to set the order of checking containers. Experiments were conducted in static, dynamic, and real-world environments, and the experimental results are shown in Tables 3 to 6.

[0074] Table 3. Item search accuracy and efficiency in static environments

[0075]

[0076] Wherein, SR represents the task success rate; SPL represents the path length weighted success rate; OI represents the number of interactions; and DT represents the path length.

[0077] Table 4 Success rate and efficiency of searching for newly appearing items in dynamic environments (scene information is partially known)

[0078]

[0079] Among them, Level 1-3 represent the level of knowledge about the scene from less to more.

[0080] Table 5. Success rate and efficiency of item relocation in dynamic environments (scene information partially known).

[0081]

[0082] The experiments in the real-world scenario include three tasks: IOS (Item Search), NTS (New Target Search), and TR (Target Relocation), which correspond to the same three tasks in the simulation environment.

[0083] Table 6 Experimental results in real-world scenarios

[0084]

[0085] Experimental results show that the method of this invention significantly outperforms existing methods in most metrics, including container classification, room segmentation, interactive target search, new target search, target relocalization, and real robot experiments, especially demonstrating a substantial improvement over SoTA's MoMa-LLM. In simulated environments, the accuracy of environmental understanding, item search success rate, and efficiency are significantly improved. In real-world environments, it also significantly reduces the number of interactions and task time. Ablation experiments further illustrate that each module, such as region optimization and inspection order strategy, substantially contributes to the final performance.

[0086] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0087] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. A dynamic environment object search method based on common sense and memory-enhanced reasoning, characterized in that, Includes the following steps: Step 1: The robot acquires the current user's item search task and historical interaction data; Step 2: The robot obtains its own position, uses its onboard sensing devices to collect RGB-D images of the surrounding environment, identifies and segments various object instances from the images, and obtains the attributes of each identified object instance, including semantic labels and 3D point cloud coordinates. Step 3: The robot creates a functional 3D scene map (F3DSG) containing container layers, movable object layers, and region layers based on the detected environment and object information; First, the robot constructs a BEV occupancy map and generates a Voronoi navigation map. Then, the robot uses LLM to classify objects into two categories: containers and movable objects, and identifies containers and movable objects with support relationships. Then, LLM is used to perform region clustering based on spatial proximity and semantic relationships to obtain the containers contained in each region and the region's functional description. Optimize the region division results, including: merging single container regions to reduce the total number of regions, and correcting container location classification errors; Step 4: Based on the designed hierarchical motion space and the current F3DSG information, input the macro motion planning prompts into the LLM, and the LLM outputs the robot's best next action; The layered action space includes macro-level and micro-level actions; among which, the macro-level action instructions include: (1) explore(room_name), which means navigating to an unexplored front point in a specified room; (2) open(room_name, door_name), which means navigating to the location of a specified door in a specified room, performing the door opening operation, and entering the space behind the door; (3) inspect(room_name, area_name), which means navigating to each container in a specified area of ​​a specified room in the order of inspection, and performing the opening operation after navigation if the container is closed and operable; (4) done(), which means terminating the task execution when the target object is successfully located or the continuous action fails; the micro-level action instructions refer to determining the inspection order of each container in the specified area when performing the inspect action; Step 5: After the robot completes each macroscopic action, repeat steps 2-4; when the target object is detected, the robot stops moving and the task is successful; if no target object is found, and the maximum number of interactions is reached or there is no next action to be performed, the task fails. Here, BEV represents bird's-eye view, Voronoi represents Thiessen polygon, and LLM represents large language model.

2. The method according to claim 1, characterized in that, Step 2 includes: the robot uses an image segmentation model to identify and segment various object instances from each frame of RGB image acquired by the sensing device to obtain a semantically meaningful object instance mask image; then, based on the mask image and the corresponding depth information, the pixel coordinates of each object instance are converted into three-dimensional point cloud coordinates, and a feature-based point cloud registration method is used to align and fuse the object point clouds from different perspectives to construct a complete three-dimensional model of the object, and the structured data of each object is recorded, including object identifier, semantic label, center coordinates and bounding box vertex coordinates.

3. The method according to claim 1, characterized in that, Step 3 includes: Step 31: The robot constructs a BEV occupancy map based on the detected environment and object information, and generates a Voronoi navigation map; for each object in the scene, it calculates the Voronoi node closest to the object; the BEV occupancy map identifies whether each grid belongs to an obstacle, free space, or an unexplored area; Step 32: Use LLM to classify all objects identified in the current scene into containers. and movable objects Two categories are identified, and the support relationships between objects are then determined through spatial geometric relationships. Container-movable object pairs with support relationships are stored in a set. In the middle, the remaining containers are classified into the set of isolated containers. In the process, the remaining movable items are categorized into other item sets. middle; Step 33: Input the names, spatial coordinates, and movable object information carried by each container into the LLM. The LLM performs region clustering based on spatial proximity and semantic relationship, and outputs a list of containers, functional descriptions, and reasoning explanations for grouping each region. Step 34: Optimize the region division results.

4. The method according to claim 1 or 3, characterized in that, The method for optimizing the region partitioning results in step 3 is as follows: In the first stage, the optimization decisions for all containers are obtained, including: For each container Determine whether the area should remain in the original region based on the following two retention criteria: (1) The first retention condition is: if a container The region contains more than one container, and the containers With at least one other container in the same area The distance is less than the set threshold. Then the container Retain in the original area; (2) The second retention condition is: if a container It forms its own area, and the container With all other containers The distances are all no less than the set threshold. Then the container Retain in the original area; If container If both retention conditions are not met, the container will be... Reassign it to the area containing the nearest container; In the second phase, optimization decisions for all containers are executed simultaneously. After execution, movable items that do not have a support relationship with the containers are categorized into the area of ​​the item closest to them.

5. The method according to claim 1, characterized in that, In step 4, each container to be inspected within the area is... Calculate the overall selection score Choose the container with the highest rating. As the target of the next navigation and inspection; among which, the computing container Overall selection score as follows: ； ； ； The A* algorithm is used to calculate the distance from the robot's current position to the container. Path distance to the nearest Voronoi node The path distance components are obtained after normalization. , , These represent the path distances from the robot's current position to the nearest and farthest Voronoi nodes in the navigation map, respectively; calculated using the SBERT model. Cosine similarity between container name and target object description The semantic relevance components are obtained after normalization. , , These represent the lowest and highest cosine similarity between the container name and the target name, respectively. To search for risk factors, it means that the target object does not exist except for The joint probability in other containers outside, The target object is in the container The probability of it; These are the weighted coefficients of the path distance component, the semantic relevance component, and the search risk factor, respectively. It is the collection of containers to be inspected.

6. The method according to claim 1, characterized in that, In step 4, the macroscopic motion planning prompts designed for LLM include: first, defining the robot's system role and task objectives; second, listing all available macroscopic actions and functional descriptions; then, specifying the LLM output format, which includes three parts: analysis, reasoning, and instructions; and finally, using a hierarchical structure of room-area-container-object to describe the currently discovered environmental information, including area function descriptions, container lists, and lists of movable objects. Then, historical action execution records and target inspection results are provided; Further list the currently available candidate actions, including: areas that can be inspected and their distance information, rooms with unexplored spaces, and rooms with closed doors; Finally, the best next action is requested, and execution constraints are given.

7. The method according to claim 1, characterized in that, In the method described above, when the robot acquires an item search task without initial map information in step 1, in step 2, the robot rotates in place once to obtain initial scene information based on the collected RGB-D image, and then continues to execute steps 3-5.

8. The method according to claim 1, characterized in that, In the method described above, when the robot obtains an item search task with initial map information and known initial location of the target object in step 1, the robot obtains the F3DSG of a part of the scene and the initial location of the target object from historical interaction data. First, it queries the F3DSG to find the initial observed location of the target object, including: (1) first locating the area where the target object is initially located. If there are more than one candidate area, the robot prioritizes the candidate area based on the A* path distance from the current position of the robot to the nearest container in each candidate area, and prioritizes navigation to the nearest candidate area; (2) identifying the initial container of the target object and recording the name, spatial location and association information of the container in the F3DSG; then the robot navigates to the area where the target object is initially located, checks the initial container, and sets the inspection order based on the comprehensive selection score of other containers in the area. The robot navigates to each container location one by one and checks the container; compares the currently observed movable object with the historical information recorded in the F3DSG in real time. If the movable object on the container changes, the F3DSG is dynamically updated; when the target object is checked, the task is completed. Otherwise, continue to execute steps 2-5.

Images