Household robot article searching method and system oriented to unknown environment

Abstract

The invention discloses an unknown environment-oriented household robot article searching method and system, and belongs to the technical field of robots. The method comprises the following steps: constructing a multi-dimensional map comprising a semantic map, a historical track map and a visual correlation map; evaluating and determining an optimal leading-edge exploration point based on a perception-prior-behavior multi-dimensional scoring mechanism; carrying out local path planning by using a fast marching method, and generating a navigation action to realize autonomous navigation; and judging the existence of the target object through a dual verification mechanism combining semantics and vision. According to the system, by fusing a multi-dimensional map and a multi-dimensional scoring strategy, the efficiency, the success rate and the intelligent level of article search of the robot in an unknown home environment are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a method and system for searching for items in unknown environments using a home robot. Background Technology

[0002] With the continuous development of embodied intelligence technology, robots are increasingly being used in daily family life. Item search, as one of their core functions, requires the rapid location of items needed for a task. However, the home environment is generally highly dynamic and unstructured. When a robot first enters or faces environmental changes, it often lacks complete prior knowledge of the environment, meaning it is in an unknown environment. In this scenario, the robot needs to explore and locate the target item step by step based on user instructions and real-time observation information. This places higher demands on the robot's environmental perception and decision-making capabilities, becoming a key technological challenge in the field of embodied intelligence.

[0003] Existing object search technologies for unknown environments can be broadly categorized into two types based on system modeling: The first type is the end-to-end approach. This method encodes perceived and target information into a unified model, directly outputting action decisions through training, forming an overall input-output mapping. Reinforcement learning or imitation learning algorithms are typically used to optimize model parameters. However, due to its high dependence on training samples, the model is prone to undergeneralization and overfitting in new environments. Furthermore, training is often conducted in simulation environments, which, limited by the domain differences between simulation and real-world home environments, results in insufficient transfer and adaptation capabilities in practical applications. The second type is the modular approach. This method divides the task into sub-modules such as a mapping module, a policy module, and a path planning module. First, the mapping module models the observed information, generating a scene graph or semantic map. Then, the policy module infers long-term goals, and finally, the path planning module generates local paths and executes actions. The modular approach offers interpretability and flexibility, facilitating system expansion and transfer. However, existing modular methods mostly use fixed template languages ​​or object labels as input, without considering the flexibility and ambiguity of natural language instructions in real home scenarios. In addition, related methods usually focus on high-level semantic association modeling, which makes it difficult to fully capture the complex spatial layout and occlusion details in home scenarios, resulting in insufficient extraction and utilization of fine-grained visual information.

[0004] In summary, existing technologies have three major bottlenecks: 1. Insufficient adaptability of models to new environments; 2. Limited ability to understand and process natural language instructions; 3. Insufficient utilization of fine-grained visual information. Summary of the Invention

[0005] The purpose of this invention is to provide a method for searching for objects in unknown environments using a home robot. This method constructs a multi-dimensional map, integrates fine-grained observation scores from a visual model, prior probabilities of proposed targets, geometric distance information, and historical motion trajectories, and jointly infers high-confidence frontier exploration points. Based on this, a fast-movement method is employed to achieve efficient local path planning and action execution, and a dual verification mechanism combining semantic maps and visual models is used to determine the existence of target objects. This improves the robot's autonomous search efficiency and accuracy in unknown home environments, thereby solving at least one of the technical problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows: This invention provides a method for searching for items in a home robot in an unknown environment, comprising the following steps: Step S1, Construct a multi-dimensional map: robot-based RGB-D Observation and pose information are used to construct a semantic map. M t Historical trajectory map H t Visual correlation map V t Multidimensional map; Step S2, Frontier Point Selection: Based on the perception-priority-behavior multi-dimensional scoring mechanism, select the top 4 connected regions with the largest areas from the boundaries between explored and unexplored regions. F k Calculate the centroid of each connected component. f k Perception rating Prior scoring and behavioral scores The centroid with the highest overall score is selected as the target frontier point. f ; Step S3, Local Motion Planning: Calculate the motion from the leading edge using the fast travel method. f Based on the principles of safety and navigation efficiency, a path point is selected from the navigation cost field to the robot's current position to generate robot actions. a t+1 ,in a∈A , A For the robot's motion space; Step S4, Target Object Verification: The existence of the target object in the current field of view is determined through a dual verification mechanism that combines semantics and vision.

[0007] Optionally, in step S1, a semantic map is constructed. M t include: Parsing user commands using a large language model C ,generate n Target items proposed O d,i (i≤n) ; At time step t, the RGB-D image is processed through semantic segmentation, point cloud projection, coordinate transformation, and mapping operations to construct a shape of... (n+m)×H×W semantic map, in which m The number of common static objects in a household. H×W This refers to the map size.

[0008] Optionally, in step S1, a historical trajectory map is constructed. H t include: Record time steps 0 arrive t Obstacle projections, explored areas, and robot historical coordinates; The map shape is 3×H×W The obstacle map is limited to the projection of objects within a preset height range.

[0009] Optionally, in step S1, a visual relevance map is constructed. V t include: Based on semantic segmentation results, region bounding rectangles and numbers are labeled on the RGB image to form a labeled augmented image. I m ; Will I m Relevance alert information P and user instructions C Input a multimodal large language model and output the relevance score at image region l. s l ; The scores are mapped to a two-dimensional map, and the overlapping areas are weighted by area ratio and updated before each frontier point selection.

[0010] Optionally, in step S2, connected components are filtered. F k include: Obstacle maps are inflated to increase safe distances; Extract the largest connected component in the explored area, and obtain the frontier area by the difference between the explored area and the obstacle map; Connectivity analysis was performed on the frontier region, and the top four connected components with the largest areas were selected. F k .

[0011] Optionally, in step S2, perception scoring The calculation method is as follows: ; That is, take the connected components F k The highest score in the visual correlation map.

[0012] Optionally, in step S2, prior scoring The calculation method is as follows: Based on observed static objects O e,j (j≤m) By combining ontology knowledge of room-object relationships, connected components are inferred. F k Most likely room type R ; Based on the ConceptNet knowledge base, the co-occurrence probability N(·) of objects is calculated using Bayes' theorem: ; ; ; The final scoring formula is: ; in s This is a smoothing factor.

[0013] Optionally, in step S2, behavioral scoring The calculation method is as follows: ; in d k argmax(·) is the normalized centroid distance, and argmax(·) is the normalized maximum value of the sum of grid points within the connected domain of the trajectory map.

[0014] Optionally, in step S4, the dual authentication mechanism includes: Semantic determination: Check if there are non-empty values ​​in the first n channels of the semantic map; Visual Judgment: The scene description is generated through reflective chain-based prompting engineering, the existence of the target is analyzed, and reflective location details are verified.

[0015] The present invention also provides a home robot item search system for performing the method in an unknown environment, comprising: Multidimensional map building module: used for real-time generation of semantic maps M t Historical trajectory map H t and visual correlation maps V t; Frontier Decision Module: Outputs target frontier points based on a perception-prior-behavior multi-dimensional scoring mechanism. f ; Path planning module: Uses a rapid movement method to generate navigation paths and robot motion commands; Target verification module: Performs both semantic and visual verification to confirm the existence of the target object.

[0016] Compared with the prior art, the advantages of this invention are as follows: 1. This invention comprehensively represents environmental information by constructing a multi-dimensional map structure that integrates semantic maps, historical trajectory maps, and visual relevance maps. Specifically, the semantic map resolves the ambiguity of natural language commands (e.g., "I'm thirsty" corresponds to targets like water cups or beverages that meet the command requirements); the historical trajectory map records obstacles and exploration trajectories, ensuring safe and efficient path construction; and the visual relevance map utilizes a multimodal large language model to capture the correlation between scene visual details and commands at a fine-grained level (e.g., region-weighted score mapping). The integration of these three significantly enhances the representation of regions related to user commands, improving the robot's accuracy in understanding environmental states and task objectives.

[0017] 2. This invention is based on a perception-priority-behavior multi-dimensional scoring mechanism to comprehensively evaluate the exploration value of connected components: perception score ( Extract the maximum regional correlation value from the visual correlation map; prior score ( Combining ontology knowledge of room-object relationships with ConceptNet to calculate the co-occurrence probability of proposed target objects (introducing a smoothing factor to avoid the zero-probability problem); behavior scoring ( It integrates distance scores and trajectory scores to encourage exploration of new areas. By using joint reasoning to select the optimal frontier point, it significantly improves the accuracy of path planning and the efficiency of item search.

[0018] 3. This invention designs a dual verification mechanism combining semantics and vision: Semantic determination: based on the semantic map... n The system detects the existence of proposed targets through channel detection; visual judgment involves generating scene descriptions, analyzing target matching, and verifying location details through reflective chain-like prompting engineering. This mechanism accurately determines the existence of target objects, effectively avoiding missing valid targets and improving the success rate and reliability of searches in unknown home environments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 A flowchart illustrating a method for searching items in an unknown environment using a home robot, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a home robot item search method for unknown environments provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] Please see Figure 1 and Figure 2 As shown, this embodiment of the invention provides a method for searching for items in a home robot in an unknown environment, including the following steps: Step S1, Construct a multidimensional map: Based on the robot's RGB-D observations and pose information, construct a map that includes semantic information. M t Historical trajectory map H t Visual correlation map V t Multidimensional map; Step S2, Frontier Point Selection: Based on the perception-priority-behavior multi-dimensional scoring mechanism, select the top 4 connected regions with the largest areas from the boundaries between explored and unexplored regions. F k Calculate the centroid of each connected component. f k Perception rating Prior scoring and behavioral scores The centroid with the highest overall score is selected as the target frontier point. f ; Step S3, Local Motion Planning: Calculate the motion from the leading edge using the fast travel method. f Based on the principles of safety and navigation efficiency, a path point is selected from the navigation cost field to the robot's current position to generate robot actions. a t+1 ,in a∈A , A Provide the robot's motion space; Step S4, Target Object Verification: The existence of the target object in the current field of view is determined through a dual verification mechanism that combines semantics and vision.

[0023] In step S1, a semantic map is constructed. M t include: Parsing user commands using a large language model C ,generate n Target items proposed O d,i (i≤n) For example, if the user's instruction is "I'm thirsty," the system might suggest various target objects such as water cups, beverages, and milk. n indivual; At time step t The RGB-D image is processed through semantic segmentation, point cloud projection, coordinate transformation, and mapping operations to construct a shape of... (n+m)×H×W semantic map, in which m The number of common static objects in a household. H×W This refers to the map size.

[0024] Building a historical trajectory map H t include: Record time steps 0 arrive t Obstacle projections, explored areas, and robot historical coordinates; The map shape is 3×H×W The obstacle map is limited to the projection of objects within a preset height range.

[0025] Constructing a visual relevance map V t include: Based on semantic segmentation results, region bounding rectangles and numbers are labeled on the RGB image to form a labeled augmented image. I m ; Will I m Relevance alert information P and user instructions C Input a multimodal large language model and output the relevance score at image region l. sl The process is as follows: ; Where VLM(·) is a multimodal large language model; rl For image I m The l-th marked region, s l for r l Visual relevance score corresponding to the region; The scores are mapped to a two-dimensional map, and the overlapping areas are weighted by area ratio and updated before each frontier point selection.

[0026] In step S2, connected components are filtered. F k include: Obstacle maps are inflated to increase safe distances; Extract the largest connected component in the explored area, and obtain the frontier area by the difference between the explored area and the obstacle map; Connectivity analysis was performed on the frontier region, and the top four connected components with the largest areas were selected. F k .

[0027] Perception rating The calculation method is as follows: ; That is, take the connected components F k The highest score in the visual correlation map.

[0028] Prior scoring The calculation method is as follows: Based on observed static objects O e,j (j≤m) By combining ontology knowledge of room-object relationships, connected components are inferred. F k Most likely room type R ; Based on the ConceptNet knowledge base, the co-occurrence probability N(·) of objects is calculated using Bayes' theorem: ; ; The final scoring formula is: ; in s As a smoothing factor, by introducing a smoothing factors To prevent uncertainty caused by a probability of zero.

[0029] Behavioral rating The calculation method is as follows: ; in d k argmax(·) is the normalized centroid distance, and argmax(·) is the normalized maximum value of the sum of grid points within the connected domain of the trajectory map.

[0030] In step S3, robot actions are generated. a t+1 At time, action space A include: movehead: Move forward 0.25m; RotateLeft / RotateRight: Turn left / Turn right; LookDown / LookUp: looking down / looking up; Stop: to cease.

[0031] In step S4, the dual verification mechanism includes: Semantic determination: Before checking the semantic map n If each channel has a non-empty value, it indicates that there is at least one proposed target object related to the user command within the current field of view. Visual Judgment: The scene description is generated through reflective chain-based prompting engineering, the existence of the target is analyzed, and reflective location details are verified.

[0032] The present invention also provides a home robot object search system for unknown environments for implementing the method described above, including a multi-dimensional map construction module, a frontier point decision module, a path planning module, and a target verification module.

[0033] The multidimensional map building module is used to generate semantic maps in real time. M t Historical trajectory map H t and visual correlation maps V t。

[0034] The frontier decision-making module outputs the target frontier point based on a perception-priority-behavior multi-dimensional scoring mechanism. f .

[0035] The path planning module uses a rapid movement method to generate navigation paths and robot action commands.

[0036] The target verification module performs both semantic and visual verification to confirm the existence of the target object.

[0037] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0038] Furthermore, it should be noted that the scope of the methods and systems in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0039] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for searching for items in a home robot in an unknown environment, characterized in that, Includes the following steps: Step S1, Construct a multi-dimensional map: Based on the robot's RGB-D observations and pose information, construct a map that includes semantic information. M t Historical trajectory map H t Visual correlation map V t Multidimensional map; Step S2, Frontier Point Selection: Based on the perception-priority-behavior multi-dimensional scoring mechanism, select the top 4 connected regions with the largest areas from the boundaries between the explored and unexplored regions. F k Calculate the centroid of each connected component. f k Perception rating Prior scoring and behavioral scores The centroid with the highest overall score is selected as the target frontier point. f ; Step S3, Local Motion Planning: Calculate the motion from the leading edge using the fast travel method. f Based on the principles of safety and navigation efficiency, a path point is selected from the navigation cost field to the robot's current position to generate robot actions. a t+1 ,in a∈A , A Provide the robot's motion space; Step S4, Target Object Verification: The existence of the target object in the current field of view is determined through a dual verification mechanism that combines semantics and vision.

2. The method according to claim 1, characterized in that, In step S1, a semantic map is constructed. M t include: Parsing user commands using a large language model C ,generate n Target items proposed O d,i (i≤n) ; At time step t The RGB-D image is processed through semantic segmentation, point cloud projection, coordinate transformation, and mapping operations to construct a shape of... (n+ m)×H×W semantic map, in which m The number of common static objects in a household. H×W This refers to the map size.

3. The method according to claim 1, characterized in that, In step S1, a historical trajectory map is constructed. H t include: Record time steps 0 arrive t Obstacle projections, explored areas, and robot historical coordinates; The map shape is 3×H×W The obstacle map is limited to the projection of objects within a preset height range.

4. The method according to claim 1, characterized in that, In step S1, a visual relevance map is constructed. V t include: Based on semantic segmentation results, region bounding rectangles and numbers are labeled on the RGB image to form a labeled augmented image. I m ; Will I m Relevance alert information P and user instructions C Input a multimodal large language model and output the relevance score at image region l. s l ; The scores are mapped to a two-dimensional map, and the overlapping areas are weighted by area ratio and updated before each frontier point selection.

5. The method according to claim 1, characterized in that, In step S2, connected components are filtered. F k include: Obstacle maps are inflated to increase safe distances; Extract the largest connected component in the explored area, and obtain the frontier area by the difference between the explored area and the obstacle map; Connectivity analysis was performed on the frontier region, and the top four connected components with the largest areas were selected. F k .

6. The method according to claim 1, characterized in that, In step S2, perception scoring The calculation method is as follows: ； That is, take the connected components F k The highest score in the visual correlation map.

7. The method according to claim 1, characterized in that, In step S2, prior scoring The calculation method is as follows: Based on observed static objects O e,j (j≤m) By combining ontology knowledge of room-object relationships, connected components are inferred. F k Most likely room type R ; Based on the ConceptNet knowledge base, the co-occurrence probability N(·) of objects is calculated using Bayes' theorem: ； ； ； The final scoring formula is: ； in s This is a smoothing factor.

8. The method according to claim 1, characterized in that, In step S2, behavioral scoring The calculation method is as follows: ； in d k argmax(·) is the normalized centroid distance, and argmax(·) is the normalized maximum value of the sum of grid points within the connected domain of the trajectory map.

9. The method according to claim 1, characterized in that, In step S4, the dual verification mechanism includes: Semantic determination: Check if there are non-empty values ​​in the first n channels of the semantic map; Visual Judgment: The scene description is generated through reflective chain-based prompting engineering, the existence of the target is analyzed, and reflective location details are verified.

10. A home robot item search system for performing the method as described in any one of claims 1-9 in an unknown environment, characterized in that, include: Multidimensional map building module: used for real-time generation of semantic maps M t Historical trajectory map H t and visual correlation maps V t ; Frontier Decision Module: Outputs target frontier points based on a multi-dimensional scoring mechanism of perception-priority-behavior. f ; Path planning module: Uses a rapid movement method to generate navigation paths and robot motion commands; Target verification module: Performs both semantic and visual verification to confirm the existence of the target object.

Images