A robot navigation method, device, equipment and medium based on a question and answer system
By employing a question-and-answer system-based robot navigation method, utilizing an RGBD camera and a reinforcement learning model, the problem of cumbersome robot navigation path planning is solved, achieving efficient and accurate navigation path planning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 浙江人形机器人创新中心有限公司
- Filing Date
- 2025-05-27
- Publication Date
- 2026-07-21
AI Technical Summary
The current robot navigation path planning process is cumbersome, consumes a lot of human resources and time, and results in low planning efficiency.
A question-answering system-based approach is adopted, which uses the robot's RGBD camera to collect environmental data, construct a topology map, receive user questions to obtain semantic guidance information, and use reinforcement learning models and fast exploration random tree algorithms for path planning, thereby reducing human intervention.
It improves the efficiency of navigation route planning, enhances the continuity and accuracy of routes, and reduces planning time.
Smart Images
Figure CN120538531B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot navigation technology, and in particular to a robot navigation method, apparatus, device and medium based on a question-and-answer system. Background Technology
[0002] Embodied navigation refers to the process by which a robot navigates its physical environment by perceiving and interacting with its surroundings. Indoor environments often contain complex layouts, dynamic obstacles, and diverse functional areas, making robots highly susceptible to disorientation and getting lost. Therefore, robots need navigation paths to minimize this problem.
[0003] However, the current robot navigation path planning process is cumbersome, which hinders the improvement of navigation path planning efficiency. This is because existing technologies primarily employ manual planning methods for robot navigation paths, which consume significant human and time resources, increasing the planning time and thus hindering the improvement of navigation path planning efficiency. Summary of the Invention
[0004] This application provides a robot navigation method, apparatus, device, and medium based on a question-and-answer system to solve the technical problem that the existing robot navigation path planning process is cumbersome and not conducive to improving the planning efficiency of navigation paths.
[0005] In a first aspect, embodiments of this application provide a robot navigation method based on a question-answering system, applied to an electronic device, the robot navigation method based on a question-answering system comprising: The robot's RGBD camera collects color and depth images of the indoor environment, and converts these images into point cloud data. A topology map of the indoor environment is constructed using point cloud data. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Semantic guidance information corresponding to the indoor targets is then played. The system obtains user responses that provide semantic guidance information for indoor targets. Based on these responses, it filters out multiple candidate targets with multimodal features of indoor targets from color images. It then obtains the recognition score for each candidate target and selects the candidate target with the highest recognition score as the final target for the robot to grasp. In the area already explored by the robot, the target location of the final target is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-target is obtained. Multiple intermediate nodes are obtained from the topological path, and a sub-map corresponding to each intermediate node is obtained. The sub-maps corresponding to each intermediate node are stitched together to obtain a stitched map. Path planning is performed on the stitched map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path.
[0006] In one possible implementation of the first aspect, the acquisition of color and depth images of the indoor environment via the robot's RGBD camera, and the conversion of the color and depth images of the indoor environment into point cloud data, includes: The robot's RGBD camera captures color and depth images of the indoor environment, which are then converted into point cloud data using point cloud processing tools.
[0007] In one possible implementation of the first aspect, the step of constructing a topological map of the indoor environment using point cloud data, receiving user questions through a question-and-answer system, obtaining indoor targets captured by the robot in the user questions, and playing semantic guidance information corresponding to the indoor targets includes: A topology map of the indoor environment is constructed using point cloud data and saved. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Preset files are accessed, and semantic guidance information corresponding to the indoor targets is obtained from the preset files and played.
[0008] In one possible implementation of the first aspect, the step of obtaining user response information corresponding to semantic guidance information of indoor targets, filtering multiple candidate targets with multimodal features of indoor targets from color images based on user response information, obtaining a recognition score for each candidate target, and selecting the candidate target with the highest recognition score as the final target to be grasped by the robot includes: The system obtains user responses that provide semantic guidance information for indoor targets. It then extracts the location and shape information of indoor targets from these responses. The system performs feature extraction on the location information to obtain the location features of the indoor targets, and extracts the shape features of the indoor targets. Finally, it concatenates the location and shape features of the indoor targets to obtain the multimodal features of the indoor targets. Multiple candidate targets with multimodal features of indoor targets are selected from color images. The image of each candidate target is input into the object detection model. The object detection model processes the image of each candidate target to obtain the recognition score of each candidate target. The candidate target with the highest recognition score is selected as the final target to be grasped by the robot.
[0009] In one possible implementation of the first aspect, the steps of obtaining multiple intermediate nodes from the topological path, obtaining a sub-map corresponding to each intermediate node, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, performing path planning on the stitched map to obtain a navigation path for the robot from its current position to the sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path include: Obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, and stitch the sub-maps corresponding to each intermediate node together to obtain the stitched map. The fast exploration random tree algorithm is used to plan the path on the stitched map, obtain the navigation path from the robot's current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
[0010] In one possible implementation of the first aspect, after obtaining multiple intermediate nodes from the topological path, obtaining a sub-map corresponding to each intermediate node, stitching the sub-maps corresponding to each intermediate node to obtain a stitched map, performing path planning on the stitched map to obtain a navigation path for the robot from its current position to the sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path, the robot navigation method includes: After the robot reaches the sub-target, path planning is performed on the topology map to obtain the forward path of the robot from the sub-target to the target position. The forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path.
[0011] In one possible implementation of the first aspect, after the robot reaches the sub-target, path planning is performed on the topology map to obtain a forward path for the robot to reach the target position from the sub-target, the forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path, the robot navigation method includes: Obtain the preset storage area and store the indoor target, navigation path, and forward path in the storage area.
[0012] Secondly, embodiments of this application provide a robot navigation device based on a question-answering system, applied to electronic devices, including: The acquisition module is used to acquire color and depth images of the indoor environment through the robot's RGBD camera, and convert the color and depth images of the indoor environment into point cloud data. The first acquisition module is used to construct a topology map of the indoor environment through point cloud data, receive user questions through a question-and-answer system, obtain the indoor targets captured by the robot in the user questions, and play the semantic guidance information corresponding to the indoor targets. The second acquisition module is used to acquire user response information corresponding to semantic guidance information of indoor targets. Based on the user response information, multiple candidate targets with multimodal features of indoor targets are selected from the color image. The recognition score of each candidate target is acquired, and the candidate target with the highest recognition score is selected as the final target to be grasped by the robot. The third acquisition module is used to acquire the target location of the final target in the area that the robot has already explored. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is acquired, and the topological path from the current position to the sub-target is acquired. The navigation module is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, stitch the sub-maps corresponding to each intermediate node to obtain a stitched map, perform path planning on the stitched map to obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot navigation method based on a question-and-answer system as described in any of the first aspects above.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot navigation method based on a question-answering system as described in any of the first aspects above.
[0015] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the robot navigation method based on a question-and-answer system as described in any of the first aspects.
[0016] The beneficial effects of this application's embodiments are twofold. Firstly, by obtaining multiple intermediate nodes from the topological path, acquiring a sub-map corresponding to each intermediate node, and stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, path planning is performed on the stitched map to obtain the navigation path for the robot from its current position to the sub-target. The navigation path is then sent to the robot, controlling it to reach the sub-target according to the navigation path. Since there is no need for manual planning of the robot's navigation path, the planning time for the robot's navigation path is reduced, thus improving the planning efficiency of the navigation path. Secondly, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, and performing path planning on the stitched map, can enhance the continuity and accuracy of the navigation path. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This application scenario diagram illustrates the robot navigation method based on a question-answering system provided in the embodiments of this application. Figure 2 This is a flowchart illustrating the robot navigation method based on a question-answering system provided in an embodiment of this application; Figure 3 A flowchart for obtaining a navigation path provided in an embodiment of this application; Figure 4 A schematic block diagram of a robot navigation device based on a question-answering system provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0020] The robot navigation method based on a question-and-answer system provided in this application can be applied to mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, personal computers, netbooks, and personal digital assistant electronic devices. This application does not impose any restrictions on the specific type of electronic device.
[0021] Please see Figure 1 , Figure 1 The application scenario diagram of the robot navigation method based on the question-answering system provided in the embodiments of this application is described in detail below: Electronic devices connect to the robot, which uses its RGBD camera to capture color and depth images of the indoor environment.
[0022] An RGBD camera is mounted on the robot, outputting color and depth images of the indoor environment. The color image provides pixel-level semantic features of color and texture, while the depth image records the spatial distance information corresponding to each pixel.
[0023] The full Chinese name for an RGBD camera is: Color Depth Camera.
[0024] In this embodiment of the application, the electronic device is connected to the robot, and the robot's RGBD camera quickly acquires color and depth images of the indoor environment.
[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating a robot navigation method based on a question-and-answer system provided in an embodiment of this application. This method can be applied to electronic devices.
[0026] like Figure 2 As shown in the embodiments of this application, the robot navigation method based on a question-answering system includes the following steps, which are detailed below: S201 uses the robot's RGBD camera to collect color and depth images of the indoor environment and converts them into point cloud data. The step of acquiring color and depth images of the indoor environment using the robot's RGBD camera and converting these images into point cloud data includes: The robot's RGBD camera captures color and depth images of the indoor environment, which are then converted into point cloud data using point cloud processing tools.
[0027] Point cloud data can simultaneously describe the three-dimensional position and appearance attributes of an object.
[0028] S202 constructs a topology map of the indoor environment using point cloud data, receives user questions through a question-and-answer system, obtains the indoor targets that the robot grabs in the user questions, and plays semantic guidance information corresponding to the indoor targets. The process of constructing a topology map of the indoor environment using point cloud data, receiving user questions through a question-and-answer system, obtaining indoor targets captured by the robot in the user questions, and playing semantic guidance information corresponding to the indoor targets includes: A topology map of the indoor environment is constructed using point cloud data and saved. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Preset files are accessed, and semantic guidance information corresponding to the indoor targets is obtained from the preset files and played.
[0029] While point cloud data can accurately describe the geometric details of a 3D scene, direct processing often involves high computational costs and struggles to extract high-level semantics. Constructing a topological map of an indoor environment using point cloud data allows key features to be abstracted into nodes, with edges representing their connections, forming a lightweight network structure. This representation not only reduces storage and computational overhead but also enhances the interpretability of the scene, such as identifying semantic units like rooms and corridors.
[0030] S203, obtain user response information for semantic guidance information corresponding to indoor targets, and based on the user response information, select multiple candidate targets with multimodal features of indoor targets from the color image, obtain the recognition score of each candidate target, and select the candidate target with the highest recognition score as the final target to be grasped by the robot. The process of obtaining user response information corresponding to the semantic guidance information of the indoor target, filtering multiple candidate targets with multimodal features of the indoor target from the color image based on the user response information, obtaining the recognition score of each candidate target, and selecting the candidate target with the highest recognition score as the final target to be grasped by the robot includes: The system obtains user responses that provide semantic guidance information for indoor targets. It then extracts the location and shape information of indoor targets from these responses. The system performs feature extraction on the location information to obtain the location features of the indoor targets, and extracts the shape features of the indoor targets. Finally, it concatenates the location and shape features of the indoor targets to obtain the multimodal features of the indoor targets. Multiple candidate targets with multimodal features of indoor targets are selected from color images. The image of each candidate target is input into the object detection model. The object detection model processes the image of each candidate target to obtain the recognition score of each candidate target. The candidate target with the highest recognition score is selected as the final target to be grasped by the robot.
[0031] For ease of explanation, the following example is provided: For example, the indoor target is a television. The semantic guidance information for the television would be: what is the probability that the current observation point is located in the room where the television is located, and whether the television is flat or curved.
[0032] The user's response was: The probability that the current observation point is located in the room where the TV is located is 0; the TV is in the living room. Television is flat.
[0033] In this context, "the television is in the living room" refers to the television's location. "The television is flat" refers to the television's shape.
[0034] The shape information of the television is extracted to obtain the shape features of the television. The position features and shape features of the television are then combined to obtain the multimodal features of the television. Multiple candidate targets with the multimodal features of the television are selected from the color image.
[0035] For example, the indoor target is a container for storing ketchup, and the container for storing ketchup is simply called a ketchup container.
[0036] The semantic guidance information for the ketchup container is: what is the probability that the current observation point is located in the room where the ketchup container is located, and whether the ketchup container is a glass bottle or a flat plastic bag.
[0037] The user's answer is: The probability that the current observation point is located in the room where the ketchup container is located is 0. The ketchup container is in the kitchen, and the ketchup container is a glass bottle.
[0038] In this context, "ketchup container in the kitchen" refers to the location of the ketchup bottle. "If the ketchup container is a glass bottle" refers to its shape.
[0039] The shape information of the ketchup container is extracted to obtain the shape features of the ketchup container. The position features and shape features of the ketchup container are spliced together to obtain the multimodal features of the ketchup container. Multiple candidate targets with the multimodal features of the ketchup container are selected from the color image.
[0040] For example, multiple candidate targets with multimodal features of indoor targets are selected from color images. The image of each candidate target is input into an object detection model. The object detection model processes the image of each candidate target to obtain a recognition score for each candidate target. The candidate target with the highest recognition score is selected as the final target to be grasped by the robot, including: Multiple candidate targets with multimodal features of indoor targets are selected from color images. The image of each candidate target is input into an object detection model. The object detection model processes the image of each candidate target to obtain a recognition score for each candidate target. Candidate targets with recognition scores greater than a preset score are selected as recommended candidate targets. Text recognition is performed on the image corresponding to each recommended candidate target to obtain the text content corresponding to each recommended candidate target. Keywords corresponding to indoor targets are obtained. Recommended candidate targets whose text content contains keywords are selected as the final targets for robot grasping.
[0041] For ease of explanation, the following example is provided: The indoor target is a container for storing tomato sauce. The container for storing tomato sauce is simply referred to as a tomato sauce container. The keyword corresponding to the tomato sauce container is tomato. Several recommended candidate targets with multimodal features of ketchup containers are: a red beverage bottle in the room, a red hot sauce bottle in the room, and a ketchup bottle in the room. Among them, the red beverage bottle in the room is shaped like a ketchup bottle, the text corresponding to the red beverage bottle in the room does not contain the word "tomato", and the red beverage bottle in the room is not the final target of the robot to grab. The text corresponding to the red chili sauce bottle in the room did not mention tomatoes, indicating that the red chili sauce bottle was not the robot's final target.
[0042] The text corresponding to the ketchup bottle in the room contains the word "tomato," and the ketchup bottle in the room is selected as the final target for the robot to grab.
[0043] S204. In the area that the robot has already explored, obtain the target location of the final target. Input the topology map into the policy network in the reinforcement learning model. Process the topology map through the policy network to obtain the confidence score of each candidate node on the topology map. Select the candidate node with the highest confidence score as the sub-target associated with the target location. Obtain the current position of the robot and obtain the topological path from the current position to the sub-target. For example, candidate nodes are candidate path nodes.
[0044] The confidence score combines the importance and reachability of candidate nodes. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. This can avoid the robot from ineffectively exploring low-value areas and reduce detour time.
[0045] S205: Obtain multiple intermediate nodes from the topology path, obtain the sub-map corresponding to each intermediate node, stitch the sub-maps corresponding to each intermediate node together to obtain a stitched map, perform path planning on the stitched map to obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
[0046] Among them, sub-goals can serve as periodic verification points. If a sub-goal deviates due to environmental changes, only the subsequent path needs to be locally corrected, without restarting the entire task planning, which significantly reduces the risk of task failure due to a single point of error.
[0047] The robot navigation method includes the following steps: First, obtaining multiple intermediate nodes from the topological path; second, obtaining a sub-map corresponding to each intermediate node; third, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map; fourth, performing path planning on the stitched map to obtain a navigation path for the robot from its current position to the sub-target; fifth, sending the navigation path to the robot; and sixth, controlling the robot to reach the sub-target according to the navigation path. After the robot reaches the sub-target, path planning is performed on the topology map to obtain the forward path of the robot from the sub-target to the target position. The forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path.
[0048] In this process, after the robot reaches the target location according to the forward path, it uses a robotic arm to grab the final target.
[0049] In this process, the robot moves from its current position to a sub-target, and then from the sub-target to the target position. This phased control allows the robot to adjust its speed in real time. For example, the robot can increase its speed in open areas to reduce travel time, while it can reduce its speed in areas with dense obstacles to ensure safety, thus improving overall performance.
[0050] The robot navigation method, which involves performing path planning on the topology map after the robot reaches the sub-target to obtain a forward path for the robot to reach the target location, sending the forward path to the robot, and controlling the robot to reach the target location according to the forward path, includes: Obtain the preset storage area and store the indoor target, navigation path, and forward path in the storage area.
[0051] The beneficial effects of this application's embodiments are twofold. Firstly, by obtaining multiple intermediate nodes from the topological path, acquiring a sub-map corresponding to each intermediate node, and stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, path planning is performed on the stitched map to obtain the navigation path for the robot from its current position to the sub-target. The navigation path is then sent to the robot, controlling it to reach the sub-target according to the navigation path. Since there is no need for manual planning of the robot's navigation path, the planning time for the robot's navigation path is reduced, thus improving the planning efficiency of the navigation path. Secondly, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, and performing path planning on the stitched map, can enhance the continuity and accuracy of the navigation path.
[0052] Please see Figure 3 , Figure 3 The flowchart for obtaining the navigation path provided in the embodiments of this application is described in detail below: S301, obtain multiple intermediate nodes from the topology path, obtain the sub-map corresponding to each intermediate node, and stitch the sub-maps corresponding to each intermediate node together to obtain the stitched map. Among them, stitched maps ensure the continuity of map information in large-scale or complex environments.
[0053] S302 uses a fast exploration random tree algorithm to perform path planning on the stitched map, obtains the navigation path from the robot's current position to the sub-target, sends the navigation path to the robot, and controls the robot to reach the sub-target according to the navigation path.
[0054] Rapidly-exploring Random Tree (RRT) is a sampling-based path planning algorithm primarily used for robot motion planning in high-dimensional spaces.
[0055] In this embodiment of the application, the sub-maps corresponding to each intermediate node are stitched together to obtain a stitched map. Path planning on the stitched map can enhance the continuity and accuracy of the navigation path.
[0056] For the robot navigation method based on a question-answering system described in the above embodiments, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic block diagram of a robot navigation device based on a question-answering system provided in an embodiment of this application. Figure 4 The robot navigation device 400 based on the question-answering system shown can be applied to, for example... Figure 1 The application scenario diagram shows electronic devices. The following section uses electronic devices as an example to illustrate this. Figure 4The robot navigation device 400 based on the question-and-answer system shown will be described in detail. The robot navigation device 400 based on the question-and-answer system may include a data acquisition module 401, a first acquisition module 402, a second acquisition module 403, a third acquisition module 404, and a navigation module 405.
[0057] The acquisition module 401 is used to acquire color and depth images of the indoor environment through the robot's RGBD camera, and convert the color and depth images of the indoor environment into point cloud data. The first acquisition module 402 is used to construct a topology map of the indoor environment through point cloud data, receive user questions through a question-and-answer system, acquire the indoor targets captured by the robot in the user questions, and play semantic guidance information corresponding to the indoor targets. The second acquisition module 403 is used to acquire user response information corresponding to semantic guidance information of indoor targets, and based on the user response information, select multiple candidate targets with multimodal features of indoor targets from the color image, acquire the recognition score of each candidate target, and select the candidate target with the highest recognition score as the final target grasped by the robot. The third acquisition module 404 is used to acquire the target location of the final target in the area that the robot has already explored, input the topology map into the policy network in the reinforcement learning model, process the topology map through the policy network to obtain the confidence score of each candidate node on the topology map, select the candidate node with the highest confidence score as the sub-target associated with the target location, acquire the current position of the robot, and acquire the topological path from the current position to the sub-target. The navigation module 405 is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, stitch the sub-maps corresponding to each intermediate node to obtain a stitched map, perform path planning on the stitched map to obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
[0058] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0059] The beneficial effects of this application's embodiments are twofold. Firstly, by obtaining multiple intermediate nodes from the topological path, acquiring a sub-map corresponding to each intermediate node, and stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, path planning is performed on the stitched map to obtain the navigation path for the robot from its current position to the sub-target. The navigation path is then sent to the robot, controlling it to reach the sub-target according to the navigation path. Since there is no need for manual planning of the robot's navigation path, the planning time for the robot's navigation path is reduced, thus improving the planning efficiency of the navigation path. Secondly, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, and performing path planning on the stitched map, can enhance the continuity and accuracy of the navigation path.
[0060] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0061] like Figure 5 As shown, Figure 5 The electronic device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 executes the computer program 22 to implement the steps in any of the above method embodiments.
[0062] The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 2 and does not constitute a limitation on electronic device 2. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.
[0063] The processor 20 is used to run a computer program 22 stored in the memory 21, and performs the following steps when executing the computer program 22: The robot's RGBD camera collects color and depth images of the indoor environment, and converts these images into point cloud data. A topology map of the indoor environment is constructed using point cloud data. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Semantic guidance information corresponding to the indoor targets is then played. The system obtains user responses that provide semantic guidance information for indoor targets. Based on these responses, it filters out multiple candidate targets with multimodal features of indoor targets from color images. It then obtains the recognition score for each candidate target and selects the candidate target with the highest recognition score as the final target for the robot to grasp. In the area already explored by the robot, the target location of the final target is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-target is obtained. Multiple intermediate nodes are obtained from the topological path, and a sub-map corresponding to each intermediate node is obtained. The sub-maps corresponding to each intermediate node are stitched together to obtain a stitched map. Path planning is performed on the stitched map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path.
[0064] In some embodiments, the processor 20 is configured to implement: The robot's RGBD camera captures color and depth images of the indoor environment, which are then converted into point cloud data using point cloud processing tools.
[0065] In some embodiments, the processor 20 is configured to implement: A topology map of the indoor environment is constructed using point cloud data and saved. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Preset files are accessed, and semantic guidance information corresponding to the indoor targets is obtained from the preset files and played.
[0066] In some embodiments, the processor 20 is configured to implement: The system obtains user responses that provide semantic guidance information for indoor targets. It then extracts the location and shape information of indoor targets from these responses. The system performs feature extraction on the location information to obtain the location features of the indoor targets, and extracts the shape features of the indoor targets. Finally, it concatenates the location and shape features of the indoor targets to obtain the multimodal features of the indoor targets. Multiple candidate targets with multimodal features of indoor targets are selected from color images. The image of each candidate target is input into the object detection model. The object detection model processes the image of each candidate target to obtain the recognition score of each candidate target. The candidate target with the highest recognition score is selected as the final target to be grasped by the robot.
[0067] In some embodiments, the processor 20 is configured to implement: Obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, and stitch the sub-maps corresponding to each intermediate node together to obtain the stitched map. The fast exploration random tree algorithm is used to plan the path on the stitched map, obtain the navigation path from the robot's current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
[0068] In some embodiments, the processor 20 is configured to implement: After the robot reaches the sub-target, path planning is performed on the topology map to obtain the forward path of the robot from the sub-target to the target position. The forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path.
[0069] In some embodiments, the processor 20 is configured to implement: Obtain the preset storage area and store the indoor target, navigation path, and forward path in the storage area.
[0070] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0071] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may be an external storage device of the electronic device 2, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 2. Furthermore, the memory 21 may include both internal and external storage units of the electronic device 2. The memory 21 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.
[0072] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0073] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.
[0074] The computer-readable storage medium stores program code that can be called by a processor to execute the robot navigation method based on the question-answering system described in the above method embodiments.
[0075] Computer-readable storage media have storage space for program code.
[0076] The program code includes the code for any step in the robot navigation method based on the question-and-answer system described in the above method embodiments.
[0077] For example, when program code is invoked by the processor, it can perform the following steps: The robot's RGBD camera collects color and depth images of the indoor environment, and converts these images into point cloud data. A topology map of the indoor environment is constructed using point cloud data. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Semantic guidance information corresponding to the indoor targets is then played. The system obtains user responses that provide semantic guidance information for indoor targets. Based on these responses, it filters out multiple candidate targets with multimodal features of indoor targets from color images. It then obtains the recognition score for each candidate target and selects the candidate target with the highest recognition score as the final target for the robot to grasp. In the area already explored by the robot, the target location of the final target is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-target is obtained. Multiple intermediate nodes are obtained from the topological path, and a sub-map corresponding to each intermediate node is obtained. The sub-maps corresponding to each intermediate node are stitched together to obtain a stitched map. Path planning is performed on the stitched map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path.
[0078] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0079] The computer-readable storage medium can also be an external storage device for a robot navigation device or electronic device based on a question-and-answer system, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, or non-transitory computer-readable storage medium equipped on a robot navigation device or electronic device based on a question-and-answer system.
[0080] Since the computer program stored in the computer-readable storage medium can execute any of the question-and-answer system-based robot navigation methods provided in the embodiments of this application, the computer-readable storage medium can achieve the beneficial effects that any of the question-and-answer system-based robot navigation methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.
[0081] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the aforementioned robot navigation method based on a question-and-answer system.
[0082] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0083] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A robot navigation method based on a question-answering system, characterized in that, The robot navigation method based on a question-answering system, applied to electronic devices, includes: The robot's RGBD camera collects color and depth images of the indoor environment, and converts these images into point cloud data. A topology map of the indoor environment is constructed using point cloud data. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Semantic guidance information corresponding to the indoor targets is then played. The system obtains user responses that provide semantic guidance for indoor targets. It then extracts the location and shape information of the indoor targets from these responses. Feature extraction is performed on the location information to obtain the location features, and on the shape information to obtain the shape features. These location and shape features are then concatenated to obtain the multimodal features of the indoor targets. Multiple candidate targets with these multimodal features are selected from color images. The image of each candidate target is input into an object detection model, which processes the image to obtain a recognition score. Candidate targets with recognition scores greater than a preset score are selected as recommended candidate targets. Text recognition is performed on the image corresponding to each recommended candidate target to obtain the corresponding text content. Keywords corresponding to the indoor targets are then extracted. Finally, recommended candidate targets whose text content contains the keywords are selected as the final targets for robot grasping. In the area already explored by the robot, the target location of the final target is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-target is obtained. Multiple intermediate nodes are obtained from the topological path, and a sub-map corresponding to each intermediate node is obtained. The sub-maps corresponding to each intermediate node are stitched together to obtain a stitched map. Path planning is performed on the stitched map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path.
2. The robot navigation method according to claim 1, characterized in that, The process of acquiring color and depth images of the indoor environment using the robot's RGBD camera and converting these images into point cloud data includes: The robot's RGBD camera captures color and depth images of the indoor environment, which are then converted into point cloud data using point cloud processing tools.
3. The robot navigation method according to claim 1, characterized in that, The process of constructing a topology map of the indoor environment using point cloud data, receiving user questions through a question-and-answer system, obtaining indoor targets captured by the robot in the user questions, and playing semantic guidance information corresponding to the indoor targets includes: A topology map of the indoor environment is constructed using point cloud data and saved. User questions are received through a question-and-answer system, and the indoor targets captured by the robot in the user questions are obtained. Preset files are accessed, and semantic guidance information corresponding to the indoor targets is obtained from the preset files and played.
4. The robot navigation method according to claim 1, characterized in that, The process of obtaining multiple intermediate nodes from the topological path, acquiring a sub-map corresponding to each intermediate node, stitching together the sub-maps corresponding to each intermediate node to obtain a stitched map, performing path planning on the stitched map to obtain a navigation path for the robot from its current position to the sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path includes: Obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, and stitch the sub-maps corresponding to each intermediate node together to obtain the stitched map. The fast exploration random tree algorithm is used to plan the path on the stitched map, obtain the navigation path from the robot's current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
5. The robot navigation method according to claim 1, characterized in that, The robot navigation method includes: obtaining multiple intermediate nodes from the topological path, obtaining a sub-map corresponding to each intermediate node, stitching the sub-maps corresponding to each intermediate node together to obtain a stitched map, performing path planning on the stitched map to obtain a navigation path for the robot from its current position to the sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path. After the robot reaches the sub-target, path planning is performed on the topology map to obtain the forward path of the robot from the sub-target to the target position. The forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path.
6. The robot navigation method according to claim 5, characterized in that, After the robot reaches the sub-target, path planning is performed on the topology map to obtain the robot's forward path from the sub-target to the target position. The forward path is then sent to the robot, and the robot is controlled to reach the target position according to the forward path. The robot navigation method further includes: Obtain the preset storage area and store the indoor target, navigation path, and forward path in the storage area.
7. A robot navigation device based on a question-and-answer system, characterized in that, Applied to electronic devices, including: The acquisition module is used to acquire color and depth images of the indoor environment through the robot's RGBD camera, and convert the color and depth images of the indoor environment into point cloud data. The first acquisition module is used to construct a topology map of the indoor environment through point cloud data, receive user questions through a question-and-answer system, obtain the indoor targets captured by the robot in the user questions, and play the semantic guidance information corresponding to the indoor targets. The second acquisition module is used to acquire user response information corresponding to semantic guidance information of indoor targets. It acquires the location information and shape information of indoor targets from the user response information. It extracts features from the location information of indoor targets to obtain the location features of indoor targets, and extracts features from the shape information of indoor targets to obtain the shape features of indoor targets. It concatenates the location features and shape features of indoor targets to obtain the multimodal features of indoor targets. It selects multiple candidate targets with multimodal features of indoor targets from color images. It inputs the image of each candidate target into an object detection model. The object detection model processes the image of each candidate target to obtain the recognition score of each candidate target. It selects candidate targets with recognition scores greater than a preset score as recommended candidate targets. It performs text recognition on the image corresponding to each recommended candidate target to obtain the text content corresponding to each recommended candidate target. It acquires the keywords corresponding to indoor targets. It selects recommended candidate targets whose text content contains keywords as the final target for robot grasping. The third acquisition module is used to acquire the target location of the final target in the area that the robot has already explored. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the highest confidence score is selected as the sub-target associated with the target location. The current position of the robot is acquired, and the topological path from the current position to the sub-target is acquired. The navigation module is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, stitch the sub-maps corresponding to each intermediate node to obtain a stitched map, perform path planning on the stitched map to obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot navigation method based on a question-and-answer system as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot navigation method based on a question-and-answer system as described in any one of claims 1 to 6.