Robot navigation method, device and equipment based on question and answer system and medium

Abstract

The invention relates to the technical field of robot navigation, and discloses a robot navigation method and device based on a question and answer system, electronic equipment and a storage medium, the method comprises the following steps: inputting a topological graph into a strategy network in a reinforcement learning model, processing the topological graph through the strategy network to obtain a confidence score of each candidate node on the topological graph, selecting the candidate node with the maximum confidence score as a sub-target, obtaining the current position of the robot, and obtaining a topological path from the current position to the sub-target; acquiring a plurality of intermediate nodes from the topological path, acquiring a sub-map corresponding to each intermediate node, splicing the sub-maps corresponding to each intermediate node to obtain a spliced map, performing path planning on the spliced map to obtain a navigation path of the robot from the current position to the sub-target, and sending the navigation path to the robot. And controlling the robot to reach the sub-target according to the navigation path. The planning efficiency of the navigation path of the robot can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot navigation technology, and in particular to a robot navigation method, device, equipment and medium based on a question-answering system. Background Art

[0002] Embodied navigation refers to the process by which robots navigate in a physical environment through perception and interaction with their surroundings. Indoor environments are characterized by complex layouts, dynamic obstacles, and diverse functional areas. Robots can easily become lost and take detours. Therefore, robots need navigational paths to minimize these detours.

[0003] However, the existing robot navigation path planning process is cumbersome and not conducive to improving navigation path planning efficiency. This is because existing technologies mainly use manual planning to plan the robot's navigation path. Manual planning consumes a lot of human resources and time resources, increasing the robot's navigation path planning time, and therefore is not conducive to improving navigation path planning efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a robot navigation method, apparatus, device and medium based on a question-answering system to solve the technical problem that the navigation path planning process of the above-mentioned existing robots is cumbersome and not conducive to improving the planning efficiency of the navigation path.

[0005] In a first aspect, an embodiment of the present application provides a robot navigation method based on a question-answering system, which is applied to an electronic device. The robot navigation method based on the question-answering system includes: The robot's RGBD camera collects color images and depth images of the indoor environment and converts them into point cloud data; Build a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor targets grasped by the robot in the user's questions, and play the semantic guidance information corresponding to the indoor targets; Obtain user responses to semantic guidance information corresponding to indoor targets. Based on these responses, select multiple candidate targets with multimodal characteristics of indoor targets from the color image. Obtain a recognition score for each candidate target, and select the candidate with the highest recognition score as the final target for the robot to grasp. In the area that the robot has explored, the target location of the final goal is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the largest confidence score is selected as the sub-goal associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-goal is obtained. Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, splice the submap corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0006] In a possible implementation of the first aspect, capturing a color image and a depth image of an indoor environment by an RGBD camera of the robot, and converting the color image and the depth image of the indoor environment into point cloud data includes: The robot's RGBD camera collects color images and depth images of the indoor environment, and the point cloud processing tools convert the color images and depth images of the indoor environment into point cloud data.

[0007] In a possible implementation of the first aspect, constructing a topological map of an indoor environment using point cloud data, receiving a user question through a question-answering system, obtaining an indoor target grasped by a robot in the user's question, and playing semantic guidance information corresponding to the indoor target include: Build a topological map of the indoor environment through point cloud data, save the topological map, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, access the preset file, obtain the semantic guidance information corresponding to the indoor target from the preset file, and play the semantic guidance information corresponding to the indoor target.

[0008] In a possible implementation of the first aspect, obtaining user response information regarding semantic guidance information corresponding to the indoor target, screening multiple candidate targets having multimodal features of the indoor target from the color image based on the user response information, obtaining a recognition score for each candidate target, and selecting the candidate target with the highest recognition score as the final target to be grasped by the robot includes: Obtaining user response information for semantic guidance information corresponding to an indoor target, obtaining location information and shape information of the indoor target from the user response information, performing feature extraction on the location information of the indoor target to obtain location features of the indoor target, performing feature extraction on the shape information of the indoor target to obtain shape features of the indoor target, and concatenating the location features and shape features of the indoor target to obtain multimodal features of the indoor target; Multiple candidate targets with multimodal features of indoor targets are screened out from the color image, and the image of each candidate target is input into the object detection model. The image of each candidate target is processed by the object detection model to obtain the recognition score of each candidate target. The candidate target with the largest recognition score is selected as the final target to be grasped by the robot.

[0009] In a possible implementation of the first aspect, obtaining multiple intermediate nodes from a topological path, obtaining a submap corresponding to each intermediate node, splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map, performing path planning on the spliced ​​map, obtaining a navigation path for the robot from a current position to a subtarget, sending the navigation path to the robot, and controlling the robot to reach the subtarget according to the navigation path include: Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, and splice the submaps corresponding to each intermediate node to obtain a spliced ​​map; The fast exploration random tree algorithm is used to perform path planning on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot and the robot is controlled to reach the sub-target according to the navigation path.

[0010] In a possible implementation of the first aspect, after obtaining multiple intermediate nodes from a topological path, obtaining a submap corresponding to each intermediate node, splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map, performing path planning on the spliced ​​map, obtaining a navigation path for the robot from a current position to a sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path, the robot navigation method includes: After the robot reaches the sub-goal, path planning is performed on the topological map to obtain the forward path of the robot from the sub-goal to the target position, and the forward path is sent to the robot to control the robot to reach the target position according to the forward path.

[0011] In a possible implementation of the first aspect, after the robot reaches the sub-goal, path planning is performed on the topological map to obtain a forward path for the robot from the sub-goal to the target location, the forward path is sent to the robot, and the robot is controlled to reach the target location according to the forward path. The robot navigation method includes: Obtain a preset storage area and store the indoor target, navigation path, and forward path in the storage area.

[0012] In a second aspect, an embodiment of the present application provides a robot navigation device based on a question-answering system, which is applied to an electronic device, including: The acquisition module is used to collect color images and depth images of the indoor environment through the robot's RGBD camera and convert the color images and depth images of the indoor environment into point cloud data; The first acquisition module is used to construct a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, and play the semantic guidance information corresponding to the indoor target; The second acquisition module is used to obtain user responses to the semantic guidance information corresponding to the indoor target. Based on the user responses, multiple candidate targets with multimodal characteristics of the indoor target are screened from the color image, and the recognition score of each candidate target is obtained. The candidate with the highest recognition score is selected as the final target to be grasped by the robot. The third acquisition module is used to obtain the target location of the final goal in the area that the robot has explored, input the topology map into the policy network in the reinforcement learning model, process the topology map through the policy network, obtain the confidence score of each candidate node on the topology map, select the candidate node with the largest confidence score as the sub-goal associated with the target location, obtain the current position of the robot, and obtain the topological path from the current position to the sub-goal; The navigation module is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, splice the sub-map corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot navigation method based on the question-answering system according to any one of the first aspects mentioned above is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the robot navigation method based on the question-answering system of any one of the above-mentioned first aspects.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the robot navigation method based on the question-answering system as described in any one of the above-mentioned first aspects.

[0016] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, multiple intermediate nodes are obtained from the topological path, and the sub-map corresponding to each intermediate node is obtained. The sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path. Since there is no need to manually plan the navigation path of the robot, the planning time of the robot's navigation path is reduced, which is conducive to improving the planning efficiency of the navigation path. On the other hand, the sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map, which can enhance the continuity and accuracy of the navigation path. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A diagram illustrating an application scenario of the robot navigation method based on a question-answering system provided in an embodiment of the present application; Figure 2 1 is a flow chart of a robot navigation method based on a question-answering system provided in an embodiment of the present application; Figure 3 A flowchart for obtaining a navigation path provided in an embodiment of the present application; Figure 4 A schematic block diagram of a robot navigation device based on a question-answering system provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] The robot navigation method based on the question-answering system provided in the embodiment of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, personal computers, netbooks, and personal digital assistant electronic devices. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0021] See also Figure 1 , Figure 1 The application scenario diagram of the robot navigation method based on the question-answering system provided in the embodiment of the present application is detailed as follows: The electronic device is connected to the robot and collects color images and depth images of the indoor environment through the robot's RGBD camera.

[0022] An RGBD camera is mounted on the robot and outputs color and depth images of the indoor environment. The color image provides pixel-level semantic features of color and texture, while the depth image records the spatial distance information corresponding to each pixel.

[0023] The full Chinese name of RGBD camera is: Color Depth Camera.

[0024] In an embodiment of the present application, the electronic device is connected to a robot, and the robot's RGBD camera quickly captures color images and depth images of the indoor environment.

[0025] See also Figure 2 , Figure 2 This is a flow chart of a robot navigation method based on a question-answering system provided in an embodiment of the present application, which can be applied to electronic devices.

[0026] like Figure 2 As shown, the robot navigation method based on the question-answering system provided in the embodiment of the present application includes the following steps, which are detailed as follows: S201, collecting a color image and a depth image of the indoor environment through the robot's RGBD camera, and converting the color image and the depth image of the indoor environment into point cloud data; The method of collecting the color image and depth image of the indoor environment by the RGBD camera of the robot and converting the color image and depth image of the indoor environment into point cloud data includes: The robot's RGBD camera collects color images and depth images of the indoor environment, and the point cloud processing tools convert the color images and depth images of the indoor environment into point cloud data.

[0027] Among them, point cloud data can simultaneously describe the three-dimensional position and appearance attributes of an object.

[0028] S202, constructing a topological map of the indoor environment using point cloud data, receiving user questions through the question-answering system, obtaining the indoor target grasped by the robot in the user's question, and playing semantic guidance information corresponding to the indoor target; The method of constructing a topological map of the indoor environment using point cloud data, receiving user questions through a question-answering system, obtaining the indoor target grasped by the robot in the user's question, and playing semantic guidance information corresponding to the indoor target includes: Build a topological map of the indoor environment through point cloud data, save the topological map, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, access the preset file, obtain the semantic guidance information corresponding to the indoor target from the preset file, and play the semantic guidance information corresponding to the indoor target.

[0029] While point cloud data can accurately describe the geometric details of a 3D scene, direct processing is often computationally intensive and difficult to extract high-level semantics. Constructing a topological map of an indoor environment using point cloud data abstracts key features of the point cloud data into nodes and represents their connections through edges, forming a lightweight network structure. This representation not only reduces storage and computational overhead but also enhances scene interpretability. For example, semantic units such as rooms and corridors can be identified.

[0030] S203: Obtaining user response information regarding semantic guidance information corresponding to the indoor target, screening multiple candidate targets having multimodal characteristics of the indoor target from the color image based on the user response information, obtaining a recognition score for each candidate target, and selecting the candidate target with the highest recognition score as the final target to be grasped by the robot; The method of obtaining user response information of semantic guidance information corresponding to the indoor target, screening multiple candidate targets having multimodal characteristics of the indoor target from the color image based on the user response information, obtaining a recognition score for each candidate target, and selecting the candidate target with the largest recognition score as the final target to be grasped by the robot includes: Obtaining user response information for semantic guidance information corresponding to an indoor target, obtaining location information and shape information of the indoor target from the user response information, performing feature extraction on the location information of the indoor target to obtain location features of the indoor target, performing feature extraction on the shape information of the indoor target to obtain shape features of the indoor target, and concatenating the location features and shape features of the indoor target to obtain multimodal features of the indoor target; Multiple candidate targets with multimodal features of indoor targets are screened out from the color image, and the image of each candidate target is input into the object detection model. The image of each candidate target is processed by the object detection model to obtain the recognition score of each candidate target. The candidate target with the largest recognition score is selected as the final target to be grasped by the robot.

[0031] For ease of explanation, the following examples are given: For example, if the indoor target is a TV, the semantic guidance information of the TV includes: the probability that the current observation point is located in the room where the TV is located, and whether the TV is flat or curved.

[0032] The user's answer is: The probability that the current observation point is in the room where the TV is is 0, and the TV is in the living room. The TV is flat screen.

[0033] Among them, the TV being in the living room is the location information of the TV, and the TV being flat is the shape information of the TV.

[0034] The shape information of the TV is extracted to obtain the shape features of the TV. The position features and shape features of the TV are spliced ​​together to obtain the multimodal features of the TV. Multiple candidate targets with the multimodal features of the TV are screened out from the color image.

[0035] For example, if the indoor target is a container for storing ketchup, the container for storing ketchup is referred to as: ketchup container.

[0036] The semantic guidance information of the ketchup container is: what is the probability that the current observation point is located in the room where the ketchup container is located, and whether the ketchup container is a glass bottle or a flat plastic bag.

[0037] The user's answer is: The probability that the current observation point is in the room where the ketchup container is located is 0, the ketchup container is in the kitchen, and the ketchup container is a glass bottle.

[0038] The ketchup container in the kitchen is the location information of the ketchup bottle, and the ketchup container being a glass bottle is the shape information of the ketchup container.

[0039] The shape information of the ketchup container is extracted to obtain the shape features of the ketchup container. The position features and shape features of the ketchup container are then combined to obtain the multimodal features of the ketchup container. Multiple candidate targets with the multimodal features of the ketchup container are screened out from the color image.

[0040] For example, multiple candidate targets with multimodal features of indoor targets are screened from a color image, the image of each candidate target is input into an object detection model, the image of each candidate target is processed by the object detection model to obtain a recognition score for each candidate target, and the candidate target with the highest recognition score is selected as the final target to be grasped by the robot, including: Multiple candidate targets with multimodal features of indoor targets are screened out from the color image, the image of each candidate target is input into the object detection model, the image of each candidate target is processed by the object detection model to obtain the recognition score of each candidate target, and the candidate targets with recognition scores greater than the preset score are selected as recommended candidate targets. Text recognition is performed on the image corresponding to each recommended candidate target to obtain the text content corresponding to each recommended candidate target, the keywords corresponding to the indoor targets are obtained, and the recommended candidate targets whose text content contains the keywords are selected as the final targets to be captured by the robot.

[0041] For ease of explanation, the following examples are given: The indoor target is a container for storing ketchup. The container for storing ketchup is referred to as: ketchup container. The keyword corresponding to the ketchup container is tomato. The multiple recommended candidate targets with the multimodal features of the ketchup container are: the red beverage bottle in the room, the red chili sauce bottle in the room, and the ketchup bottle in the room; The red drink bottle in the room is shaped like a ketchup bottle, and the text corresponding to the red drink bottle in the room does not contain tomato. Therefore, the red drink bottle in the room is not the final target of the robot's grasping. Among them, the text content corresponding to the red chili sauce bottle in the room does not include tomato, and the red chili sauce bottle in the room is not the ultimate target of the robot's grasping.

[0042] Among them, the text content corresponding to the ketchup bottle in the room contains tomato, and the ketchup bottle in the room is selected as the final target of the robot's grasping.

[0043] S204: In the area that the robot has already explored, the target location of the final goal is obtained, the topology map is input into the policy network in the reinforcement learning model, the topology map is processed by the policy network, and the confidence score of each candidate node on the topology map is obtained. The candidate node with the largest confidence score is selected as the sub-goal associated with the target location, the current location of the robot is obtained, and the topological path from the current location to the sub-goal is obtained; Exemplarily, the candidate node is a candidate path node.

[0044] Among them, the confidence score comprehensively considers the importance and reachability of the candidate node, and selects the candidate node with the largest confidence score as the sub-target associated with the target location. This can avoid the robot's ineffective exploration in low-value areas and reduce detour time.

[0045] S205, obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, splice the sub-map corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0046] Among them, sub-goals can be used as stage verification points. If the sub-goals deviate due to environmental changes, only the subsequent path needs to be partially corrected without restarting the entire task planning, which significantly reduces the risk of task failure due to single point errors.

[0047] The robot navigation method includes: obtaining a plurality of intermediate nodes from the topological path, obtaining a submap corresponding to each intermediate node, splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map, performing path planning on the spliced ​​map, obtaining a navigation path from the current position of the robot to the subtarget, sending the navigation path to the robot, and controlling the robot to reach the subtarget according to the navigation path. After the robot reaches the sub-goal, path planning is performed on the topological map to obtain the forward path of the robot from the sub-goal to the target position, and the forward path is sent to the robot to control the robot to reach the target position according to the forward path.

[0048] Among them, after the robot reaches the target position according to the forward path, it grasps the final target through the robotic arm.

[0049] The robot moves from its current position to a sub-goal, and then from the sub-goal to the target position. This phased control allows the robot to adjust its speed in real time. For example, the robot can increase speed in open areas to save time, while slowing down in areas with dense obstacles to ensure safety, thus improving overall performance.

[0050] Wherein, after the robot reaches the sub-goal, path planning is performed on the topological map to obtain a forward path for the robot to reach the target position from the sub-goal, the forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path. The robot navigation method includes: Obtain a preset storage area and store the indoor target, navigation path, and forward path in the storage area.

[0051] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, multiple intermediate nodes are obtained from the topological path, and the sub-map corresponding to each intermediate node is obtained. The sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path. Since there is no need to manually plan the navigation path of the robot, the planning time of the robot's navigation path is reduced, which is conducive to improving the planning efficiency of the navigation path. On the other hand, the sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map, which can enhance the continuity and accuracy of the navigation path.

[0052] See also Figure 3 , Figure 3 The flowchart for obtaining the navigation path provided in the embodiment of the present application is detailed as follows: S301, obtaining multiple intermediate nodes from a topological path, obtaining a submap corresponding to each intermediate node, and splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map; Among them, mosaic maps ensure the consistency of map information in large-scale or complex environments.

[0053] S302, using a fast exploration random tree algorithm to perform path planning on the spliced ​​map, obtain a navigation path for the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0054] Rapidly-exploring Random Tree (RRT) is a sampling-based path planning algorithm, which is mainly used for robot motion planning in high-dimensional space.

[0055] In an embodiment of the present application, the submaps corresponding to each intermediate node are spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map, which can enhance the continuity and accuracy of the navigation path.

[0056] For the robot navigation method based on the question-answering system described in the above embodiment, please refer to Figure 4 , Figure 4 A schematic block diagram of a robot navigation device based on a question-answering system provided in an embodiment of the present application is provided. Figure 4 The robot navigation device 400 based on the question-answering system shown can be applied to Figure 1 The electronic device in the application scenario diagram shown below takes the electronic device as an example. Figure 4The robot navigation device 400 based on the question-answering system shown is described in detail. The robot navigation device 400 based on the question-answering system may include a collection module 401 , a first acquisition module 402 , a second acquisition module 403 , a third acquisition module 404 , and a navigation module 405 .

[0057] An acquisition module 401 is configured to acquire a color image and a depth image of an indoor environment through an RGBD camera of the robot, and convert the color image and the depth image of the indoor environment into point cloud data; The first acquisition module 402 is configured to construct a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, and play semantic guidance information corresponding to the indoor target; The second acquisition module 403 is configured to obtain user responses to the semantic guidance information corresponding to the indoor target, select multiple candidate targets having the multimodal characteristics of the indoor target from the color image based on the user responses, obtain a recognition score for each candidate target, and select the candidate target with the highest recognition score as the final target to be grasped by the robot; The third acquisition module 404 is configured to obtain the target location of the final goal in the area that the robot has explored, input the topology map into the policy network in the reinforcement learning model, process the topology map through the policy network, obtain the confidence score of each candidate node on the topology map, select the candidate node with the largest confidence score as the sub-goal associated with the target location, obtain the current location of the robot, and obtain the topological path from the current location to the sub-goal; Navigation module 405 is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, splice the sub-maps corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain a navigation path from the robot's current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0058] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0059] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, multiple intermediate nodes are obtained from the topological path, and the sub-map corresponding to each intermediate node is obtained. The sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot, and the robot is controlled to reach the sub-target according to the navigation path. Since there is no need to manually plan the navigation path of the robot, the planning time of the robot's navigation path is reduced, which is conducive to improving the planning efficiency of the navigation path. On the other hand, the sub-map corresponding to each intermediate node is spliced ​​together to obtain a spliced ​​map, and path planning is performed on the spliced ​​map, which can enhance the continuity and accuracy of the navigation path.

[0060] See also Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0061] like Figure 5 As shown, Figure 5 The electronic device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned method embodiments when executing the computer program 22.

[0062] The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5 This is merely an example of the electronic device 2 and does not constitute a limitation on the electronic device 2 . The electronic device 2 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 2 may also include input and output devices, network access devices, etc.

[0063] The processor 20 is configured to run a computer program 22 stored in the memory 21 and implement the following steps when executing the computer program 22: The robot's RGBD camera collects color images and depth images of the indoor environment and converts them into point cloud data; Build a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor targets grasped by the robot in the user's questions, and play the semantic guidance information corresponding to the indoor targets; Obtain user responses to semantic guidance information corresponding to indoor targets. Based on these responses, select multiple candidate targets with multimodal characteristics of indoor targets from the color image. Obtain a recognition score for each candidate target, and select the candidate with the highest recognition score as the final target for the robot to grasp. In the area that the robot has explored, the target location of the final goal is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the largest confidence score is selected as the sub-goal associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-goal is obtained. Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, splice the submap corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0064] In some embodiments, the processor 20 is configured to implement: The robot's RGBD camera collects color images and depth images of the indoor environment, and the point cloud processing tools convert the color images and depth images of the indoor environment into point cloud data.

[0065] In some embodiments, the processor 20 is configured to implement: Build a topological map of the indoor environment through point cloud data, save the topological map, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, access the preset file, obtain the semantic guidance information corresponding to the indoor target from the preset file, and play the semantic guidance information corresponding to the indoor target.

[0066] In some embodiments, the processor 20 is configured to implement: Obtaining user response information for semantic guidance information corresponding to an indoor target, obtaining location information and shape information of the indoor target from the user response information, performing feature extraction on the location information of the indoor target to obtain location features of the indoor target, performing feature extraction on the shape information of the indoor target to obtain shape features of the indoor target, and concatenating the location features and shape features of the indoor target to obtain multimodal features of the indoor target; Multiple candidate targets with multimodal features of indoor targets are screened out from the color image, and the image of each candidate target is input into the object detection model. The image of each candidate target is processed by the object detection model to obtain the recognition score of each candidate target. The candidate target with the largest recognition score is selected as the final target to be grasped by the robot.

[0067] In some embodiments, the processor 20 is configured to implement: Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, and splice the submaps corresponding to each intermediate node to obtain a spliced ​​map; The fast exploration random tree algorithm is used to perform path planning on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot and the robot is controlled to reach the sub-target according to the navigation path.

[0068] In some embodiments, the processor 20 is configured to implement: After the robot reaches the sub-goal, path planning is performed on the topological map to obtain the forward path of the robot from the sub-goal to the target position, and the forward path is sent to the robot to control the robot to reach the target position according to the forward path.

[0069] In some embodiments, the processor 20 is configured to implement: Obtain a preset storage area and store the indoor target, navigation path, and forward path in the storage area.

[0070] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0071] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 2. Furthermore, the memory 21 may include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0072] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0073] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0074] The computer-readable storage medium stores program codes, which can be called by a processor to execute the robot navigation method based on the question-answering system described in the above method embodiment.

[0075] The computer-readable storage medium has a storage space for program codes.

[0076] The program code includes the code of any step in the robot navigation method based on the question-answering system described in the above method embodiment.

[0077] For example, the program code is called by the processor and can execute the following steps: The robot's RGBD camera collects color images and depth images of the indoor environment and converts them into point cloud data; Build a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor targets grasped by the robot in the user's questions, and play the semantic guidance information corresponding to the indoor targets; Obtain user responses to semantic guidance information corresponding to indoor targets. Based on these responses, select multiple candidate targets with multimodal characteristics of indoor targets from the color image. Obtain a recognition score for each candidate target, and select the candidate with the highest recognition score as the final target for the robot to grasp. In the area that the robot has explored, the target location of the final goal is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the largest confidence score is selected as the sub-goal associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-goal is obtained. Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, splice the submap corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

[0078] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0079] Among them, the computer-readable storage medium can also be an external storage device of a robot navigation device or electronic device based on the question-answering system, for example, a plug-in hard disk equipped on a robot navigation device or electronic device based on the question-answering system, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), a non-volatile computer-readable storage medium (non-transitory computer-readable storage medium), etc.

[0080] Since the computer program stored in the computer-readable storage medium can execute any one of the robot navigation methods based on the question-answering system provided in the embodiments of the present application, the computer-readable storage medium can achieve the beneficial effects that can be achieved by any one of the robot navigation methods based on the question-answering system provided in the embodiments of the present application. Please see the previous embodiments for details and will not be repeated here.

[0081] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned robot navigation method based on the question-answering system.

[0082] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0083] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A robot navigation method based on a question-answering system, characterized in that: Applied to electronic equipment, the robot navigation method based on the question-answering system includes: The robot's RGBD camera collects color images and depth images of the indoor environment and converts them into point cloud data; Build a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor targets grasped by the robot in the user's questions, and play the semantic guidance information corresponding to the indoor targets; Obtain user responses to semantic guidance information corresponding to indoor targets. Based on these responses, select multiple candidate targets with multimodal characteristics of indoor targets from the color image. Obtain a recognition score for each candidate target, and select the candidate with the highest recognition score as the final target for the robot to grasp. In the area that the robot has explored, the target location of the final goal is obtained. The topology map is input into the policy network in the reinforcement learning model. The topology map is processed by the policy network to obtain the confidence score of each candidate node on the topology map. The candidate node with the largest confidence score is selected as the sub-goal associated with the target location. The current position of the robot is obtained, and the topological path from the current position to the sub-goal is obtained. Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, splice the submap corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

2. The robot navigation method according to claim 1, characterized in that: The method of collecting a color image and a depth image of an indoor environment by using the robot's RGBD camera and converting the color image and the depth image of the indoor environment into point cloud data includes: The robot's RGBD camera collects color images and depth images of the indoor environment, and the point cloud processing tools convert the color images and depth images of the indoor environment into point cloud data.

3. The robot navigation method according to claim 1, characterized in that: The method of constructing a topological map of the indoor environment using point cloud data, receiving user questions through a question-answering system, obtaining the indoor target grasped by the robot in the user's question, and playing semantic guidance information corresponding to the indoor target includes: Build a topological map of the indoor environment through point cloud data, save the topological map, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, access the preset file, obtain the semantic guidance information corresponding to the indoor target from the preset file, and play the semantic guidance information corresponding to the indoor target.

4. The robot navigation method according to claim 1, characterized in that: The method of obtaining user response information of semantic guidance information corresponding to the indoor target, screening multiple candidate targets having multimodal characteristics of the indoor target from the color image based on the user response information, obtaining a recognition score for each candidate target, and selecting the candidate target with the largest recognition score as the final target to be grasped by the robot includes: Obtaining user response information for semantic guidance information corresponding to an indoor target, obtaining location information and shape information of the indoor target from the user response information, performing feature extraction on the location information of the indoor target to obtain location features of the indoor target, performing feature extraction on the shape information of the indoor target to obtain shape features of the indoor target, and concatenating the location features and shape features of the indoor target to obtain multimodal features of the indoor target; Multiple candidate targets with multimodal features of indoor targets are screened out from the color image, and the image of each candidate target is input into the object detection model. The image of each candidate target is processed by the object detection model to obtain the recognition score of each candidate target. The candidate target with the largest recognition score is selected as the final target to be grasped by the robot.

5. The robot navigation method according to claim 1, characterized in that: The method includes obtaining multiple intermediate nodes from a topological path, obtaining a submap corresponding to each intermediate node, splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map, performing path planning on the spliced ​​map, obtaining a navigation path for the robot from the current position to the subtarget, sending the navigation path to the robot, and controlling the robot to reach the subtarget according to the navigation path, including: Obtain multiple intermediate nodes from the topological path, obtain the submap corresponding to each intermediate node, and splice the submaps corresponding to each intermediate node to obtain a spliced ​​map; The fast exploration random tree algorithm is used to perform path planning on the spliced ​​map to obtain the navigation path of the robot from the current position to the sub-target. The navigation path is sent to the robot and the robot is controlled to reach the sub-target according to the navigation path.

6. The robot navigation method according to claim 1, characterized in that: After obtaining multiple intermediate nodes from the topological path, obtaining a submap corresponding to each intermediate node, splicing the submaps corresponding to each intermediate node to obtain a spliced ​​map, performing path planning on the spliced ​​map, obtaining a navigation path from the current position of the robot to the sub-target, sending the navigation path to the robot, and controlling the robot to reach the sub-target according to the navigation path, the robot navigation method includes: After the robot reaches the sub-goal, path planning is performed on the topological map to obtain the forward path of the robot from the sub-goal to the target position, and the forward path is sent to the robot to control the robot to reach the target position according to the forward path.

7. The robot navigation method according to claim 6, characterized in that: After the robot reaches the sub-goal, path planning is performed on the topological map to obtain a forward path for the robot to reach the target position from the sub-goal, the forward path is sent to the robot, and the robot is controlled to reach the target position according to the forward path. The robot navigation method includes: Obtain a preset storage area and store the indoor target, navigation path, and forward path in the storage area.

8. A robot navigation device based on a question-answering system, characterized in that: Used in electronic equipment, including: The acquisition module is used to collect color images and depth images of the indoor environment through the robot's RGBD camera and convert the color images and depth images of the indoor environment into point cloud data; The first acquisition module is used to construct a topological map of the indoor environment using point cloud data, receive user questions through the question-answering system, obtain the indoor target grasped by the robot in the user's question, and play the semantic guidance information corresponding to the indoor target; The second acquisition module is used to obtain user responses to the semantic guidance information corresponding to the indoor target. Based on the user responses, multiple candidate targets with multimodal characteristics of the indoor target are screened from the color image, and the recognition score of each candidate target is obtained. The candidate with the highest recognition score is selected as the final target to be grasped by the robot. The third acquisition module is used to obtain the target location of the final goal in the area that the robot has explored, input the topology map into the policy network in the reinforcement learning model, process the topology map through the policy network, obtain the confidence score of each candidate node on the topology map, select the candidate node with the largest confidence score as the sub-goal associated with the target location, obtain the current position of the robot, and obtain the topological path from the current position to the sub-goal; The navigation module is used to obtain multiple intermediate nodes from the topological path, obtain the sub-map corresponding to each intermediate node, splice the sub-map corresponding to each intermediate node to obtain a spliced ​​map, perform path planning on the spliced ​​map, obtain the navigation path of the robot from the current position to the sub-target, send the navigation path to the robot, and control the robot to reach the sub-target according to the navigation path.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the robot navigation method based on the question-answering system according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot navigation method based on the question-answering system according to any one of claims 1 to 7 is implemented.

Images