Autonomous exploration and navigation method and system for aircraft indoors

Abstract

A method and system for autonomous indoor exploration and navigation of an aircraft, belonging to the field of robotics, includes: S1: the aircraft receives a navigation target; S2: the memory module is searched to determine whether the navigation target exists in the memory module: if not, go to S3; otherwise go to S6; S3: the aircraft obtains RGB images and depth images of surrounding objects, adds the objects to the memory module and updates the object map, and determines whether the navigation target exists in the memory module; if not, go to S4; otherwise go to S6; S4: skeletonize the two-dimensional grid map to obtain candidate points and paths; S5: score the candidate points, and use the candidate point with the highest score as the target point; S6: complete the navigation of the aircraft from the current position to the target point, and determine whether the navigation target has been found; if yes, the navigation ends; otherwise go to S3. The method of the present invention realizes the target navigation task in indoor scenes by using a large language model.

Description

Technical Field

[0001] The present invention belongs to the field of robot technology, and in particular relates to an autonomous exploration and navigation method and system for an aircraft room. Background Art

[0002] With the rapid development of the drone industry, the demand for indoor drone applications in industrial, commercial and personal fields is growing. However, the complexity and variability of indoor space puts higher demands on the autonomous exploration and navigation capabilities of drones.

[0003] Traditional drone navigation technology mainly relies on sensors and complex algorithms, and often only involves low-level mapping and control, solving the problem of point-to-point navigation and obstacle avoidance, but is unable to cope with high-level navigation planning and reasoning that combines visual information (such as autonomously setting the starting point and end point based on environmental information). In addition, they generally have certain requirements for the form of input instructions, and cannot complete navigation tasks according to natural language instructions issued by humans.

[0004] In recent years, with the rapid development of artificial intelligence technologies such as computer vision and natural language processing, people have begun to turn their attention to pre-trained large language models (LLMs). Since large models such as pre-trained language models (PLMs) have been pre-trained in large-scale corpora, they contain a lot of knowledge and have strong multi-task learning and transfer learning capabilities, which can effectively solve cross-domain and cross-task generalization application problems. Therefore, how to use large language models to improve the reliability of drone navigation and achieve more flexible navigation strategies has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an indoor autonomous exploration and navigation method for an aircraft, comprising the following steps:

[0006] Step S1: The aircraft receives the user's navigation target in natural language or picture form;

[0007] Step S2: Search the memory module to determine whether the navigation target exists in the memory module: if not, go to step S3; otherwise, go to step S6; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image;

[0008] Step S3: The aircraft first completes the action of rotating in place at the current position, obtains the RGB image and depth image of the surrounding objects through the depth camera it carries, and respectively completes the recognition of the object and the calculation of its spatial coordinates; adds the object to the memory module, and updates the object map, which includes: a two-dimensional grid map and corresponding object information; determines whether the navigation target exists in the memory module; if not, goes to step S4; otherwise, goes to step S6;

[0009] Step S4: skeletonizing the two-dimensional grid map to obtain candidate points and paths;

[0010] Step S5: using the large language model and the object map information to score the candidate points based on the object and the path, the scores include: unknown area exploration score, historical trajectory score and navigation target finding score; the candidate point with the highest score is used as the target point;

[0011] Step S6: Based on the existing RGBD SLAM solution, the aircraft is navigated from the current position to the target point. At the same time, the memory module and the object map are updated during the navigation process to determine whether the navigation target has been found. If so, the navigation ends. Otherwise, go to step S3.

[0012] Beneficial effects:

[0013] 1. The present invention discloses a method for autonomous exploration and navigation of an aircraft indoors. With the help of the currently mature target detection model, the function of identifying objects within the field of view of the aircraft is completed. At the same time, the spatial information of the object is calculated by combining entity segmentation with depth images, so that it can be used as the basis for the aircraft to conduct navigation exploration and whether the mission is completed.

[0014] 2. The present invention is based on navigation targets in natural language form or picture form, and with the help of the common sense reasoning ability of the large language model, it greatly improves the exploration and navigation efficiency of the aircraft in complex indoor environments. The large language model first cooperates with the memory module to complete the efficient matching of the navigation targets by reading the existing object categories in the memory module. Secondly, it completes the summary of the candidate point path information and combines the navigation target information to complete the scoring of the candidate points, providing a basis for the selection of the next navigation target point.

[0015] 3. In the navigation process of the present invention, the aircraft uses the graph structure RVG to systematize the planning process into navigation subtasks across graph nodes. If the navigation target is matched in the field of view or the memory module, the minimum cost node between the aircraft position and the navigation position is planned, and the common path planning algorithm is applied to complete the local path planning, and finally the sub-paths are connected together to obtain the final path; if the navigation target is not matched, the candidate points are scored based on the large language model and the comprehensive object and path information, including the contribution score to the exploration of unknown areas, the possibility of finding the target, and whether the three parts have been passed, and the candidate point with the highest score is taken as the exploration target point, and the object map and the map module are continuously updated in the process of reaching the navigation target point until the navigation target is matched. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of an indoor autonomous exploration and navigation method for an aircraft according to the present invention;

[0017] Figure 2 Schematic diagram of target detection results;

[0018] Figure 3 Schematic diagram of object mask image;

[0019] Figure 4 It is a schematic diagram of a two-dimensional grid map;

[0020] Figure 5 It is a schematic diagram of the skeleton diagram;

[0021] Figure 6 It is a schematic diagram of node categories;

[0022] Figure 7 The present invention is a structural block diagram of an indoor autonomous exploration and navigation system for an aircraft. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] Embodiment 1

[0025] like Figure 1 As shown, an embodiment of the present invention provides an autonomous exploration and navigation method for an aircraft indoors, comprising the following steps:

[0026] Step S1: The aircraft receives the user's navigation target in natural language or picture form;

[0027] Step S2: Search the memory module to determine whether the navigation target exists in the memory module: if not, go to step S3; otherwise, go to step S6; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image;

[0028] Step S3: The aircraft first completes the action of rotating in place at the current position, obtains the RGB image and depth image of the surrounding objects through the depth camera it carries, and completes the recognition of the object and the calculation of its spatial coordinates respectively; adds the object to the memory module, and updates the object map, which includes: a two-dimensional grid map and corresponding object information; determines whether the navigation target exists in the memory module; if not, goes to step S4; otherwise, goes to step S6;

[0029] Step S4: skeletonize the two-dimensional grid map to obtain candidate points and paths;

[0030] Step S5: Use the large language model and object map information to score the candidate points based on the object and the path. The scores include: score for exploring unknown areas, score for historical trajectories, and score for finding navigation targets. The candidate point with the highest score is used as the target point.

[0031] Step S6: Based on the existing RGBD SLAM solution, the aircraft is navigated from the current position to the target point. At the same time, the memory module and the object map are updated during the navigation process to determine whether the navigation target has been found. If so, the navigation ends. Otherwise, go to step S3.

[0032] In one embodiment, the above step S1: the aircraft receives the user's navigation target in natural language or in picture form, specifically includes:

[0033] The aircraft is in an unexplored indoor environment, and obtains a navigation target in the form of natural language or picture input by the user on a web page or APP. For example, the user inputs: "Please find a sofa", which is sent to the aircraft system through data transmission and other means.

[0034] In one embodiment, the above step S2: searching the memory module to determine whether the navigation target exists in the memory module: if not, go to step S3; otherwise, go to step S6; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image, specifically including:

[0035] Step S21: using the common sense reasoning ability of the large language model to find out the possible object category to which the navigation target belongs in the memory module;

[0036] The memory module first merges the detected objects by object category, and then merges them according to whether they are the same individual within the same object category. For example, when the navigation goal is "please find a sofa", the object categories in the memory module are ["fruit", "furniture", "people", "TV"]. The large language model will infer that the sofa belongs to "furniture". Therefore, the large language model will output "furniture", thereby reducing the scope of the next search.

[0037] Step S22: Calculate the similarity between all individuals contained in the possible object types and the navigation target. If the maximum similarity is greater than a preset threshold, it means that the navigation target exists in the memory module, otherwise, it does not exist.

[0038] Since the RGB images of all individuals have been stored in the memory module, a suitable similarity calculation method can be used. For example, if the input navigation target is in text form, the cosine similarity of the clip vector can be used to measure the similarity. If the input navigation target is an image, the image similarity can be used for measurement.

[0039] In the embodiment of the present invention, similarity calculation is performed between the sofa and all objects in the furniture category in the memory module to determine whether the sofa exists in the memory module.

[0040] In one embodiment, in the above step S3: the aircraft first completes the rotation in place at the current position, obtains the RGB image and depth image of the surrounding objects through the depth camera it carries, and respectively completes the recognition of the object and the calculation of its spatial coordinates; adds the object to the memory module, and updates the object map, the object map includes: a two-dimensional grid map and corresponding object information; determines whether the navigation target exists in the memory module; if not, goes to step S4; otherwise, goes to step S6, which specifically includes:

[0041] Step S31: the aircraft first completes an in-situ rotation at the current position, and acquires RGB images and depth images of surrounding objects through the depth camera carried by the aircraft;

[0042] Step S32: Use the target detection network to perform target detection based on the RGB image to achieve object recognition, and obtain the pixel coordinates and category of the object; then use the instance segmentation network to perform instance segmentation on the RGB image and the depth image to obtain the mask image of the object, and then combine the depth image to obtain the spatial coordinates of the object;

[0043] The system in the embodiment of the present invention uses the target detection network YOLOv8, the instance segmentation network SAM, and the aircraft is equipped with a RealSense depth camera. First, YOLOv8 is used to perform bounding box level target detection on the RGB image, such as Figure 2The following figure shows the target detection result. At the same time, SAM performs instance segmentation based on the bounding box and estimates the depth information of the object by projecting the depth image on the instance segmentation result. Figure 3 The figure shows the object mask image after the depth image is projected, and then combined with the camera's internal and external parameter matrices to obtain the spatial coordinates in the world coordinate system.

[0044] Step S33: adding the object detected in step S31 to the memory module, including the object type, object space coordinates and object RGB image;

[0045] The basis for distinguishing whether the objects are the same object in the memory module in the embodiment of the present invention is the spatial coordinates of the object, that is, objects of the same type within a certain threshold range are regarded as the same object. At the same time, when the RGB images of objects appearing within a certain threshold range are too similar, they will be determined as duplicate images and will not be stored.

[0046] Step S34: updating the object map according to the object detected in step S31, wherein the two-dimensional grid map is a two-dimensional array, wherein each element has three states, namely occupied, unoccupied and unknown, generated by the SLAM algorithm, and adding object information at the corresponding position of the two-dimensional grid map according to the spatial coordinates of the object;

[0047] like Figure 4 Shown is a schematic diagram of a two-dimensional grid map, where white areas represent unoccupied areas, black areas represent occupied areas, and gray areas represent unknown areas.

[0048] Based on the type of the object and its spatial position information obtained in step S32 and combined with the two-dimensional grid map, the object position can be added and the object map can be updated.

[0049] Step S35: Calculate the similarity between the navigation target and the object detected in step S31. If the similarity is greater than a preset threshold, go to step S6; otherwise, go to step S4.

[0050] According to the input form of the navigation target, a suitable similarity calculation method is used to determine whether the navigation target exists in the newly detected object in step S3. If so, go to step S6, otherwise go to step S4.

[0051] In one embodiment, the above step S4: skeletonizing the two-dimensional grid map to obtain candidate points and paths specifically includes:

[0052] Step S41: using a skeleton algorithm to skeletonize the unoccupied part of the two-dimensional grid map to obtain a skeleton map;

[0053] The skeleton graph of the embodiment of the present invention is also called the generalized Voronoi graph (GVD), which refers to a special map where the points on it are equidistant from the two nearest obstacle points, representing the central axis path of the unoccupied space outside obstacles of any shape. Subsequently, in order to reflect the connectivity and accessibility of the uncovered map, the GVD can be processed into an RVG graph, that is, a graph with nodes V and edges E. The nodes correspond to the intersections or endpoints of the GVD, and the line segments directly connecting two adjacent nodes are identified as edges. The original graph is then preprocessed by operations such as merging approximate nodes and eliminating trivial bifurcations. Using the RVG graph, the planning process is systematized into navigation subtasks across graph nodes.

[0054] like Figure 5 As shown, Figure 4 The skeleton graph corresponds to the unoccupied part in the two-dimensional grid map, where the hollow nodes represent the nodes extracted from the skeleton graph.

[0055] Step S42: The endpoints and bifurcation points leading to the unknown area in the skeleton graph are taken as nodes, and the nearest node is found according to the current position of the aircraft as the aircraft node, all neighboring nodes of the aircraft node are taken as candidate points, and the shortest path from the aircraft node to each endpoint is taken as the path.

[0056] In an embodiment of the present invention, the endpoints and bifurcation points leading to unknown areas in the skeleton graph are taken as nodes, and they are divided into four categories according to the node positions: aircraft nodes, neighbor nodes, exploration nodes and ordinary nodes. Specifically, the node closest to the aircraft is designated as an aircraft node, representing the current decision position of the aircraft; the nodes directly connected to the aircraft node are regarded as neighbor nodes, which are used as candidate nodes for subsequent planning; nodes adjacent to unexplored areas and with only one connecting edge are classified as exploration nodes. All other nodes with multiple connecting edges are classified as ordinary nodes. The path refers to the shortest path from the aircraft node to each exploration node, and the neighbor nodes that appear in the path are candidate nodes.

[0057] like Figure 6 As shown in the figure, the hollow five-pointed star represents the actual position of the current aircraft, the black solid dot represents the aircraft node, the hollow triangle represents the neighbor node, the hollow diamond represents the exploration node, and the other hollow dots represent ordinary nodes.

[0058] In one embodiment, the above step S5: using the large language model and object map information to score the candidate points based on the object and the path, the scores include: exploring unknown area score, historical trajectory score and finding navigation target score; taking the candidate point with the highest score as the target point, specifically including:

[0059] Step S51: For each candidate point, whenever there is a shortest path from the aircraft node to the endpoint, the score of exploring the unknown area increases;

[0060] In the present invention, the Dijkstra algorithm is used to complete the calculation of the shortest path from the aircraft node to the exploration node, wherein the connectivity is guaranteed by the result of the skeleton graph, that is, if two nodes are connected in the skeleton graph, the distance between the two is the distance in the skeleton graph, otherwise it is infinite.

[0061] Step S52: For each candidate point, if the aircraft has passed the point before, the history score is reduced;

[0062] Step S53: for each candidate point, use the large language model to calculate its navigation target finding score;

[0063] For each candidate point, the aircraft will first turn to the candidate point to obtain the corresponding RGBD image of the candidate point, perform target detection on it, obtain the object information contained in the candidate point and input it into the large language model to obtain a reasonable description of the candidate point; then, based on all objects on the shortest path passing through the candidate point, a reasonable description of the path of the candidate point is obtained through the large language model, and finally the navigation target, the picture description corresponding to each candidate point and the path description are input into the large language model to obtain the final target finding score corresponding to each candidate point.

[0064] In this step, the large language model is used to describe the candidate points based on the result of target detection. For example, when the result of target detection is '["TV", "person", "table", "cup", "fruit"]', the large model may describe it as "the candidate point may be in a living room". At the same time, based on the historical object detection data, the objects that each path may pass through can be obtained. Assuming that the path passing through candidate point 1 includes objects such as people and TV, the large model may get the following input: "{candidate point: point 1, path objects: ["TV", "person", "table", "cup", "fruit"], description: "the candidate point may be in a living room"}, {candidate point: point 2, path objects: ["chair", "refrigerator", "fork"], description: "the candidate point may be in a restaurant"}", and if the navigation target is a sofa, because it is more relevant to the living room, candidate point 1 will get a higher score.

[0065] Step S54: The sum of the unknown area exploration score, history score and navigation target finding score of each candidate point is taken as its final score, and the candidate point with the highest final score is taken as the target point.

[0066] In one embodiment, the above step S6: completes the navigation of the aircraft from the current position to the target point based on the existing RGBD SLAM solution, and completes the update of the memory module and the object map during the navigation process, and determines whether the navigation target has been found; if so, the navigation ends; otherwise, go to step S3.

[0067] Embodiment 2

[0068] like Figure 7 As shown, an embodiment of the present invention provides an indoor autonomous exploration and navigation system for an aircraft, including the following modules:

[0069] Obtaining navigation target module 71, used for the aircraft to receive the user's navigation target in natural language form or picture form;

[0070] The memory module search module 72 is used to search the memory module to determine whether the navigation target exists in the memory module: if not, it goes to the update memory module and the object map module; otherwise, it goes to the navigation module; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image;

[0071] The memory module and object map module 73 are updated to firstly rotate the aircraft in place at the current position, obtain the RGB image and depth image of the surrounding objects through the depth camera carried by the aircraft, and respectively complete the object recognition and the calculation of the spatial coordinates; add the object to the memory module, and update the object map, which includes: a two-dimensional grid map and the corresponding object information; determine whether the navigation target exists in the memory module; if not, go to the candidate point acquisition module; otherwise, go to the navigation module;

[0072] A candidate point acquisition module 74 is used to skeletonize the two-dimensional grid map to obtain candidate points and paths;

[0073] The candidate point scoring module 75 is used to score the candidate points based on the object and the path using the large language model and the object map information. The scores include: the score of exploring unknown areas, the score of historical trajectories, and the score of finding navigation targets; the candidate point with the highest score is used as the target point;

[0074] The navigation module 76 completes the navigation of the aircraft from the current position to the target point based on the existing RGBD SLAM solution, and completes the update of the memory module and the object map during the navigation process to determine whether the navigation target has been found; if so, the navigation ends; otherwise, it goes to the update of the memory module and the object map module.

Claims

1. A method for autonomous exploration and navigation of an aircraft indoors, characterized in that: The steps include: Step S1: The aircraft receives the user's navigation target in natural language or picture form; Step S2: Search the memory module to determine whether the navigation target exists in the memory module: if not, go to step S3; Otherwise, go to step S6; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image; Step S3: The aircraft first completes the action of rotating in place at the current position, obtains the RGB image and depth image of the surrounding objects through the depth camera it carries, and respectively completes the recognition of the object and the calculation of its spatial coordinates; adds the object to the memory module, and updates the object map, which includes: a two-dimensional grid map and corresponding object information; determines whether the navigation target exists in the memory module; if not, goes to step S4; otherwise, goes to step S6; Step S4: skeletonizing the two-dimensional grid map to obtain candidate points and paths; Step S5: using the large language model and the object map information to score the candidate points based on the object and the path, the scores include: unknown area exploration score, historical trajectory score and navigation target finding score; the candidate point with the highest score is used as the target point; Step S6: Based on the existing RGBD SLAM solution, the aircraft is navigated from the current position to the target point. At the same time, the memory module and the object map are updated during the navigation process to determine whether the navigation target has been found. If so, the navigation ends. Otherwise, go to step S3.

2. The method for autonomous exploration and navigation of an aircraft indoors according to claim 1, characterized in that: The step S2: searching the memory module to determine whether the navigation target exists in the memory module; if not, go to step S3; Otherwise, go to step S6; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image, specifically including: Step S21: using the common sense reasoning ability of the large language model to find out the possible object category to which the navigation target belongs in the memory module; Step S22: Calculate the similarity between all individuals included in the possible object types and the navigation target. If the maximum similarity is greater than a preset threshold, it means that the navigation target exists in the memory module, otherwise, it does not exist.

3. The method for autonomous exploration and navigation of an aircraft indoors according to claim 2, characterized in that: Step S3: the aircraft first completes an in-situ rotation at the current position, obtains RGB images and depth images of surrounding objects through the depth camera carried by it, and respectively completes the recognition of the objects and the calculation of their spatial coordinates; Adding an object to the memory module and updating an object map, the object map including: a two-dimensional grid map and corresponding object information; determining whether the navigation target exists in the memory module; if not, going to step S4; otherwise going to step S6, specifically including: Step S31: the aircraft first completes an in-situ rotation at the current position, and acquires RGB images and depth images of surrounding objects through the depth camera carried by the aircraft; Step S32: using a target detection network to perform target detection based on the RGB image to achieve object recognition, and obtain pixel coordinates and categories of the object; then using an instance segmentation network to perform instance segmentation on the RGB image and the depth image to obtain a mask image of the object, and then combining the depth image to obtain the spatial coordinates of the object; Step S33: adding the object detected in step S31 to the memory module, including the object type, object space coordinates and object RGB image; Step S34: updating the object map according to the object detected in step S31, wherein the two-dimensional grid map is a two-dimensional array, wherein each element has three states, namely occupied, unoccupied and unknown, generated by the SLAM algorithm, and adding object information at the corresponding position of the two-dimensional grid map according to the spatial coordinates of the object; Step S35: Calculate the similarity between the navigation target and the object detected in step S31. If the similarity is greater than a preset threshold, go to step S6; otherwise, go to step S4.

4. The method for autonomous exploration and navigation of an aircraft indoors according to claim 3, characterized in that: The step S4: skeletonizing the two-dimensional grid map to obtain candidate points and paths, specifically includes: Step S41: using a skeleton algorithm to skeletonize the unoccupied part of the two-dimensional grid map to obtain a skeleton map; Step S42: The endpoints and bifurcation points leading to the unknown area in the skeleton graph are taken as nodes, and the nearest node is found according to the current position of the aircraft as the aircraft node, all neighboring nodes of the aircraft node are taken as candidate points, and the shortest path from the aircraft node to each of the endpoints is taken as the path.

5. The method for autonomous exploration and navigation of an aircraft indoors according to claim 4, characterized in that: The step S5: using the large language model and the object map information to score the candidate points based on the object and the path, the scores include: unknown area exploration score, historical trajectory score and navigation target finding score; taking the candidate point with the highest score as the target point, specifically including: Step S51: For each candidate point, whenever there is a shortest path from the aircraft node to the endpoint, the score of exploring the unknown area increases; Step S52: For each candidate point, if the aircraft has passed the point before, the history score is reduced; Step S53: for each candidate point, using the large language model to calculate its navigation target finding score; Step S54: The sum of the unknown area exploration score, the history score and the navigation target finding score of each candidate point is taken as its final score, and the candidate point with the highest final score is taken as the target point.

6. An autonomous exploration and navigation system for an aircraft indoors, characterized in that: Includes the following modules: The navigation target acquisition module is used for the aircraft to receive the user's navigation target in the form of natural language or pictures; A memory module retrieval module is used to retrieve the memory module to determine whether the navigation target exists in the memory module: if not, it goes to the update memory module and object map module; Otherwise, go to the navigation module; wherein the information stored in the memory module includes: object type, object space coordinates and object RGB image; The memory module and the object map module are updated, and the aircraft first completes the rotation in place at the current position, obtains the RGB image and the depth image of the surrounding objects through the depth camera carried by the aircraft, and respectively completes the recognition of the object and the calculation of its spatial coordinates; adds the object to the memory module, and updates the object map, and the object map includes: a two-dimensional grid map and corresponding object information; determines whether the navigation target exists in the memory module; if not, turns to the candidate point acquisition module; otherwise, turns to the navigation module; A candidate point acquisition module is used to skeletonize the two-dimensional grid map to obtain candidate points and paths; A candidate point scoring module is used to score the candidate points based on the object and the path using a large language model and object map information, the scores including: score for exploring unknown areas, score for historical trajectories, and score for finding navigation targets; the candidate point with the highest score is used as the target point; The navigation module completes the navigation of the aircraft from the current position to the target point based on the existing RGBD SLAM solution, and completes the update of the memory module and the object map during the navigation process, and determines whether the navigation target has been found; if so, the navigation ends; otherwise, it goes to update the memory module and the object map module.

Images