A target navigation method and system driven by directional gestures and language instructions

Abstract

The present disclosure relates to a target navigation method and system driven by directional gestures and language instructions, including: calculating the direction of the human body directional gesture according to the acquired human body initialization image; constructing a local multi-layer map in combination with the direction of the human body directional gesture and the panoramic depth image, fusing it with the global multi-layer map, updating the global multi-layer map, and judging whether there is a candidate target object, selecting an action execution position, and determining the shortest path between the robot's current position and the action execution position; reaching the action execution position according to the shortest path, and executing the action; if performing an exploration action, updating the global multi-layer map; if performing an observation action, determining whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; repeating the above steps until the target navigation end point has been reached. The disclosed method improves the accuracy of the definition of target attributes and spatial relationships in target navigation, and has stronger generalization and flexibility.

Description

Technical Field

[0001] The present disclosure relates to the field of mobile robot navigation, and in particular to a target navigation method and system driven by directional gestures and language instructions. Background Art

[0002] Target navigation is an important basic task in the field of mobile robot research. The robot makes corresponding action decisions by perceiving the environment and interacting with humans, thereby exploring unknown environments and moving to the vicinity of pre-defined targets in order to perform subsequent terminal tasks.

[0003] The definition of navigation targets is generally to directly input the target category and attributes to the agent as the initial parameters of the task, or to indirectly input the target category and surrounding environment and other related information to the agent in the form of natural language, requiring the agent to understand the target attributes from the language. However, when the task space contains multiple objects of the same category, it is often easy to have problems such as unclear spatial relationship description and unclear target definition.

[0004] The existing target navigation methods mostly recognize target objects through target detection, semantic segmentation, instance segmentation and other technologies. However, these methods have made advance limitations on the categories of target objects and can only identify target objects within the predetermined category range, which is not conducive to generalizing the target navigation methods to various application scenarios. Summary of the invention

[0005] In order to solve the problems that traditional target navigation methods are prone to unclear spatial relationship description, unclear target definition, and insufficient generalization when the task space contains multiple objects of the same category, the present disclosure proposes a target navigation method driven by directional gestures and language commands to solve the above problems.

[0006] According to one aspect of the present disclosure, a target navigation method driven by directional gestures and language instructions is provided, comprising:

[0007] S10, obtaining a human body initialization image, and calculating the direction of the human body directional gesture according to the human body initialization image;

[0008] S20, constructing a local multi-layer map by combining the direction of the human directional gesture and the panoramic depth image, fusing the local multi-layer map with the global multi-layer map, and updating the global multi-layer map;

[0009] S30, selecting an action execution location by determining whether a candidate target object exists in the updated global multi-layer map;

[0010] S40, determining the shortest path between the current position of the robot and the action execution position according to the action execution position;

[0011] S50, reaching the action execution location according to the shortest path, and executing the action;

[0012] S60. When performing the exploration action, return to step S20 and update the global multi-layer map; when performing the observation action, determine whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; repeat the above steps until the target navigation end point has been reached.

[0013] Preferably, calculating the direction of the directional gesture of the human body according to the initialization image of the human body includes:

[0014] The human key points are extracted from the human initialization image through the human key point detection model, the coordinates of the human key points are fitted into the human body area of ​​the panoramic depth image, and the spatial coordinates p of each human key point are calculated. i , expressed as:

[0015]

[0016] In the formula, is the homogeneous coordinate of the pixel coordinate of the human key point i in the panoramic depth image, z i is the corresponding depth value, K is the camera intrinsic parameter matrix;

[0017] The shoulder key point among the key points of the human body is used as the starting point of the directional gesture direction, and the elbow, wrist, index finger and little finger among the key points of the human body form a point set. A point in the point set is selected as the end point of the directional gesture direction. The direction of the directional gesture is:

[0018]

[0019] Where v is the direction of the directional gesture, α is the shoulder key point among the key points of the human body, and β is the end point of the direction of the directional gesture.

[0020] Preferably, the global multi-layer map includes: an occupancy layer, a wall-door layer, a node layer and / or a gesture direction layer;

[0021] The occupied layer includes: an obstacle area, a passable area, an unknown area and / or an edge area.

[0022] Preferably, the gesture direction layer is: projecting the direction v of the directional gesture onto a two-dimensional plane to obtain a plane direction vector v 2d , the plane direction vector v 2d Rotate 8° clockwise and counterclockwise respectively, and the angle area of ​​the two rays is used as the effective area of ​​the gesture direction layer.

[0023] Preferably, the action execution position is selected by judging whether there is a candidate target object in the updated global multi-layer map, including:

[0024] When there is a candidate target object in the global multi-layer map, the optimal observation point p near the candidate target object is calculated. o , and use it as the action execution location;

[0025] When there is no candidate target object in the global multi-layer map, calculate the best exploration point p on the edge area e , and use it as the action execution location, where the edge area is the boundary area between the traversable area and the unknown area in the global multi-layer map.

[0026] Preferably, reaching the action execution location according to the shortest path and executing the action includes:

[0027] Determine the properties of the action according to the action execution position. e When the action execution position is p o , perform the observation action.

[0028] Preferably, determining whether the target navigation destination has been reached by judging whether the current location image matches the natural language instruction includes:

[0029] The current position image and natural language instructions are respectively input into the image encoder and the text encoder to obtain a feature vector, which is normalized. The dot product calculation is performed on the normalized feature vector to obtain the cosine similarity. The cosine similarity is multiplied by the scaling factor and binarized to obtain a matching result. When the matching result is 1, it indicates a match, and the target navigation end point is reached, and the navigation ends; when the matching result is 0, it indicates a mismatch, and the target navigation end point has not been reached, and the navigation continues, jumping to step S30.

[0030] According to one aspect of the present disclosure, a target navigation system driven by directional gestures and language instructions is provided, comprising:

[0031] The directional gesture direction acquisition module acquires the human body initialization image and calculates the direction of the human body directional gesture according to the human body initialization image;

[0032] The global multi-layer map update module combines the direction of human directional gestures and the panoramic depth image to build a local multi-layer map, merges the local multi-layer map with the global multi-layer map, and updates the global multi-layer map;

[0033] The action execution location selection module selects the action execution location by judging whether there is a candidate target object in the updated global multi-layer map;

[0034] The shortest path determination module determines the shortest path between the robot's current position and the action execution position according to the action execution position;

[0035] An action execution module, which reaches the action execution position according to the shortest path and executes the action;

[0036] The target navigation module, when performing exploration actions, returns to the global multi-layer map update module to update the global multi-layer map; when performing observation actions, it determines whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; the above process is repeated until the target navigation end point is reached.

[0037] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the above-mentioned target navigation method driven by directional gestures and language instructions.

[0038] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the target navigation method driven by the directional gesture and language instructions is implemented.

[0039] Compared with the prior art, the beneficial effects of the present disclosure are:

[0040] 1) The present invention constructs a multimodal human-computer interaction channel integrating gestures and languages ​​for real application scenarios, which enhances the accuracy of the definition of target attributes and spatial relationships in target navigation and improves the density of interactive information.

[0041] 2) The present disclosure constructs a multi-layer two-dimensional map, aggregating a variety of spatial information such as obstacles, traversable areas, edges, walls, waypoints and gesture directions, so that navigation planning can be more flexible. The displayed stored historical path information promotes long-term navigation planning.

[0042] 3) The present invention introduces the CLIP encoder to construct a language understanding model, removes the target category limitation for target navigation, realizes open target navigation, and has the advantages of zero sample and training-free, making the target navigation method more generalizable and flexible.

[0043] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0044] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0046] Figure 1 A flow chart of a target navigation method driven by directional gestures and language instructions is shown;

[0047] Figure 2 A flow chart showing a target navigation method in an example of the present disclosure is shown;

[0048] Figure 3 A flow chart of directional gesture direction calculation in an embodiment of the present disclosure is shown;

[0049] Figure 4 A multi-layer map layer structure diagram in an embodiment of the present disclosure is shown;

[0050] Figure 5 The structure diagram of the language understanding module in the embodiment of the present disclosure is shown;

[0051] Figure 6 The block diagram of the target navigation system driven by directional gestures and language commands is shown. DETAILED DESCRIPTION

[0052] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0053] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0054] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0055] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] Example 1

[0058] Based on the above ideas, the present invention proposes a target navigation method driven by directional gestures and language instructions. Figure 1 A flow chart showing a target navigation method driven by directional gestures and language instructions is shown. The method comprises:

[0059] S10, obtaining a human body initialization image, and calculating the direction of the human body directional gesture according to the human body initialization image;

[0060] S20, constructing a local multi-layer map by combining the direction of the human directional gesture and the panoramic depth image, fusing the local multi-layer map with the global multi-layer map, and updating the global multi-layer map;

[0061] S30, selecting an action execution location by determining whether a candidate target object exists in the updated global multi-layer map;

[0062] S40, determining the shortest path between the current position of the robot and the action execution position according to the action execution position;

[0063] S50, reaching the action execution location according to the shortest path, and executing the action;

[0064] S60. When performing the exploration action, return to step S20 and update the global multi-layer map; when performing the observation action, determine whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; repeat the above steps until the target navigation end point has been reached.

[0065] The present disclosure provides a target navigation method driven by directional gestures and language instructions. The specific flow chart is as follows: Figure 2 As shown, the following steps are included:

[0066] S10: Acquire a human body initialization image, and calculate the direction of the human body directional gesture according to the human body initialization image.

[0067] In this embodiment, a region growing algorithm is used to segment the human body region in the depth map, and the human body key points are extracted from the RGB image through the human body key point detection model in the open source Mediapipe framework.

[0068] The direction of the human body directional gesture is calculated according to the human body initialization image, including: extracting human body key points from the human body initialization image through a human body key point detection model, fitting the coordinates of the human body key points to the human body area of ​​the panoramic depth image, and calculating the spatial coordinates p of each human body key point i , expressed as:

[0069]

[0070] In the formula, is the homogeneous coordinate of the pixel coordinate of the human key point i in the panoramic depth image, z i is the corresponding depth value, K is the camera intrinsic parameter matrix;

[0071] Furthermore, the shoulder key point among the key points of the human body is used as the starting point of the directional gesture direction, and the elbow, wrist, index finger and little finger among the key points of the human body form a point set A. A point in the point set A is selected as the end point of the directional gesture direction. Then the direction of the directional gesture is:

[0072]

[0073] In the formula, v is the direction of the directional gesture, α is the shoulder key point among the key points of the human body, and β is the end point of the directional gesture. The key point close to the fingertips is preferred as the end point β of v. When the preferred key point is invalid, it is replaced in sequence from A according to the priority. Knowing the spatial coordinates of the key points α and β, the angle θ of the vector v can be easily obtained. The direction calculation process of the directional gesture is as follows: Figure 3 shown.

[0074] In the region growing algorithm, for a given panoramic depth image D and a set of human key points The depth value corresponding to each key point is The calibration points are grouped by the depth difference threshold τ1, and two calibration points (x i ,y i ) and (x j ,y j ) belong to the same group if Take out the group with the largest number of key points As the initial seed point set From the initial seed point set Starting from each point (x, y) in the graph, check the neighboring points (x, y) around it to determine whether their depth difference meets the growth condition. The growth condition of the neighboring points is Where τ2 is the depth difference threshold, and the adjacent points (x′, y′) that meet the conditions are added to the new seed point set Right now: This process is iterated repeatedly until there are no new adjacent points to add. at this time is the desired human body segmentation area.

[0075] S20: construct a local multi-layer map by combining the direction of the human body's directional gestures and the panoramic depth image, fuse the local multi-layer map with the global multi-layer map, and update the global multi-layer map.

[0076] In this embodiment, the panoramic depth image is converted into a three-dimensional point cloud, and the three-dimensional point cloud is projected onto a horizontal plane to form a two-dimensional plane map. The two-dimensional projection of the directional gesture direction is fused to construct a local multi-layer map. The local multi-layer map is fused with the global multi-layer map, and the global multi-layer map is updated. The global multi-layer map is initially empty.

[0077] Specifically, the initial position of the robot is taken as the center point of the global multi-layer map, and the initial orientation of the robot is taken as the forward direction of the global multi-layer map, such as Figure 4 As shown, the global multi-layer map includes: an occupancy layer, a wall-door layer, a node layer and / or a gesture direction layer. Among them:

[0078] 1) Occupancy layer: including obstacle area, passable area, unknown area and edge area.

[0079] The obstacle area refers to a non-ground and non-empty area in a three-dimensional space. The generation process is: converting the depth map into a three-dimensional point cloud, filtering the part of the point cloud that is higher than the ground, projecting it onto a two-dimensional plane, and generating the obstacle area.

[0080] The passable area refers to the area in the explored space in the environment where there are no obstacles. The generation process is: use the Bresenham algorithm to calculate all pixels on the line connecting the obstacle pixel point of this round and the robot's current position pixel point, and mark the non-obstacle pixels as the passable area.

[0081] The unknown area refers to a non-obstruction area and a non-passable area in a preset range of the occupied layer.

[0082] The edge area represents the boundary area between the passable area and the unknown area. After finding the above three areas, the edge area can be found. Suppose the mask image Contains three values Indicates unknown area. Indicates the passable area. represents the obstacle area, and the sliding window algorithm is used to calculate the edge area:

[0083]

[0084] In the formula, represents the edge area, r is the half length of the sliding window, δ is the Kronecker function, N m (x, y) represents the number of points with value m in the sliding window of point (x, y), T = (2r + 1) 2 is the number of points in the sliding window.

[0085] 2) Wall-door layer: The RANSAC algorithm is used to fit the wall plane in the point cloud data, and the point y with the largest y coordinate value in the 3D point cloud is used as the max As the benchmark, intercept the y coordinate belonging to [y max -l,y max ] is used as the sampling area of ​​the RANSAC algorithm. The y coordinate axis is perpendicular to the horizontal plane, l is the set threshold, and the number of sampling points is set to 3 points. When the plane a formed by the three points i x i +b i y i +c i z i +d i =0 normal vector (a i , b i , c i ) is perpendicular to the y-axis, plane a i x i +b i y i +c i z i +d i = 0 is a valid plane, and the points whose distance from the plane is less than the threshold are selected as the inner points of the wall. Set the sampling times to 1000 times, and repeat the above wall sampling process until the preset sampling times are reached.

[0086] The door area is calculated based on the wall area. At the same viewing angle, if there is a continuous area on the wall plane with a depth value greater than the wall depth and this area is close to the floor area, then this area can be judged as a door. The filtered point cloud is projected in two dimensions to obtain the wall-door layer.

[0087] 3) Node layer: The traversable area of ​​the occupied layer is divided into triangles through an incremental algorithm to generate a navigation grid. The midpoints of the common edges of two triangles are taken as valid nodes to form a node layer, which can ensure that the intelligent agent will not collide when moving between nodes. The node layer helps to simplify the path planning algorithm and improve computational efficiency.

[0088] 4) Gesture direction layer: Project the direction v of the directional gesture onto a two-dimensional plane to obtain the plane direction vector v 2d , the plane direction vector v 2dThe angle area between the two rays obtained by rotating 8° clockwise and counterclockwise is used as the effective area of ​​the gesture direction layer, where 8° is the measured directional gesture direction calculation error.

[0089] S30. Selecting an action execution location by determining whether there is a candidate target object in the updated global multi-layer map.

[0090] In this embodiment, a location for executing an action in this round is selected based on the global multi-layer map. The action execution location is selected by judging whether there is a candidate target object in the updated global multi-layer map, including:

[0091] When there is a candidate target object in the global multi-layer map, the optimal observation point p near the candidate target object is calculated. o , and use it as the action execution position, where the candidate target object is the object represented by the intersection of the effective area of ​​the gesture direction layer and the obstacle mask layer in the global multi-layer map;

[0092] When there is no candidate target object in the global multi-layer map, calculate the best exploration point p on the edge area e , and use it as the action execution location, where the edge area is the boundary area between the traversable area and the unknown area in the global multi-layer map.

[0093] Specifically, the best observation point p o The node closest to the candidate target object on the node layer is used as the starting point, and the first passable coordinate point less than 1m away from the candidate target object is found by using the BFS algorithm. It is required that there is no wall between the coordinate point and the node and the candidate target object; the best exploration point p e The distance from the point in the edge area to the effective area of ​​the gesture direction layer and the current robot position is used as the judgment standard, and the closer the distance, the better.

[0094] S40: Determine the shortest path between the current position of the robot and the action execution position according to the action execution position.

[0095] In this embodiment, the shortest path between the current position of the robot and the action execution position is planned by using the Dijkstra algorithm and the node layer.

[0096] S50: Arrive at the action execution location according to the shortest path, and execute the action.

[0097] In this embodiment, the attribute of the action is determined according to the action execution position. The action is divided into two types: observation and exploration. When the action execution position is p oWhen the action execution position is p, the observation action is to call the language understanding module to predict the degree of matching between the natural language instruction and the RGB image at the current position, and then judge whether the target navigation end point has been reached; when the action execution position is p e The exploration action is to rotate the depth camera, obtain the panoramic depth image, and update the global multi-layer map.

[0098] S60. When performing the exploration action, return to step S20 and update the global multi-layer map; when performing the observation action, determine whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; repeat the above steps until the target navigation end point has been reached.

[0099] In this embodiment, updating the global multi-layer map means acquiring a new panoramic depth image at the current position of the robot, and then optimizing and improving the global multi-layer map according to step S20.

[0100] By judging whether the current position image matches the natural language instruction, it is determined whether the target navigation end point has been reached, including: inputting the current position image and the natural language instruction into the image encoder and the text encoder respectively to obtain a feature vector, and normalizing it, performing dot product calculation on the normalized feature vector to obtain cosine similarity, multiplying the cosine similarity with a scaling factor, and performing binarization to obtain a matching result. When the matching result is 1, it indicates a match, and the target navigation end point has been reached, and navigation ends; when the matching result is 0, it indicates a mismatch, and the target navigation end point has not been reached, navigation continues, and jumps to step S30.

[0101] The language understanding module structure in this embodiment is as follows Figure 5 As shown in the figure, the single RGB image and natural language instruction at the current position are input into the CLIP image encoder and the CLIP text encoder to obtain the image feature vector and the text feature vector, and the normalized image feature vector φ rgb and the text feature vector φ txt Dot product calculation is performed to obtain their cosine similarity. Cosine similarity represents the degree of match between image and text. It is multiplied by the trainable scaling factor and binarized with 0.5 as the dividing line. The result is 1 for match and 0 for mismatch. Among them, the single image is an RGB view captured at the best observation point and should contain potential navigation target objects; the form of natural language instructions is similar to: "Go to the table in that bedroom. There is a laptop on the table."

[0102] In order to enhance the adaptability to specific tasks, the encoder of CLIP is fine-tuned, and the convergence of the model is accelerated by the trainable scaling factor. BCELoss is used as the loss function, and the calculation formula of the training loss is:

[0103]

[0104] In the formula, φ rgb and φ txt are the normalized image feature vector and text feature vector respectively, s is the trainable scaling factor, and the Limit function is used to limit the number to between 0 and 1, that is, if it is greater than 1, it takes 1, and if it is less than 0, it takes 0. is the predicted value of the model, y is the label of the text image pair, 0 is mismatch, 1 is match, and L is the loss value.

[0105] This embodiment provides a target navigation method driven by directional gestures and language instructions, constructs a multimodal human-computer interaction channel integrating gestures and language for real application scenarios, enhances the accuracy of the definition of target attributes and spatial relationships in target navigation, and improves the interactive information density. The CLIP encoder is introduced to build a language understanding model, removes the target category limitation for target navigation, and realizes open target navigation. At the same time, it has the advantages of zero sample and training-free, making the target navigation method more generalized and more flexible.

[0106] Example 2

[0107] As another aspect of the embodiment of the present disclosure, a target navigation system 100 driven by directional gestures and language instructions is also provided. Figure 6 As shown, including:

[0108] The directional gesture direction acquisition module 1 acquires a human body initialization image and calculates the direction of the human body directional gesture according to the human body initialization image;

[0109] The global multi-layer map updating module 2 constructs a local multi-layer map by combining the direction of the human directional gesture and the panoramic depth image, merges the local multi-layer map with the global multi-layer map, and updates the global multi-layer map;

[0110] The action execution location selection module 3 selects the action execution location by judging whether there is a candidate target object in the updated global multi-layer map;

[0111] The shortest path determination module 4 determines the shortest path between the current position of the robot and the action execution position according to the action execution position;

[0112] An action execution module 5 reaches the action execution position according to the shortest path and executes the action;

[0113] The target navigation module 6, when performing the exploration action, returns to the global multi-layer map update module to update the global multi-layer map; when performing the observation action, it determines whether the target navigation end point has been reached by judging whether the current position image matches the natural language instruction; repeat the above process until the target navigation end point is reached.

[0114] In the absence of any contradiction, the above modules in the system of the embodiment of the present disclosure can implement any implementation of the above method.

[0115] Based on the description of the above embodiments, it can be seen that the embodiments of the present disclosure can achieve the following technical effects:

[0116] 1) The present invention constructs a multimodal human-computer interaction channel integrating gestures and languages ​​for real application scenarios, which enhances the accuracy of the definition of target attributes and spatial relationships in target navigation and improves the density of interactive information.

[0117] 2) The present disclosure constructs a multi-layer two-dimensional map, aggregating a variety of spatial information such as obstacles, traversable areas, edges, walls, waypoints and gesture directions, so that navigation planning can be more flexible. The displayed stored historical path information promotes long-term navigation planning.

[0118] 3) The present invention introduces the CLIP encoder to construct a language understanding model, removes the target category limitation for target navigation, realizes open target navigation, and has the advantages of zero sample and training-free, making the target navigation method more generalizable and flexible.

[0119] The disclosed embodiment also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned target navigation method driven by directional gestures and language instructions. The electronic device can be provided as a terminal, a server, or other forms of devices.

[0120] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the target navigation method driven by the directional gesture and language instructions is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0121] Those skilled in the art will understand that in the above-mentioned target navigation method and system driven by directional gestures and language instructions in a specific implementation manner, the writing order of the steps does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0122] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0123] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A target navigation method driven by directional gestures and language instructions, characterized in that: The steps include: S10, obtaining a human body initialization image, and calculating the direction of the human body directional gesture according to the human body initialization image; S20, constructing a local multi-layer map by combining the direction of the human directional gesture and the panoramic depth image, fusing the local multi-layer map with the global multi-layer map, and updating the global multi-layer map; S30, selecting an action execution location by determining whether a candidate target object exists in the updated global multi-layer map; S40, determining the shortest path between the current position of the robot and the action execution position according to the action execution position; S50, reaching the action execution location according to the shortest path, and executing the action; S60, when performing the exploration action, return to step S20 to update the global multi-layer map; when performing the observation action, determine whether the target navigation destination has been reached by judging whether the current position image matches the natural language instruction; Repeat the above steps until you reach the target navigation destination.

2. The method according to claim 1, characterized in that The direction of the directional gesture of the human body is calculated according to the initialization image of the human body, including: The human key points are extracted from the human initialization image through the human key point detection model, the coordinates of the human key points are fitted into the human body area of ​​the panoramic depth image, and the spatial coordinates p of each human key point are calculated. i , expressed as: In the formula, is the homogeneous coordinate of the pixel coordinate of the human key point i in the panoramic depth image, z i is the corresponding depth value, K is the camera intrinsic parameter matrix; The shoulder key point among the key points of the human body is used as the starting point of the directional gesture direction, and the elbow, wrist, index finger and little finger among the key points of the human body form a point set. A point in the point set is selected as the end point of the directional gesture direction. The direction of the directional gesture is: Where v is the direction of the directional gesture, α is the shoulder key point among the key points of the human body, and β is the end point of the direction of the directional gesture.

3. The method according to claim 1, characterized in that: The global multi-layer map includes: an occupancy layer, a wall-door layer, a node layer and / or a gesture direction layer; The occupied layer includes: an obstacle area, a passable area, an unknown area and / or an edge area.

4. The method according to claim 3, characterized in that The gesture direction layer is: project the direction v of the directional gesture onto a two-dimensional plane to obtain the plane direction vector v 2d , the plane direction vector v 2d Rotate 8° clockwise and counterclockwise respectively, and the angle area of ​​the two rays is used as the effective area of ​​the gesture direction layer.

5. The method according to claim 4, characterized in that By judging whether there is a candidate target object in the updated global multi-layer map, the action execution location is selected, including: When there is a candidate target object in the global multi-layer map, the best observation point p near the candidate target object is calculated. o , and use it as the action execution location; When there is no candidate target object in the global multi-layer map, calculate the best exploration point p on the edge area e , and use it as the action execution location, where the edge area is the boundary area between the traversable area and the unknown area in the global multi-layer map.

6. The method according to claim 1, characterized in that Arriving at the action execution location along the shortest path and executing the action, including: Determine the properties of the action according to the action execution position. e When the action execution position is p o , perform the observation action.

7. The method according to claim 1, characterized in that By judging whether the current position image matches the natural language instruction, it is determined whether the target navigation destination has been reached, including: The current position image and natural language instructions are respectively input into the image encoder and the text encoder to obtain a feature vector, which is normalized. The dot product calculation is performed on the normalized feature vector to obtain the cosine similarity. The cosine similarity is multiplied by the scaling factor and binarized to obtain a matching result. When the matching result is 1, it indicates a match, and the target navigation end point is reached, and the navigation ends; when the matching result is 0, it indicates a mismatch, and the target navigation end point has not been reached, and the navigation continues, jumping to step S30.

8. A target navigation system driven by directional gestures and language instructions, characterized in that: include: The directional gesture direction acquisition module acquires the human body initialization image and calculates the direction of the human body directional gesture according to the human body initialization image; The global multi-layer map update module combines the direction of human directional gestures and the panoramic depth image to build a local multi-layer map, merges the local multi-layer map with the global multi-layer map, and updates the global multi-layer map; The action execution location selection module selects the action execution location by judging whether there is a candidate target object in the updated global multi-layer map; The shortest path determination module determines the shortest path between the robot's current position and the action execution position according to the action execution position; An action execution module, which reaches the action execution position according to the shortest path and executes the action; The target navigation module, when performing exploration actions, returns to the global multi-layer map update module to update the global multi-layer map; when performing observation actions, it determines whether the target navigation destination has been reached by judging whether the current position image matches the natural language instruction; Repeat the above process until the target navigation end point is reached.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the target navigation method driven by directional gestures and language instructions as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the target navigation method driven by directional gestures and language instructions as described in any one of claims 1 to 7 is implemented.

Images