Robot semi-autonomous navigation method and system based on gesture and voice interaction assistance

Abstract

The application belongs to the technical field of robot autonomous navigation, and discloses a robot semi-autonomous navigation method and system based on gesture and voice interaction assistance; the method comprises the following steps: step 1, generating gesture and voice control instructions; step 2, robot task training based on gesture and voice interaction; and step 3, robot online motion planning based on gesture and voice interaction. The application performs gesture recognition and silent voice recognition through an augmented reality device, and continuously updates a knowledge base based on the offline task training of the robot from visual gesture interaction and silent voice interaction, the trajectory planning and optimization algorithm of the prior knowledge base and itself, and the continuous online adjustment of the motion trajectory, so as to realize the semi-autonomous navigation of the robot. The application can more intuitively perform offline task training and online task planning on the robot, and the combination of gesture and silent voice recognition can effectively improve the autonomous navigation performance of the robot and optimize the planned trajectory.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot autonomous navigation, and particularly relates to a robot semi-autonomous navigation method and system based on gesture and voice interaction assistance. BACKGROUND

[0002] At present, the autonomous navigation technology of robots is increasingly mature, but the existing online motion planning method is difficult to ensure that the robot moves efficiently and robustly in unknown and highly complex environments.

[0003] Chinese patent CN114463833A discloses an Android human-computer interaction method based on a MediaPipe gesture recognition model. The hand motion image obtained is input into the gesture recognition model to identify the current gesture action in real time. However, only the offset information of the palm moving left and right is considered, and the offset angle of the palm movement is not considered.

[0004] Chinese patent CN107150347B discloses a robot perception and understanding method based on human-computer cooperation. The operator describes the target online through gesture and voice interaction, and the robot learns the task teaching trajectory by understanding the operator's gesture and voice instructions. Based on the reinforcement learning model, the operator gives the robot behavior evaluation online, and continuously corrects the robot's behavior through reward and punishment signals. However, only the results of subtask execution are corrected, and the local minimum value in the process of the robot executing the task teaching trajectory is not considered.

[0005] Chinese patent CN106095109B discloses a method of robot online teaching through gesture coarse adjustment and voice fine adjustment. The voice instruction is divided into control command and command command, and the operator starts or ends the robot online teaching process or switches the gesture instruction and the voice instruction through the control command. The gesture instruction and the voice instruction are used to control the robot to perform large amplitude motion and small amplitude motion respectively, or the two are combined to realize the more flexible and natural motion of the robot. However, the specific situation of the robot action in the process of the robot online task teaching and the specific cooperation strategy of the gesture coarse adjustment and the voice fine adjustment are not considered.

[0006] Zhou B, Gao F, Pan J, et al. Robust real-time UAV replanning using guided gradient-based optimization and topological paths. IEEE International Conference on Robotics and Automation (ICRA), 2020: 1208-1214, proposed a gradient-based trajectory optimization method (GTO), designed a path-guided optimization (PGO) method to solve the infeasible local minimum problem, thereby improving the success rate of trajectory replanning, in addition, a topological path search algorithm was proposed to capture different available paths in three-dimensional environment, thereby realizing real-time topological trajectory replanning. Benchmark evaluation shows that the proposed method is superior to the state-of-the-art method in terms of replanning success rate and optimality, and in the aggressive flight experiment, the effectiveness of the proposed topological path search method is proved. However, for some complex environments, the adaptability of this method is poor, and sometimes the generated trajectory cannot guarantee the smooth and safe flight of the UAV.

[0007] Zhou X, Wang Z, Ye H, et al. Ego-planner: An ESDF-free gradient-based local planner for quadrotors. IEEE Robotics and Automation Letters, 2020, 6(2): 478-485, proposed an ESDF-free gradient-based local planner, which achieved comparable performance to the current state-of-the-art ESDF planner, and reduced more than an order of magnitude of time, through the comparison of benchmark experiments and real experiments, the robustness and efficiency of the proposed method were proved. However, the proposed method uses A* search, which leads to local optimal trajectory, and due to the use of uniform time allocation, the trajectory is biased towards conservatism.

[0008] In summary, in the field of robot autonomous navigation, most of the research focuses on using the on-board resources of the system itself to complete the online motion planning of the robot through some path search algorithms and trajectory planning algorithms, there is a problem of local optimal solution, and there is no method of task training of the robot by external intervention to optimize the motion trajectory. Therefore, developing and designing a gesture and voice interaction assisted robot semi-autonomous navigation system and method has very important research significance for improving the autonomous navigation performance of the robot, which can effectively solve the problem of local minimum value encountered by the robot in the motion planning process. SUMMARY

[0009] The present application aims to provide a gesture and voice interaction assisted robot semi-autonomous navigation method and system, which trains the robot offline through gesture and voice interaction, and uses gesture and voice interaction as a publisher of motion planning waypoint information during online motion planning of the robot, and the robot continuously adjusts the motion trajectory according to the established knowledge base and its own motion planning algorithm.

[0010] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0011] A gesture and voice interaction assisted robot semi-autonomous navigation method, comprising the following steps:

[0012] Step 1. Generating gesture and voice control instructions;

[0013] The original gesture image information and lip image information are captured through an augmented reality helmet, and the gesture instructions and voice instructions are recognized by the intelligent device;

[0014] Step 2. Robot task training based on gesture and voice interaction;

[0015] The robot is taught to perform tasks offline, and the robot will continuously update the experience base and knowledge base in the reinforcement learning process, thereby providing support for online obstacle avoidance of the robot;

[0016] Step 3. Robot online motion planning based on gesture and voice interaction;

[0017] Based on the established knowledge base, the waypoint information of the robot is published through voice instructions to control the robot to move in a small range, and the robot is controlled to move in a large range through gesture instructions, the voice and gesture instructions are cooperatively controlled, the direction of movement of the robot is controlled according to the gesture instructions, and the displacement of the robot is read according to the voice instructions; when the robot encounters an obstacle, the trajectory planning and optimization algorithm of the robot is called, and the trajectory is continuously corrected according to the established knowledge base, and finally the optimal motion trajectory is generated.

[0018] Preferably, step 1 comprises the following steps:

[0019] Step 1.1: The generation of gesture control commands includes the generation of the direction the gesture is pointing and the gesture's displacement;

[0020] Step 1.2: The generation of voice control commands includes the generation of control commands and command commands.

[0021] Step 1.1 is as follows: When a gesture action is generated, the key point information of the knuckles in each frame is obtained in a loop. The offset displacement of the key points between two frames is taken as the displacement of the gesture movement, and the offset angle of the key points between two frames is taken as the direction of the gesture. The calculation is repeated n times, and the offset displacement and offset angle are continuously superimposed to obtain the displacement value and direction information of the current gesture.

[0022] The formula for calculating the superimposed offset is as follows:

[0023]

[0024] The formula for calculating the superimposed offset angle is as follows:

[0025]

[0026] in, This is the offset displacement. For the first The key points of each joint For the first The offset angle corresponding to each joint.

[0027] Preferably, in step 1.2, the control command input selection parameter ( ), Method selection Switching between gestures and voice. Gestures and voice commands specify the direction the robot will move. Gestures and voice commands specify the robot's step size. : Displacement unit, command-based issuance of waypoint information for the robot ( ).

[0028] Preferably, step 2 includes the following steps:

[0029] Step 2.1. Issuance of gesture and voice commands;

[0030] Step 2.2. Task teaching for the robot;

[0031] Step 2.3. Robot's task learning;

[0032] Step 2.4 Robot obstacle avoidance teaching based on gesture and voice interaction.

[0033] Preferably, in step 2.1, when the robot needs to make a large motion, the operator switches the gesture and voice by voice instruction is set to gesture, and the direction of the robot movement is issued by gesture action and fixed movement step and displacement unit ; when the robot needs to make a small motion, the operator switches the gesture and voice by voice instruction is set to voice, and the direction of the robot movement is issued by voice input and movement step and displacement unit ; when the gesture and voice instructions need to be controlled cooperatively, the gesture and voice are switched is always set to 1, and the direction of the robot movement is the direction indicated by the current gesture , the current issued voice instruction sets the step of the robot movement and unit .

[0034] Preferably, step 2.2 is specifically:

[0035] Based on the established experience library, the operator uses gestures and voice to state task names, the number of subtasks, and the properties of the operation object, and sends the entire task analysis to the robot, and the robot uses a motion planning algorithm to generate an initial task teaching trajectory; during the motion of the robot, the operator observes the reproduced trajectory of the robot, and for some teaching points with an error greater than the expected error, adjusts through voice instruction, and continuously updates the experience library of task teaching until the task teaching is completed.

[0036] Preferably, step 2.3 is specifically:

[0037] Observe the working environment through a binocular camera, determine the poses of key obstacles and targets, and form an actual working trajectory by applying the task trajectory learned by itself to a given task plan; use a reinforcement learning method for modeling, and the operator continuously evaluates the behavior of the robot, and continuously corrects the behavior of the robot through reward and punishment signals; in the process of task learning, the robot continuously improves its knowledge base.

[0038] Preferably, step 2.4 specifically includes the following steps:

[0039] Step 2.4.1: Estimate the distance from the robot motion trajectory to the obstacle

[0040] Before the robot starts trajectory optimization, an initial B-spline curve that satisfies the terminal constraint and does not consider collision will be generated For each collision segment detected during the iteration process, a collision-free path is generated. Each control point in the collision segment Assign an anchor point on the collision surface. and the corresponding repulsive force direction ,in Indicates the index of the control point. express Correct index, therefore control point To the The distance to the obstacle will be defined as:

[0041] ;

[0042] Step 2.4.2: Apply collision penalty to the robot.

[0043] The safe clearance between the robot and the obstacle is defined as... In the collision penalty function Punishment factors incorporating gestures and voice intervention and reward factors .

[0044]

[0045]

[0046] in, For safety clearance With control points To the The difference in distance between obstacles, To be at the control point superior For the resulting collision costs, each control point The collision costs are assessed independently and from the corresponding Accumulate the values ​​in the middle, and finally obtain the result added to the first position. Cost at each control point:

[0047]

[0048] in, Control point superior The correct number, all control points The costs are combined to obtain the total cost:

[0049] .

[0050] Preferably, the trajectory planning and optimization algorithm in step 3 specifically includes:

[0051] Step 3.1: the topological path search method based on informed sampling provides an initial path, and a geometric guiding method is used to deform the initial trajectory; a smooth trajectory, i.e. a hot trajectory, is output near the geometric guiding path; finally, a gradient-based optimization method is used to further improve the trajectory quality;

[0052] Step 3.2: the B-spline optimization method is used to further optimize the trajectory into a smooth, safe and dynamically feasible trajectory; the objective function includes three parts, i.e. a collision cost evaluated in the Euclidean symbol field, an infeasible speed penalty term and an infeasible acceleration penalty term; the three parts of the objective function are optimized based on the convex hull property of the B-spline.

[0053] Step 3.3: trajectory re-planning, a topological path search method based on informed sampling is combined with a path-guided trajectory optimization method to realize real-time trajectory re-planning.

[0054] A robot semi-autonomous navigation system based on gesture and voice interaction assistance, comprising an augmented reality helmet and a ROS server; the ROS server comprises a gesture and voice recognition model, a gesture and voice recognition client, a robot controller server, a database and a visualization interface;

[0055] The monocular camera carried by the augmented reality helmet captures raw gesture image information and lip image information, which are input into the gesture and voice recognition model, and the recognition results of the gesture and voice are stored in the database in a serialized form; the gesture and voice recognition client is used to publish gesture and voice instructions; based on the ROS service and topic communication mechanism, the robot controller server defines corresponding service names and topic names, as well as service message types and topic message types;

[0056] When publishing instructions, the gesture and voice recognition client reads instruction parameters from the database, maps the instruction parameters into the robot's waypoint information, and publishes the instruction parameters to the robot controller server in the form of a service; after receiving the message, the robot controller server immediately feeds back to the gesture and voice recognition client, and publishes the received waypoint information to the robot's corresponding subscriber in the form of a topic, to drive the robot's actuator to perform actions.

[0057] Preferably, the visualization interface is used to display gesture recognition and voiceless speech recognition instructions, and is used in the offline stage to set up task training, task demonstration and task learning for robot motion planning, and to intuitively correct the robot's planning trajectory; in the online stage, the visualization interface is used to assist the robot's autonomous navigation process, to real-time feedback the robot's motion state, and to facilitate the use of gestures and voice to publish tasks, or to correct emergency error conditions.

[0058] Advantages

[0059] The present application carries out gesture recognition and voiceless speech recognition through an augmented reality device, starting from visual gesture interaction and voiceless speech interaction, and based on the interaction characteristics of both, the robot is trained offline, the knowledge base is constantly updated, in the offline task training stage, gesture recognition and voiceless speech recognition demonstrate the task to the robot, and assist the robot in obstacle avoidance as external intervention factors; in the task learning process, the robot constantly updates the knowledge base and records the trajectory optimization factor; in the online motion planning stage, gesture recognition and voiceless speech recognition serve as a waypoint information publisher, and realize large-scale and small-scale actions of the robot in a cooperative manner, and start and end the motion planning process of the robot, based on the prior knowledge base and the trajectory planning and optimization algorithm of the robot, the motion trajectory is constantly adjusted online, so as to realize semi-autonomous navigation of the robot.

[0060] The present application can more intuitively train and plan the task of the robot offline, and the combination of gesture and voiceless speech recognition can effectively improve the autonomous navigation performance of the robot, optimize the planning trajectory, and make the actual moving trajectory of the robot more consistent with dynamic feasibility, smoothness and collision-free. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 A flowchart of a semi-autonomous navigation method of a robot assisted by gesture and speech interaction is provided for an embodiment of the present application;

[0062] Figure 2 A robot trajectory planning schematic diagram is provided for an embodiment of the present application;

[0063] Figure 3 A structure block diagram of a semi-autonomous navigation system of a robot assisted by gesture and speech interaction is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0065] As shown in Figure 1 The semi-autonomous navigation method of the robot assisted by gesture and speech interaction of the present application embodiment; includes the following steps:

[0066] Step 1. Generate gesture and speech control instructions.

[0067] The operator recognizes gesture and voice commands by wearing an augmented reality headset and an Android device. The augmented reality headset captures raw gesture and lip-reading images, while the Android device, acting as a data processing and model computation unit, recognizes gesture and silent voice commands. The specific steps include:

[0068] Step 1.1. The generation of gesture control commands mainly includes the generation of the direction the gesture is pointing and the displacement of the gesture movement. Based on the MediaPipe gesture recognition framework, the image information of the hand is acquired using the monocular camera mounted on the augmented reality headset and input into the gesture recognition model, which returns the key point position information of the hand in real time. When a gesture action is generated, the key point information of the knuckles in each frame is acquired in a loop. The offset displacement of the key points between two frames is taken as the displacement of the gesture movement, and the offset angle of the key points between two frames is taken as the direction the gesture is pointing. This calculation is repeated 20 times, and the offset displacement and offset angle are continuously superimposed to obtain the displacement value and direction information of the current gesture.

[0069] The formula for calculating the superimposed offset is as follows:

[0070]

[0071] The formula for calculating the superimposed offset angle is as follows:

[0072]

[0073] in, This is the offset displacement. For the first The key points of each joint For the first The offset angle corresponding to each joint.

[0074] Step 1.2. The generation of voice control commands mainly includes the generation of control commands and command commands. Control commands mainly input selection parameters ( ), Method selection Switching between gestures and voice. Gestures and voice commands specify the direction the robot will move. Gestures and voice commands specify the robot's step size. Displacement units, while command instructions primarily issue waypoint information for the robot. ). The operator's lip image information is collected by a monocular camera mounted on an augmented reality helmet. The original lip image sequence is subjected to video frame separation and RGB color image to grayscale image operation. The processed grayscale image is input into an end-to-end lip reading model. A three-dimensional convolutional neural network extracts spatial feature information of the lip image sequence. A residual network ResNet34 optimizes neural network training. High-dimensional lip image convolution feature sequences are reduced in dimension. A gated recurrent unit GRU further extracts time sequence information of the lip image feature sequence. Finally, a Softmax classifier identifies the lip reading result.

[0075] Step 2. Robot task training based on gesture and voice interaction.

[0076] The robot task training process mainly issues instructions through gestures and voices, and off-line demonstrates the robot to perform tasks. The robot will continuously update the experience base and knowledge base during the task learning process, thereby providing support for online obstacle avoidance of the robot. Specifically, the following steps are included:

[0077] Step 2.1. Gesture and voice instruction issuing

[0078] Since the voice instruction misrecognition rate is relatively low, the voice instruction is considered to be used to control the instruction issuing and selection of the robot task demonstration. When the robot task demonstration is performed, the is set to 1, and the mode selection state is entered. When the operator issues a start voice command, the robot is in standby state after receiving the command, and is ready to receive new instructions at any time. When the robot needs to perform a large amplitude action, the operator sets the to 1 through a voice instruction, and the operator can issue the direction and the fixed moving step and the displacement unit of the robot movement through a gesture action. When the robot needs to perform a small amplitude action, the operator sets the to 0 through a voice instruction, and the operator issues the direction and the moving step of the robot movement through a voice input.

[0079] Further, since the gesture instruction often presents a trend or direction, which is suitable for functional control instructions, and the voice instruction is inclined to be a fine control, which is suitable for specific command instructions, the two are considered to be combined to demonstrate the robot task. When the gesture and voice instructions need to be cooperatively controlled, the is always set to 1, the direction of the robot movement is the direction indicated by the current gesture , and the step and the unit of the robot movement are set by the currently issued voice instruction Finally, the whole task teaching process is ended by voice instruction. The robot's movement is controlled by gesture and voice, which is more flexible and natural.

[0080] Step 2.2. Task teaching of robot

[0081] Based on the established experience base, the operator uses gestures and voice to state the task name, the number of sub-tasks, and the properties of the operating object, etc. information, and sends the whole task analysis to the robot. The robot generates an initial task teaching trajectory using motion planning algorithm. During the movement of the robot, the operator observes the reproduced trajectory of the robot, and for some teaching points with error greater than the expected error, adjusts through voice instruction, and continuously updates the experience base of task teaching, until the task teaching is completed.

[0082] Step 2.3. Task learning of robot

[0083] Observe the working environment through binocular camera, determine the pose of key obstacles and targets, and form the actual working trajectory by applying the learned task trajectory to the given task plan. Use reinforcement learning method for modeling, and the operator continuously evaluates the behavior of the robot, and continuously corrects the behavior of the robot through reward and punishment signals. The robot continuously improves its knowledge base in the process of task learning.

[0084] Step 2.4. Gesture and voice interaction based robot teaching obstacle avoidance

[0085] As shown in Figure 2 , the method based on Euclidean symbol field is easy to fall into local minimum and cannot escape from obstacles. In order to avoid this situation, offline teaching obstacle avoidance is considered before online motion planning of the robot. Gesture and voice are used as intervention means to continuously correct the deviation points in the robot motion planning process, so that the final trajectory is far away from the obstacles. The specific steps include:

[0086] Step 2.4.1. Estimate the distance from the robot motion trajectory to the obstacle.

[0087] Before the robot starts trajectory optimization, an initial B-spline curve that satisfies the terminal constraint and does not consider collision will be generated . For each collision segment detected in the iteration process, a collision-free path will be generated. Each control point of the collision segment will be assigned an anchor point and a corresponding repulsive force direction , where represents the index of the control point, index of the pair, each The pair belongs to only one specific control point, thus the control point to the distance to the obstacle will be defined as:

[0088] ;

[0089] Step 2.4.2. Collision penalty for the robot.

[0090] The safety gap between the robot and the obstacle is defined as In order to guarantee obstacle avoidance during the robot movement, it is necessary to ensure In addition, consider incorporating the penalty factor and the reward factor of gesture and voice intervention into the collision penalty function .

[0091]

[0092]

[0093] where, is the safety gap and the difference between the distance of the control point to the obstacle, is the collision cost generated at the control point The collision cost of each control point is evaluated independently and accumulated from the corresponding pair. The final cost added to the control point is:

[0094]

[0095] where, is the number of pairs at the control point The total cost is obtained by combining the costs of all control points :

[0096] .

[0097] Step 3. Online motion planning of robots based on gesture and voice interaction.

[0098] ​​After the task training process of the robot, the robot can basically adapt to the changing unknown and complex environment. Based on the established knowledge base, the operator starts or ends the online motion planning process of the robot through the control voice instruction. When the robot motion planning process starts, the waypoint information of the robot is issued through the voice instruction to control the robot to move slightly, or switch into the gesture control mode to control the robot to move greatly, or switch into the voice and gesture collaborative control mode to control the robot to move in the direction according to the gesture instruction, and read the voice instruction to control the robot to move in displacement. When the robot encounters an obstacle, the trajectory planning and optimization algorithm of the robot is called, and the trajectory is continuously corrected according to the established knowledge base, so that an optimal motion trajectory is finally generated, which meets the requirements of safety, obstacle avoidance and dynamic feasibility of the robot motion planning.

[0099] The trajectory planning and optimization algorithm of the robot specifically includes:

[0100] Step 3.1. A transition hot trajectory is generated. A topological path search method based on informed sampling provides an initial path, and a geometric guide method is used to geometrically deform the initial trajectory. The objective function is defined as the smooth cost of the trajectory and the distance between the guide path and the B-spline trajectory. Therefore, the solution of the hot trajectory is converted into an unconstrained quadratic programming problem, and an optimal closed-form solution will be obtained, so that a smooth trajectory, i.e. the hot trajectory, is output near the geometric guide path. Finally, a gradient-based optimization method is used to further improve the trajectory quality;

[0101] Step 3.2. The B-spline optimization method is used to further refine the trajectory into a smooth, safe and dynamically feasible trajectory. The objective function includes three parts, the collision cost evaluated in the Euclidean symbol field, the infeasible velocity penalty term and the infeasible acceleration penalty term. Based on the convex hull property of the B-spline, the three parts of the objective function are simplified, so that the control points of the B-spline can be constrained to ensure safety and dynamic feasibility.

[0102] Step 3.3. Trajectory re-planning of the robot.

[0103] The method based on step 3.1 can plan a locally optimal trajectory, but this trajectory does not contain high-order information, so it cannot reflect the real robot motion. It is considered to combine a topological path search method based on informed sampling with a path-guided trajectory optimization method to realize real-time trajectory re-planning.

[0104] As shown in Figure 3 Fig. 1, the gesture and voice interaction assisted robot semi-autonomous navigation system based on the embodiment of the application includes an augmented reality helmet and a ROS (Robot Operation System, ROS) server.

[0105] The ROS server includes gesture and speech recognition model, gesture and speech recognition client, robot controller server, database and visualization interface.

[0106] The monocular camera on the augmented reality helmet captures raw gesture image information and lip image information, which are input into the gesture and speech recognition model. The recognition results of gestures and speech are stored in the database in a serialized form. The gesture and speech recognition client is used to issue gesture and speech instructions. Based on the ROS service and topic communication mechanism, the robot controller server defines corresponding service name and topic name, as well as service message type and topic message type.

[0107] When an instruction needs to be issued, the gesture and speech recognition client reads the instruction parameters from the database, maps the instruction parameters into the waypoint information of the robot, and issues the instruction to the robot controller server in the form of a service. After receiving the message, the robot controller server gives feedback to the gesture and speech recognition client immediately, and publishes the received waypoint information to the corresponding subscriber of the robot in the form of a topic. The robot controller server and the robot drive the robot actuator to move through WiFi communication.

[0108] The visualization interface is used to display gesture recognition and silent speech recognition instructions, and to present the actual motion state of the robot. The offline stage is used to set the task training, task demonstration and task learning of the robot motion planning, and to intuitively correct the robot planning trajectory. The online stage is used to assist the autonomous navigation process of the robot, to provide real-time feedback of the robot motion state, and to facilitate the issuance of tasks by gestures and speech, or to correct for emergency error conditions.

[0109] The above has described the present application in detail in combination with specific embodiments, which are preferred embodiments of the present application, and the present application should not be limited to the content disclosed in the embodiments and the drawings. Any equivalent or modification made without departing from the disclosed spirit of the present application falls within the scope of the present application.

Claims

1. A semi-autonomous navigation method for robots based on gesture and voice interaction assistance, characterized in that... Includes the following steps: Step 1. Generate gesture and voice control commands; By capturing raw gesture and lip-reading images with an augmented reality headset, smart devices can recognize gesture and voice commands. Step 2. Robot task training based on gesture and voice interaction; including the following steps: Step 2.

1. Issuance of gesture and voice commands; When the robot needs to perform large movements, the operator switches between gestures and voice commands. Set as a gesture to indicate the direction of robot movement via hand gestures. and fixed movement step size and displacement units ; When the robot needs to perform small movements, the operator switches between gestures and voice commands. Set to voice input to give the robot the direction of movement. and moving step size and displacement units ; When gestures and voice commands are needed for coordinated control, switch between gestures and voice commands. Always keep it set to 1, and use the direction of the current gesture as the direction the robot moves. The currently released voice commands set the robot's step size. and unit ; Step 2.

2. Task teaching for the robot; specifically: Based on the established experience base, the operator uses gestures and voice to state the task name, the number of subtasks, and the attributes of the operation object, and sends the entire task analysis to the robot. The robot uses a motion planning algorithm to generate an initial task teaching trajectory. During the robot's movement, the operator observes the robot's reproduced trajectory and makes adjustments through voice commands for some teaching points where the error is greater than the expected error, and continuously updates the task teaching experience base until the task teaching is completed. Step 2.

3. Robot's task learning; specifically: By observing the working environment through binocular cameras, the poses of key obstacles and targets are determined, and the task trajectory learned by itself is applied to the given task plan to form the actual working trajectory. By using reinforcement learning to model the robot, the operator continuously evaluates the robot's behavior and corrects it through reward and punishment signals; during the task learning process, the robot continuously improves its knowledge base. Step 2.4 Robot obstacle avoidance teaching based on gestures and voice interaction; The robot is taught offline to perform tasks. During the reinforcement learning process, the robot continuously updates its experience and knowledge base, thereby supporting online obstacle avoidance. This includes the following steps: Step 2.4.1: Estimate the distance from the robot's trajectory to the obstacle to be avoided; Before the robot begins trajectory optimization, an initial B-spline curve that satisfies the terminal constraints and does not consider collisions will be generated. For each collision segment detected during the iteration process, a collision-free path is generated. Each control point in the collision segment Assign an anchor point on the collision surface. and the corresponding repulsive force direction ,in Indicates the index of the control point. express Correct index, therefore control point To the The distance to the obstacle will be defined as: ； Step 2.4.2: Apply collision penalties to the robot; The safe clearance between the robot and the obstacle is defined as... In the collision penalty function Punishment factors incorporating gestures and voice intervention and reward factors ; in, For safety clearance With control points To the The difference in distance between obstacles, To be at the control point superior For the resulting collision costs, each control point The collision costs are assessed independently and from the corresponding Accumulate the values ​​in the middle, and finally obtain the result added to the first position. Cost at each control point: in, Control point superior The correct number, all control points The costs are combined to obtain the total cost: ； Step 3. Online motion planning for the robot based on gestures and voice interaction; Based on the established knowledge base, the robot's waypoint information is issued via voice commands to control small movements, while larger movements are controlled via gesture commands. Voice and gesture commands work in tandem; gesture commands control the robot's direction of movement, while voice commands control its displacement. When the robot encounters an obstacle, it invokes its own trajectory planning and optimization algorithms, continuously refining its trajectory based on the established knowledge base to ultimately generate the optimal trajectory. The trajectory planning and optimization algorithms specifically include: Step 3.1: The informed sampling-based topology path search method provides an initial path, and uses a geometric guidance method to geometrically deform the initial trajectory; a smooth trajectory, i.e., a hot trajectory, is output near the geometrically guided path; finally, a gradient-based optimization method is used to further improve the trajectory quality. Step 3.2: The trajectory is further optimized into a smooth, safe, and dynamically feasible trajectory using the B-spline optimization method. The objective function consists of three parts: the collision cost evaluated in the Euclidean symbol field, the infeasible velocity penalty term, and the infeasible acceleration penalty term. The three parts of the objective function are optimized based on the convex hull properties of B-splines. Step 3.3 Trajectory replanning: Real-time trajectory replanning is achieved by combining a topology path search method based on informed sampling with a path-guided trajectory optimization method.

2. The semi-autonomous navigation method for robots based on gesture and voice interaction assistance according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1: The generation of gesture control commands includes the generation of the direction the gesture is pointing and the displacement of the gesture movement; Step 1.2: The generation of voice control commands includes the generation of control commands and command commands.

3. The semi-autonomous navigation method for robots based on gesture and voice interaction assistance according to claim 2, characterized in that: Step 1.1 is as follows: When a gesture action is generated, the key point information of the knuckles in each frame is obtained in a loop. The offset displacement of the key points between two frames is taken as the displacement of the gesture movement, and the offset angle of the key points between two frames is taken as the direction of the gesture. The calculation is repeated n times, and the offset displacement and offset angle are continuously superimposed to obtain the displacement value and direction information of the current gesture. The formula for calculating the superimposed offset is as follows: The formula for calculating the superimposed offset angle is as follows: in, This is the offset displacement. For the first The key points of each joint For the first The offset angle corresponding to each joint.

4. A semi-autonomous navigation system for robots based on gesture and voice interaction assistance, using the semi-autonomous navigation method for robots based on gesture and voice interaction assistance as described in any one of claims 1 to 3; It includes an augmented reality headset and a ROS server; the ROS server includes gesture and speech recognition models, gesture and speech recognition clients, a robot controller server, a database, and a visualization interface; its features are: The monocular camera mounted on the augmented reality helmet captures raw gesture image information and lip reading image information, which are then input into the gesture and speech recognition model. The recognition results of gesture and speech are stored in the database in a serialized form. The gesture and speech recognition client is used to issue gesture and speech commands. Based on the ROS service and topic communication mechanism, the robot controller server defines the corresponding service name and topic name, as well as the service message type and topic message type. When a command is issued, the gesture and voice recognition client reads the command parameters from the database, maps the command parameters into waypoint information for the robot, and publishes it to the robot controller server as a service. After receiving the message, the robot controller server immediately feeds back to the gesture and voice recognition client and publishes the received waypoint information to the robot's corresponding subscribers in the form of a topic, driving the robot actuator to perform actions.

5. The semi-autonomous navigation system for robots based on gesture and voice interaction assistance according to claim 4, characterized in that: The visualization interface is used to display gesture recognition and silent voice recognition commands. In the offline stage, it is used to set up task training, task teaching and task learning for robot motion planning, and intuitively correct the robot's planned trajectory. The online phase assists the robot in its autonomous navigation process, providing real-time feedback on the robot's motion status, facilitating the issuance of tasks via gestures and voice, or correcting urgent errors.

Images