Strange area man-machine collaborative search method based on tactile feedback device

Abstract

The invention discloses an unfamiliar area man-machine collaborative search method based on a tactile feedback device. The method comprises the following steps that S1, a robot obtains relative poses between man-machine formation members and the position between the robot and an obstacle; s2, the computer calculates and issues a motion control instruction of the robot according to the relative pose acquired by the robot in combination with a cooperative control algorithm; s3, according to the evaluated task target point and the obstacle information acquired by the sensor, the computer generates a planning path by adopting a search target evaluation method and utilizing an A * path planning algorithm, and further generates motion guidance information in real time; and S4, the wearable tactile feedback device communicates with the computer through the ROS, executes corresponding actions, and provides tactile feedback for the human members to enable the human members to adjust their own movement. The method does not depend on environmental prior information, a map or a centralized control system does not need to be built in advance, and a solution is provided for scenes such as city search and rescue.

Description

Technical Field

[0001] This invention relates to the field of electronic information technology, and in particular to a human-machine collaborative search method for unfamiliar areas based on a haptic feedback device. Background Technology

[0002] Multi-robot formations, through the complementary capabilities and coordinated actions of individual robots, can overcome the limitations faced by single robots in complex tasks, significantly improving system efficiency. Furthermore, multi-robot formations possess advantages such as high fault tolerance and robustness, leading to their widespread application in various fields including patrol reconnaissance, terrain exploration, disaster search and rescue, and material transportation.

[0003] While multi-robot searches offer significant advantages, relying solely on robots for searches still presents limitations. First, when a robot's perception systems (such as LiDAR and cameras) are subject to visual occlusion or sensor accuracy limitations, they may fail to accurately identify detailed information in complex environments. Second, robots often lack the flexibility and autonomy to make autonomous decisions during task execution, and human intervention may still be necessary in response to unforeseen circumstances. Therefore, relying solely on robots for environmental searches may face problems such as insufficient information, limited adaptability, and a lack of emergency response capabilities.

[0004] To overcome the aforementioned limitations, human-robot collaboration remains necessary to improve the efficiency of search tasks. In dynamic and highly complex scenarios, limitations in visual channels and non-line-of-sight conditions lead to a decline in human situational awareness. To enhance real-time sensitivity to the status of robotic collaborative units, potential environmental threats, and task objectives, it is urgent to construct a multimodal feedback mechanism that supplements other sensory channels. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a human-machine collaborative search method for unfamiliar areas based on a tactile feedback device. By combining a collaborative control algorithm with tactile feedback, it overcomes the limitations of weak human situational awareness caused by factors such as limited visual channels and non-line-of-sight conditions, thereby improving the efficiency of human-machine collaborative search tasks and the emergency handling capability of heterogeneous formations.

[0006] Technical Solution: A human-machine collaborative search method for unfamiliar areas based on haptic feedback devices, wherein the human-machine collaborative search system includes a physical verification platform and wearable haptic feedback devices; the physical verification platform includes human members and robots equipped with computers, forming a human-machine team; each member of the human-machine team is equipped with a UWB module and an IMU module, moving and performing pre-defined tasks in a physical scene, which includes a room and multiple obstacles; the robots are also equipped with lidar to acquire scene information by combining angle and distance measurement data, and the human members wear haptic feedback devices; through the collaborative control of multiple robots in the physical scene, the human members cooperate according to the prompts of the haptic feedback devices, and the multiple robots complete the search of the unfamiliar area; the steps include the following:

[0007] S1, the robot acquires the relative poses of the human-robot formation members and the position of the robot relative to obstacles;

[0008] S2, the computer calculates and issues motion control commands to the robot based on the robot's relative pose acquired and in conjunction with the cooperative control algorithm;

[0009] S3, the computer, based on the evaluated task target point and obstacle information obtained by the sensors, uses a search target evaluation method and the A* path planning algorithm to generate a planned path from the current position to the next task target point, and then generates motion guidance information in real time;

[0010] S4, through feedback information encoding, sends the formation status and motion guidance information to the wearable haptic feedback device according to different priorities; the wearable haptic feedback device communicates with the computer through ROS, executes corresponding actions, and provides haptic feedback to human members to adjust their own movement.

[0011] Furthermore, in step S1, the robot calculates the relative poses between robots and between the robot and the human based on the pairwise ranging results between the UWB modules and the angle difference of the IMU; and obtains the position of the obstacle relative to the robot based on the ranging results of the lidar and the angle corresponding to the ranging value.

[0012] Furthermore, in the cooperative control algorithm, the human members in the formation are also regarded as robots. Each robot calculates the difference between the relative distance and the yaw angle between itself and other members within the communication range. The difference between the relative distance and the yaw angle is used as the parameters of the artificial potential field method to calculate the expected moving speed of each robot.

[0013] Furthermore, the search target evaluation method obtains the boundaries between known and unknown areas through the occupied grid map constructed in real time by SLAM. Based on the trade-off between the information gain G that can be obtained to reach different boundaries and the distance cost C between the current position and the corresponding boundary, the search efficiency of each major boundary is estimated according to the search efficiency E=GC. Then, the A* path planning algorithm is used to generate the planned path from the current position to the next optimal target point.

[0014] The information gain is the length of the boundary line in the occupied raster map.

[0015] Furthermore, the working modes of the wearable haptic feedback device include squeeze feedback mode and vibration feedback mode: when the robot becomes disconnected from other formation members and exceeds the range of stable communication, it is mapped to squeeze feedback mode; when the disconnected robot returns to the range of stable communication with other formation members, it is mapped to vibration feedback mode.

[0016] During the search task, the squeezing feedback mode should have a higher priority than the vibration feedback mode.

[0017] Compared with the prior art, the significant advantages of this invention are as follows:

[0018] 1. This invention designs a method for calculating the relative pose between robots and humans in real-world scenarios and a method for evaluating search targets. This enables the estimation of the relative pose between humans and multiple robots under real-world conditions using onboard sensors. It does not require the prior construction of base stations or the creation of prior maps, thus meeting the needs of conducting search tasks in unfamiliar areas of different environments.

[0019] 2. This invention combines collaborative control algorithms with tactile feedback, enabling the robot to have a certain degree of autonomous movement. Robot formation information is transmitted to the wearer through a wearable tactile feedback device. By utilizing the tactile feedback channel, the limitations of weak situational awareness of human members in the formation caused by factors such as limited visual channels and non-line-of-sight conditions are overcome, thereby improving the efficiency of human-machine collaborative search tasks and the formation's ability to handle emergency situations. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention;

[0021] Figure 2 A schematic diagram of the human-machine collaborative search area;

[0022] Figure 3 This is a schematic diagram of the relative pose estimation process;

[0023] Figure 4 A model diagram of a vibration actuator;

[0024] Figure 5 This is a schematic diagram of the extrusion arm belt structure. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 The diagram shows a flowchart of a human-machine collaborative search method for unfamiliar areas based on a haptic feedback device. The human-machine collaborative search system includes a physical verification platform and a wearable haptic feedback device.

[0027] The physical verification platform comprises a wheeled robot equipped with a microcomputer and a human-robot formation. Each member of the human-robot formation is equipped with an ultra-wideband (UWB) positioning module and an inertial measurement unit (IMU) module to estimate relative pose and calculate motion control commands for the robot. The human-robot formation moves and executes pre-defined tasks in a physical environment, including a room and multiple obstacles, such as… Figure 2 As shown. In addition to UWB and IMU, the physical wheeled robot is also equipped with LiDAR to acquire scene information by combining angle and distance measurement data. Besides motion control commands, the robot's onboard computer also needs to calculate the state information of the heterogeneous formation and motion guidance information related to the task objective based on sensor data, the pose estimation results of the human members, and the current path planning direction. At the same time, it sends information to the wearable haptic feedback device; the wearable haptic feedback device executes the corresponding action to remind the human members to adjust their movement.

[0028] Preferably, the scene information includes an occupied grid map and task target points. The occupied grid map is acquired in real time by multi-robot simultaneous localization and mapping (SLAM), and the task target points are evaluated and optimized in real time by the computer based on the occupied grid map information, taking into account benefits and costs.

[0029] Wearable haptic feedback devices have multiple haptic feedback modes such as vibration feedback and squeeze feedback. They communicate with computers via ROS and execute corresponding actions based on the feedback information encoding corresponding to the received formation status information and motion guidance information, providing haptic feedback to users to adjust their own movement.

[0030] Wearable haptic feedback devices include vibratory actuators and squeeze armbands; such as Figure 4 As shown, the vibration actuator includes a vibration motor, a TPU fastener, and a PCB board. The vibration motor is secured to the TPU fastener, which has pre-drilled holes for a fabric strap to pass through, thus fixing it to the user's forearm. Figure 5As shown, the extrusion arm includes a nylon strap, a TPU nylon cloth, a TPU film, and a TPU nozzle. The TPU nylon cloth and the TPU film are heat-sealed together, with a liquid cavity in the middle. A silicone tube connects the TPU nozzle to the syringe. The syringe piston is controlled by a stepper motor to move, thereby controlling the sealed transmission of liquid in the silicone tube. The nylon strap is fixedly connected to the TPU nylon cloth.

[0031] The robot is equipped with sensors including LiDAR, inertial measurement unit (IMU), and ultra-wideband ranging module (UWB); human crew members, in addition to wearing haptic feedback devices, also need to wear IMU and UWB.

[0032] A human-robot collaborative search method for unfamiliar areas based on haptic feedback devices is disclosed. This method utilizes a human-robot collaborative search system to achieve collaborative control. Through the collaborative control of multiple robots in a real-world scenario, human personnel cooperate based on prompts from wearable haptic feedback devices, enabling the robots to complete the search for unfamiliar areas. The steps include the following:

[0033] Step 1: In the real-world scenario, the robot calculates the relative pose between robots and between the robot and the human based on the pairwise ranging results between the UWB modules and the angle difference of the IMU; and obtains the position of the obstacle relative to the robot based on the ranging results of the LiDAR and the angle corresponding to the ranging value.

[0034] The sensors on the robot can acquire the relative poses between members of the human-robot formation and between the robot and obstacles. For example... Figure 3 As shown, each robot has a UWB module installed at a distance d on each of its left and right sides. The four modules installed on the two robots form an equilateral quadrilateral. Within the same local area network, the IDs of the different UWB modules are known, and the distance measurements between the modules are z. 11 , z 12 , z 21 , z 22 Then, the distance z between the module and the other robot can be calculated first using the formula for the length of the centerline. a1 or z a2 This allows us to calculate the distance between the robots ||p AB ||:

[0035]

[0036] z a1 or z a2 With d, ||p AB Substituting into the cosine theorem formula, we can obtain p AB Direction α in robot A coordinate system ABAn IMU containing a magnetometer can acquire the magnetic heading of the carrier, i.e., the absolute angle. In planar positioning, the absolute heading angles of the yaw axes of the two robots can be acquired first, and then the relative angle θ between the robots can be calculated by subtracting the values. AB .

[0037] Step 2: Based on the robot's sensor information and in conjunction with the cooperative control algorithm, the computer calculates and issues motion control commands to the robot.

[0038] In the multi-robot cooperative control algorithm, the human members in the formation are also regarded as robots. Each robot calculates the difference between the relative distance and yaw angle between itself and other members within the communication range. The difference between the relative distance and yaw angle is used as the parameters of the artificial potential field method to calculate its own expected moving speed.

[0039] The artificial potential field method is used as the basic architecture of the cooperative control algorithm in this invention. During the search mission in an unfamiliar area, the human's movement is entirely determined by themselves. When the human moves, the potential force on the robot changes. The desired speed of the robot is calculated based on the artificial potential field method, and the actual moving speed of the robot is adjusted by a PID algorithm to achieve cooperative movement of the human-robot formation. Specifically, the potential field generated by obstacles is modeled as a smooth decreasing concave function, while the potential field between formation members is modeled as a smooth concave function with one and only one minimum point. The gradient of the potential field function is used as the potential force function to calculate the direction and magnitude of the robot's desired speed. Therefore, obstacles always exert a repulsive force on formation members, and when the distance between formation members is less than a certain threshold, a repulsive force is generated between them; conversely, an attractive force is generated. Furthermore, since the human in the human-robot formation can move autonomously, when the human moves, it causes a change in the potential force on each robot, thereby enabling the robots to coordinate with the human's movement within the communication range.

[0040] In the process of controlling robot formation using the artificial potential field method, individual robots may prioritize obstacle avoidance, causing them to become disconnected from other formation members once the distance between them and the obstacle reaches a safe threshold. This means the distance becomes too great, or even close to exceeding the stable communication range. Although multi-robot systems are highly fault-tolerant, the absence of a single robot has a relatively small impact on the overall system. To ensure all robots remain within the stable communication range, this situation is mapped to a squeeze feedback mode of a haptic feedback device. By controlling the rotation of a stepper motor, the syringe piston is displaced, thereby delivering liquid and tightening the squeeze arm. Different motor displacements are designed to generate different squeezing forces based on the user's need to slow down or stop, allowing the user to wait for the disconnected robot to regain stable communication with the other formation members.

[0041] Step 3: Based on the evaluated task target point and obstacle information obtained by the sensors, the computer uses a search target evaluation method and the A* path planning algorithm to generate a planned path from the current position to the next task target point, and then generates motion guidance information in real time.

[0042] In the search target evaluation method, each robot performs localization and mapping (SLAM) while moving. Combining the estimated relative pose, the occupied grid maps constructed by each robot are stitched together. The grid map contains the boundaries of known and unknown areas. The task target point will be evaluated and selected from these boundaries in real time, and then the path to the target point will be planned.

[0043] In practical applications, the search target evaluation method obtains the boundaries between known and unknown areas through the occupancy grid map constructed in real time by SLAM. Based on the trade-off between the information gain G that a person can obtain by reaching different boundaries and the distance cost C between the person's current position and the corresponding boundary, the search benefit of each major boundary is estimated according to the search benefit E=GC, that is, the difference between information gain and distance cost. Then, the A* path planning algorithm is used to generate the planned path from the current position to the next optimal target point.

[0044] Information gain is defined as the length of the boundary line occupied in a grid map. The longer the boundary line, the wider the "field of view" after reaching the boundary line, and the more areas can be observed, thus having a higher information gain. The computer combines the person's current heading angle and updates and publishes motion direction guidance information to the haptic feedback device in real time. This information will be mapped into vibration feedback modes.

[0045] Step 4: By encoding feedback information, the formation status and motion guidance are sent to the wearable haptic feedback device according to different priorities; the wearable haptic feedback device communicates with the computer through ROS (Robot Operating System) to execute corresponding actions and provide haptic feedback to human members to adjust their own movement.

[0046] The feedback information encoding separately encodes the formation's motion state and path guidance information, sending them to the wearable haptic feedback device according to different priorities, causing it to execute corresponding actions. In this way, the feedback priority for preventing individual robots from becoming disconnected from the main formation is set higher than that for motion direction guidance, avoiding misunderstandings of the feedback information by the device wearer.

[0047] Furthermore, to prevent users from being confused by the haptic feedback device's prompts, different feedback priorities need to be set for different haptic modes; that is, at most one haptic mode can be executed at any given time. This invention proposes that during the search task, preventing a robot from separating from the main human-robot formation should have a higher priority than motion direction guidance; that is, the squeeze feedback mode should have a higher priority than the vibration feedback mode. When an individual robot becomes disconnected from other formation members, and the user's forward direction deviates significantly from the motion guidance direction, the feedback device will only activate the squeeze armband to remind the user to slow down or stop. Once the disconnected robot regains stable communication with other formation members, the vibration motor will then activate to remind the user to adjust their direction.

[0048] The methods described above can complete the search of unfamiliar areas without relying on the prior construction of base stations or the creation of prior maps, thereby improving the efficiency of task execution and minimizing losses.

[0049] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the present invention.

Claims

1. A human-machine collaborative search method for unfamiliar areas based on a haptic feedback device, wherein the human-machine collaborative search system includes a physical verification platform and a wearable haptic feedback device; the physical verification platform includes human members and a robot equipped with a computer, the robot and human members forming a human-machine team; each member of the human-machine team is equipped with a UWB module and an IMU module, moves in a physical scene and performs a set task, the physical scene including a room and multiple obstacles; the robot is also equipped with a lidar to obtain scene information by combining angle and distance measurement data, and the human members wear haptic feedback devices; characterized in that, Through the collaborative control of multiple robots in a real-world scenario, human members cooperate based on the prompts from the tactile feedback device, and the multiple robots complete the search of unfamiliar areas; The steps include the following: S1, the robot acquires the relative poses of the human-robot formation members and the position of the robot relative to obstacles; S2, the computer calculates and issues motion control commands to the robot based on the robot's relative pose acquired and in conjunction with the cooperative control algorithm; S3, the computer, based on the evaluated task target point and obstacle information obtained by the sensors, uses a search target evaluation method and the A* path planning algorithm to generate a planned path from the current position to the next task target point, and then generates motion guidance information in real time; S4, through feedback information encoding, sends the formation status and motion guidance information to the wearable haptic feedback device according to different priorities; Wearable haptic feedback devices communicate with computers via ROS to execute corresponding actions, providing haptic feedback to human users to adjust their movements.

2. The human-machine collaborative search method for unfamiliar areas based on a haptic feedback device according to claim 1, characterized in that, In step S1, the robot calculates the relative poses between robots and between the robot and the human based on the pairwise ranging results between the UWB modules and the angle difference of the IMU; and obtains the position of the obstacle relative to the robot based on the ranging results of the lidar and the angle corresponding to the ranging value.

3. The human-machine collaborative search method for unfamiliar areas based on a haptic feedback device according to claim 1, characterized in that, In the cooperative control algorithm, human members in the formation are also regarded as robots. Each robot calculates the difference between the relative distance and yaw angle between itself and other members within its communication range. The difference between the relative distance and yaw angle is used as the parameters of the artificial potential field method to calculate the expected moving speed of each robot.

4. The human-machine collaborative search method for unfamiliar areas based on a haptic feedback device according to claim 1, characterized in that, The search target evaluation method obtains the boundaries between known and unknown areas through an occupied grid map constructed in real time by SLAM. Based on a trade-off between the information gain G that can be obtained to reach different boundaries and the distance cost C between the current position and the corresponding boundary, the search efficiency of each major boundary is estimated according to the search efficiency E=GC. Then, the A* path planning algorithm is used to generate a planned path from the current position to the next optimal target point. The information gain is the length of the boundary line in the occupied raster map.

5. The human-machine collaborative search method for unfamiliar areas based on a haptic feedback device according to claim 1, characterized in that, The working modes of wearable haptic feedback devices include squeeze feedback mode and vibration feedback mode: when the robot becomes disconnected from other formation members and exceeds the range of stable communication, it is mapped to squeeze feedback mode; when the disconnected robot returns to the range of stable communication with other formation members, it is mapped to vibration feedback mode. During the search task, the squeezing feedback mode should have a higher priority than the vibration feedback mode.

Images