Haptic feedback system for embodied robots

Abstract

To facilitate the embodiment of swarm robots, we provide a control system for swarm robots that incorporates a system that provides tactile feedback to the operator. [Solution] Multiple robots, each capable of moving freely within a set field, are assigned target position coordinates. Each robot moves toward its target position coordinate. Characteristic points on the operator's body are detected, and a value equivalent to the amount of movement of the position coordinate of each characteristic point is assigned to the target position coordinate of the corresponding robot. As a result, each robot behaves in a way that follows its corresponding characteristic point. If any robot makes contact or collides, this is detected, and a module attached to the corresponding characteristic point on the robot outputs force, vibration, movement, or a combination thereof, providing tactile feedback to the operator.

Description

Technical Field

[0001] The present invention relates to a swarm robot control system with tactile feedback. Background Art

[0002] Due to its plasticity and flexibility, the human brain can recognize an object as a part of the body and operate it intuitively under specific conditions. This phenomenon is called embodiment. Utilizing this cognitive ability, various embodied systems with intuitive operation and immersive feeling for applications such as the remote control of virtual avatars and robot arms have been developed. The sense of embodiment, in essence, is a sensation composed of "sense of agency", which is a subjective sense of control and awareness that one's own self causes a certain movement or event, and "sense of ownership", which is the sensation of the connection between oneself and an object.

[0003] There are swarm robots in which a plurality of robots are made to perform cooperative operations for a specific purpose. Due to their flexibility, scalability / reducibility, and robustness, swarm robots are expected to be widely used in various applications ranging from commercial use to entertainment, and this is a field where research and development have been continuously conducted in recent years. The present inventors focused on the combination of embodiment and swarm robots.

[0004] It has already been proven by operation experiments on swarm robots that humans can embody swarm robots. However, due to individual differences in sensation and delays caused by collisions between individual robots, swarm robot control systems that can be embodied by most people have by no means reached a practical level. Accordingly, when swarm robots controlled by movement of feature points on the operator's body come into contact or collide, the present invention applies force, vibration, or movement, or a combination of the foregoing to the aforementioned feature points, thereby providing tactile feedback from the swarm robot to the operator. Tactile feedback as used herein refers to feedback that stimulates the operator's sense of touch among feedback from the robot to the operator. Prior Art Literature Patent Literature

[0005] [Patent Document 1] Japanese Patent Publication No. 2012-194948

[0006] [Patent Document 2] Japanese Patent Publication No. 2019-209425 [Overview of the project] [Problems that the invention aims to solve]

[0007] The problem we are trying to solve is that, due to delays caused by collisions between robot swarms and individual differences in perception, it has not been possible to realize swarm robots that many people can embody. Specifically, the only means of understanding the situation of the swarm robots is visual information. [Means for solving the problem]

[0008] The present invention provides a control system for a swarm of robots that is controlled by changes in the position coordinates of the operator's body. This system comprises multiple robots that can move freely within a set field, a module that detects characteristic points on the operator's body and acquires their position coordinates, a module that controls the multiple robots based on changes in the position coordinates of the characteristic points, a module that acquires the position coordinates of each of the multiple robots, a module that detects contact or collision between each robot and an object in the field or between the robots themselves, and a module that, when the module detects a collision or contact, provides tactile feedback by outputting force, vibration, movement, or a combination thereof to the characteristic points on the operator's body corresponding to each of the robots that have come into contact or collided. In a swarm of robots controlled by changes in the position coordinates of the operator's body, the system provides a control system for a swarm of robots that, when contact or collision between each robot is detected, outputs force, vibration, movement, or a combination thereof to the characteristic points on the robots that have come into contact or collided, thereby providing tactile feedback to the operator. [Effects of the Invention]

[0009] The present invention provides a swarm robot operation system using tactile feedback, which enhances the sense of embodying the robot and makes embodying easier by providing tactile feedback to the operator by outputting force, vibration, movement, or a combination thereof, in accordance with the status of the swarm robot controlled by changes in the position coordinates of characteristic points on the operator's body. In this invention, a module that detects contact or collision between each robot sends a signal when it detects contact or collision between each robot, which is transmitted via a computer to a module that provides tactile feedback to the operator, attached to the operator's characteristic points. The module outputs force, vibration, movement, or a combination thereof, allowing the operator to receive tactile feedback and understand the status of the swarm robot based on tactile information, even when visual information is unavailable. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing the modules necessary for controlling a swarm of robots and the swarm of robots initially positioned at arbitrary locations. [Figure 2] This diagram illustrates the flow of information necessary to set target position coordinates for rearranging a swarm of robots into a hand shape. [Figure 3] A diagram showing the robot's movement method and possible movement and rotation directions. [Figure 4] This diagram shows how the target position coordinates for each robot to line up in the shape of a hand are set within the field. [Figure 5] A diagram showing a group of robots lined up in the shape of a hand. [Figure 6] This diagram shows how each feature point moves, and how each target position coordinate moves accordingly, with each robot moving towards each target position coordinate. [Figure 7] This figure shows the amount of movement of the x-axis and y-axis coordinates of P-1 due to the movement of P-1. [Figure 8] This figure shows the amount of movement of the x and y coordinates of T-1 in accordance with the movement of P-1. [Figure 9]Figure showing a flowchart illustrating feedback control until R-1 reaches T-1 [Figure 10] Figure showing a top view of a robot and a pressure sensor mounted on the robot [Figure 11] Figure showing a scenario where two robots collide with an object in a field, wirelessly transmit the information to a control PC, and the control PC controls vibration motors respectively attached to two feature points corresponding to the two colliding robots [Figure 12] Figure showing the positions of an operator, an object A in field 2, and a swarm of robots [Description of Reference Signs]

[0011] 1. Control PC 2. State of information transmission from C-1 to the control PC 3. State of information transmission from C-2 to the control PC C-1. Camera that detects a left-hand feature point and transmits the detection result to the control PC C-2. Camera that acquires position coordinates of each robot F-1. Detection range of C-1 F-1. Area detectable by C-2 and movable for the swarm of robots P-1. Detected feature point of the little finger P-2. Detected feature point of the ring finger P-3. Detected feature point of the middle finger P-4. Detected feature point of the index finger P-5. Detected feature point of the thumb R-1. One robot in the swarm of robots that corresponds to P-1 R-2. One robot in the swarm of robots that corresponds to P-2 R-3. One robot in the swarm of robots that corresponds to P-3 R-4. One robot in the swarm of robots that corresponds to P-4 R-5. One robot in the swarm of robots that corresponds to P-5 4. Bottom view of one robot 5. Front view of one robot 6. Two wheels mounted on a robot 7.6 Rotation direction 8. Possible directions of movement and rotation of the robot Target position coordinates of T-1.R-1 T-2.R-2 target position coordinates Target position coordinates of T-3.R-3 T-4.R-4 target position coordinates T-5.R-5 target position coordinates W-1. Control signals are being transmitted wirelessly from the control PC to each robot. AP-1. Coordinates of P-1 after movement AP-2. Coordinates of P-2 after movement AP-3. Coordinates of P-3 after movement AP-4. Coordinates of P-4 after movement AP-5. Coordinates of P-5 after movement BP-1. Coordinates of P-1 before movement BP-2. Coordinates of P-2 before movement BP-3. Coordinates of P-3 before movement BP-4. Coordinates of P-4 before movement BP-5. Coordinates of P-5 before movement dx-1. Any difference in the x-axis coordinates of AP-1 and BP-1, and any displacement of the x-axis coordinate of P-1. dy-1. Any difference in the y-axis coordinates of AP-1 and BP-1, and any displacement of the y-axis coordinate of P-1. AT-1. Coordinates of T-1 after movement BT-1. Coordinates of T-1 before movement Fx-1.F-1 x-axis size Fx-2. x-axis size of F-2 Fx-2 / Fx-1 represents the ratio of the x-axis of F-1 and F-2 as a fraction. dx-1 × (Fx-2 / Fx-1). The x-axis movement of the target position coordinate of T-1 when the x-axis movement of P-1 is dx-1. Fy-1.Y-axis size of F-1 Fy-2.Y-axis size of F-2 Fy-2 / Fy-1: The fractional expression of the ratio of F-1 and F-2 on the y-axis. dy-1 × (Fy-2 / Fy-1). The y-axis movement of the target position coordinate of T-1 when the y-axis movement of P-1 is dy-1. 9.1 robots viewed from above 10.1 Four pressure sensors installed in each robot M-1. Vibration motor attached to the little finger M-2. Vibration motor attached to the ring finger M-3. Vibration motor attached to the middle finger M-4. Vibration motor attached to the index finger M-5. Vibration motor attached to the thumb Objects inside OF-2 Collision between SO and R-1,2 W-2. Information is wirelessly transmitted from the pressure sensor that detected the collision to the control PC. 11. Control signals from the control PC to M-1 and M-2. 12. Pilot R. Swarm Robots A. Objects inside the F-2 that can be moved by swarm robots [Modes for carrying out the invention]

[0012] Due to individual differences in embodied sensation and the possibility of contact or collision between robots, a swarm of robots that anyone can embody has not yet been put into practical use. To solve this problem, we have invented a swarm robot control system equipped with a module that provides tactile feedback to the operator and controls the swarm robot by changing the position coordinates of characteristic points of the body. By providing tactile feedback to the operator, it becomes possible to create mutual interaction between the operator and the swarm robot, rather than just one-way control from the operator to the swarm robot, and enables more intuitive and immersive operation, in other words, the sense of embodied sensation is improved.

[0013] A module is used to detect arbitrary feature points on the body of one or more operators and control a swarm of robots, thereby detecting arbitrary feature points on the body and obtaining their position coordinates. Here, a feature point is defined as any point on the body, not just fingertips or joints. A one-to-one correspondence is established between each feature point and each swarm of robots using a PC. At this time, the number of feature points detected matches that of the swarm of robots. The number is matched to either the number of swarm of robots being handled or the number of detectable body feature points, whichever is more convenient. Furthermore, a system is used to obtain the absolute position coordinates of any swarm of robots, so that the absolute position coordinates of each swarm of robots can be obtained, and the field is defined as the range in which the swarm of robots can move using any movement method. The arbitrary coordinates where each robot is placed are set as the initial absolute position coordinates and initial target position coordinates of each swarm of robots.

[0014] A control PC is connected to a module that detects arbitrary body feature points and controls the swarm of robots, and to a module that acquires the absolute position coordinates of each robot in the swarm. Once the swarm of robots is positioned, it can be controlled. Changes in arbitrary position coordinates and the amount of movement are recorded, and an equivalent amount of movement is assigned to the target position coordinate of each robot in the swarm corresponding to each feature point, causing the target position coordinates to move. After that, the swarm of robots moves in a straight line to the target position coordinates using an arbitrary movement method. At this time, a system that acquires the absolute value coordinates of each robot in the swarm is used to calculate the difference between each robot and each target position coordinate, and feedback control is performed to confirm whether each target position coordinate has been reached. When each robot in the swarm reaches the target position coordinate, it stops in place and waits until the target position coordinate moves again. As each robot moves toward the target position coordinate corresponding to each feature point, each robot behaves in a way that follows the corresponding feature point.

[0015] Modules are attached to each feature point to provide haptic feedback by outputting arbitrary forces, vibrations, or movements, or a combination thereof. When a group of robots or a robot collides with an object in the field, a module that detects contact or collision between any robot group and an object detects this and transmits it to the control PC in any way. The control PC then activates the aforementioned haptic feedback modules at each feature point corresponding to the robot that made contact or collided, providing haptic feedback to the corresponding feature points of the operator. The robot that made contact or collided continues to move toward its respective target position coordinates. Furthermore, even after providing feedback, the operator continues to control the robot until the objective using the group of robots is completed. [Examples]

[0016] An embodiment will be explained using drawings. Figure 1, part 1 is the control PC. Figure 1, part C-1 is a module for detecting characteristic points of the operator's body, such as a camera (hereinafter referred to as camera 1), and Figure 1, part 2 shows the flow of information transmission from camera 1 to the control PC via a wire. Figure 1, parts P-1, 2, 3, 4, and 5 are, for example, five characteristic points detected by camera 1 on the fingertips of the left hand. P-1 corresponds to the tip of the little finger, P-2 to the tip of the ring finger, P-3 to the tip of the middle finger, P-4 to the tip of the index finger, and P-5 to the tip of the thumb. These characteristic points will hereafter be referred to as P-1, 2, 3, 4, and 5, respectively. Figure 1, part F-1 is the detection range of camera 1 (hereinafter referred to as field 1), and when P-1, 2, 3, 4, and 5 are within field 1, their respective position coordinates are obtained and transmitted to the control PC as shown in Figure 1, part 2. R-1, R-2, R-3, R-4, and R-5 in Figure 1 represent the five robots being operated. Hereafter, the entire group of five robots will be referred to as the robot swarm, and each robot will be referred to as R-1, R-2, R-3, R-4, R-4, and R-5, respectively. R-1 corresponds to P-1, R-2 to P-2, R-3 to P-3, R-4 to P-4, and R-5 to P-5. The robot swarm is initially placed in an arbitrary location. C-2 in Figure 1 is a camera (hereinafter referred to as camera 2) for acquiring the position coordinates of the robot swarm. Camera 2 is positioned vertically downwards to observe the robot swarm from above. F-2 in Figure 1 is the area detectable by camera 2 and in which the robot swarm can move. Hereafter referred to as field 2. Figure 13 shows the flow of information transmitted via wire from camera 2 to the control PC.

[0017] Figure 2 illustrates the robots being discussed. Each robot's locomotion mechanism is, for example, a two-wheeled, opposing-wheeled type, allowing it to move forward, backward, rotate, and travel on a flat surface. Figures 2-4 and 2-5 show the robots from the underside and front, respectively. Figure 2-6 shows the robot's wheels, and Figure 2-7 shows the direction of wheel rotation. Figure 2-8 shows the directions of movement by forward and backward, the direction of rotation, and the combined direction of movement by forward, backward, and rotation. Thus, each robot is capable of rotation, forward, backward, and combinations thereof through the movement of its wheels.

[0018] As shown in Figure 3, camera 1 transmits the position coordinates of each feature point and the size of field 1 (xy plane), and camera 2 transmits the size of field 2 (xy plane) to the control PC. Then, the ratio of the size of field 1 to the size of field 2 from camera 1 is calculated, and each target position coordinate is set on field 2 so that the hand shape is arranged based on the position coordinates of each feature point in field 1 and that ratio. T-1, 2, 3, 4, 5 in Figure 4 are the target position coordinates of R-1, 2, 3, 4, 5. Hereafter, each target position coordinate will be referred to as T-1, 2, 3, 4, 5. Information is also transmitted wirelessly from the control PC to the swarm of robots to set each robot's target position coordinate. W-1 in Figure 4 shows the wireless transmission of control signals from the control PC to each robot. After setting each target position coordinate, each swarm of robots moves in a straight line toward its respective target position coordinate. Figure 5 shows the state when each swarm of robots has arrived at its respective target position coordinate and is arranged in the shape of a hand. When each robot reaches its designated target coordinate and the swarm of robots forms a hand shape, the operator can control the swarm. By aligning the swarm of robots in the same hand shape as the operator, the robots can perform movements that more closely resemble human movements, enabling intuitive and immersive control.

[0019] The operator controls the swarm robots by changing the position coordinates of the feature points, that is, by moving the feature points. When the operator is able to control the swarm robots, the position coordinates of each target are the same as the position coordinates of each robot. In Figure 6, AP-1, 2, 3, 4, and 5 are the position coordinates of P-1, 2, 3, 4, and 5 after movement, respectively, and BP-1, 2, 3, 4, and 5 are the position coordinates of P-1, 2, 3, 4, and 5 before movement, respectively. In Figure 7, dx-1 is the difference between any x coordinates of BP-1 and AP-1, that is, the amount of movement of any x coordinate of P-1. Similarly, dy-1 in Figure 7 is the difference between any y coordinates of BP-1 and AP-1, that is, the amount of movement of any y coordinate of P-1. In Figure 8, BT-1 and AT-1 are the position coordinates of T-1 before and after movement, respectively. In Figure 8, Fx-1 and Fy-1 are the size of the detection range of camera 1 in the x-axis direction and the size in the y-axis direction, respectively. In Figure 8, Fx-2 and Fy-2 similarly represent the x-axis and y-axis dimensions of the field size, respectively. Using Fx-1, Fy-1, Fx-2, and Fy-2, the x-axis and y-axis dimensions of the field size can be expressed as (Fx-2 / Fx-1) times and (Fy-2 / Fy-1) times the x-axis and y-axis dimensions of the detection range of camera 1, respectively. When the x-axis and y-axis displacements of the feature points are dx-1 and dy-1, respectively, the target position coordinates of R-1 corresponding to P-1 move dx-1 × (Fx-2 / Fx-1) in the x-axis direction and dy-1 × (Fy-2 / Fy-1) in the y-axis direction. After that, R-1 moves in a straight line towards the moved T-1. Similarly, for the other robots R-2, 3, 4, and 5, after moving their respective feature points P-2, 3, 4, and 5 in any direction and distance, the amount of movement of their x and y axis coordinates is measured. Then, the corresponding target position coordinates T-2, 3, 4, and 5 are moved in the x and y directions by (Fx-2 / Fx-1) times and (Fy-2 / Fy-1) times the amount of movement of their respective x and y axis coordinates, respectively. Each robot corresponding to each target position coordinate is then controlled by moving in a straight line. Figure 9 is a flowchart showing R-1 traveling towards T-1 and reaching the target position coordinate. When R-1 reaches T-1, it stops traveling and remains stationary, waiting until T-1 moves again. Whether the coordinates of R-1 and T-1 match is checked using camera 2. The same applies to the other robots.By moving feature points, then using that movement to move the target position coordinates, and having the robot move towards the target position coordinates, and repeating this process at shorter intervals, each robot can perform actions with a greater degree of freedom.

[0020] A pressure sensor, for example, is used as a module to detect contact or collision between each robot. Figure 10-9 shows a top view of one robot, and Figure 10-10 is a pressure sensor, one mounted on each of the four sides of the robot. The other four robots are similar. Figure 11 shows M-1, 2, 3, 4, and 5 as vibration motors (hereinafter referred to as M-1, 2, 3, 4, and 5, respectively) attached to the fingertips. M-1 is attached to the little finger, M-2 to the ring finger, M-3 to the middle finger, M-4 to the index finger, and M-5 to the thumb. M-1, 2, 3, 4, and 5 are controlled by a control PC, for example, via a wired connection. Figure 11-O represents an object on the field (hereinafter referred to as an object). Figure 11-S represents a collision between the object and R-1 and R-2. When the pressure sensors mounted on R-1 and R-2 detect a collision, the information is transmitted wirelessly to the control PC. Figure 11, labeled W-2, shows how information is wirelessly transmitted from the pressure sensor that detected the collision to the control PC. Figure 11, labeled 11, shows the wired transmission of control signals from the control PC to M-1 and M-2, corresponding to R-1 and R-2. After detecting the collision between R-1 and R-2, the information is transmitted to the control PC, which activates the vibration motor and outputs vibration. This allows the operator to receive tactile feedback. Even after the collision, R-1 and R-2 continue to travel towards T-1 and T-2.

[0021] This robot swarm is controlled by this control system with the primary objective of pushing and moving objects within Field 2. Whether the robot swarm can move an object is determined by its weight. In Figure 12, A is an object that can be moved by the robot swarm within Field 2 (hereinafter referred to as Object A). In Figure 12, R is the robot swarm. In Figure 12, 12 is the operator. When the robot swarm and Object A come into contact or collide, the operator is given tactile feedback. When the operator, Object A, and robot swarm are positioned as in Figure 12, the robot swarm can move Object A below the operator by moving its feature points towards the operator so that the target position coordinates of the robot swarm are below the operator.

Claims

1. Multiple robots that can move freely within a designated field (called Field A), Module A detects characteristic points of the pilot's body and obtains their position coordinates, Module B controls the multiple robots based on changes in the position coordinates of the aforementioned feature points, Module C acquires the position coordinates of each of the aforementioned multiple robots, Module D detects contact or collision between each robot and an object in the field A or between each robot and itself, When module D detects a collision or contact, module E, which is attached to the characteristic points of the operator's body corresponding to each of the multiple robots that have come into contact or collided with it, provides haptic feedback by outputting force, vibration, movement, or a combination thereof to the characteristic points of the operator's body. A control system for swarm robots equipped with haptic feedback for the embodiment of swarm robots, characterized by having the following features.

2. The pilot's physical characteristics are located within a designated field (called Field B). Module A can obtain the position coordinates of the feature point within field B, Module B is capable of moving the corresponding robot in accordance with the movement of the aforementioned feature point. A control system for a swarm of robots equipped with tactile feedback for the embodiment of the swarm robots as described in feature 1.

3. The control system for a swarm of robots equipped with haptic feedback for the embodiment of a swarm of robots according to claim 2, characterized in that the position coordinates of the feature points acquired by module A within field B are sent to module B, and module B moves the corresponding robot in accordance with the movement of the feature points.

4. A control system for a swarm of robots with haptic feedback for the embodiment of a swarm of robots, characterized in that module A is a positioned camera capable of acquiring the position coordinates within field B of characteristic points of the operator's body positioned within field B.

5. The control system for a swarm of robots with haptic feedback for the embodiment of a swarm of robots according to claim 1, characterized in that module C is a positioned camera capable of acquiring the position coordinates of each of the multiple robots positioned within field A.

6. A control system for a swarm of robots with tactile feedback for the embodiment of a swarm of robots, characterized in that module D is a pressure sensor, as described in claim 1.

7. A swarm robot control system with haptic feedback for the embodiment of a swarm robot, characterized in that when module D detects a collision or contact, the signal indicating that module D has detected a collision or contact is sent to module E, which in turn causes module E to apply force, vibration, or movement to characteristic points on the operator's body.

8. A control system for a swarm of robots and a bot, characterized in that the physical features are the fingertips or joints of the fingers, as described in claim 1, with tactile feedback for the embodiment of the swarm robot.

Images