Control device, control method, and control program

Abstract

This control device according to one aspect of the present invention: acquires observational data of an operator's body parts; dynamically allocates each of a plurality of subgoal positions, which correspond to the body parts represented in the acquired observational data, to each of a plurality of robot devices; and controls operation of each of the plurality of robot devices with respect to each of the respective plurality of subgoal positions that have been allocated. Provided thereby is a technology for controlling swarm robots, which reduces the potential that a plurality of robot devices collide with one another when the robot devices are being controlled in accordance with body parts.

Description

Control device, control method, and control program

[0001] The present invention relates to a control device, a control method, and a control program.

[0002] In recent years, various methods for controlling robotic devices have been developed. For example, Non-Patent Document 1 proposes a method for embodied robots by controlling the movements of multiple robotic devices (swarm robots) in accordance with the state of the body (e.g., hands). This method allows multiple robotic devices to be controlled by intuitive operations.

[0003] Masato Nakagawa, Yoshihiro Kashino, Shigeaki Yoshida, Masahiko Inami, “Preliminary Study on Embodied Swarm Robots”, [online], [Retrieved November 29, 2023], Internet <URL: https: / / conference.vrsj.org / ac2021 / program / doc / 2C1-5.pdf>

[0004] The present inventors have found that the above-mentioned conventional method has the following problem: In the above-mentioned conventional method, the subgoal position of each robot device is statically assigned to a feature point of the body in a one-to-one correspondence. This fixes the relationship between each robot device and the feature point of the body, which can lead to problems such as collisions between robot devices when the robots change direction or flip over, making the control for collision avoidance complicated.

[0005] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide a control technique for reducing the possibility of collisions between robot devices when controlling multiple robot devices in accordance with body parts.

[0006] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0007] A control device according to one aspect of the present invention includes a control unit configured to acquire observation data of body parts of an operator, dynamically assign a plurality of subgoal positions to a plurality of robotic devices according to the body parts appearing in the acquired observation data, and control the operation of each of the plurality of robotic devices toward the assigned subgoal positions.

[0008] In this configuration, each subgoal position corresponding to a body part is dynamically assigned to each robot device. As a result, the relationship between each robot device and a body part is not fixed, so that collisions between robot devices can be avoided even by a simple method such as assigning the nearest subgoal position. Therefore, with this configuration, the possibility of collisions between robot devices can be reduced when multiple robot devices are controlled according to body parts.

[0009] In the control device according to the above aspect, dynamically allocating the plurality of subgoal positions to the plurality of robotic devices may be configured by identifying a plurality of target points of the body part in the acquired observation data, determining the plurality of subgoal positions according to the identified plurality of target points, and dynamically allocating the determined plurality of subgoal positions to the plurality of robotic devices. With this configuration, it is possible to appropriately determine each subgoal position and allocate it to each robotic device.

[0010] In the control device according to the above aspect, the identification of the plurality of target points may be configured by identifying the plurality of target points according to a silhouette of the body part. With this configuration, it is possible to appropriately set each sub-goal position according to the observed body part.

[0011] In the control device according to the above aspect, the identification of the plurality of target points may be configured by identifying the plurality of target points according to characteristics of the body parts. With this configuration, it is possible to appropriately set each sub-goal position according to the observed body part.

[0012] In the control device according to the above aspect, controlling the movement of each of the plurality of robotic devices may be configured by planning a path for each of the robotic devices to move to each of the sub-goal positions, and controlling the movement of each of the robotic devices according to the planned path. With this configuration, it is possible to appropriately drive each of the robotic devices toward each of the sub-goal positions.

[0013] In the control device according to the above aspect, the observation data may include image data. With this configuration, when multiple robotic devices are controlled by body parts via images, the possibility of collisions between the robotic devices can be reduced.

[0014] In the control device according to the above aspect, the observation data may include motion capture data. With this configuration, when multiple robotic devices are controlled by body parts via motion capture, the possibility of collision between the robotic devices can be reduced.

[0015] In the control device according to the above aspect, each of the robotic devices may be an autonomous robot capable of moving. With this configuration, when controlling multiple autonomous robots, it is possible to reduce the possibility of the robotic devices colliding with each other.

[0016] In the control device according to the above aspect, each of the robot devices may be a mobile object capable of moving. With this configuration, when controlling a plurality of mobile objects, it is possible to reduce the possibility of collisions between the robot devices.

[0017] In the control device according to the above aspect, the moving object may be a drone. With this configuration, when controlling multiple drones, it is possible to reduce the possibility of collisions between the robotic devices.

[0018] In the control device according to the above aspect, the body part may include a hand. With this configuration, when multiple robotic devices are controlled by hands, the possibility of the robotic devices colliding with each other can be reduced.

[0019] In the control device according to the above aspect, the body part may include at least a part of a limb. With this configuration, when multiple robotic devices are controlled by at least a part of a limb, it is possible to reduce the possibility of collision between the robotic devices.

[0020] In the control device according to the above aspect, the body part may include a face. With this configuration, when multiple robotic devices are controlled by faces, the possibility of collisions between the robotic devices can be reduced.

[0021] In the control device according to the above aspect, the body part may be configured by a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. With this configuration, when multiple robotic devices are controlled by at least a part of the body and an object, the possibility of collision between the robotic devices can be reduced.

[0022] In the control device according to the above aspect, the control unit may be further configured to determine at least one of the number and size of the robotic devices according to at least one of a task type and an operation target for the task. Dynamically allocating each of the plurality of subgoal positions to each of the plurality of robotic devices may be configured by dynamically allocating each of the plurality of subgoal positions to each of the plurality of robotic devices whose at least one of the number and size has been determined. With this configuration, at least one of the number and size of the robotic devices can be optimized according to the task.

[0023] In the control device according to the above aspect, the control unit may be further configured to determine the range of the body part according to at least one of a task type and an operation target in the task. The plurality of subgoal positions according to the body part may be configured by the plurality of subgoal positions according to the body part whose range has been determined. With this configuration, the range of the body part to be used for operation can be optimized according to the task.

[0024] The present invention is not limited to the control device (information processing device) described above. As another embodiment of the control device according to each of the above aspects, one aspect of the present invention may be an information processing method or program that realizes all or part of each of the above configurations, or may be a storage medium readable by a machine such as a computer that stores such a program. A storage medium readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0025] For example, a control method according to one aspect of the present invention may be an information processing method in which a computer acquires observation data of an operator's body parts, dynamically assigns a plurality of subgoal positions corresponding to the body parts appearing in the acquired observation data to a plurality of robotic devices, and controls the operation of each of the plurality of robotic devices toward each of the assigned subgoal positions.

[0026] Furthermore, for example, a control program according to one aspect of the present invention may be a program for causing a computer to acquire observation data of an operator's body parts, dynamically assign a plurality of subgoal positions corresponding to the body parts appearing in the acquired observation data to a plurality of robotic devices, and control the operation of each of the plurality of robotic devices toward each of the assigned subgoal positions.

[0027] According to the present invention, when a plurality of robotic devices are controlled in accordance with body parts, the possibility of collision between the robotic devices can be reduced.

[0028] FIG. 1 schematically illustrates an example of a scenario to which the present invention is applied. FIG. 2 schematically illustrates an example of a body part according to an embodiment. FIG. 3 schematically illustrates an example of a method for determining multiple subgoal positions according to an embodiment. FIG. 4 schematically illustrates an example of a method for determining multiple subgoal positions according to an embodiment. FIG. 5 schematically illustrates an example of a hardware configuration of a control device according to an embodiment. FIG. 6 schematically illustrates an example of a software configuration of a control device according to an embodiment. FIG. 7 is a flowchart illustrating an example of a processing procedure of a control device according to an embodiment. FIG. 8 illustrates subgoal positions adopted in examples and comparative examples under various conditions of size and density of the robot device. FIG. 9A illustrates the calculation results of the body ownership score in the first experimental example. FIG. 9B illustrates the calculation results of the body ownership score in the first experimental example. FIG. 9C illustrates the calculation results of the actor agency score in the first experimental example. FIG. 9D illustrates the calculation results of the actor agency score in the first experimental example. FIG. 10 illustrates the configuration of a robot device used in a second experimental example. FIG. 11 illustrates the experimental environment for the second experimental example. 12A and 12B show experimental results of the second experimental example.

[0029] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it is specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0030] §1 Application Example Fig. 1 schematically shows an example of a situation in which the present invention is applied. A control device 1 according to this embodiment acquires observation data 20 of a body part BP of an operator P. The control device 1 dynamically assigns, to each of a plurality of robot devices 30, a plurality of sub-goal positions SG corresponding to the body parts BP appearing in the acquired observation data 20. The control device 1 controls the movement of each of the plurality of robot devices 30 toward each of the plurality of assigned sub-goal positions SG.

[0031] In this embodiment, each sub-goal position SG corresponding to a body part BP is dynamically assigned to each robot device 30. As a result, the relationship between each robot device 30 and a body part BP is not fixed (i.e., the assignment can be freely changed). Therefore, even a simple method such as assigning the nearest sub-goal position SG can be expected to avoid collisions between robot devices 30. Therefore, according to this embodiment, when multiple robot devices 30 are controlled in accordance with the body parts BP, the possibility of collisions between the robot devices 30 can be reduced.

[0032] [Observation Data] The type of observation data 20 is not particularly limited as long as it can capture the body part BP, and may be selected appropriately depending on the embodiment. The observation data 20 may be obtained by any method. In one example, the observation data 20 may be obtained by one or more sensors S. The sensor S may be selected appropriately depending on the embodiment. The sensor S may be, for example, an imaging device, a motion capture device, etc. The imaging device may include any device that acquires data in the form of an image or an image representation, such as an RGB camera, a depth camera, an infrared camera, radar, LiDAR (Light Detection and Ranging), etc. The type of motion capture is not particularly limited and may be selected appropriately depending on the embodiment.

[0033] In one example, the sensor S may include an imaging device, and the observation data 20 may include image data obtained by the imaging device. In this example of the present embodiment, in a situation where a plurality of robotic devices 30 are controlled by body parts BP via images, the possibility of collisions between the robotic devices 30 can be reduced.

[0034] In another example, the sensor S may include a motion capture device, and the observation data 20 may include motion capture data obtained by the motion capture device. In one example of this embodiment, in a situation where multiple robotic devices 30 are controlled by body parts BP via motion capture, the possibility of collisions between the robotic devices 30 can be reduced. Note that, as a typical example, optical motion capture that tracks the positions of markers may be used for the motion capture device. In this case, the obtained motion capture data may be composed of trajectory data of the markers.

[0035] [Body Part] The body part BP is not particularly limited and may be the whole body or a part of the body. In one example, the body part BP may be appropriately selected from the hand, arm, limb, face, torso, whole body, etc. The body part BP may refer to a divided area (e.g., hand, arm, face, etc.) or may refer to a different area (e.g., part of the hand and part of the arm, etc.).

[0036] For example, the body part BP may include a hand. This makes it possible to control the operations of multiple robot devices 30 in accordance with the movement of the hand, and also reduces the possibility of collisions between the robot devices 30 in such a situation. In one example, the hand may be used as the body part BP in a situation where a task is performed by hand, such as carrying an object.

[0037] In the example of FIG. 1 , the hand is used as the body part BP, and a scene is assumed in which the hand changes from paper to scissors. In this scene, when the hand changes to scissors, the thumb and ring finger may cross. Therefore, if each robot device is statically assigned to each part of the hand in a one-to-one correspondence, there is a possibility that the robot devices assigned to the thumb and ring finger may collide with each other. In contrast, in this embodiment, collisions between the robot devices can be easily avoided by changing these assignments using any index, such as assigning the nearest subgoal position SG.

[0038] Furthermore, for example, the body part BP may include at least a part of a limb. The limb includes a hand and a foot. This makes it possible to control the operation of multiple robot devices 30 in accordance with the movement of at least a part of the limb, and also reduces the possibility of collisions between the robot devices 30 in such a situation. In one example, at least a part of the limb may be employed as the body part BP in a situation where remote communication is performed using limb gestures.

[0039] Furthermore, for example, the body part BP may include a face. This makes it possible to control the operations of the multiple robotic devices 30 in accordance with facial movements (e.g., facial expressions, head movements), and also reduces the possibility of collisions between the robotic devices 30 in such situations. In one example, in a situation where remote communication is being performed, the face may be adopted as the body part BP in order to convey the facial expression of the operator P by controlling the operations of the multiple robotic devices 30 in accordance with the facial expressions.

[0040] In addition, in one example, when an object such as a tool is under the control of a body part BP, the body part BP may include the range of this object. In other words, the controlled object may be considered a body part BP. "Under control" may refer to a state in which some relationship exists between the body part BP and the object, such as the object being attached to the body part BP or the behavior of the object being controlled by the body part BP. The body part BP may be composed of a part or all of the body, an object under the control of at least a part of the body, or a combination of these. This makes it possible to control the operations of multiple robot devices 30 in accordance with the movements of at least a part of the body and the object, and also reduces the possibility of collisions between the robot devices 30 in such situations.

[0041] "Wearing" may refer to any state in which a tool is linked to a body part, such as by holding it in the hand, wearing it, wrapping it around the body part, or attaching it to the body part. An object may be attached in close contact with a body part BP, or may be attached away from the body part BP. "Controlling" may also refer to any state in which an object is not attached to a body part but is influenced by the body part. States influenced by a body part may include, for example, dribbling a ball, throwing a ball or baton into the air, manipulating a rhythmic gymnastics ribbon, or remotely controlling a moving object (e.g., a mobile autonomous robot, drone, etc.). "Controlling" may include, for example, directly controlling the behavior of an object with a body part, such as dribbling a ball. In addition, "controlling" may include indirectly controlling the behavior of an object, such as autonomously performing at least some of its actions (e.g., self-driving). In other words, "controlling" may include states in which, for example, there is a moment of influence from a body part, such as turning on a power source, pressing a start button, or giving voice instructions for an action, but the object is not controlled by a body part at all other times.

[0042] The type of object is not particularly limited and may be appropriately selected depending on the embodiment. The tool may be an object that is homeomorphic to a one-dimensional line segment, a two-dimensional sphere (S ２ ), a torus (T ２), and objects of any shape, such as objects homeomorphic to a one-dimensional line segment. Tools homeomorphic to a one-dimensional line segment may include, for example, a pole, a flag, a writing implement, a baton, a rhythmic gymnastics ribbon, etc. Tools homeomorphic to a two-dimensional sphere may include, for example, a ball, etc. Tools homeomorphic to a torus may include, for example, a Möbius strip, a hula hoop, etc. Tools of other shapes may include, for example, a costume, a cup, a moving object (e.g., an autonomous robot, a drone, etc.), etc. When using a tool, the body part BP may be composed of a part or all of the body, the tool, or a combination thereof.

[0043] 2 is a schematic diagram illustrating an example of a body part BP according to this embodiment. In the example illustrated in FIG. 2, the body part BP is a hand, the object T is a rod-shaped tool, and a scene is assumed in which the rod-shaped tool is being held by the hand. In such a scene, at least some of the subgoal positions SG may be determined based on the object T.

[0044] 2 assumes a scene in which the subgoal position SG is determined based on the range of both the hand (body part BP) and the tool (object T). In one example, the subgoal position SG may be determined based on both the body part BP and the object T in this way. However, the range in which the subgoal position SG is located is not limited to this example. In another example, even in a scene in which the object T is worn, the subgoal position SG may be determined based only on the body part BP (i.e., the object T may be ignored). In yet another example, in a scene in which the object T is worn, the subgoal position SG may be determined based only on the object T.

[0045] (Number of Operators) In the example of FIG. 1 , there is one operator P. However, the number of operators P is not limited to one, and may be multiple. In one example, the control device 1 may be used in a situation where multiple operators P control the operations of multiple robot devices 30. In this case, the body parts BP may be determined for each of the multiple operators P.

[0046] [Dynamic Allocation] In this embodiment, each sub-goal position SG is dynamically allocated to each robot device 30. That is, the correspondence between the body parts BP and each robot device 30 is not fixed one-to-one, but can change at any timing, for example, the allocation destination can change from the thumb to another part. As long as such a change is allowed, the processing content of the dynamic allocation is not particularly limited and can be determined appropriately depending on the embodiment. In one example, as the dynamic allocation, the reallocation of the sub-goal position SG can be executed every one or more control cycles.

[0047] Furthermore, the method of dynamic allocation is not particularly limited and may be selected appropriately depending on the embodiment. In one example, dynamic allocation may include assigning the closest sub-goal position SG to each robotic device 30. This minimizes the overall movement amount of each robotic device 30 and reduces the possibility of collisions.

[0048] In another example, dynamically allocating may include assigning to each robotic device 30 the subgoal position SG with the smallest cost. The cost may be configured to indicate the degree to which the allocation is recommended using any index (the higher the cost, the less recommended). Indexes for evaluating the degree of recommendation may include, for example, the presence or absence of a collision, the movement distance, the amount of rotation, the power consumption, constraints, or a combination thereof. Constraints may be assigned according to, for example, human perceptual quality, the level of agency, the level of body ownership, etc. Agency refers to whether the operator P feels the movement of each robotic device 30 as his or her own movement. Body ownership refers to whether the operator P feels each robotic device 30 as part of his or her body. Human perceptual quality, the level of agency, or the level of body ownership may be evaluated using indexes such as a sense of unity, a sense of smoothness, or whether devices that are farther apart should move faster. In one example, the cost may be calculated so that it is higher the greater the possibility of a collision and lower the greater the likelihood of a collision. When reducing the travel distance, the cost may be calculated to be higher the longer the travel distance and lower the shorter the travel distance. When reducing rotation, the cost may be calculated to be higher the greater the amount of rotation and lower the smaller the amount of rotation. When reducing the power consumption of each robot device 30, the cost may be calculated to be higher the greater the power consumption and lower the less the power consumption. When obtaining an allocation that satisfies given constraints, the cost may be calculated to be higher the more the constraints are not satisfied and lower the more the constraints are satisfied. The cost may be replaced with a reward (profit), and an allocation that minimizes the cost may be replaced with an allocation that maximizes the reward. The numerical expression of the cost or reward may be set appropriately. The cost may be expressed as being proportional to the numerical value (i.e., the larger the numerical value, the higher the cost) or inversely proportional to the numerical value (i.e., the smaller the numerical value, the higher the cost). The same applies to the reward.Other dynamic allocation methods may include known methods such as those described in Reference 1 (Saurav Agarwal et al., “Simultaneous Optimization of Assignments and Goal Formations for Multiple Robots”, [online], [searched November 29, 2023], Internet <URL: https: / / par.nsf.gov / servlets / purl / 10179280>).

[0049] (Method of determining subgoal position) Each subgoal position SG is a provisional goal position of each robot device 30. Each subgoal position SG is determined according to the observed body part BP. The method of determining each subgoal position SG is not particularly limited and may be selected appropriately depending on the embodiment.

[0050] In one example, dynamically assigning each of the subgoal positions SG to each of the robotic devices 30 may include identifying a plurality of target points of the body part BP in the acquired observation data 20, determining a plurality of subgoal positions SG according to the identified target points, and dynamically assigning each of the determined subgoal positions SG to each of the robotic devices 30. Determining the subgoal positions SG according to the target points may include adopting the target points as the subgoal positions SG directly and determining the subgoal positions SG from the target points according to a predetermined rule. The predetermined rule may be determined appropriately depending on the embodiment. The predetermined rule may include, for example, providing an offset (determining a position away from the target point by the offset distance as the subgoal position SG). According to this example embodiment, the subgoal positions SG can be appropriately determined and assigned to each robotic device 30.

[0051] The method for identifying the plurality of target points in the body part BP is not particularly limited and may be appropriately selected depending on the embodiment. In one example, at least one of the following two methods may be adopted.

[0052] (1) Silhouette-based In one example, identifying a plurality of target points may be configured by identifying a plurality of target points according to the silhouette of the body part BP. Identifying target points according to the silhouette may include dividing the silhouette (range) of the body part BP into regions and identifying any point within the divided region (e.g., a central point, a center of gravity, etc.) as a target point. The method of dividing the region is not particularly limited and may be determined appropriately depending on the embodiment.

[0053] The ratio by which the silhouette of the body part BP is divided is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the silhouette of the body part BP may be divided evenly. In another example, the silhouette of the body part BP may be divided at least partially using different ratios. The ratio may be defined in advance or may be dynamically determined depending on, for example, the shape of the body part BP, the sign (e.g., gesture) of the body part BP, the size of the robotic device 30 to be assigned, the shape of the robotic device 30, etc.

[0054] FIG. 3 schematically shows an example of a method for determining a subgoal position SG based on a silhouette. In the example shown in FIG. 3, a scene is assumed in which the body part BP is a hand. First, the control device 1 extracts the body part BP from the observation data 20. The observation data 20 is, for example, image data. The control device 1 then meshes the silhouette (area) of the extracted body part BP. The meshing may be performed by any method. Meshing methods include, for example, Reference 2 (Xingyu Chen et al., "Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration", [online], retrieved November 29, 2023, Internet URL: https: / / arxiv.org / abs / 2103.02845), Reference 3 (Xingyu Chen et al., "MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image", [online], retrieved November 29, 2023, Internet URL: https: / / arxiv.org / abs / 2112.02753), Reference 4 ("Skinned Mesh Renderer Component", [online], retrieved November 29, 2023), The control device 1 may use the Internet (URL: https: / / docs.unity3d.com / ja / 2021.3 / Manual / class-SkinnedMeshRenderer.html). The control device 1 may cluster the vertices of the obtained mesh using a method such as the k-means algorithm. This allows the silhouette of the body part BP to be divided into regions. The control device 1 may appropriately identify target points in each obtained region. The control device 1 may then determine a subgoal position SG from the identified target points.

[0055] This method may also be applied to body parts other than hands. Furthermore, the method of determining target points based on a silhouette is not limited to this example, and may be modified as appropriate depending on the embodiment. According to one example of this embodiment, each sub-goal position SG can be appropriately set depending on the observed body part BP.

[0056] (2) Feature-Based In one example, identifying the plurality of interest points may comprise identifying the plurality of interest points according to features of the body part BP. The features of the body part BP may include any extractable element.

[0057] The control device 1 may extract the features of the body part BP and use the obtained feature points (feature points) as target points. The feature points of the body part BP may be, for example, feature points of the skeleton, muscles, face, object, etc. The feature points of the skeleton may be, for example, joints, tips, etc. The feature points of the muscles may be, for example, connecting parts, centers, etc. The feature points of the face may be, for example, end points, centers, etc. of features. The features may be, for example, organs (mouth, eyes, ears, nose), other parts (eyebrows, forehead, etc.), etc. The feature points of the object may be, for example, end points, midpoints, center of gravity, etc.

[0058] The features may be extracted directly from the observation data 20, or may be extracted indirectly through information processing such as skeletal estimation. When the observation data 20 is image data, the control device 1 may extract the features of the body parts BP through image processing. When a motion capture device that tracks markers is used as the sensor S, the control device 1 may extract the markers directly as feature points. The feature points to be extracted may be determined in advance or may be determined dynamically. The method for extracting feature points is not particularly limited and may be selected appropriately depending on the embodiment.

[0059] FIG. 4 schematically illustrates an example of a feature-based method for determining a subgoal position SG. In the example illustrated in FIG. 4, the body part BP is a hand, the observation data 20 is image data, and a skeleton is assumed to be extracted as a feature. First, the control device 1 extracts the body part BP from the observation data 20. The control device 1 then aligns the skeleton with the extracted body part BP (skeleton estimation). The position of the skeleton may be estimated by any method, such as a machine learning-based method. The method for estimating the position of the skeleton may be, for example, Reference 5 (Tomas Simon et al., "Hand Keypoint Detection in Single Images using Multiview Bootstrapping," [online], searched November 29, 2023, Internet URL: https: / / arxiv.org / abs / 1704.07809). The control device 1 may identify pre-specified feature points on the obtained skeleton as target points (rule-based). Then, the control device 1 may determine the subgoal position SG from the identified target points.

[0060] This method may also be applied to body parts other than hands. By replacing the skeleton with another feature, this method may also be applied to features other than the skeleton. Furthermore, the method of determining target points based on features is not limited to this example, and may be modified as appropriate depending on the embodiment. According to one example of this embodiment, each subgoal position SG can be appropriately set depending on the observed body part BP.

[0061] [Movement Control] The method for controlling the movement of each robot device 30 toward each sub-goal position SG is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the control device 1 may determine a route to each sub-goal position SG by path planning. That is, controlling the movement of each of the multiple robot devices 30 may be configured by planning a path for each robot device 30 to move to each sub-goal position SG, and controlling the movement of each robot device 30 according to the planned path. Path planning may be performed by any method. The movement control algorithm may employ known techniques such as those described in Reference 6 (Jur van den Berg et al., "Reciprocal Velocity Obstacles for Real-Time Multi-Agent Navigation," [online], [searched November 29, 2023], internet <URL: https: / / gamma.cs.unc.edu / RVO / icra2008.pdf>) and Reference 7 (Jamie Snape et al., "Smooth and Collision-Free Navigation for Multiple Robots Under Differential-Drive Constraints," [online], [searched November 29, 2023], internet <URL: http: / / gamma-web.iacs.umd.edu / ORCA-DD / ORCA-DD.pdf>). According to this embodiment, each robot device 30 can be appropriately driven toward each subgoal position SG. The control device 1 may directly control each robot device 30, or may indirectly control each robot device 30 via an external computer such as a controller.

[0062] [Robot Device] The type of robot device 30 is not particularly limited and may be selected appropriately depending on the embodiment. The robot device 30 may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile body configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, or an orthogonal robot. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a nursing robot, a security robot, or a transport robot (including a disaster relief robot). The content of the autonomous processing may be selected appropriately depending on the embodiment. The mobile body may include, for example, a cleaning robot, the above-mentioned mobile autonomous robot (including a mobile robot), a vehicle configured to be autonomously driven (including dedicated vehicles such as an ambulance, a fire engine, or a construction vehicle), or an autonomous flying body (such as a drone). The fire engine may be, for example, a rescue vehicle, a vehicle equipped with a portable fire pump, or the like.

[0063] For example, each robotic device 30 may be an autonomous robot capable of moving. In this case, it is possible to reduce the possibility of collisions in situations where multiple autonomous robots are controlled by physical manipulation. Furthermore, for example, each robotic device 30 may be a mobile body capable of moving. In this case, it is possible to reduce the possibility of collisions in situations where multiple mobile bodies are controlled by physical manipulation. Furthermore, the mobile body may be a drone. In this case, it is possible to reduce the possibility of collisions in situations where multiple drones are controlled by physical manipulation.

[0064] Each robotic device 30 may move on at least one of land, water, and air. The space in which it moves may be real space or virtual space. That is, each robotic device 30 may exist in real space or virtual space. Note that a plurality of robotic devices 30 controlled by physical manipulation may also be referred to as a "swarm robot."

[0065] [Usage Scenarios] The control device 1 according to this embodiment may be used in any situation where a robotic device 30 in a real space or a virtual space is controlled. In one example, the control device 1 according to this embodiment may be used for real-time physical telepresence or operation.

[0066] For example, the control device 1 according to this embodiment may be used for communication with another person who is remote from the operator P. The control device 1 may realize remote communication between the operator P and the other person by having multiple robot devices 30 present at the other person perform gestures or facial expressions that appear on body parts BP. In other words, the control device 1 may cause each robot device 30 to execute a communication task.

[0067] Furthermore, for example, the control device 1 may realize interactions between an operator P and another person by having multiple robotic devices 30 present at another person perform interactions such as pointing, touching, carrying an object, grabbing an object, and writing with a writing implement. That is, the control device 1 may cause each robotic device 30 to execute an interaction task. Interactions such as carrying an object and grabbing an object may be performed for collaborative work with another person present at a remote location. Collaborative work may also include cooperative work at a disaster site (e.g., removing rubble). In this case, because each robotic device 30 is configured to be smaller than a hand, each robotic device 30 can be physically manipulated to enter gaps that a human hand cannot reach.

[0068] In each of the above cases, when controlling the robotic devices 30 in real space, the control device 1 may be located either on the operator P's side or on another person's side. In one example, the control device 1 may be located on the operator P's side and transmit instructions to each robotic device 30 to a computer located on the other person's side via a network. In another example, the control device 1 may be located on the other person's side and receive observation data 20 from the computer on the operator P's side via a network, and use the received observation data 20 to execute the above-mentioned arithmetic processing, thereby controlling the operation of each robotic device 30. Additionally, the control device 1 according to this embodiment may be used to control the robotic devices 30 in a virtual space.

[0069] (Feedback Method) The method of feeding back the operation of the robot device 30 to the operator P is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the operator P may directly visually recognize the operation of the robot device 30 by driving the robot device 30 near the operator P. In another example, the operator P may visually recognize the operation of the robot device 30 in real space or virtual space via at least one of a display, a projector display, and a virtual reality (VR) display. This feedback may be performed by the control device 1 or an external computer other than the control device 1. For example, the control device 1 or the external computer may project the status of each robot device 30 near the body part BP of the operator P via a projector (e.g., at the hand if the body part BP is a hand). When each robot device 30 exists in real space, the status of each robot device 30 may be appropriately observed by a sensor such as an imaging device. The control device 1 or the external computer may feed back the obtained sensing data to the operator P. As a feedback method, using a projector to project near the body part BP or using VR display can be expected to improve the sense of agency and body ownership. Also, for example, each robot device 30 may exist in a VR space, and the operator P may control each robot device 30 by manipulating their body in the VR space. In this way, the control device 1 may operate as a user interface in the VR space.

[0070] (Number and Size of Robotic Devices) The number and size of the robotic devices 30 are not particularly limited and may be determined appropriately depending on the embodiment. At least one of the number and size of the robotic devices 30 may be given in advance, may be determined in preprocessing, or may be determined dynamically during execution of the control process of the robotic devices 30.

[0071] In one example, the control device 1 may determine at least one of the number and size of the robot devices 30 to be used based on the type of task (including gestures) to be performed by the robot devices 30, the operation target in the task, the size of the body part BP, or a combination thereof. For example, in the case of a task requiring force, such as carrying an object or pressing a button, the control device 1 may determine at least one of increasing the number of robot devices 30 to be used and using a larger robot device 30. On the other hand, in the case of a task requiring no force, such as simply moving for communication, the control device 1 may determine at least one of reducing the number of robot devices 30 to be used and using a smaller robot device 30. Similarly, for example, if the operation target is heavy, the control device 1 may determine at least one of increasing the number of robot devices 30 to be used and using a larger robot device 30. On the other hand, if the operation target is light, the control device 1 may determine at least one of reducing the number of robot devices 30 to be used and using a smaller robot device 30. Furthermore, for example, when the target range of the body part BP is wide (e.g., when the hand is made into an open hand), the control device 1 may determine at least one of increasing the number of robot devices 30 to be used and using a larger robot device 30. On the other hand, when the target range of the body part BP is narrow (e.g., when the hand is made into a fist), the control device 1 may determine at least one of decreasing the number of robot devices 30 to be used and using a smaller robot device 30. The relationship between the type of task, etc. and the number of robot devices 30, etc. may be determined as appropriate depending on the embodiment, for example, by any method such as a rule-based method.

[0072] When each robotic device 30 is used to perform a task in real space, events (including objects) related to the task appear in the environment in which each robotic device 30 exists. Therefore, before performing dynamic allocation, the control device 1 may acquire sensing data from the sensors described above for feedback on the state of each robotic device 30 and estimate at least one of the type of task and the operation target from the acquired sensing data using any method. In one example, the control device 1 may estimate at least one of the type of task and the operation target based on an object detected from the sensing data. For example, the control device 1 may estimate that the task to be performed is to transport the object based on the detection of an object to be transported. When the operator P and each robotic device 30 are located nearby, the sensor S observing the operator P may include a sensor observing each robotic device 30, and the observation data 20 may include this sensing data.

[0073] The range of the body part BP may be determined based on at least one of the type of task and the operation target of the task. Accordingly, the subgoal positions SG corresponding to the body part BP may be composed of the subgoal positions SG corresponding to the body part BP whose range has been determined. The control device 1 may increase or decrease the range of the body part BP based on at least one of the type of task and the operation target of the task. For example, the control device 1 may change the range of the body part BP from only the hands to the hands and arms when the task is carrying a heavy load, change the range of the body part BP from one hand to two hands when the task is carrying multiple loads, or change the range of the body part BP from both hands to one hand when the task is carrying a light load. The control device 1 may at least either increase the number of robot devices 30 to be used or use a larger robot device 30 when the range of the body part BP is expanded. Furthermore, the control device 1 may at least either reduce the number of robot devices 30 to be used or use a smaller robot device 30 when the range of the body part BP is narrowed. According to one example of this embodiment, the range of body parts BP used for operation can be optimized depending on the task.

[0074] The method for determining the type of task, the operation target in the task, and the range of body parts BP is not limited to the above method. In a simple example, at least one of the type of task, the operation target in the task, and the range of body parts BP may be determined according to a selection by a user (including operator P). The control device 1 may appropriately accept input of the selection via an input device before executing dynamic allocation.

[0075] When a task is continuously executed, the control device 1 may periodically or irregularly repeatedly determine at least one of the number and size of the robotic devices 30 to be used using the above-described method. As a result, at least one of the number and size of the robotic devices 30 to be used may be changed while the task is being executed. Furthermore, when multiple tasks are continuously executed, the control device 1 may determine at least one of the number and size of the robotic devices 30 to be used for the next task using the above-described method when switching the task to be executed (i.e., after completing the current task and before starting the next task). As a result, at least one of the number and size of the robotic devices 30 to be used may be changed when switching the task to be executed. A task is an operation to be executed by at least one of the multiple robotic devices 30. The type of task is not particularly limited and may be selected appropriately depending on the embodiment. Tasks may include, for example, the above-described communication, interaction, etc.

[0076] When the control device 1 determines at least one of the number and size of the robot devices 30 before performing dynamic allocation, dynamically assigning each subgoal position SG to each robot device 30 includes assigning each subgoal position SG to each robot device 30 whose number and / or size have been determined. When the number of robot devices 30 is determined before assigning each subgoal position SG to each robot device 30, in the step of determining the subgoal positions SG, the control device 1 determines the same number of subgoal positions SG as the determined number of robot devices 30. For example, when the silhouette-based approach is adopted, the control device 1 performs the same number of region divisions as the determined number of robot devices 30. Furthermore, when the feature-based approach is adopted, the control device 1 extracts the same number of feature points as the determined number of robot devices 30. The feature points to be used may be determined in advance depending on the number of robot devices 30. According to an example of the present embodiment, at least one of the number and size of the robot devices 30 can be optimized depending on the task.

[0077] In another example, the control device 1 may dynamically change the number of sub-goal positions SG in the process of identifying the sub-goal positions SG. That is, the control device 1 may identify a variable number of sub-goal positions SG in the process. In this case, the control device 1 may determine the number of robot devices 30 to be used in accordance with the number of identified sub-goal positions SG. Thus, the control device 1 may determine the number of robot devices 30 to be used after determining the number of sub-goal positions SG (i.e., during dynamic allocation).

[0078] In response to this, the control device 1 may also determine the size of the robot device 30 to be used. For example, the control device 1 may select to use a smaller robot device 30 in response to an increase in the number of robot devices 30. In addition, for example, the control device 1 may select to use a smaller robot device 30 when the intervals between the identified sub-goal positions SG are narrow, and may select to use a larger robot device 30 when the intervals between the sub-goal positions SG are wide.

[0079] At least one of the size and type of the robot devices 30 used may be unified (i.e., identical) or may be at least partially different. The control device 1 may determine at least one of the size and type of the robot devices 30 to be assigned depending on the location of the body part BP, such as assigning a small robot device 30 to the fingertips and a large robot device 30 to the parts other than the fingertips.

[0080] When at least one of the number and size of the robotic devices 30 is allowed to be changed in real space, candidate robotic devices to be used may be pooled in any location. The pool may include, for example, multiple types of robotic devices with different attributes such as size and type. The pool may be a fixed area, or an unspecified and arbitrary location away from the operating robotic devices 30. The control device 1 may select a robotic device 30 to be used from the pooled robotic devices depending on the result of determining at least one of the number and size of the robotic devices 30 to be used through the above processing. The control device 1 may control the operations of the selected multiple robotic devices 30 to move each robotic device 30 to a location where the task is to be performed.

[0081] Furthermore, during the continuation of a task or when switching tasks, a process for determining at least one of the number and size of robotic devices 30 to be used may be executed, resulting in a change in the robotic devices 30 to be used. In this case, the control device 1 may select a robotic device from the pooled robotic devices that is compatible with the robotic device newly added as the robotic device 30 to be used due to the change in the robotic device 30 to be used, and control the operation of the selected robotic device 30 to move the selected robotic device 30 to the location where the task is to be performed. In this way, the new robotic device 30 may be added to the performance of the task. Furthermore, the control device 1 may evacuate the robotic device 30 that is no longer in use due to the change in the robotic device 30 to be used to the pool, and remove it from the control targets after the evacuation is complete.

[0082] When changes in at least one of the number and size of the robot devices 30 are permitted in the virtual space, the control device 1 may represent the change in the robot devices 30 in any manner. In one example, the control device 1 may represent a change in the robot devices 30 to be used in the virtual space in the same manner as in the real space. In another example, the control device 1 may make a newly added robot device 30 to be used appear at any timing and add it to the performance of a task. Furthermore, the control device 1 may erase from the display at any timing a robot device 30 that is no longer being used due to a change in the robot devices 30 to be used.

[0083] (Robot Device Movement Range) The relationship between the movement range of the body part BP and the movement range of each robot device 30 is not particularly limited and may be determined appropriately depending on the embodiment. Furthermore, the scale between the movement range of the body part BP and each robot device 30 may also be determined appropriately depending on the embodiment. The scale between the movement range of the body part BP and each robot device 30 may be the same or different. The control device 1 may determine the scale of the movement range of each robot device 30 depending on the width of the range of the observed body part BP and control the operation of each robot device 30 within the determined movement range. For example, the control device 1 may set the movement range of each robot device 30 to be wide depending on the range of the body part BP, or may set the movement range of each robot device 30 to be narrow depending on the range of the body part BP. Conversely, for example, making a fist narrows the range of the body part BP, which may make avoidable path planning difficult if the movement range is narrowed. Therefore, the control device 1 may set a wider range of motion for each robot device 30 in response to a narrower range of the body part BP. The control device 1 may also determine the scale of the range of motion in response to at least one of the type of task and the operation target in the task. Furthermore, the control device 1 may determine at least one of the number and size of the robot devices 30 in response to the scale of the range of motion. According to one example of this embodiment, the body expression formed by the robot device 30 can be freely changed. This allows the body expression to be optimized in response to the environment, which can be expected to result in an improvement in at least one of body ownership and agency.

[0084] §2 Configuration Example [Hardware Configuration] Fig. 5 shows a schematic example of the hardware configuration of the control device 1 according to this embodiment. In the example shown in Fig. 5, the control device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0085] The control unit 11 includes a hardware processor such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0086] The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, or the like. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as a control program 81. The control program 81 is a program for causing the control device 1 to execute information processing (see FIG. 7 described below) related to the motion control of each robot device 30 through body control. The control program 81 includes a series of commands for this information processing.

[0087] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, or the like. The type and number of external interfaces 13 may be determined appropriately depending on the embodiment. If the external interface 13 includes a communication port, the communication standard of the communication port may be selected arbitrarily. In this embodiment, the control device 1 may be connected to at least one of a sensor S that observes an operator P and a sensor that observes each robot device 30 via the external interface 13.

[0088] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a projector, a VR device, a speaker, etc. A user, including an operator P, can operate the control device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrated into one device, for example, a touch panel display, etc. Note that the operator P who controls each robot device 30 and the user who operates the control device 1 may be the same person or different people.

[0089] The drive 16 is a device for reading various information such as programs stored in a storage medium 91. The control program 81 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as stored programs) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The control device 1 may obtain the control program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface 13.

[0090] Note that, with regard to the specific hardware configuration of the control device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The control device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may be the same or at least partially different. Furthermore, the control device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a mobile device, a terminal device, or the like.

[0091] 6 schematically illustrates an example of the software configuration of the control device 1 according to this embodiment. The control unit 11 of the control device 1 loads the control program 81 stored in the storage unit 12 into RAM and executes instructions included in the control program 81 using the CPU. As a result, the control device 1 operates as a computer including a data acquisition unit 111, a dynamic allocation unit 112, and an operation control unit 113 as software modules. That is, in this embodiment, each software module of the control device 1 is realized by the control unit 11 (CPU).

[0092] The data acquisition unit 111 is configured to acquire observation data 20 of a body part BP of an operator P. The dynamic allocation unit 112 is configured to dynamically allocate, to each of the plurality of robot devices 30, a plurality of sub-goal positions SG corresponding to the body part BP appearing in the acquired observation data 20. The movement control unit 113 is configured to control the movement of each of the plurality of robot devices 30 toward each of the plurality of allocated sub-goal positions SG.

[0093] In this embodiment, an example is described in which each software module of the control device 1 is implemented by a general-purpose CPU. However, some or all of the software modules may be implemented by one or more dedicated processors or chipsets. Each module may also be implemented as a hardware module. Regarding the software configuration of the control device 1, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0094] §3 Operational Example Figure 7 is a flowchart showing an example of the processing procedure of the control device 1 according to this embodiment. The following processing procedure is an example of a control method executed by a computer. However, the following processing procedure of the control device 1 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0095] (Step S101) In step S101, the control unit 11 operates as the data acquisition unit 111 and acquires the observation data 20 of the body part BP of the operator P.

[0096] In one example, the body part BP may include a hand, an arm, a limb, a face, a torso, or the entire body. The body part BP may include an object attached to the body part BP. The observation data 20 may include at least one of image data and motion capture data. The body part BP may be observed by a sensor S, and the control unit 11 may acquire the observation data 20 directly or indirectly from the sensor S. Upon acquiring the observation data 20, the control unit 11 proceeds to the next step S102.

[0097] (Step S102) In step S102, the control unit 11 operates as the dynamic allocation unit 112 and dynamically allocates a plurality of sub-goal positions SG corresponding to the body parts BP appearing in the acquired observation data 20 to each of the plurality of robot devices 30.

[0098] In one example, the control unit 11 may identify multiple target points of the body part BP in the acquired observation data 20, determine multiple sub-goal positions SG according to the identified multiple target points, and dynamically assign each of the determined multiple sub-goal positions SG to each of the multiple robot devices 30. As an example of the process of identifying target points, the control unit 11 may identify multiple target points according to the silhouette of the body part BP. As another example, the control unit 11 may identify multiple target points according to the characteristics of the body part BP. After dynamically assigning each sub-goal position SG to each robot device 30, the control unit 11 proceeds to the next step S103.

[0099] (Step S103) In step S103, the control unit 11 operates as the movement control unit 113 and controls the movement of each of the plurality of robot devices 30 toward each of the plurality of assigned sub-goal positions SG.

[0100] In one example, in the process of step S103, the control unit 11 may plan a path for moving to each sub-goal position SG for each robotic device 30, and control the operation of each robotic device 30 according to the planned path. In one example, each robotic device 30 may be at least one of an autonomous robot and a mobile object. When a mobile object is used, each robotic device 30 may be a drone. After controlling the operation of each robotic device 30, the control unit 11 proceeds to the next step S104.

[0101] (Step S104) In step S104, the control unit 11 determines whether or not to end control of each robot device 30. The criteria for this determination may be set arbitrarily. In one example, the control unit 11 may determine not to end control of each robot device 30 until an arbitrary end instruction (e.g., an end operation by the user via the input device 14) is given. If it is determined not to end control of each robot device 30, the control unit 11 returns to step S101 and executes the process again from step S101. On the other hand, if an arbitrary end instruction is given, the control unit 11 may determine to end control of each robot device 30. If it is determined to end control of each robot device 30, the control unit 11 ends the processing procedure of the control device 1 according to this operation example. The processes of steps S101 to S104 may be executed in real time. Furthermore, the timing of executing the process of step S104 is not limited to this example. The control device 1 may end its operation at any timing.

[0102] In one example, the control unit 11 may provide feedback on the status of each robot device 30 to the operator P at any timing. The feedback may be provided by at least one of a display, a projector, and a VR display.

[0103] In one example, before executing the process of step S102, the control unit 11 may determine at least one of the number and size of the robot devices 30 to be used depending on the type of task, the operation target in the task, the size of the body part BP, or a combination thereof. In step S102, the control unit 11 may assign a plurality of sub-goal positions SG to each of the plurality of robot devices 30 whose number and / or size have been determined.

[0104] In one example, the control unit 11 may determine the range of the body part BP at any timing depending on at least one of the type of task and the operation target in the task. Accordingly, in step S102, the control unit 11 may assign, to each of the robot devices 30, a plurality of sub-goal positions SG corresponding to the determined range of the body part BP.

[0105] In one example, in step S102, the control unit 11 may identify a variable number of subgoal positions SG. In response to this, the control unit 11 may determine the number of robot devices 30 to be used according to the number of identified subgoal positions SG. The control unit 11 may also determine the size of the robot devices 30 to be used.

[0106] In one example, the control unit 11 may execute the loop of step S104 repeatedly to execute steps S101 to S103, thereby continuously executing a task or consecutively executing multiple tasks. While continuously executing a task or when switching the task to be performed, the control unit 11 may execute a process to determine at least one of the number and size of the robot devices 30 to be used. As a result, the control unit 11 may change at least one of the number and size of the robot devices 30 to be used.

[0107] [Features] In the control device 1 according to this embodiment, by the processing of step S102, each sub-goal position SG corresponding to a body part BP is dynamically assigned to each robot device 30. As a result, the relationship between each robot device 30 and the body part BP is not fixed, so collisions between the robot devices 30 can be avoided even with a simple method. Therefore, according to this embodiment, when multiple robot devices 30 are controlled in accordance with the body parts BP, the possibility of collisions between the robot devices 30 can be reduced. In addition, according to one example of this embodiment, the operator P can feel that each robot device 30 is a part of his or her own body.

[0108] §4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs. Various improvements or modifications may be made to the above embodiments as appropriate.

[0109] §5 Experimental Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

[0110] [First Experimental Example] In the first experimental example, a VR device (MetaQuest 2) was used to create a task environment in a VR space in which participants had to move multiple robot devices to target positions using specified hand signs, and participants were asked to complete the task. Ten participants (6 men, 4 women, average age: 24.20, standard deviation of age: 2.57) participated in the first experimental example. Four types of hand signs were prepared: rock, scissors, paper, and reverse paper. Two types of target positions were prepared. In other words, 4 x 2 = 8 tasks were set. The right hand was used as the body part used for operation.

[0111] The shape of the robot device was set to be cylindrical. Two diameter sizes of the robot device were prepared (20 mm and 30 mm). Three density (number of robot devices) conditions were prepared for each size. Specifically, for the 20 mm size, three conditions were prepared: 6 devices (sparse), 18 devices (medium), and 27 devices (dense). For the 30 mm size, three conditions were prepared: 6 devices (sparse), 8 devices (medium), and 12 devices (dense).

[0112] In the first embodiment, the feature-based (skeleton-based) method shown in FIG. 4 was used to identify subgoal positions, and each subgoal position was dynamically assigned to each robot device by assigning the closest subgoal position (bone-dynamic). In the second embodiment, the silhouette-based method shown in FIG. 3 was used to identify subgoal positions (silhouette-dynamic). Other conditions in the second embodiment were the same as those in the first embodiment. Meanwhile, in the first comparative example, the feature-based (skeleton-based) method shown in FIG. 4 was used to identify subgoal positions, as in the first embodiment, but each subgoal position was statically assigned to each robot device on a one-to-one basis (bone-static). After each subgoal position was assigned to each robot device, the operation of each robot device was controlled using the same path planning method in the first, second, and first comparative examples.

[0113] Figure 8 shows the subgoal positions (feature points) adopted in the first example and the first comparative example under each condition of robot device size and density. As described above, 2 x 3 x 3 = 18 conditions were prepared for the robot device size (20 mm, 30 mm), density (sparse, medium, dense), and subgoal position allocation method (first example, second example, first comparative example). Each participant was asked to perform the above eight tasks for each condition and to answer the following questionnaire after completing each of the eight tasks. Each response was scored on a 7-point Likert scale.

[0114] (Survey) (1) Questions about body ownership (1-1) It felt like the swarm robot was my body. (1-2) It felt like some of the robots were my fingers. (1-3) It felt like the swarm robots belonged to me. (1-4) The swarm robot felt like a human hand. (2) Questions about agency (2-1) The movements of the swarm robot felt like they were my movements. (2-2) I felt like I was controlling the movements of the swarm robot. (2-3) I felt like I was causing the movements of the swarm robot. (2-4) The movements of the swarm robot were in sync with my The swarm robots' movements were synchronized with my own movements.

[0115] Each response was scored on a 7-point Likert scale. Scores for the first example, second example, and second comparative example were calculated by calculating the average of the responses to the four questions regarding body ownership and agency under each of the robot device size and density conditions.

[0116] (Experimental Results) Figures 9A and 9B show the calculation results for body ownership scores for sizes of 20 mm and 30 mm. Figures 9C and 9D show the calculation results for agency scores for sizes of 20 mm and 30 mm. In each figure, the calculation results for the scores of the first comparative example, the first example, and the second example are shown from left to right for each density condition. Comparing the first example and the first comparative example, the scores of the first example were higher than those of the first comparative example in both body ownership and agency. It was speculated that this was due to the effect of employing dynamic allocation, which made it possible to avoid collisions between the robot devices.

[0117] Furthermore, for the second example, the scores for body ownership and agency were low under the sparse condition. This was presumably due to the fact that the silhouette-based approach makes it easier for the shape of the swarm robot to deviate from the body shape compared to the skeleton-based approach. On the other hand, under the dense condition, the scores for both body ownership and agency for the second example increased. Under the condition where the robot device size was 20 mm, the score for the second example was higher than that of the first comparative example. These results demonstrate that even the silhouette-based approach can produce sufficiently beneficial effects under the dense condition.

[0118] [Second Experimental Example] In the second experimental example, a robot device with a diameter of 30 mm was fabricated, and the task of the first experimental example was reproduced in real space using the 30 mm diameter robot device.

[0119] Figure 10 shows the configuration of the robot device used in the second experimental example. As shown in Figure 10, the robot device was equipped with a printed circuit board, a Li-Po battery (40 mAh), a housing, a motor holder, a motor (Pololu 2357), wheels, and caster wheels. The robot device also included a microcontroller unit (STMicroelectronics STM32G071KBU6), a motor driver (Texas Instruments DRV8837DSGR), an RF module (RF2401F20), and two photodiodes (Everlight Electronics PD15-22C / TR8).

[0120] Figure 11 shows the experimental environment for the second experimental example. A cradle was connected to the host computer (host PC) via USB 2.0. The robot and cradle were equipped with RF modules and communicated via wireless communication in the 2.4 GHz ISM band. A high-speed projector (Texas Instruments DLP LightCrafter 4500) projected a Gray code pattern onto the table. The robot received the projected and coded pattern light with two photodiodes. The robot's microcontroller decoded the pattern information into position information, calculated its orientation from the positions of the two photodiodes, and broadcast the calculated position and orientation information to the host computer. The host computer determined the position and orientation of each robot by receiving the information broadcast from each robot. The host computer sent commands to each robot via the cradle every 100 milliseconds to move to the subgoal position.

[0121] The task settings were the same as in the first experimental example (i.e., eight tasks were set). The robot device density conditions were also the same as in the first experimental example (sparsely spaced six robots, medium spaced eight robots, and densely spaced twelve robots). Ten participants (four males, six females, mean age: 28.89, standard deviation of age: 13.83) participated in the second experimental example.

[0122] In the third example, the subgoal position assignment method was a bone-dynamic method, similar to the first example. In the fourth example, the subgoal position assignment method was a silhouette-dynamic method, similar to the second example. Meanwhile, in the second comparative example, the bone-static method was used, similar to the first comparative example. Three x three = nine conditions were prepared for the robot device density and subgoal position assignment method. Each participant performed a task of operating a robot device placed on a table with their right hand. The other conditions were the same as in the first experimental example. That is, after assigning each subgoal position to each robot device, the movement of each robot device was controlled using the same path planning method in the third example, fourth example, and second comparative example. Each participant performed the eight tasks described above for each condition and answered the same questionnaire as in the first experimental example after completing each of the eight tasks.

[0123] (Experimental Results) Figures 12A and 12B show the calculation results of the scores for body ownership and agency. Each figure shows, from left to right, the calculation results for the scores of the second comparative example, the third example, and the fourth example for each density condition. As shown in each figure, in the second experimental example, as in the first experimental example, the score of the third example, which adopted skeleton-based dynamic allocation, was generally high. Furthermore, the score of the fourth example, which adopted silhouette-based dynamic allocation, increased under dense conditions.

[0124] [Summary] As described above, in both the first and second experimental examples, the skeleton-based allocation method employed dynamic allocation, resulting in higher scores for body ownership and agency (first example, third example, first comparative example, and second comparative example). From these results, it was inferred that employing dynamic allocation made it easier to avoid collisions between robot devices, thereby achieving the effect of improving body ownership and agency. Furthermore, it was found that the silhouette-based method also produced advantageous results by increasing the density of robot devices. This demonstrated the usefulness of the above-described embodiment.

[0125] This specification includes the following disclosure: [Supplementary Note 1] A control device (1) including a control unit (11), wherein the control unit (11) is configured to: acquire observation data (20) of a body part (BP) of an operator (P), dynamically assign a plurality of subgoal positions (SG) corresponding to the body part (BP) appearing in the acquired observation data (20) to each of a plurality of robotic devices (30), and control the movement of each of the plurality of robotic devices (30) toward the assigned subgoal positions (SG). [Supplementary Note 2] The control device (1) according to Supplementary Note 1, wherein dynamically allocating each of the plurality of subgoal positions (SG) to each of the plurality of robotic devices (30) comprises: identifying a plurality of target points of the body part (BP) in the acquired observation data (20); determining the plurality of subgoal positions (SG) according to the identified plurality of target points; and dynamically allocating each of the determined plurality of subgoal positions (SG) to each of the plurality of robotic devices (30). [Supplementary Note 3] The control device (1) according to Supplementary Note 2, wherein identifying the plurality of target points comprises identifying the plurality of target points according to a silhouette of the body part (BP). [Supplementary Note 4] The control device (1) according to Supplementary Note 2, wherein identifying the plurality of target points comprises identifying the plurality of target points according to a feature of the body part (BP). [Supplementary Note 5] The control device (1) according to any one of Supplementary Notes 1 to 4, wherein controlling the movement of each of the plurality of robot devices (30) comprises planning a path for moving to each of the subgoal positions (SG) for each of the robot devices (30), and controlling the movement of each of the robot devices (30) according to the planned path. [Supplementary Note 6] The control device (1) according to any one of Supplementary Notes 1 to 5, wherein the observation data (20) includes image data. [Supplementary Note 7] The control device (1) according to any one of Supplementary Notes 1 to 5, wherein the observation data (20) includes motion capture data.[Supplementary Note 8] The control device (1) according to any one of Supplementary Notes 1 to 7, wherein each of the robotic devices (30) is an autonomous robot capable of moving. [Supplementary Note 9] The control device (1) according to any one of Supplementary Notes 1 to 8, wherein each of the robotic devices (30) is a moving body capable of moving. [Supplementary Note 10] The control device (1) according to Supplementary Note 9, wherein the moving body is a drone. [Supplementary Note 11] The control device (1) according to any one of Supplementary Notes 1 to 10, wherein the body part (BP) includes a hand. [Supplementary Note 12] The control device (1) according to any one of Supplementary Notes 1 to 11, wherein the body part (BP) includes at least a part of a limb. [Supplementary Note 13] The control device (1) according to any one of Supplementary Notes 1 to 12, wherein the body part (BP) includes a face. [Supplementary Note 14] The control device (1) according to any one of Supplementary Notes 1 to 10, wherein the body part (BP) is composed of a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. [Supplementary Note 15] The control device (1) according to any one of Supplementary Notes 1 to 14, wherein the control unit (11) is further configured to determine at least one of the number and size of the robot devices (30) according to at least one of a task type and an operation target in the task, and wherein dynamically allocating each of the plurality of subgoal positions (SG) to each of the plurality of robot devices (30) is configured by dynamically allocating the plurality of subgoal positions (SG) to each of the plurality of robot devices (30) for which at least one of the number and size has been determined. [Supplementary Note 16] The control device (1) according to any one of Supplementary Notes 1 to 15, wherein the control unit (11) is further configured to determine a range of the body part (BP) according to at least one of a type of task and an operation target in the task, and the plurality of sub-goal positions (SG) according to the body part (BP) are composed of the plurality of sub-goal positions (SG) according to the body part (BP) whose range has been determined.[Supplementary Note 17] A control method in which a computer (1) executes the following: acquiring observation data (20) of a body part (BP) of an operator (P), dynamically assigning a plurality of sub-goal positions (SG) corresponding to the body parts (BP) appearing in the acquired observation data (20) to a plurality of robot devices (30), respectively, and controlling the movement of each of the plurality of robot devices (30) toward the respective assigned sub-goal positions (SG). [Supplementary Note 18] A control program (81) for causing a computer (1) to execute the following: acquiring observation data (20) of a body part (BP) of an operator (P), dynamically assigning a plurality of sub-goal positions (SG) corresponding to the body parts (BP) appearing in the acquired observation data (20) to a plurality of robot devices (30), respectively, and controlling the movement of each of the plurality of robot devices (30) toward the respective assigned sub-goal positions (SG).

[0126] REFERENCE SIGNS LIST 1...control device, 11...control unit, 12...storage unit, 13...external interface, 14...input device, 15...output device, 16...drive, 81...control program, 91...storage medium, 111...data acquisition unit, 112...dynamic allocation unit, 113...motion control unit, 20...observation data, 30...robot device, P...operator, BP...body part, SG...subgoal position, S...sensor

Claims

1. A control device having a control unit, the control unit being configured to execute the following: acquiring observation data of an operator's body parts; dynamically assigning, to each of a plurality of robotic devices, a plurality of sub-goal positions corresponding to the body parts appearing in the acquired observation data; and controlling the operation of each of the plurality of robotic devices toward each of the assigned sub-goal positions.

2. The control device of claim 1, wherein dynamically allocating each of the plurality of sub-goal positions to each of the plurality of robotic devices is comprised of: identifying a plurality of target points of the body part in the acquired observation data; determining the plurality of sub-goal positions according to the identified plurality of target points; and dynamically allocating each of the determined sub-goal positions to each of the plurality of robotic devices.

3. The control device of claim 2, wherein identifying the plurality of target points comprises identifying the plurality of target points according to a silhouette of the body part.

4. The control device according to claim 2, wherein identifying the plurality of target points comprises identifying the plurality of target points according to characteristics of the body part.

5. The control device according to claim 1, wherein controlling the movement of each of the plurality of robotic devices is comprised of: planning a path for moving to each of the sub-goal positions for each of the robotic devices; and controlling the movement of each of the robotic devices in accordance with the planned path.

6. The control device according to claim 1, wherein the observation data includes image data.

7. The control device of claim 1, wherein the observation data includes motion capture data.

8. The control device of claim 1, wherein each of the robotic devices is an autonomous robot capable of locomotion.

9. The control device according to claim 1, wherein each of the robot devices is a mobile body capable of moving.

10. The control device according to claim 9, wherein the moving object is a drone.

11. The control device of claim 1, wherein the body part includes a hand.

12. The control device of claim 1, wherein the body part includes at least a part of a limb.

13. The control device of claim 1, wherein the body part includes a face.

14. The control device according to claim 1, wherein the body part is constituted by a part or all of the body, an object under the control of at least a part of the body, or a combination thereof.

15. The control device according to claim 1, wherein the control unit is further configured to determine at least one of the number and size of the robot devices depending on at least one of the type of task and the operation target in the task, and dynamically assigning each of the plurality of sub-goal positions to each of the plurality of robot devices is configured by dynamically assigning each of the plurality of sub-goal positions to each of the plurality of robot devices for which at least one of the number and size has been determined.

16. The control device of claim 1, wherein the control unit is further configured to determine a range of the body part depending on at least one of a type of task and an operation target in the task, and the plurality of sub-goal positions corresponding to the body part are composed of the plurality of sub-goal positions corresponding to the body part whose range has been determined.

17. A control method in which a computer executes the following: acquiring observation data of an operator's body parts; dynamically assigning, to each of a plurality of robotic devices, a plurality of subgoal positions corresponding to the body parts appearing in the acquired observation data; and controlling the movement of each of the plurality of robotic devices toward each of the plurality of assigned subgoal positions.

18. A control program for causing a computer to execute the following: acquiring observation data of an operator's body parts; dynamically assigning, to each of a plurality of robotic devices, a plurality of subgoal positions corresponding to the body parts appearing in the acquired observation data; and controlling the movement of each of the plurality of robotic devices toward each of the assigned subgoal positions.

Images