Control device, control method, and control program

Abstract

To provide a control technique for group robots for reducing the possibility of the collision between robotic systems when controlling multiple robotic systems according to a body part.SOLUTION: A control device according to one aspect of the present invention acquires observation data of an operator's body part, dynamically assigns multiple sub-goal positions corresponding to the body parts indicated by the acquired observation data to each of multiple robotic systems, and controls the operation of each of the multiple robotic systems toward the assigned multiple sub-goal positions.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a control device, a control method, and a control program.

Background Art

[0002] In recent years, various methods for controlling robot devices have been developed. For example, Non-Patent Document 1 proposes a method of embodying a group of robots by controlling the operations of a plurality of robot devices (group robots) according to the state of the body (e.g., hand). According to such a method, a plurality of robot devices can be controlled by an intuitive operation.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventors of the present case have found that the above conventional method has the following problems. That is, in the above conventional method, the sub-goal positions of each robot device are statically assigned one-to-one to the feature points of the body. As a result, since the relationship between each robot device and the feature points of the body is fixed, in a scene where the orientation of the body changes, it turns over, etc., collisions between robot devices are likely to occur, and there may arise a problem that the control for avoiding collisions becomes complicated.

[0005] The present invention is made in view of such circumstances, and an object thereof is to provide a control technique for reducing the possibility of collision between robot devices when controlling a plurality of robot devices according to a body part.

Means for Solving the Problems

[0006] In order to solve the above-described problems, the present invention employs the following configuration. Note that the following configurations of the invention can be combined as appropriate.

[0007] A control device according to an aspect of the present invention includes a control unit. The control unit is configured to acquire observation data of a body part of an operator, dynamically assign each of a plurality of sub-goal positions corresponding to the body part represented in the acquired observation data to each of a plurality of robot devices, and control operations of each of the plurality of robot devices toward each of the assigned plurality of sub-goal positions.

[0008] In this configuration, each sub-goal position corresponding to the body part is dynamically assigned to each robot device. As a result, the relationship between each robot device and the body part is not fixed. Therefore, for example, even with a simple method such as assigning the closest sub-goal position, it is possible to expect avoidance of collision between the robot devices. Therefore, according to this configuration, when controlling a plurality of robot devices according to a body part, the possibility of collision between the robot devices can be reduced.

[0009] In the control device according to the above aspect, dynamically assigning each of the plurality of sub-goal positions to each of the plurality of robot devices includes identifying a plurality of target points of the body part in the acquired observation data, and according to the identified plurality of target points, the It may be configured by determining a plurality of sub-goal positions and dynamically assigning each of the determined plurality of sub-goal positions to each of the plurality of robot devices. According to this configuration, it is possible to appropriately determine each sub-goal position and assign it to each robot device.

[0010] In the control device according to the above aspect, specifying the plurality of target points may be configured by specifying the plurality of target points according to the silhouette of the body part. According to this configuration, each sub-goal position can be appropriately set according to the observed body part.

[0011] In the control device according to the above aspect, specifying the plurality of target points may be configured by specifying the plurality of target points according to the characteristics of the body part. According to this configuration, each sub-goal position can be appropriately set according to the observed body part.

[0012] In the control device according to the above aspect, controlling the operations of each of the plurality of robot devices may be configured by planning a path for each robot device to move to each sub-goal position and controlling the operations of each robot device according to the planned path. According to this configuration, each robot device can be appropriately driven toward each sub-goal position.

[0013] In the control device according to the above aspect, the observation data may include image data. According to this configuration, in a scenario where a plurality of robot devices are controlled by a body part via an image, it is possible to reduce the possibility of collision between the robot devices.

[0014] In the control device according to the above aspect, the observation data may include motion capture data. According to this configuration, in a scenario where a plurality of robot devices are controlled by a body part via motion capture, it is possible to reduce the possibility of collision between the robot devices.

[0015] In the control device according to the above aspect, each of the robot devices may be an autonomous robot having the ability to move. According to this configuration, in a scenario of controlling a plurality of autonomous robots, it is possible to reduce the possibility of collision between the robot devices.

[0016] In the control device according to the above aspect, each of the robot devices may be a moving body having the ability to move. According to this configuration, in a scenario of controlling a plurality of moving bodies, it is possible to reduce the possibility of collision between the robot devices.

[0017] In the control device according to the above aspect, the moving body may be a drone. According to this configuration, in a scenario of controlling a plurality of drones, it is possible to reduce the possibility of collision between the robot devices.

[0018] In the control device according to the above aspect, the body part may include a hand. According to this configuration, in a scenario of controlling a plurality of robot devices with a hand, it is possible to reduce the possibility of collision between the robot devices.

[0019] In the control device according to the above aspect, the body part may include at least a part of a limb. According to this configuration, in a scenario of controlling a plurality of robot devices with at least a part of a limb, it is possible to reduce the possibility of collision between the robot devices.

[0020] In the control device according to the above aspect, the body part may include a face. According to this configuration, in a scenario of controlling a plurality of robot devices with a face, it is possible to reduce the possibility of collision between the robot devices.

[0021] In the control device according to the above aspect, the body part may be constituted by a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. According to this configuration, in a scenario of controlling a plurality of robot devices with at least a part of the body and the object, it is possible to reduce the possibility of collision between the robot devices.

[0022] In the control device according to the above aspect, the control unit may be further configured to determine at least one of the number and size of the robot devices according to at least one of the type of task and the operation target in the task. Dynamically allocating each of the plurality of sub-goal positions to each of the plurality of robot devices may be configured by dynamically allocating the plurality of sub-goal positions to each of the plurality of robot devices when at least one of the number and size is determined. According to this configuration, at least one of the number and size of the robot devices can be optimized according to the task.

[0023] In the control device according to the above aspect, the control unit may be further configured to determine the range of the body part according to at least one of the type of task and the operation target in the task. The plurality of sub-goal positions corresponding to the body part may be configured by the plurality of sub-goal positions corresponding to the body part when the range is determined. According to this configuration, the range of the body part used for the operation can be optimized according to the task.

[0024] Note that the form of the present invention may not be limited to the above control device (information processing device). As another aspect of the control device according to each of the above aspects, one aspect of the present invention may be an information processing method for realizing all or a part of each of the above configurations, or a program, or a machine-readable storage medium such as a computer storing such a program. A machine-readable storage medium such as a computer is a medium that stores information such as a program by an electrical, magnetic, optical, mechanical, or chemical action.

[0025] For example, a control method according to an aspect of the present invention may be an information processing method in which a computer executes acquiring observation data of a body part of an operator, dynamically allocating each of a plurality of sub-goal positions corresponding to the body part represented in the acquired observation data to each of a plurality of robot devices, and controlling operations of each of the plurality of robot devices toward each of the allocated plurality of sub-goal positions.

[0026] Further, for example, a control program according to an aspect of the present invention may be a program for causing a computer to execute acquiring observation data of a body part of an operator, dynamically allocating each of a plurality of sub-goal positions corresponding to the body part represented in the acquired observation data to each of a plurality of robot devices, and controlling operations of each of the plurality of robot devices toward each of the allocated plurality of sub-goal positions.

Advantages of the Invention

[0027] According to the present invention, when controlling a plurality of robot devices in accordance with body parts, it is possible to reduce the possibility of collision between the robot devices.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10

Figure 11

Figure 12A

Figure 12B

MODE FOR CARRYING OUT THE INVENTION

[0029] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as "this embodiment") will be described with reference to the drawings. However, the embodiment described below is merely an exemplification of the present invention in every respect. Various improvements or modifications may be made without departing from the scope of the present invention. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that the data appearing in this embodiment is described in natural language, but more specifically, it is specified by a quasi-language, command, parameter, machine language, etc. that can be recognized by a computer.

[0030] §1 Application Example FIG. 1 schematically shows an example of a scene to which the present invention is applied. The control device 1 according to the present embodiment acquires observation data 20 of the body part BP of the operator P. The control device 1 dynamically assigns each of a plurality of sub-goal positions SG corresponding to the body part BP represented in the acquired observation data 20 to each of the plurality of robot devices 30. The control device 1 controls the operations of each of the plurality of robot devices 30 toward each of the assigned plurality of sub-goal positions SG.

[0031] In the present embodiment, each sub-goal position SG corresponding to the body part BP is dynamically assigned to each robot device 30. Thereby, the relationship between each robot device 30 and the body part BP is not fixed (that is, the assignment can be freely changed). Therefore, for example, even by a simple method such as assigning the nearest sub-goal position SG, it is possible to expect avoidance of collision between the robot devices 30. Therefore, according to the present embodiment, when controlling the plurality of robot devices 30 in accordance with the body part BP, it is possible to reduce the possibility of collision between the robot devices 30.

[0032] [Observation Data] The type of the observation data 20 does not particularly have to be limited as long as the body part BP can be captured, and may be appropriately selected according to the embodiment. The observation data 20 may be obtained by any method. In one example, the observation data 20 may be obtained by one or more sensors S. The sensor S may be appropriately selected according to the embodiment. The sensor S may be, for example, an imaging device, a motion capture, etc. The imaging device may include any device that acquires data in the form of an image or an image representation, such as an RGB camera, a depth camera, an infrared camera, a radar, a LiDAR (Light Detection And Ranging), etc. The type of the motion capture does not particularly have to be limited and may be appropriately selected according to the embodiment.

[0033] In one example, the sensor S may include an imaging device, and the observation data 20 may include image data obtained by the imaging device. In an example of the present embodiment, in a scenario where a plurality of robot devices 30 are controlled by a body part BP via an image, it is possible to reduce the possibility of collision between the robot devices 30.

[0034] Also, in another example, the sensor S may include motion capture, and the observation data 20 may include motion capture data obtained by the motion capture. In an example of the present embodiment, in a scenario where a plurality of robot devices 30 are controlled by a body part BP via motion capture, it is possible to reduce the possibility of collision between the robot devices 30. Note that, as a typical example, an optical motion capture that tracks the positions of markers may be used for the motion capture. In this case, the obtained motion capture data may be constituted by the trajectory data of the markers.

[0035] [Body part] The body part BP may not be particularly limited and may be the whole body or a part of the body. In one example, the body part BP may be appropriately selected from a hand, an arm, a limb, a face, a torso, the whole body, etc. The body part BP may refer to a divided range (for example, a hand, an arm, a face, etc.) or a non-divided range (for example, a part of a hand and a part of an arm, etc.).

[0036] For example, the body part BP may include a hand. Thereby, it becomes possible to control the operations of the plurality of robot devices 30 in accordance with the movement of the hand, and in such a scenario, it is possible to reduce the possibility of collision between the robot devices 30. In one example, the hand may be adopted as the body part BP in a scenario where a task performed by the hand such as carrying an object is executed.

[0037] In the example of FIG. 1, a hand is adopted as the body part BP, and a scenario where the hand is changed from a par to a choki is assumed. In this scenario, when making the hand into a choki, the thumb and the ring finger may cross each other. Therefore, if each robot device is statically assigned to each part of the hand on a one-to-one basis, there is a possibility that the robot devices assigned to the thumb and the ring finger may collide with each other. In contrast, in the present embodiment, for example, by changing these assignments with any index such as assigning the closest sub-goal position SG, the collision of the robot devices can be easily avoided.

[0038] Also, for example, the body part BP may include at least a part of a limb. The limb includes a hand and a foot. Thereby, it becomes possible to control the operations of the plurality of robot devices 30 in accordance with the movement of at least a part of the limb, and in such a scenario, it is possible to reduce the possibility of collision between the robot devices 30. In one example, in a scenario where remote communication is performed with a gesture of a limb, at least a part of the limb may be adopted as the body part BP.

[0039] Also, for example, the body part BP may include a face. Thereby, it becomes possible to control the operations of the plurality of robot devices 30 in accordance with the movement of the face (for example, expression, movement of the head), and in such a scenario, it is possible to reduce the possibility of collision between the robot devices 30. In one example, in a scenario where remote communication is performed, the face may be adopted as the body part BP in order to convey the expression of the operator P by controlling the operations of the plurality of robot devices 30 in accordance with the expression of the face.

[0040] Also, in one example, when an object such as a tool is under the control of a body part BP, the body part BP may include the range of this object. That is, the object under control may be regarded as the body part BP. Being under control may refer to a state in which there is some relationship between the body part BP and the object, such as the object being attached to the body part BP or the behavior of the object being controlled by the body part BP. The body part BP may be composed of a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. Thereby, it becomes possible to control the operations of the plurality of robot devices 30 in accordance with at least a part of the movements of the body and the object, and in such a scenario, it is possible to reduce the possibility of collision between the robot devices 30.

[0041] Wearing may refer to any state in which a tool such as holding with the hand, wearing, winding, attaching, etc. is linked to the body part. The object may be worn in a state of being in close contact with the body part BP, or may be worn in a state of being separated from the body part BP. Also, controlling may refer to any state in which the object is not worn on the body part but is affected by the body part. The state of being affected by the body part may include, for example, acts such as dribbling a ball, throwing a ball or a baton into the air, manipulating a ribbon in rhythmic gymnastics, remotely operating a moving body (for example, an autonomous robot configured to be movable, a drone, etc.). Controlling may include, for example, directly controlling the behavior of the object by the body part as in dribbling the above-mentioned ball. In addition, controlling may include indirectly controlling the behavior of the object, such as autonomously executing at least a part of the operations (for example, automatic driving). That is, controlling may include a state in which there is a time affected by the body part even for an instant, such as turning on the power, pressing the start button, instructing the operation by voice, etc., and not being under the control of the body part at other times.

[0042] The type of the object may not be particularly limited and may be appropriately selected according to the embodiment. The tool is an object in phase with a one-dimensional line segment, a two-dimensional spherical surface (S 2)An object that is homeomorphic to, a torus (T 2 ) may include objects of any shape such as an object that is homeomorphic to, and objects of other shapes. Tools having a shape homeomorphic to a one-dimensional line segment may include, for example, a rod, a flag, a writing instrument, a baton, a ribbon for rhythmic gymnastics, etc. Tools having a shape homeomorphic to a two-dimensional sphere may include, for example, a ball, etc. Tools having a shape homeomorphic to a torus may include, for example, a Möbius strip, a hula hoop, etc. Tools of other shapes may include, for example, a stuffed toy, a cup, a moving body (e.g., an autonomous robot, a drone, etc.), etc. When using a tool, the body part BP may be constituted by a part or all of the above body, the above tool, or a combination thereof.

[0043] FIG. 2 schematically shows an example of the body part BP according to the present embodiment. In an example of FIG. 2, the body part BP is a hand, the object T is a rod-shaped tool, and a scene where the rod-shaped tool is grasped by the hand is assumed. In such a scene, at least some of the sub-goal positions SG may be determined according to the object T.

[0044] Note that, in the example of FIG. 2, a scene is assumed in which the sub-goal position SG is determined according to the ranges of both the hand (body part BP) and the tool (object T). In one example, in this way, the sub-goal position SG may be determined according to both the body part BP and the object T. However, the range in which the sub-goal position SG is arranged is not limited to such an example. In another example, even in a scene where the object T is worn, the sub-goal position SG may be determined according to only the body part BP (that is, the object T may be ignored). In yet another example, in a scene where the object T is worn, the sub-goal position SG may be determined according to only the object T.

[0045] (Number of operators) In the example of FIG. 1, the operator P is one person. However, the number of operators P is not limited to one person and may be a plurality of people. In one example, the control device 1 is an operator for a plurality of people It may be used in a scenario where the operation of a plurality of robot devices 30 is controlled by the operator P. In this case, the body part BP may be determined by each of a plurality of operators P.

[0046] [Dynamic Assignment] In the present embodiment, each sub-goal position SG is dynamically assigned to each robot device 30. That is, the correspondence between the body part BP and each robot device 30 is not fixed one-to-one, and can vary at any timing, for example, the assignment destination changes from the thumb to another location. If such variations are allowed, the processing content of dynamic assignment may not be particularly limited and may be appropriately determined according to the embodiment. In one example, as dynamic assignment, re-assignment of the sub-goal position SG may be executed every one or more control cycles.

[0047] Also, the method of dynamic assignment may not be particularly limited and may be appropriately selected according to the embodiment. In one example, dynamic assignment may include assigning the sub-goal position SG that is closest in distance to each robot device 30. Thereby, the overall movement amount of each robot device 30 can be minimized, and the possibility of collision can be reduced.

[0048] In another example, dynamically allocating may include allocating a sub-goal position SG that minimizes the cost to each robot device 30. The cost may be configured to indicate, by any metric, the degree to which the allocation is recommended (the higher the cost, the less recommended). The metrics for evaluating the degree of recommendation may include, for example, the presence or absence of collisions, the moving distance, the amount of rotation, the power consumption, the constraints, or a combination thereof. The constraints may be given, for example, according to the human perceptual goodness, the high level of behavioral agency, the high level of body ownership, etc. Behavioral agency refers to the operator P feeling the operation of each robot device 30 as his own operation. Body ownership refers to the operator P feeling each robot device 30 as a part of his own body. The human perceptual goodness, the high level of behavioral agency, or the high level of body ownership may be evaluated by metrics such as feeling a sense of unity, feeling smooth, and it is better for a device farther away to move faster. In one example, the cost may be calculated such that the higher the possibility of a collision occurring, the higher the cost, and the lower the possibility, the lower the cost. In a scenario where the moving distance is suppressed, the cost may be calculated such that the longer the moving distance, the higher the cost, and the shorter the moving distance, the lower the cost. In a scenario where the rotation is suppressed, the cost may be calculated such that the larger the amount of rotation, the higher the cost, and the smaller the amount of rotation, the lower the cost. In a scenario where the power consumption of each robot device 30 is suppressed, the cost may be calculated such that the higher the power consumption, the higher the cost, and the lower the power consumption, the lower the cost. In a scenario where an allocation that satisfies the given constraints is obtained, the cost may be calculated such that the less the constraints are satisfied, the higher the cost, and the more the constraints are satisfied, the lower the cost. The cost may be replaced by a reward (benefit), and the allocation that minimizes the cost may be replaced by the allocation that maximizes the reward. The numerical representation of the cost or reward may be set as appropriate. The cost may be expressed as being proportional to the numerical value (i.e., the larger the numerical value, the higher the cost), or as being inversely proportional to the numerical value (i.e., the smaller the numerical value, the higher the cost). The same applies to the reward.Other methods of dynamic allocation may use known methods such as Reference 1 (Saurav Agarwal et al., “Simultaneous Optimization of Assignments and Goal Formations for Multiple Robots”, [online], [searched on November 29, Reiwa 5], Internet <URL:https: / / par.nsf.gov / servlets / purl / 10179280>), etc.

[0049] (Method for determining sub-goal positions) Each sub-goal position SG is a provisional goal position of each robot device 30. Each sub-goal position SG is determined according to the observed body part BP. The method for determining each sub-goal position SG may not be particularly limited and may be appropriately selected according to the embodiment.

[0050] In one example, dynamically allocating each of the plurality of sub-goal positions SG to each of the plurality of robot devices 30 may be configured by identifying a plurality of target points of the body part BP in the acquired observation data 20, determining a plurality of sub-goal positions SG according to the identified plurality of target points, and dynamically allocating each of the determined plurality of sub-goal positions SG to each of the plurality of robot devices 30. Determining the sub-goal position SG according to the target point may include adopting the target point as the sub-goal position SG as it is, and determining the sub-goal position SG from the target point according to a predetermined rule. The predetermined rule may be appropriately determined according to the embodiment. The predetermined rule may include, for example, giving an offset (determining a position separated by the offset distance from the target point as the sub-goal position SG), etc. According to an example of this embodiment, the determination of each sub-goal position SG and the allocation to each robot device 30 can be appropriately performed.

[0051] Note that the method for specifying a plurality of target points in the body part BP is not particularly limited and may be appropriately selected according to the embodiment. In one example, at least one of the following two methods may be adopted.

[0052] (1) Silhouette-based In one example, specifying a plurality of target points may be constituted by specifying a plurality of target points according to the silhouette of the body part BP. Specifying target points according to the silhouette may include dividing the silhouette (range) of the body part BP into regions and specifying an arbitrary point (for example, the central point, the center of gravity, etc.) within the divided region as the target point. The method of dividing the region is not particularly limited and may be appropriately determined according to the embodiment.

[0053] The ratio of dividing the silhouette of the body part BP is not particularly limited and may be appropriately determined according to the embodiment. In one example, the silhouette of the body part BP may be evenly divided. In another example, the silhouette of the body part BP may be divided at least partially at different ratios. The ratio may be predefined or may be dynamically determined according to, for example, the shape of the body part BP, the sign of the body part BP (for example, gesture), the size of the robot device 30 to be assigned, the shape of the robot device 30, etc.

[0054] FIG. 3 schematically shows an example of a method for determining the sub-goal position SG based on the silhouette. In the example of FIG. 3, a scene where the body part BP is a hand is assumed. First, the control device 1 extracts the body part BP from the observation data 20. The observation data 20 is, for example, image data. The control device 1 meshes the silhouette (range) of the extracted body part BP. The meshing may be performed by any method. Examples of the method of meshing include Reference 2 (Xingyu Chen et al., "Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration", [online], retrieved on November 29, Reiwa 5 years], in The Internet <URL:https: / / arxiv.org / abs / 2103.02845>, Reference 3 (Xingyu Chen et al., "MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image", [online], retrieved on November 29, Reiwa 5 years, Internet <URL:https: / / arxiv.org / abs / 2112.02753>), Reference 4 ("Skinned Mesh Renderer component", [online], retrieved on November 29, Reiwa 5 years, Internet <URL:https: / / docs.unity3d.com / ja / 2021.3 / Manual / class-SkinnedMeshRenderer.html>), etc. may be used. The control device 1 clusters the vertices of the obtained mesh by a method such as the k-means algorithm, for example. As a result, the silhouette of the body part BP can be regionally segmented. The control device 1 may appropriately specify target points in each of the obtained regions. Then, the control device 1 may determine the sub-goal position SG from the specified target points.

[0055] Note that this method may also be applied to body parts other than the hand. Also, the method of determining target points based on the silhouette is not limited to such an example and may be appropriately changed according to the embodiment. According to an example of this embodiment, each sub-goal position SG can be appropriately set according to the observed body part BP.

[0056] (2) Feature-based In one example, specifying a plurality of target points may be configured by specifying a plurality of target points according to the features of the body part BP. The features of the body part BP may include any extractable elements.

[0057] The control device 1 may extract the features of the body part BP and adopt the obtained feature points (feature points) as the target points. The feature points of the body part BP may be, for example, the feature points of the skeleton, muscles, face, object, etc. The feature points of the skeleton may be, for example, joints, tips, etc. The feature points of the muscles may be, for example, connection parts, centers, etc. The feature points of the face may be, for example, the endpoints of parts, centers, etc. The parts may be, for example, organs (mouth, eyes, ears, nose), other parts (eyebrows, forehead, etc.), etc. The feature points of the object may be, for example, endpoints, midpoints, centers of gravity, etc.

[0058] The features may be directly extracted from the observation data 20 or may be indirectly extracted, for example, through information processing such as skeleton estimation. When the observation data 20 is image data, the control device 1 may extract the features of the body part BP by image processing. When using a motion capture of the type that tracks markers as the sensor S, the control device 1 may directly extract the markers as feature points. The feature points to be extracted may be determined in advance or may be determined dynamically. The method of extracting the feature points is not particularly limited and may be appropriately selected according to the embodiment.

[0059] FIG. 4 schematically shows an example of a method for determining the sub-goal position SG based on features. In the example of FIG. 4, the body part BP is the hand, the observation data 20 is image data, and a scene where the skeleton is extracted as a feature is assumed. First, the control device 1 extracts the body part BP from the observation data 20. The control device 1 aligns the skeleton with the extracted body part BP (skeleton estimation). The position of the skeleton may be estimated by any method such as a machine learning-based method. Examples of the method for estimating the position of the skeleton include Reference 5 (Tomas Simon et al., "Hand Keypoint Detection in Single Images using Multiview Bootstrapping", [online], retrieved on November 29, Reiwa 5], Internet <URL:https: / / arxiv.org / abs / 1704.07809> ) may be used. The control device 1 may identify, as target points, feature points specified in advance on the obtained skeleton (rule-based). Then, the control device 1 may determine the sub-goal positions SG from the identified target points.

[0060] Note that this method may also be applied to other body parts other than the hand. Instead of the skeleton, with other features, this method may also be applied to other features other than the skeleton. Also, the method of determining the target points based on features is not limited to such examples and may be appropriately changed according to the embodiment. According to an example of this embodiment, each sub-goal position SG can be appropriately set according to the observed body part BP.

[0061] [Operation Control] The method of controlling the operation of each robot device 30 toward each sub-goal position SG is not particularly limited and may be appropriately selected according to the embodiment. In one example, the control device 1 may determine a path to each sub-goal position SG by path planning. That is, controlling the operations of each of the plurality of robot devices 30 may be configured by planning a path to move to each sub-goal position SG for each robot device 30 and controlling the operations of each robot device 30 according to the planned path. Path planning may be performed by any method. Also, for the algorithm for controlling the movement, for example, Reference 6 (Jur van den Berg et al., “Reciprocal Velocity Obstacles for Real-Time Multi-Agent Navigation”, [online], [searched on November 29, Reiwa 5], Internet <URL:https: / / gamma Well-known methods such as those in [“Reciprocal Velocity Obstacles: Theory and Practice”, [online], [searched on November 29, Reiwa 5], Internet <URL: http: / / gamma-web.iacs.umd.edu / RVO / icra2008.pdf>] and reference 7 (Jamie Snape et al., “Smooth and Collision-Free Navigation for Multiple Robots Under Differential-Drive Constraints”, [online], [searched on November 29, Reiwa 5], Internet <URL: http: / / gamma-web.iacs.umd.edu / ORCA-DD / ORCA-DD.pdf>) may be adopted. According to an example of this embodiment, each robot device 30 can be appropriately driven toward each sub-goal position SG. Note that the control device 1 may directly control each robot device 30 or may indirectly control each robot device 30 via an external computer such as a controller.

[0062] [Robot Device] The type of the robot device 30 is not particularly limited and may be appropriately selected according to the embodiment. The robot device 30 may be, for example, an industrial robot in a production line, an autonomous robot configured to be operable autonomously, a moving body configured to be movable, or the like. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (scalar robot), a parallel link robot, an orthogonal robot, or the like. The autonomous robot may be, for example, a humanoid robot, a guiding robot, an agricultural robot, a caregiving robot, a security robot, a transportation robot (including a disaster rescue robot), or the like. The content of the autonomous processing may be appropriately selected according to the embodiment. The moving body may include, for example, a cleaning robot, the above-mentioned autonomous robot configured to be movable (including a mobile robot), a vehicle configured to be capable of autonomous driving (including dedicated vehicles such as an ambulance, a fire truck, and a construction vehicle), an airframe capable of autonomous flight (such as a drone), or the like. The fire truck may include, for example, a rescue work vehicle, a vehicle loaded with a portable fire pump, or the like.

[0063] For example, each robot device 30 may be an autonomous robot having the ability to move. In this case, in a scenario where a plurality of autonomous robots are controlled by physical operations, it is possible to reduce the possibility of collision. Also, for example, each robot device 30 may be a moving body having the ability to move. In this case, in a scenario where a plurality of moving bodies are controlled by physical operations, it is possible to reduce the possibility of collision. Further, the moving body may be a drone. In this case, in a scenario where a plurality of drones are controlled by physical operations, it is possible to reduce the possibility of collision.

[0064] Each robot device 30 may move in at least any one of land, water, and air. The space to move may be a real space or a virtual space. That is, each robot device 30 may be an existence in a real space or a virtual space. Note that the plurality of robot devices 30 controlled by physical operations may also be referred to as "group robots".

[0065] [Usage Scenario] The control device 1 according to the present embodiment may be used in any scenario for controlling the robot device 30 in a real space or a virtual space. In one example, the control device 1 according to the present embodiment may be used for real-time physical telepresence or operation.

[0066] For example, the control device 1 according to the present embodiment may be used for communication with others remotely separated from the operator P. The control device 1 may realize remote communication between the operator P and others by causing a gesture or expression appearing on the body part BP to be demonstrated to a plurality of robot devices 30 existing in front of others. That is, the control device 1 may cause each robot device 30 to execute a communication task.

[0067] Also, for example, the control device 1 may realize the interaction between the operator P and others by causing interactions such as pointing, touching, carrying an object, grasping an object, and writing with a writing instrument held to be demonstrated to a plurality of robot devices 30 existing in front of others. That is, the control device 1 may cause each robot device 30 to execute an interaction task. For an object Interactions such as carrying or grasping an object may be performed for cooperation with others existing remotely. Further, the cooperation may include cooperative work at a disaster site (for example, carrying rubble, etc.). At this time, since each robot device 30 is configured to be smaller than a human hand, each robot device 30 can be physically operated and entered into a gap where a human hand cannot enter.

[0068] In each of the above cases, when controlling the robot device 30 in the real space, the control device 1 may be arranged on either the operator P or the other person. In one example, the control device 1 may exist on the operator P side and transmit an instruction to each robot device 30 to a computer existing on the other person side via a network. In another example, the control device 1 may exist on the other person side, receive the observation data 20 from the computer on the operator P side via a network, and use the received observation data 20 to execute the above arithmetic processing to control the operation of each robot device 30. In addition, the control device 1 according to the present embodiment may be used for controlling the robot device 30 in the virtual space.

[0069] (Feedback method) The method of feeding back the operation of the robot device 30 to the operator P may not be particularly limited and may be appropriately selected according to the embodiment. In one example, by driving the robot device 30 in the vicinity of the operator P, the operator P may directly visually recognize the operation of the robot device 30. In another example, the operator P may display the operation of the robot device 30 in the real space or the virtual space on at least one of a display, a projector display, and a VR (Virtual Reality) display. It may also be visually recognized through any of them. This feedback may be performed by the control device 1 or by an external computer other than the control device 1. For example, the control device 1 or the external computer may project the status of each robot device 30 near the body part BP of the operator P (for example, at the operator's hand when the hand is adopted as the body part BP) through a projector. When each robot device 30 exists in the real space, the status of each robot device 30 may be appropriately observed by a sensor such as an imaging device. The control device 1 or the external computer may feedback the obtained sensing data to the operator P. As a feedback method, by using projection near the body part BP by the projector or VR display, improvement of the sense of agency and the sense of body ownership can be expected. Further, for example, each robot device 30 may exist in the VR space, and the operator P may control each robot device 30 by body operation in the VR space. Thereby, the control device 1 may operate as a user interface in the VR space.

[0070] (Number and size of robot devices) The number and size of the robot devices 30 may not be particularly limited and may be appropriately determined according to the embodiment. At least either the number or the size of the robot devices 30 may be given in advance, may be determined by preprocessing, or may be dynamically determined during the execution of the control process of the robot devices 30.

[0071] In one example, the control device 1 may determine at least one of the number and size of the robot devices 30 to be used based on the type of task (including gestures) to be performed by the robot device 30, the object to be manipulated in the task, the size of the body part BP, or a combination thereof. For example, in the case of a task that requires force, such as carrying an object or pressing a button, the control device 1 may determine to increase the number of robot devices 30 to be used and / or use a large robot device 30. On the other hand, in the case of a task that does not require force, such as only moving for communication, the control device 1 may determine to reduce the number of robot devices 30 to be used and / or use a small robot device 30. Similarly, for example, when the object to be manipulated is heavy, the control device 1 may determine to increase the number of robot devices 30 to be used and / or use a large robot device 30. On the other hand, when the object to be manipulated is light, the control device 1 may determine to reduce the number of robot devices 30 to be used and / or use a small robot device 30. Further, for example, when the target range of the body part BP is wide (such as spreading the hand), the control device 1 may determine to increase the number of robot devices 30 to be used and / or use a large robot device 30. On the other hand, when the target range of the body part BP is narrow (such as making a fist), the control device 1 may determine to reduce the number of robot devices 30 to be used and / or use a small robot device 30. The relationship between the type of task and the like and the number of robot devices 30 and the like may be appropriately determined according to the embodiment by any method such as a rule-based method.

[0072] When each robot device 30 is used to perform a task in the real space, events (including objects) related to the task appear in the environment where each robot device 30 exists. Therefore, before executing the dynamic allocation, the control device 1 may acquire the sensing data of the sensors for feedback on the state of each robot device 30, and estimate at least one of the task type and the operation target from the acquired sensing data by an arbitrary method. In one example, the control device 1 may estimate at least one of the task type and the operation target according to the object detected from the sensing data. For example, the control device 1 may estimate that the task to be performed is to carry the object according to the detection of the object to be carried. When the operator P and each robot device 30 are located in the vicinity, the sensor S for observing the operator P may include the sensors for observing each robot device 30, and the observation data 20 may include this sensing data.

[0073] Also, the range of the body part BP may be determined according to at least one of the task type and the operation target in the task. Accordingly, a plurality of sub-goal positions SG corresponding to the body part BP may be constituted by a plurality of sub-goal positions SG corresponding to the body part BP whose range is determined. For example, according to the fact that the task is to carry a heavy load, the control device 1 changes the range of the body part BP from only the hand to the hand and the arm, and according to the fact that the task is to carry a plurality of loads, the control device 1 changes the range of the body part BP from the hands of one person to the hands of two people, and according to the fact that the task is to carry a light load, the control device 1 changes the range of the body part BP from both hands to one hand. The control device 1 may increase or decrease the range of the body part BP according to at least one of the task type and the operation target in the task. According to the fact that the control device 1 expands the range of the body part BP, the control device 1 may determine at least one of increasing the number of robot devices 30 to be used and using a large robot device 30. Also, according to the fact that the control device 1 narrows the range of the body part BP, the control device 1 may determine at least one of reducing the number of robot devices 30 to be used and using a small robot device 30. According to an example of the present embodiment, the range of the body part BP used for the operation can be optimized according to the task.

[0074] Note that the method for determining the type of task, the operation target in the task, and the range of the body part BP is not necessarily limited to the above method. In a simple example, at least any one of the type of task, the operation target in the task, and the range of the body part BP may be determined according to the selection of the user (including the operator P). The control device 1 may appropriately receive the input of the selection via the input device before executing the dynamic allocation.

[0075] When continuously executing a task, the control device 1 may repeatedly execute determining at least one of the number and size of the robot devices 30 to be used by the above method periodically or irregularly. Thereby, at least one of the number and size of the robot devices 30 to be used may be changed while the task is being continuously performed. Also, when continuously executing a plurality of tasks, when the control device 1 switches the task to be performed (that is, before the execution of the current task is completed and before entering the execution of the next task), at least one of the number and size of the robot devices 30 to be used for the next task may be determined by the above method. Thereby, when switching the task to be performed, at least one of the number and size of the robot devices 30 to be used may be changed. The task is work to be performed by at least any one of the plurality of robot devices 30. The type of task is not particularly limited and may be appropriately selected according to the embodiment. The task may include, for example, the above communication, interaction, etc.

[0076] When the control device 1 determines at least one of the number and size of the robot devices 30 before executing dynamic allocation, dynamically allocating each sub-goal position SG to each robot device 30 is constituted by allocating each sub-goal position SG to each robot device 30 for which at least one of the number and size has been determined. Before allocating each sub-goal position SG to each robot device 30, when the number of robot devices 30 has been determined, in the step of determining the sub-goal position SG, the control device 1 determines the same number of sub-goal positions SG as the determined number of robot devices 30. For example, when adopting the silhouette-based method, the control device 1 executes the same number of region divisions as the determined number of robot devices 30. Also, for example, when adopting the feature-based method, the control device 1 extracts the same number of feature points as the determined number of robot devices 30. The feature points to be used may be determined in advance according to the number of robot devices 30. According to an example of the present embodiment, at least one of the number and size of the robot devices 30 can be optimized according to the task.

[0077] Also, in another example, the control device 1 may dynamically change the number of sub-goal positions SG in the specific process of the sub-goal position SG. That is, the control device 1 may specify a variable number of sub-goal positions SG in the above process. In this case, the control device 1 may determine the number of robot devices 30 to be used according to the number of specified sub-goal positions SG. Thereby, the control device 1 may determine the number of robot devices 30 to be used after determining the number of sub-goal positions SG (that is, at the time of dynamic allocation).

[0078] Also, accordingly, the control device 1 may also determine the size of the robot devices 30 to be used. For example, the control device 1 may select to use small robot devices 30 in response to an increase in the number of robot devices 30. Also, for example, the control device 1 may select to use small robot devices 30 when the interval between the specified sub-goal positions SG is narrow, and may select to use large robot devices 30 when the interval between the sub-goal positions SG is wide.

[0079] At least either the size or the type of the robot device 30 to be used may be unified (i.e., the same), or may be at least partially different. The control device 1 may determine at least one of the size and the type of the robot device 30 to be assigned according to the location of the body part BP, for example, by assigning a small robot device 30 to the fingertips and a large robot device 30 to other than the fingertips.

[0080] In the real space, when allowing at least one of the number and the size of the robot device 30 to be changed, the candidate robot devices to be used may be pooled at an arbitrary location. In the pool, for example, a plurality of types of robot devices having different attributes such as size and type may be arranged in plural numbers respectively. The pool may be a fixed range, or may be an unspecified and arbitrary location away from the operating robot device 30. The control device 1 may select the robot device 30 to be used from among the pooled robot devices according to the determination result of at least one of the number and the size of the robot device 30 to be used by the above processing. The control device 1 may move each robot device 30 to the task execution location by controlling the operations of the selected plurality of robot devices 30.

[0081] Also, during the continuation of the task or at the time of task switching, when the process of determining at least one of the number and the size of the robot device 30 to be used is executed, the robot device 30 to be used may be changed. In this case, the control device 1 selects, from among the pooled robot devices, a robot device that is compatible with the robot device newly added as the robot device 30 to be used due to the change of the robot device 30 to be used, and may move the robot device 30 to the task execution location by controlling the operation of the selected robot device 30. Thereby, a new robot device 30 may be added to the execution of the task. Further, the control device 1 may evacuate the unused robot device 30 to the pool due to the change of the robot device 30 to be used, and may exclude it from the control target after the evacuation is completed.

[0082] When allowing at least one of the number and size of the robot device 30 to be changed in the virtual space, the control device 1 may represent the change of the robot device 30 in any method. In one example, the control device 1 may also represent the change of the robot device 30 used in the virtual space in the same method as in the real space. In another example, the control device 1 may cause a newly added robot device as the robot device 30 to be used to appear at any timing and add it to the execution of the task. Further, the control device 1 may erase the robot device 30 that is no longer used from the display at any timing due to the change of the robot device 30 used.

[0083] (Range of motion of the robot device) The relationship between the range of motion of the body part BP and the range of motion of each robot device 30 is not particularly limited and may be appropriately determined according to the embodiment. Also, the scale between the body part BP and the range of motion of each robot device 30 may be appropriately determined according to the embodiment. The scale between the body part BP and the range of motion of each robot device 30 may or may not match. The control device 1 may determine the scale of the range of motion of each robot device 30 according to the width of the observed range of the body part BP, and control the operation of each robot device 30 within the determined range of motion. For example, the control device 1 may set the range of motion of each robot device 30 wider according to the wide range of the body part BP, or may set the range of motion of each robot device 30 narrower according to the narrow range of the body part BP. Conversely, for example, when making a fist or the like, the range of the body part BP becomes narrower, which may make it difficult to perform path planning that can be avoided by narrowing the range of motion. Therefore, the control device 1 may set the range of motion of each robot device 30 wider according to the narrowing of the range of the body part BP. Also, the control device 1 may determine the scale of the range of motion according to at least one of the type of task and the operation target in the task. Furthermore, the control device 1 may determine at least one of the number and size of the robot devices 30 according to the scale of the range of motion. According to an example of this embodiment, the body expression constituted by the robot device 30 can be freely changed. Thereby, the body expression can be optimized according to the environment, and as a result, it is possible to expect an improvement in at least one of body ownership and agency.

[0084] §2 Configuration Example [Hardware Configuration] FIG. 5 schematically illustrates an example of the hardware configuration of the control device 1 according to this embodiment. In an example of FIG. 5, the control device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0085] The control unit 11 is a CPU (Central Processing Unit), which is a hardware processor, and includes a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0086] The storage unit 12 may be composed of, for example, a hard disk drive, a solid state drive, etc. The storage unit 12, the RAM, and the ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the control program 81. The control program 81 is a program for causing the control device 1 to execute information processing (FIG. 7 described later) related to the operation control of each robot device 30 by body control. The control program 81 includes a series of instructions for the information processing.

[0087] The external interface 13 is configured to be connected to an external device by wire or wirelessly. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, etc. The type and number of the external interface 13 may be appropriately determined according to the embodiment. When the external interface 13 includes a communication port, the communication standard of the communication port may be arbitrarily selected. In this embodiment, the control device 1 may be connected to at least one of the sensor S for observing the operator P and the sensors for observing each robot device 30 via the external interface 13.

[0088] The input device 14 is a device for performing inputs such as a mouse, a keyboard, etc. The output device 15 is a device for performing outputs such as a display, a projector, a VR device, a speaker, etc. A user including the operator P can operate the control device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via the external interface 13. The input device 14 and the output device 15 may be integrally configured by, for example, a touch panel display or the like. Note that the operator P who controls each robot device 30 and the user who operates the control device 1 may be the same or different.

[0089] The drive 16 is a device for reading various information such as programs stored in the storage medium 91. The control program 81 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store the various information (stored programs, etc.) by an electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The control device 1 may acquire the control program 81 from the storage medium 91. Note that the storage medium 91 may be a disk-type storage medium such as a CD, a DVD, etc., or a storage medium other than a disk type such as a semiconductor memory (for example, a flash memory). The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. The drive 16 may be connected via the external interface 13.

[0090] Regarding the specific hardware configuration of the control device 1, depending on the embodiment, components can be appropriately omitted, replaced, or added. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), GPU (Graphics Processing Unit), ASIC (application specific integrated circuit), etc. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. At least any one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The control device 1 may be composed of a plurality of computers. In this case, the hardware configurations of each computer may be the same or at least partially different. Also, the control device 1 may be a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a mobile device, a terminal device, etc., in addition to an information processing device designed specifically for the provided service.

[0091] [Software Configuration] FIG. 6 schematically illustrates an example of the software configuration of the control device 1 according to this embodiment. The control unit 11 of the control device 1 expands the control program 81 stored in the storage unit 12 into the RAM and executes the instructions included in the control program 81 by the CPU. As a result, the control device 1 operates as a computer including software modules such as a data acquisition unit 111, a dynamic allocation unit 112, and an operation control unit 113. That is, in this embodiment, each software module of the control device 1 is realized by the control unit 11 (CPU).

[0092] The data acquisition unit 111 is configured to acquire the observation data 20 of the body part BP of the operator P. The dynamic allocation unit 112 is configured to dynamically allocate each of a plurality of sub-goal positions SG corresponding to the body part BP appearing in the acquired observation data 20 to each of the plurality of robot devices 30. The motion control unit 113 is configured to control the motion of each of the plurality of robot devices 30 toward each of the allocated sub-goal positions SG.

[0093] In addition, in the present embodiment, an example in which each software module of the control device 1 is realized by a general-purpose CPU is described. However, part or all of the above software modules may be realized by one or more dedicated processors or chip sets. Each of the above modules may be realized as a hardware module. Regarding the software configuration of the control device 1, omission, replacement, and addition of modules may be appropriately performed according to the embodiment.

[0094] §3 Operation Example FIG. 7 is a flowchart showing an example of the processing procedure of the control device 1 according to the present embodiment. The following processing procedure is an example of a control method executed by a computer. However, the following processing procedure of the control device 1 is only an example, and each step may be changed as much as possible. Also, regarding the following processing procedure, omission, replacement, and addition of steps are possible as appropriate according to the embodiment.

[0095] (Step S101) In step S101, the control unit 11 operates as the data acquisition unit 111 and acquires the observation data 20 of the body part BP of the operator P.

[0096] In one example, the body part BP may include a hand, an arm, a limb, a face, a torso, or the entire body. The body part BP may also include an object attached to the body part BP. The observation data 20 may include at least one of image data and motion capture data. The body part BP may be observed by the sensor S, and the control unit 11 may acquire the observation data 20 directly or indirectly from the sensor S. When the observation data 20 is acquired, the control unit 11 proceeds to the next step S102 in the process.

[0097] (Step S102) In step S102, the control unit 11 operates as a dynamic allocation unit 112 and dynamically allocates each of a plurality of sub-goal positions SG corresponding to the body part BP represented in the acquired observation data 20 to each of a plurality of robot devices 30.

[0098] In one example, in the acquired observation data 20, the control unit 11 identifies a plurality of target points of the body part BP, determines a plurality of sub-goal positions SG according to the identified plurality of target points, and may dynamically allocate each of the determined plurality of sub-goal positions SG to each of a plurality of robot devices 30. As an example of the process of identifying target points, the control unit 11 may identify a plurality of target points according to the silhouette of the body part BP. As another example, the control unit 11 may identify a plurality of target points according to the characteristics of the body part BP. When each sub-goal position SG is dynamically allocated to each robot device 30, the control unit 11 proceeds to the next step S103 in the process.

[0099] (Step S103) In step S103, the control unit 11 operates as an operation control unit 113 and controls the operations of each of the plurality of robot devices 30 toward each of the allocated plurality of sub-goal positions SG.

[0100] In one example, as the process of step S103, the control unit 11 may plan a path for each robot device 30 to move to each sub-goal position SG, and control the operations of the respective robot devices 30 according to the planned path. In one example, each robot device 30 may be at least either an autonomous robot or a mobile body. When adopting a mobile body, each robot device 30 may be a drone. When controlling the operations of each robot device 30, the control unit 11 proceeds to the next step S104 for processing.

[0101] (Step S104) In step S104, the control unit 11 determines whether to end the control of each robot device 30. The criterion for determination may be set arbitrarily. In one example, until an arbitrary end instruction (for example, an end operation by the user via the input device 14) is given, the control unit 11 may determine not to end the control of each robot device 30. When it is determined not to end the control of each robot device 30, the control unit 11 returns the process to step S101 and executes the process again from step S101. On the other hand, when an arbitrary end instruction is given, the control unit 11 may determine to end the control of each robot device 30. When it is determined to end the control of each robot device 30, the control unit 11 ends the processing procedure of the control device 1 according to this operation example. The processes of steps S101 to S104 may be executed in real time. Also, the timing for executing the process of step S104 is not limited to this example. The control device 1 may end the operation at an arbitrary timing.

[0102] Note that, in one example, the control unit 11 may feedback the status of each robot device 30 to the operator P at an arbitrary timing. The feedback may be constituted by at least any one of a display on a display, a display on a projector, and a VR display.

[0103] Also, in one example, before executing the process of step S102, the control unit 11 may determine at least one of the number and size of the robot devices 30 to be used according to the type of task, the operation target in the task, the size of the body part BP, or a combination thereof. In step S102, the control unit 11 may assign each of the plurality of sub-goal positions SG to each of the plurality of robot devices 30 for which at least one of the number and size has been determined.

[0104] Also, in one example, the control unit 11 may determine the range of the body part BP according to at least one of the type of task and the operation target in the task at an arbitrary timing. Accordingly, in step S102, the control unit 11 may assign each of the plurality of sub-goal positions SG corresponding to the determined range of the body part BP to each of the plurality of robot devices 30.

[0105] Also, in one example, in step S102, the control unit 11 may specify a variable sub-goal position SG. Accordingly, the control unit 11 may determine the number of robot devices 30 to be used according to the number of the specified sub-goal positions SG. Further, the control unit 11 may also determine the size of the robot devices 30 to be used.

[0106] Also, in one example, by the loop of step S104, the control unit 11 may continuously execute a task or continuously execute a plurality of tasks by repeatedly executing steps S101 to S103. While continuously executing a task or when switching the task to be performed, the control unit 11 may execute a process of determining at least one of the number and size of the robot devices 30 to be used. Thereby, the control unit 11 may change at least one of the number and size of the robot devices 30 to be used.

[0107] [Features] In the control device 1 according to the present embodiment, by the process of step S102, each sub-goal position SG corresponding to the body part BP is dynamically assigned to each robot device 30. As a result, since the relationship between each robot device 30 and the body part BP is not fixed, it is possible to expect avoidance of collision between the robot devices 30 even by a simple method. Therefore, according to the present embodiment, when controlling a plurality of robot devices 30 in accordance with the body part BP, it is possible to reduce the possibility of collision between the robot devices 30. In addition, according to an example of the present embodiment, each robot device 30 can be made to feel to the operator P as a part of its own body.

[0108] §4 Modification Although the embodiments of the present invention have been described in detail above, the foregoing description is merely illustrative of the present invention in all respects. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs. In the above embodiments, various improvements or modifications may be appropriately made.

[0109] §5 Experimental Example In order to verify the effectiveness of the above embodiment, the following experiment was conducted. However, the present invention is not limited to the following examples.

[0110] [First Experimental Example] As a first experimental example, a task environment in which a plurality of robot devices are moved to a target position with a designated hand sign was constructed on a VR space using a VR device (Meta Quest 2), and the participants were made to perform the task. Ten participants (6 males, 4 females, average age: 24.20, standard deviation of age: 2.57) participated in the first experimental example. Four types of hand signs were prepared: rock, scissors, paper, and reverse paper. Two types of target positions were prepared. That is, 4×2 = 8 tasks were set. The right hand was adopted as the body part to be used for the operation.

[0111] The shape of the robot device was set to be cylindrical. Two types of sizes (20 mm and 30 mm) were prepared for the diameter of the robot device. For each size, three density (number of robot devices) conditions were prepared. Specifically, for the size of 20 mm, three conditions of 6 units (sparse), 18 units (medium), and 27 units (dense) were prepared. For the size of 30 mm, three conditions of 6 units (sparse), 8 units (medium), and 12 units (dense) were prepared.

[0112] Also, in the first embodiment, the feature-based (skeleton-based) method shown in FIG. 4 above was adopted for the method of specifying the sub-goal position, and each sub-goal position was dynamically assigned to each robot device by the method of assigning the closest sub-goal position (bone-dynamic). In the second embodiment, the silhouette-based method shown in FIG. 3 above was adopted for the method of specifying the sub-goal position (silhouette-dynamic). Otherwise, the conditions of the second embodiment were set the same as those of the first embodiment. On the other hand, in the first comparative example, similar to the first embodiment, the feature-based (skeleton-based) method shown in FIG. 4 above was adopted for the method of specifying the sub-goal position, but each sub-goal position was statically assigned to each robot device on a one-to-one basis (bone-static). For each robot device, each sub-goal position After assigning, in the first embodiment, the second embodiment, and the first comparative example, the operations of each robot device were controlled by the same path planning method.

[0113] FIG. 8 shows the sub-goal positions (feature points) adopted in the first embodiment and the first comparative example under the respective conditions of the size and density of the robot device. As described above, regarding the size (20 mm, 30 mm), density (sparse, medium, dense), and sub-goal position assignment method (first embodiment, second embodiment, first comparative example) of the robot device, 2×3×3 = 18 conditions were prepared. For each participant, the above eight tasks were performed for each condition, and each time the eight tasks were completed, the following questionnaire was answered. Each answer was scored on a 7-point Likert scale.

[0114] (Questionnaire) (1) Questions about body ownership (1-1) It felt like the swarm robot was my body. (I felt that the swarm robot was my body.) (1-2) It felt like some of the robots were my fingers. (I felt that some of the robots were my fingers.) (1-3) It felt like the swarm robots belonged to me. (I felt that the swarm robots belonged to me.) (1-4) The swarm robot felt like a human hand. (I felt that the swarm robot was like a human hand.) (2) Questions about agency (2-1) The movements of the swarm robot felt like they were my movements. (I felt that the movements of the swarm robot were my movements.) (2-2) I felt like I was controlling the movements of the swarm robot. (I felt like I was controlling the movements of the swarm robot.) (2-3) I felt like I was causing the movements of the swarm robot. (I felt like I was causing the movements of the swarm robot.) (2-4) The movements of the swarm robot were in sync with my own movements. (I felt that the movements of the swarm robot were in sync with my own movements.)

[0115] Each response was scored on a 7-point Likert scale. For each of the size and density conditions of the robot device, the average value of the responses to four questions regarding body ownership and behavioral agency was calculated to obtain the scores for the first embodiment, the second embodiment, and the second comparative example.

[0116] (Experimental Results) Figures 9A and 9B show the calculation results of the scores for body ownership at sizes of 20 mm and 30 mm. Figures 9C and 9D show the calculation results of the scores for behavioral agency at sizes of 20 mm and 30 mm. In each figure, for each density condition, the calculation results of the scores for the first comparative example, the first embodiment, and the second embodiment are shown in order from the left. Comparing the first embodiment and the first comparative example, the scores of the first embodiment were higher than those of the first comparative example for both body ownership and behavioral agency. This is presumably due to the effect that by adopting dynamic allocation, collisions between robot devices could be avoided.

[0117] Also, for the second embodiment, the scores for body ownership and behavioral agency were low under the sparse condition. This is presumably due to the fact that the silhouette-based method is more likely to result in a shape of the swarm robots that is farther from the body shape compared to the skeleton-based method. On the other hand, under the dense condition, the scores for both body ownership and behavioral agency of the second embodiment increased. Under the condition of the robot device size of 20 mm, the score of the second embodiment was higher than that of the first comparative example. From this result, it was found that under the dense condition, a sufficiently beneficial effect can also be obtained with the silhouette-based method.

[0118] [Second Experimental Example] In the second experimental example, a robot device with a diameter of 30 mm was fabricated, and the task of the first experimental example using the 30-mm robot device was reproduced in the real space.

[0119] Figure 10 shows the configuration of the robot device used in the second experimental example. As shown in Figure 10, a printed circuit board, a Li-Po battery (40 mAh), a housing, a motor holder, a motor (Pololu 2357 ), wheels, and caster wheels were installed on the robot device. The device includes a microcontroller unit (STMicroelectronics' STM32G071KBU6), Driver (DRV8837DSGR from Texas Instruments), RF module (RF2401F20) and It is equipped with two photodiodes (PD15-22C / TR8 from Everlight Electronics).

[0120] FIG. 11 shows the experimental environment in the second experimental example. The cradle was connected to the host computer (host PC) via USB 2.0. The robot device and the cradle had an RF module. The device was equipped with a 2.4GHz ISM band wireless communication system, and data was transmitted via wireless communication. A gray code pattern was printed on the table using a DLP LightCrafter 4500 from Digital Instruments. The robot device received the projected and coded pattern of light with two photodiodes. The microcontroller of the robot device decoded the pattern information into position information, calculated the orientation from the positions of the two photodiodes, and broadcast the calculated position information and orientation to the host computer. The host computer grasped the position and orientation of each robot device by receiving the information broadcast from each robot device. The host computer sent a command to each robot device via the cradle every 100 milliseconds to move to the subgoal position.

[0121] The task settings were the same as in the first experimental example (i.e., eight tasks were set). The density conditions of the robot devices were also the same as in the first experimental example (6 units (sparse), 8 units (medium), and 12 units (dense)). In the second experimental example, 10 participants (4 men, Six women (mean age: 28.89, standard deviation of age: 13.83) participated.

[0122] For the method of allocating sub-goal positions in the third embodiment, a bone-based dynamic allocation method (bone-dynamic) was adopted as in the first embodiment. For the method of allocating sub-goal positions in the fourth embodiment, a silhouette-based dynamic allocation method (silhouette-dynamic) was adopted as in the second embodiment. On the other hand, for the second comparative example, a bone-based static allocation method (bone-static) was adopted as in the first comparative example. Regarding the density of the robot device and the method of allocating sub-goal positions, 3×3 = 9 conditions were prepared. Each participant was made to perform a task of operating the robot device placed on the table with the right hand. Other conditions were the same as in the first experimental example. That is, after allocating each sub-goal position to each robot device, in the third embodiment, the fourth embodiment, and the second comparative example, the operations of each robot device were controlled by the same path planning method. Each participant was made to perform the above eight tasks for each condition, and each time the eight tasks were completed, the participant was made to answer the same questionnaire as in the first experimental example. Regarding the method of allocating sub-goal positions, 3×3 = 9 conditions were prepared. Each participant was made to perform a task of operating the robot device placed on the table with the right hand. Other conditions were the same as in the first experimental example. That is, after allocating each sub-goal position to each robot device, in the third embodiment, the fourth embodiment, and the second comparative example, the operations of each robot device were controlled by the same path planning method. Each participant was made to perform the above eight tasks for each condition, and each time the eight tasks were completed, the participant was made to answer the same questionnaire as in the first experimental example.

[0123] (Experimental Results) Figures 12A and 12B show the calculation results of the scores for body ownership and agency. In each figure, for each density condition, the calculation results of the scores for the second comparative example, the third embodiment, and the fourth embodiment are shown in order from the left. As shown in each figure, also in the second experimental example, as in the first experimental example, the scores of the third embodiment that adopted bone-based dynamic allocation were generally high. Also, the scores of the fourth embodiment that adopted silhouette-based dynamic allocation increased under the dense condition.

[0124] [Parentheses] As described above, in both the first and second experimental examples, in the skeleton-based allocation method, the scores of body ownership and behavioral agency were higher when dynamic allocation was adopted (first embodiment, third embodiment, first comparative example, and second comparative example). From this result, it was presumed that by adopting dynamic allocation, it became easier to avoid collisions between robot devices, and as a result, the effect of improving body ownership and behavioral agency could be obtained. Also, it was found that even in the silhouette-based method, advantageous results could be obtained by increasing the density of the robot devices. Thus, the usefulness of the above-described embodiment could be shown.

[0125] This specification includes the following disclosures. [Appendix 1] A control device (1) including a control unit (11), wherein the control unit (11) acquires observation data (20) of a body part (BP) of an operator (P), dynamically allocates each of a plurality of sub-goal positions (SG) corresponding to the body part (BP) represented in the acquired observation data (20) to each of a plurality of robot devices (30), and controls the operations of each of the plurality of robot devices (30) toward each of the allocated plurality of sub-goal positions (SG), and is configured to execute the control device (1). [Appendix 2] Dynamically allocating each of the plurality of sub-goal positions (SG) to each of the plurality of robot devices (30) includes identifying a plurality of target points of the body part (BP) in the acquired observation data (20), determining the plurality of sub-goal positions (SG) according to the identified plurality of target points, and dynamically allocating each of the determined plurality of sub-goal positions (SG) to each of the plurality of robot devices (30), and is configured by the control device (1) according to Appendix 1. [Appendix 3] Identifying the plurality of target points is constituted by identifying the plurality of target points according to the silhouette of the body part (BP). The control device (1) described in Appendix 2. [Appendix 4] Identifying the plurality of target points is constituted by identifying the plurality of target points according to the characteristics of the body part (BP). The control device (1) described in Appendix 2. [Appendix 5] Controlling the operations of each of the plurality of robot devices (30) is Planning a path for each robot device (30) to move to each sub-goal position (SG), and Controlling the operations of each robot device (30) according to the planned path, and is constituted by The control device (1) described in any one of Appendices 1 to 4. [Appendix 6] The observation data (20) includes image data. The control device (1) described in any one of Appendices 1 to 5. [Appendix 7] The observation data (20) includes motion capture data. The control device (1) described in any one of Appendices 1 to 5. [Appendix 8] Each of the robot devices (30) is an autonomous robot having the ability to move. The control device (1) described in any one of Appendices 1 to 7. [Appendix 9] Each of the robot devices (30) is a moving body having the ability to move. The control device (1) described in any one of Appendices 1 to 8. [Appendix 10] The moving body is a drone. The control device (1) described in Appendix 9. [Appendix 11] The body part (BP) includes a hand. The control device (1) according to any one of Appendices 1 to 10. [Appendix 12] The body part (BP) includes at least a part of a limb. The control device (1) according to any one of Appendices 1 to 11. [Appendix 13] The body part (BP) includes the face. The control device (1) according to any one of Appendices 1 to 12. [Appendix 14] The body part (BP) is composed of a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. The control device (1) according to any one of Appendices 1 to 10. [Appendix 15] The control unit (11) is further configured to determine at least one of the number and size of the robot devices (30) according to at least one of the type of the task and the operation target in the task. Dynamically allocating each of the plurality of sub-goal positions (SG) to each of the plurality of robot devices (30) is configured by dynamically allocating the plurality of sub-goal positions (SG) to each of the plurality of robot devices (30) when at least one of the number and size is determined. The control device (1) according to any one of Appendices 1 to 14. [Appendix 16] The control unit (11) is further configured to determine the range of the body part (BP) according to at least one of the type of the task and the operation target in the task. The plurality of sub-goal positions (SG) corresponding to the body part (BP) are composed of the plurality of sub-goal positions (SG) corresponding to the body part (BP) when the range is determined. The control device (1) according to any one of Appendices 1 to 15. [Appendix 17] A computer (1) obtains observation data (20) of the body part (BP) of an operator (P). Dynamically allocating each of a plurality of sub-goal positions (SG) corresponding to the body part (BP) represented in the obtained observation data (20) to each of a plurality of robot devices (30), and Controlling the operations of each of the plurality of robot devices (30) toward each of the allocated plurality of sub-goal positions (SG). A control method for executing the above. Control method. [Appendix 18] Causing a computer (1) to acquire observation data (20) of the body part (BP) of an operator (P), dynamically allocate each of a plurality of sub-goal positions (SG) corresponding to the body part (BP) represented in the obtained observation data (20) to each of a plurality of robot devices (30), and control the operations of each of the plurality of robot devices (30) toward each of the allocated plurality of sub-goal positions (SG). A control program (81) for causing the above to be executed. Control program.

Explanation of Reference Numerals

[0126] 1... Control device, 11... Control unit, 12... Storage unit, 13... External interface, 14... Input device, 15... Output device, 16... Drive, 81... Control program, 91... Storage medium, 111... Data acquisition unit, 112... Dynamic allocation unit, 113... Operation control unit, 20... Observation data, 30... Robot device, P... Operator, BP... Body part, SG... Sub-goal position, S... Sensor

Claims

1. A control device comprising a control unit, wherein the control unit acquires observation data of a body part of an operator, dynamically assigns each of a plurality of sub-goal positions corresponding to the body part represented in the acquired observation data to each of a plurality of robot devices, and controls the operations of each of the plurality of robot devices toward each of the assigned plurality of sub-goal positions, and is configured to execute, a control device.

2. The dynamically assigning each of the plurality of sub-goal positions to each of the plurality of robot devices includes identifying a plurality of target points of the body part in the acquired observation data, determining the plurality of sub-goal positions according to the identified plurality of target points, and dynamically assigning each of the determined plurality of sub-goal positions to each of the plurality of robot devices, and is constituted by the control device according to Claim 1.

3. The identifying the plurality of target points is constituted by identifying the plurality of target points according to the silhouette of the body part, the control device according to Claim 2.

4. The identifying the plurality of target points is constituted by identifying the plurality of target points according to the characteristics of the body part, the control device according to Claim 2.

5. The controlling the operations of each of the plurality of robot devices includes planning a path for each robot device to move to each sub-goal position, and controlling the operations of each robot device according to the planned path, and is constituted by the control device according to Claim 1.

6. The observation data includes image data, the control device according to Claim 1.

7. The observation data includes motion capture data, the control device according to Claim 1.

8. Each of the robot devices is an autonomous robot having the ability to move, the control device according to Claim 1.

9. Each of the robot devices is a moving body having the ability to move, the control device according to Claim 1.

10. The moving body is a drone, the control device according to Claim 9.

11. The body part includes a hand, the control device according to Claim 1.

12. The body part includes at least a part of a limb, the control device according to Claim 1.

13. The body part includes a face, the control device according to Claim 1.

14. The body part is composed of a part or all of the body, an object under the control of at least a part of the body, or a combination thereof. The control device according to claim 1.

15. The control unit is further configured to determine at least one of the number and size of the robot devices according to at least one of the type of task and the operation target in the task. Dynamically allocating each of the plurality of sub-goal positions to each of the plurality of robot devices is constituted by dynamically allocating the plurality of sub-goal positions to each of the plurality of robot devices for which at least one of the number and size has been determined. The control device according to claim 1.

16. The control unit is further configured to determine the range of the body part according to at least one of the type of task and the operation target in the task. The plurality of sub-goal positions corresponding to the body part are constituted by the plurality of sub-goal positions corresponding to the body part for which the range has been determined. The control device according to claim 1.

17. A computer acquires observation data of an operator's body part, dynamically allocates each of the plurality of sub-goal positions corresponding to the body part shown in the acquired observation data to each of the plurality of robot devices, and controls the operations of each of the plurality of robot devices toward each of the allocated plurality of sub-goal positions. executes a control method.

18. A computer is caused to acquire observation data of an operator's body part, dynamically allocate each of the plurality of sub-goal positions corresponding to the body part shown in the acquired observation data to each of the plurality of robot devices, and control the operations of each of the plurality of robot devices toward each of the allocated plurality of sub-goal positions. for causing the execution of a control program.

Images