Communication system

Abstract

Provided is a communication system comprising: a controller (10) that is worn on a hand of a user; a manipulated object (50) that is remotely manipulated by the controller; and a relay device (70) that relays between the controller and the manipulated object. The controller has: a first detection unit (11) that detects movements of fingers of the user; a stimulus provision unit (12) that provides an external stimulus to the fingers of the user; and a control unit (13) that controls the stimulus provision unit. The manipulated object has: a drive unit (51) that operates on the basis of received operation information; and a second detection unit (52) that detects an external stimulus to the manipulated object. The relay device transmits the operation information which is based on a detection result from the first detection unit, and transmits, to the controller and on the basis of a detection result from the second detection unit, stimulus provision information which controls the stimulus provision unit so that an external stimulus is provided to the user.

Description

Technical Field

[0001] The present invention relates to a communication system for remotely operating an operation target by a controller.

Background Art

[0002] Conventionally, various technologies have been proposed for remote communication. In remote communication, there is a lot of demand for approaching face-to-face communication including content and expressions that are difficult to convey by voice and video. For example, the application of remote operation technology as described in Patent Document 1 can be considered.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the automation control system described in Patent Document 1, the method of conventionally operating a hand puppet by a human hand is replaced by an automation control system. However, it is difficult for a simple automation control system to meet the demand for realizing content and expressions that are difficult to convey by voice and video.

[0005] One of the objects of the present disclosure is to solve at least a part of the above problems and to provide a communication system that can experience a new remote communication as if actually face-to-face.

Means for Solving the Problems

[0006] The communication system relating to this disclosure comprises a controller worn on the user's hand, an object to be operated remotely by the controller, and a relay device that relays communication between the controller and the object to be operated. The controller has a first detection unit that detects the movement of the user's fingers, a stimulus providing unit that provides external stimuli to the user's fingers, and a control unit that controls the stimulus providing unit. The object to be operated has a drive unit that operates based on received motion information and a second detection unit that detects external stimuli to the object to be operated. The relay device transmits motion information to the object to be operated based on the detection result by the first detection unit, and transmits stimulus providing information to the controller that controls the stimulus providing unit to provide external stimuli to the user based on the detection result by the second detection unit. [Brief explanation of the drawing]

[0007] [Figure 1] A schematic diagram showing the configuration of a communication system according to an embodiment. [Figure 2] A functional block diagram showing the configuration of the communication system according to the embodiment. [Figure 3] A functional block diagram showing the configuration of a communication system according to a first modified embodiment. [Figure 4] A functional block diagram showing the configuration of a communication system according to a second modified embodiment. [Figure 5] A functional block diagram showing the configuration of a communication system according to a third modified embodiment. [Figure 6] A functional block diagram showing the configuration of a communication system according to a fourth modified embodiment. [Figure 7] A functional block diagram showing the configuration of a communication system according to a fifth modified example of the embodiment. [Figure 8] A functional block diagram showing the configuration of a communication system according to a sixth modified example of the embodiment. [Figure 9] A functional block diagram showing the configuration of a communication system according to a seventh modified embodiment. [Figure 10] A functional block diagram showing the configuration of a communication system according to the eighth modified embodiment. [Modes for carrying out the invention]

[0008] The embodiments of this disclosure will be described below with reference to the drawings. [Outline of the communication system configuration] Figure 1 is a schematic diagram showing the configuration of the communication system 1 according to this embodiment. As shown in Figure 1, the communication system 1 according to this embodiment includes a controller 10 worn on the user's hand, a puppet 50 which is the object to be operated remotely by the controller 10, and a relay device 70 which relays communication between the controller 10 and the puppet 50. The following describes the various components of communication system 1.

[0009] Figure 2 is a functional block diagram showing the configuration of communication system 1. [Controller Configuration] As shown in Figure 1, the controller 10 has, for example, the shape of a glove, is worn on the user's hand, and is an operating device for remotely controlling an object. As shown in Figure 2, the controller 10 includes a first detection unit 11, a stimulus supply unit 12, and a control unit 13.

[0010] The first detection unit 11 is equipped with various trackers and sensors to detect the movement of the user's fingers when the controller 10 is attached. Any known trackers and sensors may be used. The trackers and sensors are also positioned adjacent to the user's fingers at an appropriate location when the controller 10 is attached. Some parts of the trackers and sensors, such as the generator necessary for detection by the trackers, may be provided separately from the controller 10. Operation information based on the detection results by the first detection unit 11 is transmitted to the puppet 50 by the relay device 70. Here, operation information refers to information indicating the movement of the user's fingers detected by the first detection unit 11, and may be the detected information itself, or it may be information based on the movement of the user's fingers detected by the first detection unit 11.

[0011] The stimulus providing unit 12 includes various actuators and adjustment units, and provides an external stimulus to the finger of the user wearing the controller 10. The external stimulus includes tactile sensation, force sensation, pressure sensation, temperature sensation, etc. Any known actuators and adjustment units may be used. Also, the actuators and adjustment units are arranged to be adjacent to an appropriate position on the finger of the user when the controller 10 is worn.

[0012] The control unit 13 has arithmetic processing circuits such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the stimulus providing unit 12. At this time, the control unit 13 controls the stimulus providing unit 12 based on the stimulus providing information derived from the puppet 50 received from the relay device 70. The stimulus providing information will be described later.

[0013] Note that the controller 10 is not limited to a configuration having the shape of the glove body shown in FIG. 1. For example, it may be a configuration of an exoskeleton type, or a configuration consisting of a combination of a glove body and an exoskeleton, or a component for each finger may be an independent ring type device or a wristband type device, etc., or other configurations. Also, the power supply to the controller 10 may be a configuration performed from the outside of the controller 10, or a configuration in which a battery is built into the controller 10.

[0014] [Configuration of the Puppet] As shown in FIG. 1, the puppet 50 has, for example, a head and a pair of arm parts, and is an operation target remotely operated by the controller 10. As shown in FIG. 2, the puppet 50 includes a drive unit 51 and a second detection unit 52.

[0015] The drive unit 51 includes a plurality of motors and actuators such as servo motors according to the shape of the puppet 50, and drives at least one of, for example, a part of the head, arm, etc. of the puppet 50, the upper body, lower body, or the whole of the puppet 50. At this time, the drive unit 51 drives based on the operation information (for example, a control signal for operating a servo motor) from the controller 10 received from the relay device 70.

[0016] The second detection unit 52 includes various sensors and detects external stimuli to the puppet 50. As described above, external stimuli include tactile sensation, force sensation, pressure sensation, temperature sensation, etc., and the various sensors may use any known sensors capable of detecting these external stimuli. Also, the sensors are arranged at appropriate positions on the puppet 50. The relay device 70 transmits the stimulus-providing information based on the detection result by the second detection unit 52 to the controller 10. Here, the stimulus-providing information is information for controlling the stimulus-providing unit 12 to provide an external stimulus to the user, and may be information indicating the detection result itself detected by the second detection unit 52, or may be information based on the detection result by the second detection unit 52. For example, it may be raw data with no processing performed on the output of the sensors included in the second detection unit 52, or may be data obtained by performing predetermined processing on the output of the sensors included in the second detection unit 52.

[0017] Note that the puppet 50 is not limited to a configuration having the shape of a human figure shown in FIG. 1, and may, for example, have a configuration having the shape of a human hand. Also, for example, the puppet 50 does not necessarily require joints, and may have a configuration such as a cushion-like shape, an inorganic object shape with a built-in vibration actuator or a mechanism system capable of shifting the center of gravity. Also, regarding the power supply to the puppet 50, it may be configured to be performed from outside the puppet 5, or may be configured to have a built-in battery in the puppet 50.

[0018] [Configuration of Relay Device] As shown in Figures 1 and 2, the relay device 70 is a device that relays information between the controller 10 and the puppet 50. The relay device 70 transmits operation information based on the detection result by the first detection unit 11 of the controller 10 to the puppet 50, which is the target of the operation. The relay device 70 also transmits stimulus provision information to the controller 10.

[0019] [Processing in communication systems] In the communication system 1, when a user operates the controller 10 by moving their fingers while the controller 10 is attached to their fingers, the first detection unit 11 detects the movement of the user's fingers. Then, the relay device 70 transmits operation information based on the detection result by the first detection unit 11 to the puppet 50, and the drive unit 51 of the puppet 50 operates based on the operation information from the controller 10 received from the relay device 70. In other words, when the controller 10 is operated, the drive unit 51 of the puppet 50 operates in accordance with the movement of the user's fingers, and the operation on the controller 10 is reproduced on the puppet 50 side.

[0020] On the other hand, when the puppet 50 receives an external stimulus, the second detection unit 52 detects the external stimulus to the puppet 50. Then, the relay device 70 transmits stimulus provision information based on the detection result by the second detection unit 52 to the controller 10, and the control unit 13 of the controller 10 controls the stimulus provision unit 12 based on the stimulus provision information originating from the puppet 50 that it receives from the relay device 70. In other words, when the puppet 50 receives an external stimulus, the control unit 13 of the controller 10 controls the stimulus provision unit 12 in accordance with that external stimulus, and the external stimulus received by the puppet 50 is reproduced on the controller 10 side.

[0021] The communication system 1 according to this embodiment, as described above, can achieve the following effects. The communication system 1 comprises a controller 10 worn on the user's hand, a puppet 50 which is the object of operation remotely controlled by the controller 10, and a relay device 70 which relays communication between the controller 10 and the puppet 50. The controller 10 has a first detection unit 11 that detects the movement of the user's fingers, a stimulus providing unit 12 that provides external stimuli to the user's fingers, and a control unit 13 that controls the stimulus providing unit 12. The puppet 50 has a drive unit 51 that operates based on received motion information and a second detection unit 52 that detects external stimuli to the puppet 50. The relay device 70 transmits operation information based on the detection result from the first detection unit 11 to the puppet 50, and transmits stimulus provision information to the controller 10 that controls the stimulus provision unit 12 to provide external stimuli to the user based on the detection result from the second detection unit 52.

[0022] With this configuration, information is transmitted in real time and bidirectionally between the controller 10 and the puppet 50, which is the object of control. This allows not only remote control of the puppet 50 by the controller 10, but also the feeding back of external stimuli to the puppet 50 to the controller 10. For this reason, it becomes possible to intuitively and easily transmit information and expressions that are difficult to convey through voice and video during remote control, such as the hardness or softness of an object or its temperature. Consequently, users can intuitively and realistically convey emotional and sentimental information, such as the enjoyment and comfort they feel as if they were actually meeting the puppet face-to-face, enabling a new form of remote communication.

[0023] [Modified Example of Embodiment 1] Figure 3 is a functional block diagram showing the configuration of the communication system 1A according to the first modified example. Communication system 1A has the same configuration as communication system 1 according to the embodiment described above, but the configuration of the controller 10 and the puppet 50 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0024] [Communication system configuration] As shown in Figure 3, the controller 10 of the communication system 1A includes a vibration generating unit 121 and a temperature control unit 122 in the stimulus providing unit 12.

[0025] The vibration generating unit 121 has a vibrator such as a motor and generates vibrations under the control of the control unit 13. When the vibration generating unit 121 vibrates, external stimuli such as tactile sensations are provided to the fingers of the user to whom the controller 10 is attached. The vibration generating unit 121 is positioned adjacent to an appropriate location on the user's fingers when the controller 10 is attached.

[0026] The temperature control unit 122 has a temperature-adjustable component and adjusts the temperature under the control of the control unit 13. When the temperature is adjusted by the temperature control unit 122, external stimuli such as warmth and coldness are provided to the user's fingers to which the controller 10 is attached. The temperature control unit 122 is positioned adjacent to an appropriate location on the user's fingers when the controller 10 is attached.

[0027] As shown in Figure 3, the puppet 50 of the communication system 1A includes a tactile sensor 521 and a temperature sensor 522 in the second detection unit 52.

[0028] The tactile sensor 521 detects external stimuli such as touch, force, and pressure from among the external stimuli applied to the puppet 50. The tactile sensor 521 is positioned appropriately on the puppet 50. The relay device 70 transmits stimulus provision information based on the detection results from the tactile sensor 521 to the controller 10.

[0029] The temperature sensor 522 detects external stimuli such as warmth and coldness among the external stimuli applied to the puppet 50. The temperature sensor 522 is positioned appropriately on the puppet 50. The relay device 70 transmits stimulus provision information based on the detection results from the temperature sensor 522 to the controller 10.

[0030] [Processing in communication systems] When the puppet 50 receives external stimuli such as touch, force, and pressure, the tactile-force sensor 521 of the second detection unit 52 detects the external stimuli to the puppet 50. Then, the relay device 70 transmits stimulus provision information based on the detection results from the second detection unit 52 to the controller 10, and the control unit 13 of the controller 10 operates the vibration generating unit 121 of the stimulus provision unit 12 based on the stimulus provision information originating from the puppet 50 received from the relay device 70. In other words, when the puppet 50 receives external stimuli such as touch, force, and pressure, the control unit 13 of the controller 10 operates the vibration generating unit 121 of the stimulus provision unit 12 in response to those external stimuli, and the external stimuli such as touch, force, and pressure received by the puppet 50 are reproduced on the controller 10 side.

[0031] Furthermore, when the puppet 50 receives an external stimulus such as warmth or cold, the temperature sensor 522 of the second detection unit 52 detects the external stimulus to the puppet 50. Then, the relay device 70 transmits stimulus provision information based on the detection result by the second detection unit 52 to the controller 10, and the control unit 13 of the controller 10 operates the temperature control unit 122 of the stimulus provision unit 12 based on the stimulus provision information originating from the puppet 50 received from the relay device 70. In other words, when the puppet 50 receives an external stimulus such as warmth or cold, the control unit 13 of the controller 10 operates the temperature control unit 122 of the stimulus provision unit 12 in accordance with that external stimulus, and the external stimulus such as warmth or cold received by the puppet 50 is reproduced on the controller 10 side.

[0032] In the communication system 1A according to the first modified example of this embodiment described above, the second detection unit 52 of the puppet 50 includes a tactile sensor 521, and the relay device 70 transmits stimulus provision information based on the detection result of the tactile sensor 521 to the controller 10. The stimulus provision unit 12 of the controller 10 includes a vibration generation unit 121, and the control unit 13 operates the vibration generation unit 121 based on the stimulus provision information.

[0033] With this configuration, external stimuli such as tactile, force, and pressure sensations applied to the puppet 50 can be fed back to the controller 10 and reproduced. As a result, gestures and physical contact that are closer to face-to-face communication can be reproduced. The vibration generating unit 121 is not required, and in addition to the vibration generating unit 121, or in place of the vibration generating unit 121, a resistance force generating unit, a pressure generating unit, etc., may be provided.

[0034] Furthermore, in the communication system 1A, the second detection unit 52 of the puppet 50 includes a temperature sensor 522, and the relay device 70 transmits stimulus provision information based on the detection result from the temperature sensor 522 to the controller 10. The stimulus provision unit 12 of the controller 10 includes a temperature control unit 122, and the control unit 13 operates the temperature control unit 122 based on the stimulus provision information.

[0035] With this configuration, external stimuli such as warmth and coldness to the puppet 50, which is the object of manipulation, can be fed back to the controller 10 and reproduced. Therefore, it is possible to reproduce temperature sensations such as warmth, which are closer to face-to-face communication. Note that the temperature control unit 122 is not required.

[0036] The tactile sensor 521 and temperature sensor 522 shown in the first modified example are merely examples of sensors that detect external stimuli to the puppet 50 being manipulated, and are not limited to these examples. The second detection unit 52 may also include a tactile touch / pressure sensor that detects when the arms, head, back, etc., of the puppet 50 are touched, that is, a joint angle sensor that detects external force, a temperature sensor that detects non-contact areas by temperature (body temperature, etc.), a temperature sensor for monitoring room temperature, and the like. All of these sensors are positioned appropriately on the puppet 50.

[0037] [Second modified example of the embodiment] Figure 4 is a functional block diagram showing the configuration of the communication system 1B according to the second modified example. Communication system 1B has the same configuration as communication system 1 according to the embodiment described above, but the configuration of the puppet 50 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0038] [Communication system configuration] As shown in Figure 4, the puppet 50 of the communication system 1B includes an acquisition unit 53 in addition to the drive unit 51 and the second detection unit 52. The acquisition unit 53 acquires at least one of the following: imaging information of the surroundings of the puppet 50, which is the object to be operated, and audio information of the surroundings of the puppet 50. The acquisition unit 53 is positioned appropriately on the puppet 50. The relay device 70 transmits at least one of the information acquired by the acquisition unit 53 to the controller 10.

[0039] [Processing in communication systems] When the acquisition unit 53 of the puppet 50 acquires at least one of the imaging information and audio information, the relay device 70 transmits that information to the controller 10. The controller 10 uses this information for various purposes. For example, by displaying still images or moving images based on imaging information on the controller 10 itself or on a display device provided separately from the controller 10, the sense of presence in remote communication can be enhanced, making it feel like a face-to-face interaction. Similarly, by outputting audio information from the controller 10 itself or on an audio output device provided separately from the controller 10, the sense of presence in dialogue during remote communication can be enhanced.

[0040] Furthermore, imaging information and audio information may be used for control in the controller 10. For example, when imaging information and audio information are used as stimulus information, the surrounding conditions of the puppet 50, which is the target of the operation, can be inferred based on the imaging information and audio information, and this can be reflected in the control of the stimulus provision unit 12. As an example, by analyzing changes in the volume of the audio information, if the volume is louder, it can be inferred that the surrounding conditions of the puppet 50 are exciting or lively, and the tactile sensation can be greatly changed in the vibration generation unit 121 described in the first modified example, or the thermoregulation unit 122 can be greatly changed, thereby allowing the controller 10 to reproduce external stimuli that take into account the surrounding conditions of the puppet 50, which is the target of the operation.

[0041] According to the second modified communication system 1B of this embodiment described above, imaging information and audio information from around the puppet 50, which is the object of operation, can be fed back to the controller 10, thereby improving the quality of remote communication. Furthermore, by using at least one of the imaging information and audio information as stimulus information, it is possible to realize content and expressions that are difficult to convey with voice and video alone in remote communication, as well as to expand communication.

[0042] [Third Modification of the Embodiment] Figure 5 is a functional block diagram showing the configuration of the communication system 1C according to the third modified example. Communication system 1C has a configuration similar to communication system 1 according to the embodiment described above, but it has multiple targets for operation and the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0043] [Communication system configuration] As shown in Figure 5, the communication system 1C includes a plurality of puppets 50A, 50B, 50C, etc. The relay device 70 also includes a switching operation determination unit 71, a target selection unit 72, and a transmitting / receiving unit 73, as shown in Figure 5.

[0044] The switching operation determination unit 71 determines whether the finger movement detected by the first detection unit 11 is a communication target switching operation. A communication target switching operation is a pre-set pose or gesture, which is an operation to select or switch the puppet to be used for communication from among multiple puppets 50A, 50B, 50C, etc. For example, if puppet 50A is set to a pose with one finger raised, puppet 50B is set to a pose with two fingers raised, and puppet 50C is set to a pose with three fingers raised, the switching operation determination unit 71 can determine that the finger movement detected by the first detection unit 11 is a communication target switching operation if it matches the corresponding pose. Note that the number of targets associated with a single communication target switching operation may be one or multiple. Also, the communication target switching operation may be a static pose or a series of actions involving movement, such as waving one's hand.

[0045] When the switching operation determination unit 71 determines that the finger movement is a communication target switching operation, the target selection unit 72 selects the operation target corresponding to the communication target switching operation from among the multiple puppets 50A, 50B, 50C, etc. that are the operation targets. The transmitting / receiving unit 73 transmits operation information to the communication target selected by the target selection unit 72 and receives stimulus provision information from the communication target.

[0046] [Processing in communication systems] When the switching operation determination unit 71 determines that the finger movement detected by the first detection unit 11 is a communication target switching operation, the target selection unit 72 selects the target that corresponds to the communication target switching operation from among the multiple puppets 50A, 50B, 50C, etc. that are the targets of operation, and the transmitting / receiving unit 73 transmits operation information to the communication target selected by the target selection unit 72 and receives stimulus provision information from the communication target.

[0047] According to the third modified example of this embodiment described above, the user can select and switch to a communication target from among the multiple puppets 50A, 50B, 50C, etc. included in the communication system 1C simply by performing a pre-configured communication target switching operation. Furthermore, the multiple puppets 50A, 50B, 50C, etc. included in the communication system 1C may have the same configuration, or they may have some or all different configurations.

[0048] [Fourth modified example of the embodiment] Figure 6 is a functional block diagram showing the configuration of the communication system 1D according to the fourth modified example. Communication system 1D has the same configuration as communication system 1 according to the embodiment described above, but the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0049] [Communication system configuration] As shown in Figure 6, the communication system 1D includes a relay device 70 equipped with a type acquisition unit 74 and a type-specific information conversion unit 75.

[0050] The type acquisition unit 74 indicates the type of the operation target. Separate love The report is retrieved. The type of target being operated on is information indicating the configuration of the device being operated on by controller 10.

[0051] The type-based information conversion unit 75 converts the operation information based on the type information acquired by the type acquisition unit 74. As mentioned above, the operation information is information indicating the movement of the user's fingers detected by the first detection unit 11 of the controller 10. Examples of such operation information include the following types.

[0052] (1) Operation information A detected by the first detection unit 11 of the controller 10 The motion information A is, for example, the output of the various trackers and sensors provided by the first detection unit 11, or coordinate information of feature points such as joints in the user's fingers, or motion data of feature points calculated according to the output. Regarding operation information A, it is not necessary to know the configuration of the motor and actuator of the object being operated, and it is independent of the configuration of the object being operated. Furthermore, operation information A contains a relatively large amount of information and is useful for improving reproducibility, but the amount of data transmitted is relatively large.

[0053] (2) Operation information B obtained by interpreting the time-series data of operation information A and applying it to a predetermined operation. Operation information B is data converted into predetermined poses or gestures, similar to the communication target switching operation described in the third modified example, by interpreting the time-series data of operation information A. Regarding operation information B, in order to interpret the time-series data, it is necessary to have some knowledge of the motor and actuator configuration of the object being operated. Therefore, operation information B depends to some extent on the configuration of the object being operated. Also, because operation information B is first coded through the interpretation of time-series data, the amount of information is relatively small, and the amount of data transmitted is relatively small, but its reproducibility is inferior to operation information A.

[0054] (3) Operation information C obtained by converting operation information A or operation information B into time-series control data of the configuration of the target device. Regarding operation information C, the configuration of the motor and actuator in the target device must be known in order to generate time-series control data for the motor and actuator in the target device. Therefore, operation information C depends on the configuration of the target device. Also, generally, the period for which control data for the motor and actuator of the target device is required is longer than the detection period in the first detection unit 11, so the amount of information in operation information C and the amount of data transmitted are smaller than that of operation information A.

[0055] The type-based information conversion unit 75 selects the most suitable operation information from operation information A to operation information C described above, based on the type information acquired by the type acquisition unit 74, and converts it as necessary. When the type-based information conversion unit 75 selects the most suitable operation information, it is configured to consider at least one of the following: dependence on the configuration of the target device, the amount of information required, the desired reproducibility, and the amount of data transmitted. Furthermore, the user may be able to configure what kind of conversion is performed according to the type information acquired by the type acquisition unit 74. Furthermore, if the type acquisition unit 74 cannot acquire type information, it may be predetermined which operation information to select. In addition to operation information A to operation information C described above, operation information with different conversion content may also be selected. Moreover, there may be multiple types of operation information selected by the type-specific information conversion unit 75.

[0056] [Processing in communication systems] The type acquisition unit 74 indicates the type of the operation target. Separate love Upon receiving the information, the type-specific information conversion unit 75 selects operation information based on the type information and converts it as necessary. If the operation information is converted, the converted operation information is then transmitted to the target of the operation.

[0057] According to the fourth modified example of this embodiment described above, the communication system 1D transmits suitable operation information from the controller 10 depending on the type of operation target.

[0058] [Fifth modified example of the embodiment] Figure 7 is a functional block diagram showing the configuration of the communication system 1E according to the fifth modified example. Communication system 1E has the same configuration as communication system 1 according to the above embodiment, but the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0059] [Communication system configuration] As shown in Figure 7, the communication system 1E includes a proficiency-based information conversion unit 76 in the relay device 70.

[0060] The proficiency-based information conversion unit 76 converts the operation information based on the user's proficiency level. The operation information is the same as operation information A to operation information C in the fourth modified example described above. The proficiency-based information conversion unit 76 analyzes the information indicating the user's finger movements detected by the first detection unit 11 and determines the proficiency level of the user operating the controller 10. Such analysis can be performed using known rule-based analysis or deep learning methods. For example, if the user has a high level of proficiency, that is, if the user is judged to be an expert, the proficiency-based information conversion unit 76 selects the information itself that indicates the user's finger movements detected by the first detection unit 11, such as the motion information A described above.

[0061] On the other hand, if the user's proficiency is low, that is, if it can be determined that the user is a beginner or unfamiliar with the operation, the proficiency-based information conversion unit 76 converts the operation information from predetermined poses and gestures to poses and gestures that can be inferred to correspond to the user's finger movements detected by the first detection unit 11. Such inference can be performed in the same way as the determination by the switching operation determination unit 71 described in the third modified example. Furthermore, if the user's proficiency level is low, the proficiency level-based information conversion unit 76 may perform various conversions to support the user with low proficiency, such as conversions to make the movements smoother, conversions to add nuances to the movements such as cuteness and approachability, and conversions to make the movements more distinct. In addition, the user may be able to set what kind of conversions are performed when the user's proficiency level is low. Furthermore, the user's proficiency level itself may also be able to set. Furthermore, if the proficiency-based information conversion unit 76 cannot determine the user's proficiency level, it may be pre-configured to select what kind of operation information to use. In addition to operation information A through operation information C described above, operation information with different conversion content may also be selected as an option. Moreover, there may be multiple types of operation information selected by the proficiency-based information conversion unit 76.

[0062] [Processing in communication systems] The proficiency-based information conversion unit 76 selects operation information based on the user's proficiency level and converts it as necessary. When operation information is converted, the converted operation information is sent to the target of the operation.

[0063] According to the fifth modified example of this embodiment described above, in the communication system 1E, suitable operation information is transmitted from the controller 10 according to the user's level of proficiency.

[0064] [Sixth Modification of the Embodiment] Figure 8 is a functional block diagram showing the configuration of the communication system 1F according to the sixth modified example. Communication system 1F has the same configuration as communication system 1 according to the above embodiment, but the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0065] [Communication system configuration] As shown in Figure 8, the communication system 1F includes a relay device 70 equipped with a gesture determination unit 77 and a gesture-specific information conversion unit 78. The gesture determination unit 77 determines whether the finger movement detected by the first detection unit 11 is a gesture that causes the puppet 50, which is the target of the operation, to perform a predetermined action. A gesture that causes a predetermined action to be performed is a pre-set gesture, similar to the communication target switching operation described in the third modified example, and is an action that causes a predetermined action to be performed in association with each gesture. For example, if a gesture of raising one finger is associated with a clapping motion, a gesture of raising two fingers is associated with a bowing motion, and a gesture of raising three fingers is associated with a waving motion, the gesture determination unit 77 can determine that the finger movement detected by the first detection unit 11 is a gesture that causes a predetermined action to be performed if it matches the corresponding gesture. Furthermore, when setting both a gesture to perform a predetermined operation and the communication target switching operation described in the third modified example, it is preferable to set different poses or gestures to prevent confusion.

[0066] When the gesture determination unit 77 determines that a gesture is one that performs a predetermined action, the gesture-specific information conversion unit 78 converts the action information corresponding to the movement of the fingers into action information that performs a predetermined action set in accordance with the gesture.

[0067] [Processing in communication systems] When the gesture determination unit 77 determines that the finger movement detected by the first detection unit 11 is a gesture that causes the puppet 50, which is the target of the operation, to perform a predetermined action, the gesture-specific information conversion unit 78 converts the action information corresponding to the finger movement into action information that causes the puppet 50, the target of the operation, to perform the predetermined action. If the action information has been converted, the converted action information is then transmitted to the puppet 50.

[0068] According to the sixth modified example of this embodiment described above, in the communication system 1F, the user can transmit action information to the puppet 50, which is the target of the operation, to perform a predetermined action associated with each gesture simply by performing a pre-set gesture. Therefore, by setting frequently performed or characteristic actions in association with gestures, remote communication can be performed more easily.

[0069] [Seventh modified example of the embodiment] Figure 9 is a functional block diagram showing the configuration of communication system 1G according to the seventh modified example. Communication system 1G has the same configuration as communication system 1 according to the above embodiment, but the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0070] [Communication system configuration] As shown in Figure 9, the communication system 1G includes a first relay device 81 and a second relay device 82 in the relay device 70. The first relay device 81 has a first transceiver 811 and communicates with the controller 10. The first relay device 81 is a relay device on the controller 10 side, and the controller 10 may be configured to have some or all of the functions of the first relay device 81. Alternatively, the first relay device 81 may be configured to have some or all of the functions of the controller 10. The second relay device 82 has a second transceiver unit 821, communicates with the puppet 50 which is the object being operated, and communicates with the first relay device 81 via a network. The second relay device 82 is a relay device on the puppet 50 side, and the puppet 50 may be configured to have some or all of the functions of the second relay device 82. Alternatively, the second relay device 82 may be configured to have some or all of the functions of the puppet 50.

[0071] Furthermore, if the drive unit 51 of the puppet 50 has a servo motor, the first transmitting / receiving unit 811 of the first relay device 81 may be configured to generate a control signal to operate the servo motor in accordance with the movement of the fingers detected by the first detection unit as operation information, and to transmit the generated control signal to the second relay device 82. Furthermore, the operation information A to operation information C of the fourth modified example described above may also be configured to be converted as appropriate in either the first relay device 81 or the second relay device 82.

[0072] According to the seventh modified example of this embodiment described above, the communication system 1G allows communication between the controller 10 and the puppet 50 via a network. Therefore, the degree of flexibility in the communication system configuration can be improved.

[0073] [Eighth modified example of the embodiment] Figure 10 is a functional block diagram showing the configuration of the communication system 1H according to the eighth modified example. Communication system 1H has the same configuration as communication system 1 according to the above embodiment, but the configuration of the relay device 70 is different. In the following description, parts that are the same or substantially the same as those already described are denoted by the same reference numerals and their description is omitted.

[0074] [Communication system configuration] As shown in Figure 10, the communication system 1H includes a first relay device 81 and a second relay device 82 in the relay device 70. The first relay device 81 has a first transmitting / receiving unit 811 and a communication status-based information conversion unit 812, and communicates with the controller 10. The communication state-based information conversion unit 812 converts operation information based on the communication state between the first relay device 81 and the second relay device 82. The operation information is the same as operation information A to operation information C in the fourth modified example described above. Based on the communication state between the first relay device 81 and the second relay device 82, the communication state-based information conversion unit 812 selects suitable operation information from operation information A to operation information C described above and converts it as necessary. When the communication state-based information conversion unit 812 selects suitable operation information, it is configured to consider either the quality state of the network's communication bandwidth or the processing load state of both the first relay device 81 and the second relay device 82. Furthermore, the user may configure what kind of conversion is performed depending on the communication status between the first relay device 81 and the second relay device 82. In addition to the operation information A to operation information C described above, operation information with different conversion content may also be selected. Moreover, there may be multiple types of operation information selected by the communication status-specific information conversion unit 812.

[0075] The second relay device 82 has a second transceiver unit 821, communicates with the puppet 50 which is the object being operated, and communicates with the first relay device 81 via the network. The second relay device 82 is a relay device on the puppet 50 side, and the puppet 50 may be configured to have the functions of the second relay device 82. Alternatively, the second relay device 82 may have some or all of the functions of the puppet 50.

[0076] [Processing in communication systems] The communication status information conversion unit 812 selects operation information based on the communication status between the first relay device 81 and the second relay device 82, and converts it as necessary. If the operation information is converted, the converted operation information is transmitted from the first transceiver unit 811 to the second transceiver unit 821.

[0077] According to the eighth modified example of this embodiment described above, in the communication system 1H, suitable operation information is transmitted from the controller 10 according to the communication status between the first relay device 81 and the second relay device 82. Therefore, stable and overall leveled data transmission and reception can be achieved.

[0078] [Embodiments and combinations of modified examples] Some or all of the embodiments and variations described so far may be combined and implemented. In this embodiment, an example of an asymmetric system is shown with the controller 10 and the puppet 50, but a symmetric system may also be configured. For example, both parties performing remote communication may be provided with a set of controllers 10 and puppets 50, with one controller 10 controlling the other puppet 50, and the other controller 10 controlling the first puppet 50. Alternatively, for example, both parties performing remote communication may be equipped with a controller 10, and each other's controllers 10 may be used as the target of operation. In such a case, the controller 10 may be configured to have some or all of the functions of the puppet 50. Similarly, both parties performing remote communication may be equipped with a puppet 50, and each other's puppets 50 may be used as the target of operation. In such a case, the puppet 50 may be configured to have some or all of the functions of the controller 10.

[0079] Furthermore, in the asymmetric system consisting of the controller 10 and the puppet 50 described in the embodiment, the controller 10 may be configured to have some or all of the functions of the puppet 50, or the puppet 50 may be configured to have some or all of the functions of the controller 10. For example, by equipping the controller 10 with a second detection unit 52 and the puppet 50 with a stimulus providing unit 12, when the controller 10 receives an external stimulus, the external stimulus received by the controller 10 is reproduced by the puppet 50. Furthermore, for example, by equipping the controller 10 with a drive unit 51 and the puppet 50 with a first detection unit 11, when the puppet 50 is operated, the operation on the puppet 50 is reproduced by the controller 10.

[0080] Furthermore, although the embodiments and their variations show examples in which the system includes one controller 10, it may include multiple controllers 10. Also, the number of controllers 10 and puppets 50 in the system may be the same or different. For example, the system may include multiple controllers 10 and one puppet 50, where the multiple controllers 10 all operate on the same puppet 50. Furthermore, the system may include, for example, multiple controllers 10 and multiple puppets 50, with each controller 10 controlling one or more puppets 50.

[0081] [Examples of communication system applications] The application examples of each communication system described above will now be explained. (1) A system in which puppets are placed in place of an audience Multiple puppets 50 are placed in the venue as spectators at sports or entertainment events, and a user wearing a controller 10 communicates remotely with each puppet 50. In such cases, the acquisition unit 53 of the puppet 50 acquires imaging and audio information from the surroundings of each puppet 50 and transmits it to the controller 10 that controls each puppet 50. As a result, information from the perspective of the audience in the venue is transmitted, and each user can output this information using a display device and an audio output device, thereby improving the sense of presence in remote communication. Furthermore, each controller 10 is equipped with an acquisition unit 53, and each puppet 50 is equipped with an audio output unit to output audio information, allowing for cheering during remote communication. Furthermore, if a puppet 50 consciously or accidentally comes into contact with another puppet 50 placed next to it or with any object in the venue, the external stimulus received by the puppet 50 is reproduced by the controller 10. As a result, actions such as putting arms around each other's shoulders or high-fiving can be performed through the puppets 50, creating a sense of unity and realism as if one were actually in the venue. Furthermore, by having multiple controllers 10 that control any puppet 50, the external stimuli received by that puppet 50 can be reproduced by multiple controllers 10. In such a case, for example, external stimuli given to any puppet 50 by an athlete or entertainer will be reproduced by multiple controllers 10. Therefore, with regard to so-called online audiences, athletes and entertainers can achieve remote communication with a large number of audience members, represented by controller 10 users, by only having to deal with one or a few puppets 50. Furthermore, as explained in the second modified example, by using the imaging and audio information acquired by the puppet 50 for control in the controller 10, it is possible to reproduce external stimuli that reflect the actual situation in the venue.

[0082] (2) A system in which a puppet is placed in the home of the person to whom the user is communicating remotely. A puppet 50 is placed in the home of a family member or friend who lives far away, and a user wearing a controller 10 communicates with the puppet 50 remotely. In such cases, it becomes possible to experience a new form of remote communication that feels like actual face-to-face interaction through the puppet 50. Specifically, for example, by remotely operating the puppet 50 before the interaction, one can observe the subject's condition, or by remotely operating the puppet 50 and simultaneously reproducing the external stimuli received by the puppet 50 with the controller 10, it becomes possible to interact while feeling mutual contact with the subject, such as shaking hands or high-fiving.

[0083] This disclosure is not limited to the embodiments and modifications described above, and any modifications and improvements that can achieve the purposes of this disclosure are included. [Explanation of symbols]

[0084] 1·1A·1B·1C·1D·1E·1F·1G·1H…Communication system, 10…Controller, 11…First detection unit, 12…Stimulus provision unit, 13…Control unit, 50·50A·50B·50C…Puppet, 51…Drive unit, 52…Second detection unit, 53…Acquisition unit, 70…Relay device, 71…Switching operation determination unit, 72…Target selection unit, 73…Transmit / receive unit, 74…Type acquisition unit, 75…Type-specific information conversion unit, 76…Proficiency-specific information conversion unit, 77…Gesture determination unit, 78…Gesture-specific information conversion unit, 81…First relay device, 82…Second relay device, 121…Vibration generation unit, 122…Temperature control unit, 521…Tactile sensor, 522…Temperature sensor, 811…First transmit / receive unit, 812…Communication status-specific information conversion unit.

Claims

1. A controller that is worn on the user's hand, The object to be operated remotely by the aforementioned controller, The system includes a relay device that relays information between the controller and the object being operated, The aforementioned controller, The first detection unit detects the movement of the user's fingers, A stimulation providing unit that provides external stimulation to the user's fingers, It includes a control unit that controls the stimulus providing unit, The target of the operation is, A drive unit that operates based on the received operation information, A second detection unit for detecting external stimuli to the object being operated on, It includes an acquisition unit that acquires audio information about the surroundings of the object being operated on, The relay device transmits the operation information based on the detection result by the first detection unit to the target of operation, and transmits to the controller the stimulus provision information that controls the stimulus provision unit to provide an external stimulus to the user based on the detection result by the second detection unit, and the voice information acquired by the acquisition unit. The communication system is characterized in that the control unit analyzes the change in volume corresponding to the audio information, and when the volume is increased, it strengthens the external stimulus provided by the stimulus provider compared to when the volume is not changing.

2. A controller that is worn on the user's hand, Multiple objects to be operated remotely by the aforementioned controller, The system includes a relay device that relays information between the controller and the plurality of objects to be operated, The aforementioned controller, The first detection unit detects the movement of the user's fingers, A stimulation providing unit that provides external stimulation to the user's fingers, It includes a control unit that controls the stimulus providing unit, Each of the aforementioned multiple targets for operation is, A drive unit that operates based on the received operation information, It includes a second detection unit that detects external stimuli to the object being operated on, The relay device is, A switching operation determination unit determines whether the finger movement detected by the first detection unit is a communication target switching operation, When the switching operation determination unit determines that the finger movement is a communication target switching operation, the target selection unit selects from the plurality of operation targets the operation target corresponding to the communication target switching operation as the communication target, A communication system characterized by having: an operation information based on the detection result of the first detection unit, which transmits the operation information based on the detection result of the first detection unit to the communication target; and a transmitting / receiving unit which, based on the detection result of the second detection unit, receives stimulus provision information from the communication target to control the stimulus provision unit and provide external stimuli to the user, and transmits this information to the controller.

3. A controller that is worn on the user's hand, The object to be operated remotely by the aforementioned controller, The system includes a relay device that relays information between the controller and the object being operated, The aforementioned controller, The first detection unit detects the movement of the user's fingers, A stimulation providing unit that provides external stimulation to the user's fingers, It includes a control unit that controls the stimulus providing unit, The target of the operation is, A drive unit that operates based on the received operation information, It includes a second detection unit that detects external stimuli to the object being operated on, The relay device is, A type acquisition unit that acquires type information indicating the type of the operation target, A type-specific information conversion unit converts the operation information based on the detection result by the first detection unit based on the type information, A communication system characterized by having a transmitting / receiving unit that transmits the operation information converted by the type-specific information conversion unit to the target of operation, and transmits stimulus provision information to the controller that controls the stimulus provision unit to provide an external stimulus to the user based on the detection result by the second detection unit.

4. A controller that is worn on the user's hand, The object to be operated remotely by the aforementioned controller, The system includes a relay device that relays information between the controller and the object being operated, The aforementioned controller, The first detection unit detects the movement of the user's fingers, A stimulation providing unit that provides external stimulation to the user's fingers, It includes a control unit that controls the stimulus providing unit, The target of the operation is, A drive unit that operates based on the received operation information, It includes a second detection unit that detects external stimuli to the object being operated on, The relay device is, A gesture determination unit that determines whether the finger movement detected by the first detection unit is a gesture that causes the target to perform a predetermined action, When the gesture determination unit determines that the movement of the fingers is a gesture, the gesture-specific information conversion unit converts the action information corresponding to the movement of the fingers into action information that causes the target to perform the predetermined action. A communication system characterized by having a transmitting / receiving unit that transmits the operation information converted by the gesture-specific information conversion unit to the target of operation, and transmits stimulus provision information to the controller that controls the stimulus provision unit to provide an external stimulus to the user based on the detection result by the second detection unit.

5. A controller that is worn on the user's hand, The object to be operated remotely by the aforementioned controller, The system includes a relay device that relays information between the controller and the object being operated, The aforementioned controller, The first detection unit detects the movement of the user's fingers, A stimulation providing unit that provides external stimulation to the user's fingers, It includes a control unit that controls the stimulus providing unit, The target of the operation is, A drive unit that operates based on the received operation information, It includes a second detection unit that detects external stimuli to the object being operated on, The relay device is, A first relay device that communicates with the controller, A second relay device that communicates with the target of the operation and communicates with the first relay device via a network, The first relay device is A communication state-specific information conversion unit converts the operation information based on the detection result by the first detection unit based on the communication state between the first relay device and the second relay device, The system includes a first transmitting / receiving unit that transmits the operation information converted by the communication state-specific information conversion unit to the second relay device, The second relay device transmits the operation information received from the first relay device to the target of operation, and transmits stimulus provision information to the controller via the first relay device, which controls the stimulus provision unit to provide external stimuli to the user based on the detection result by the second detection unit. The communication system is characterized in that the communication state-specific information conversion unit selects one of the following based on the communication state: first operation information obtained by converting the operation information based on the detection result of the first detection unit into a predetermined pose or gesture according to the time-series data of the user's fingers detected by the first detection unit, and second operation information obtained by converting the detection result of the first detection unit into time-series control data for operating the drive unit; and generates the selected one of the two pieces of information as the operation information to be transmitted to the second relay device by the first transmitting / receiving unit.

6. In the communication system described in Claim 5, The aforementioned drive unit has a servo motor, The communication system is characterized in that the second operation information is a control signal that operates the servo motor in accordance with the movement of the fingers detected by the first detection unit.

7. In the communication system according to any one of claims 1 to 6, The second detection unit includes a tactile sensor, The relay device transmits the stimulus provision information based on the detection result from the tactile sensor to the controller. The aforementioned stimulus-providing unit includes a vibration-generating unit, A communication system characterized in that the control unit operates the vibration generating unit based on the stimulus provision information.

8. In the communication system according to any one of claims 1 to 7, The second detection unit includes a temperature sensor, The relay device transmits the stimulus provision information based on the detection result from the temperature sensor to the controller. The aforementioned stimulus-providing unit includes a temperature control unit, A communication system characterized in that the control unit operates the temperature control unit based on the stimulus information.

9. In the communication system according to any one of claims 1 to 8, The relay device further, A proficiency-based information conversion unit determines the user's proficiency level based on the user's finger movements detected by the first detection unit, and converts the operation information based on the detection results by the first detection unit based on the determined user proficiency level. A communication system characterized by having a transmitting / receiving unit that transmits the operation information converted by the proficiency level-based information conversion unit to the target of operation, and transmits stimulus provision information to the controller that controls the stimulus provision unit to provide an external stimulus to the user based on the detection result by the second detection unit.

10. In the communication system according to any one of claims 1 to 9, The object of the operation is a puppet having a head and a pair of arms. The communication system is characterized in that the target of operation has a drive unit that moves at least a portion of the head and the pair of arms based on received operation information.

Images