Pseudo-tactile presentation system, control program, and control method

Abstract

To enable providing a new pseudo contact presentation system that presents pseudo contact.SOLUTION: A pseudo contact presentation system (10) comprises: a communication robot 12; and a HMD 20, displays a plurality of virtual agents (70) on the HMD, and displays an animation in which predetermined actions are individually or simultaneously performed. An arm of a robot corresponding to a virtual agent is operated in connection with the predetermined action. In other words, the pseudo contact presentation system presents a visual stimulus to a user 60 wearing the HMD, and presents a contact stimulus associated with this. Thus, the user obtains a body feeling in which he / she is contacting with the virtual agent. In other words, pseudo contact is presented to the user.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a pseudo-tactile feedback system, a control program, and a control method, and more particularly to a pseudo-tactile feedback system, a control program, and a control method that use, for example, an acting body that presents a tactile stimulus to a user and a head-mounted display that presents a visual stimulus to the user. [Background technology]

[0002] An example of this type of conventional pseudo-touch presentation system is disclosed in Patent Document 1. According to the pseudo-touch presentation system disclosed in Patent Document 1, a communication robot presents a touch stimulus to a user by, for example, hugging the user with arms long enough to wrap around the user's back. A virtual agent whose state changes substantially in sync with the hugging action of the communication robot is displayed on an HMD (Head Mounted Display) worn on the user's head. The communication robot hugs the user back, providing the user with a touch stimulus, and further a visual stimulus is provided to the user by the virtual agent of the HMD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2019-45928 Summary of the Invention [Problem to be solved by the invention]

[0004] In the pseudo-touch presentation system disclosed in the above-mentioned Patent Document 1, one virtual agent is assigned to one communication robot, and the movement of the virtual agent is synchronized with the movement of the communication robot. In other words, the configuration for presenting a touch stimulus to a user using a communication robot and a virtual agent is simple. Patent Document 1 does not take into consideration presenting a touch stimulus to a user by a plurality of virtual agents.

[0005] Therefore, a primary object of the present invention is to provide a novel pseudo-tactile presentation system, a control program, and a control method.

[0006] Another object of the present invention is to provide a pseudo-tactile presentation system, a control program, and a control method that are capable of providing complex tactile stimuli. [Means for solving the problem]

[0007] A first invention is a contact stimulus presentation device including a plurality of stimulus presentation units that act on a user to present the user with a contact stimulus, an allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units, and a head-mounted display device that is worn on the user's head, displays the plurality of virtual agents allocated by the allocation means, and operates some or all of the plurality of virtual agents to present a visual stimulus to the user, wherein the contact stimulus presentation device: selectively executing a first process in which, in synchronization with one virtual agent performing a first predetermined action toward a user, the stimulus presentation unit to which the virtual agent is assigned executes an action corresponding to the first predetermined action, or a second process in which two or more of the plurality of virtual agents simultaneously perform a second predetermined action toward the user, and each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned executes an action corresponding to the second predetermined action; , a pseudo-tactile presentation system.

[0008] The second invention is dependent on the first invention, Head-Mounted Display Device The present invention displays some or all of the virtual agents as animations that indicate actions for a user, and the stimulation presentation unit includes a movable unit, and the contact stimulation presentation unit presents a touch stimulation to the user. Device The moving part is operated so that a touch stimulus is presented to the user in association with animation of actions of some or all of the multiple virtual agents according to the movement of the moving part.

[0009] A third invention is a control program executed by a computer of a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to present the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, the control program causing the computer to control the plurality of stimulus presentation units of the touch stimulus presentation device to present touch stimuli to the user. Stimulus presentation unit control means an allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units; , A part or all of the multiple virtual agents assigned in this method Head-mounted display devices View The display control means functions as an execution means for selectively executing a first process for causing a stimulus presentation unit to which a virtual agent is assigned to perform an action corresponding to the first predetermined action in synchronization with one virtual agent performing a first predetermined action toward a user, or a second process for causing two or more of the plurality of virtual agents to simultaneously perform a second predetermined action toward a user, and causing each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned to perform an action corresponding to the second predetermined action. , a sensory stimulation presentation program.

[0010] A fourth invention is a control method for a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to present the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, the method including the steps of controlling the plurality of stimulus presentation units of the touch stimulus presentation device to present touch stimuli to the user, and assigning a virtual agent to each of two or more of the plurality of stimulus presentation units. , Some or all of the assigned virtual agents Head-mounted display devices Steps to display and selectively executing a first process in which, in synchronization with one virtual agent performing a first predetermined action toward a user, the stimulus presentation unit to which the virtual agent is assigned performs an action corresponding to the first predetermined action, or a second process in which two or more of the virtual agents simultaneously perform a second predetermined action toward the user, and each of the stimulus presentation units to which two or more of the virtual agents are assigned performs an action corresponding to the second predetermined action. , a control method. Effect of the Invention

[0011] According to the present invention, some or all of a plurality of virtual agents displayed on a head-mounted display device mounted on the user's head are operated, and in association with the operation, a stimulus presentation unit corresponding to some or all of the plurality of virtual agents is operated, so that a visual stimulus due to the operation of the virtual agent is presented, and in association therewith, a touch stimulus is presented by the stimulus presentation unit. In other words, a bodily sensation as if touching the virtual agent can be obtained. Thus, a new pseudo-touch presentation system, control program, and control method for presenting pseudo-touch can be presented.

[0012] The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the electrical configuration of a pseudo-tactile presentation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a front view of the external configuration of the robot shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing the electrical configuration of the robot shown in FIG. [Figure 4] FIG. 4 is a side view of an example of a state in which the pseudo tactile presentation system shown in FIG. 1 is used. [Diagram 5] FIG. 5 is a diagram showing an example of two virtual agents displayed on the HMD shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of three virtual agents displayed on the HMD shown in FIG. [Figure 7] FIG. 7 is a diagram showing another example of two virtual agents displayed on the HMD shown in FIG. [Figure 8] FIG. 8 is a diagram showing another example of the three virtual agents displayed on the HMD shown in FIG. [Figure 9] FIG. 9 is a diagram showing an example of a memory map of a RAM built into the server shown in FIG. [Figure 10] FIG. 10 is a flowchart showing a part of an example of the first pseudo-touch process of the CPU built in the server shown in FIG. [Figure 11] FIG. 11 is a flowchart showing another part of the first pseudo-contact processing by the CPU built in the server shown in FIG. 1, which follows FIG. [Figure 12] FIG. 12 is a flowchart showing a part of an example of the second pseudo-touch processing of the CPU built in the server shown in FIG. [Figure 13]FIG. 13 is a flowchart showing another part of the second pseudo-touch processing by the CPU built in the server shown in FIG. 1, which follows FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] 1, a pseudo-touch presentation system (hereinafter referred to as "system") 10 of this embodiment includes a robot 12 for presenting a physical or bodily stimulus to a user (person or participant), that is, a stimulus given to the user's sense of touch or a stimulus given to the user by contact (hereinafter referred to as "touch stimulus"). In this embodiment, the robot 12 is a communication robot that has an action of presenting a touch stimulus and generates a sound as necessary.

[0015] Fig. 2 shows an example of the external appearance of a robot 12 seen from the front as a tactile stimulus presentation device used in the embodiment of Fig. 1. However, any tactile stimulus presentation device having a different appearance and structure can be used.

[0016] The robot 12 includes three rod-shaped frames 120a, 120b, and 120c. The frames 120a, 120b, and 120c may be made of metal such as aluminum, or resin such as fiber-reinforced plastic or engineering plastic, and are basically designed with a cross-sectional shape and thickness that provide safe strength. For example, a rectangular column or rectangular tube frame may be used.

[0017] Frame 120a constitutes the bottom of robot 12, extends horizontally, and is attached to the floor or floor surface. Frame 120b constitutes the torso of robot 12, extends vertically, and has its lower end fixed to the center or nearly the center of frame 120a. Frame 120c constitutes the shoulder of robot 12, extends horizontally, and is fixed to the upper part of frame 120b.

[0018] The frame 120c is provided with robot arms (hereinafter simply referred to as "arms") 122, 124, and 126 extending from both ends thereof. Each of the arms 122, 124, and 126 has a structure that imitates a human arm, and each has joints that correspond to an elbow joint and a shoulder joint. Therefore, in the following description, for convenience, names of parts of a human arm may be used.

[0019] The arms 122 and 124 are provided at the left end of the frame 120c when the robot 12 is viewed from the front, and the arm 126 is provided at the right end of the frame 120c when the robot 12 is viewed from the front. Hereinafter, when the robot 12 or parts thereof are described using directions, up, down, left and right when the robot 12 is viewed from the front will be used.

[0020] The arm 122 includes a frame 122a corresponding to the forearm and a frame 122b corresponding to the upper arm, and further includes a joint (corresponding to an elbow joint) 122c connecting frames 122a and 122b, and a joint (corresponding to a shoulder joint) 122d connecting frames 122b and 120c.

[0021] Arm 124 includes a frame 124a corresponding to the forearm and a frame 124b corresponding to the upper arm, and further includes a joint (corresponding to an elbow joint) 124c connecting frames 124a and 124b, and a joint (corresponding to a shoulder joint) 124d connecting frames 124b and 120c.

[0022] However, as can be seen from FIG. 2, arm 122 is connected to the lower surface of the left end of frame 120c, and arm 124 is connected to the upper surface of the left end of frame 120c.

[0023] The arm 126 includes a frame 126a corresponding to the forearm and a frame 126b corresponding to the upper arm, and further includes a joint (corresponding to an elbow joint) 126c connecting the frames 126a and 126b, and a joint (corresponding to a shoulder joint) 126d connecting the frames 126b and 120c. The arm 126 is connected to the lower surface of the right end of the frame 120c.

[0024] As an example, the joints 122c, 124c, and 126c can each open the forearm to a maximum of 90° relative to the upper arm, so that the arms 122, 124, and 126 can be bent or extended by controlling the angles of the joints 122c, 124c, and 126c.

[0025] Also, as an example, the joints 122d, 124d, and 126d can open the upper arm up to 150° with respect to the plane where the joints 122d, 124d, and 126d are connected to the frame 120c. However, the angle when the frames 122b, 124b, and 126b constituting the upper arm are perpendicular to the plane where the joints 122d, 124d, and 126d are connected to the frame 120c is set to 0°. Therefore, by controlling the angles of the joints 122d, 124d, and 126d, the arms 122, 124, and 126 can be raised and lowered, or extended forward and backward.

[0026] Furthermore, in this embodiment, actuators are incorporated in the joints 122c, 122d, 124c, 124d, 126c and 126d so that the robot 12 can shake hands with the user 60 (see FIG. 4) with each of the arms 122, 124 and 126.

[0027] The tactile stimulus presentation device, i.e., the robot 12, is connected to a server 16 via a network 14 such as the Internet or a telephone communication line. The server 16 is a general-purpose computer such as a PC, a PDA, a smartphone, or a tablet terminal.

[0028] Although not shown in the figure, the server 16 includes an input device such as a touch panel and / or a keyboard, and an operator can operate such an input device. Programs and data for controlling the operation of such a server 16 are stored in a memory (e.g., HDD and RAM) 16b built into the server 16, and the overall operation of the server 16 is controlled by a CPU 16a also built in the server 16. This CPU 16a controls the movement and speech of the robot 12 through a CPU 36 (described later) of the robot 12 in accordance with the above-mentioned operation of the operator, and further controls the display of a virtual agent by the HMD 20 described later. For this reason, the server 16 includes a communication module 16c, and the CPU 16a is communicably connected to the network 14, i.e., the robot 12 and the HMD 20, via the communication module 16c, by wire or wirelessly.

[0029] 1 also includes an HMD 20, which is a head-mounted display device, and this HMD 20 is worn on the head 64 of a user 60 who receives tactile stimuli presented by the robot 12 (see FIG. 4). In the embodiment, an Oculus Rift (product name) manufactured by Oculus Inc. is used as this HMD 20, and a position detection device 18a such as an Oculus sensor (product name) is used to accurately detect the position of this HMD 20, i.e., the head position of the user 60.

[0030] That is, the output of the position detection device 18a is provided to the server 16. The position detection device 18a functions as a head position measurement unit, and the Oculus sensor used as this position detection device 18a can accurately detect the position of the head 64 of the user 60 by tracking the position of the HMD 20 (Oculus Rift) by combining a gyro sensor and acceleration.

[0031] As described above, the robot 12 is fixed to the floor, and the position of the HMD 20 in the real space, i.e., the head position of the user 60, can be detected by setting or registering the initial position of the HMD 20 relative to the robot 12. Therefore, in this embodiment, the CPU 16a controls the positions of the virtual agent and the viewpoint (virtual camera) in the virtual space using the positional relationship between the head positions of the robot 12 and the user 60 in the real space. Since the HMD 20 displays a so-called first-person viewpoint image (in this embodiment, the virtual agent), the position of the virtual camera corresponds to the head position of the user 60 in the virtual space. In addition, the gaze point of the virtual camera is set to a predetermined position of the robot 12 in the virtual space. For example, the predetermined position is set to the center of the robot 12 in the virtual space. This is just one example, and other positions may be adopted as long as virtual agents of various sizes fit into the image displayed on the HMD 20.

[0032] However, the robot 12 itself is not drawn in the virtual space, and the position of the robot 12 and the like are only used when arranging a virtual agent in the virtual space or when determining the position of a virtual camera in the virtual space. For example, the position of the robot 12 (the position of the center of the frame 120a) is set to the origin in the three-dimensional virtual space, the direction parallel to the frames 120a and 120c of the robot 12 is set to the X-axis direction, the direction parallel to the frame 120b is set to the Z-axis direction, and the direction perpendicular to both the X-axis and Z-axis is set to the Y-axis direction.

[0033] The server 16 is further connected to a microphone 18b for collecting environmental sounds and the voice of the user 60, a camera 18c for capturing images of the environment and the user 60, and the like. Voice data collected by the microphone 18b is input to the server 16, and the server 16 can use the voice data to perform voice recognition. Image data from the camera 18c is input to the server 16, and the server 16 can process the image data to perform, for example, personal identification.

[0034] In this embodiment, the robot 12 is adapted to be able to converse with a user 60. To this end, topics necessary for such a conversation with the user 60 and scenarios corresponding to those topics are set in advance in the memory 16b of the server 16. However, as described above, a remote operator can also converse with the user 60 through the server 16.

[0035] The above-mentioned HMD 20 can display high-resolution stereoscopic images with a 110° field of view at a refresh rate of, for example, 90 Hz.

[0036] The HMD 20 displays virtual agents 70 (70a, 70b, 70c) as shown in Fig. 5 to Fig. 8 described later, based on image data transmitted from the server 16. This provides a visual and / or auditory stimulus (visual stimulus) to the user 60.

[0037] For this reason, image data of a plurality of virtual agents 70 of different genders and ages is set in advance in the memory 16b of the server 16. The image data includes not only still image data of the virtual agent 70 standing upright, but also animation data of the virtual agent 70 performing predetermined actions (in this embodiment, greeting, shaking hands, and walking).

[0038] Then, the server 16 identifies the user 60 who will cooperate (interact) with the robot 12, and then selects a virtual agent 70 of a gender and age that matches the identified user 60, and sends the image data thereof to the HMD 20 via the network 14. This enables the HMD 20 to display the virtual agent 70 that matches the user 60. However, the virtual agent 70 may be registered by the user 60.

[0039] In this embodiment, multiple virtual agents 70 can be assigned to the robot 12. In this embodiment, the number of virtual agents 70 assigned to the robot 12 is equal to or less than the number of arms of the robot 12. Thus, one, two or three virtual agents 70 are assigned to correspond to the three arms 122, 124 and 126 of the robot 12.

[0040] The user 60 can decide how many virtual agents 70 to assign to the robot 12. However, when the system 10 is applied to any application such as a game, the number of virtual agents 70 can be determined either fixedly or dynamically according to the scenario of the application.

[0041] When one virtual agent 70 is assigned to the robot 12, one virtual agent 70 is assigned to the arms 122 and 126. When two virtual agents 70 are assigned to the robot 12, one of the two virtual agents 70 is assigned to the arm 122 and the other of the two virtual agents 70 is assigned to the arm 126. When three virtual agents 70 are assigned to the robot 12, one of the three virtual agents 70 is assigned to the arm 122, one of the remaining two virtual agents 70 is assigned to the arm 124, and the other of the remaining two virtual agents 70 is assigned to the arm 126.

[0042] However, this is merely an example and is not necessarily limiting. In addition, the multiple assigned virtual agents 70 are displayed on the HMD 20 so as not to overlap each other.

[0043] In addition, with regard to the case where one virtual agent 70 is assigned to the robot 12, this is not an essential content of the present invention, and since it overlaps with the content disclosed in JP 2019-45928, which was previously filed by the applicant and has already been published, a detailed explanation will be omitted.

[0044] Although omitted in Fig. 2, a speaker and a microphone are attached to the upper part of the robot 12 (for example, the upper end part of the frame 120b), so that the robot 12 can speak to the user 60 and hear the speech of the user 60. XIMERA (product name: ATR Speech and Language Research Laboratories) was adopted as the speech synthesis software for speaking.

[0045] Fig. 3 is a block diagram showing an example of the electrical configuration of the robot 12. As shown in Fig. 3, the robot 12 includes a CPU 36 that is responsible for overall control of the robot 12. The CPU 36 is connected to a communication module 40 through a bus 38, and therefore the CPU 36 is communicably connected to the network 14, i.e., the server 16, via the communication module 40 in a wired or wireless manner.

[0046] The CPU 36 can also access a memory 42 via the bus 38, and provides command values ​​to an actuator control circuit 44 via the bus 38 in accordance with the programs and data set in the memory 42, thereby controlling the operation of each of the actuators A1-An. The actuator control circuit 44 generates a number of pulse powers according to the command values ​​provided by the CPU 36, and provides them to the corresponding stepping motors to drive each of the actuators A1-An.

[0047] However, in addition to the actuator using such a stepping motor, any actuator such as an actuator using a servo motor or a fluid actuator can be used.

[0048] Here, the actuators A1-An are actuators (stepping motors) of the joints 122c, 122d, 124c, 124d, 126c, and 126d. Therefore, the actuator control circuit 44 also functions as a joint control unit.

[0049] The sensor I / F (interface) 46 is connected to the CPU 36 via the bus 38 , and receives outputs from the tactile sensor 48 and the camera 50 .

[0050] The tactile sensor 48 is, for example, a touch sensor, and constitutes a part of the sense of touch of the robot 12. In other words, the tactile sensor 48 detects whether or not a human or other object has touched the robot 12. For example, the tactile sensor 48 is provided on each of the forearms (frames 122a, 124a, and 126a) of the arms 122, 124, and 126 of the robot 12, and can detect whether or not the hand of the user 60 has touched the arms 122, 124, and 126 of the robot 12.

[0051] The tactile sensor 48 also functions as a pressure sensor. For example, the tactile sensor 48 is formed of a polymer piezoelectric sensor sheet, and a plurality of sensor sheets are appropriately distributed on the inner surface of the arm 28. The tactile sensor 48 can detect the pressure applied to the forearm (frames 122a, 124a, 126a) of the robot 12 when shaking hands with the user 60.

[0052] The output (detection data) from the tactile sensor 48 is provided to the CPU 36 via the sensor I / F 46. Therefore, the CPU 36 can detect that a human or other object has touched the robot 12 (and the strength or pressure of the touch) and that the human or other object has left the robot 12.

[0053] Furthermore, the pressure data detected by the tactile sensor 48 is output from the CPU 36 and provided to the server 16 via the communication module 40 and the network 14 .

[0054] The camera 50 is an image sensor and constitutes a part of the vision of the robot 12. That is, the camera 50 is used to detect an image or an image seen by the eyes of the robot 12. In this embodiment, data (image data) corresponding to an image (moving image or still image) captured by the camera 50 is provided to the CPU 36 via the sensor I / F 46. The CPU 36 not only detects changes in the captured image, but also transmits the image data to the server 16 via the communication module 40 and the network 14 (FIG. 1). The server 16 then outputs the received image data to a monitor (not shown). Thus, the image captured by the camera 50 is displayed on the monitor.

[0055] The speaker 54 and the microphone 56 are also connected to the input / output I / F 52. The speaker 54 outputs sound when the robot 12 speaks. When an operator of the server 16 (hereinafter sometimes referred to as a "remote operator") speaks directly, the sound is output through the network 14, the communication module 40, and the input / output I / F 52. Specifically, when the remote operator speaks through the microphone 56, corresponding voice data is provided from the server 16 to the CPU 36 via the network 14. Then, the CPU 36 outputs the voice data from the speaker 54 via the input / output I / F 52.

[0056] The microphone 56 is a sound sensor and constitutes part of the hearing of the robot 12. The microphone 56 has directionality and is mainly used to detect the voice of a user 60 (person) who interacts (communicates) with the robot 12.

[0057] The memory 42 of the robot 12 is, for example, a RAM and a HDD, and is preset with a voice recognition program and a voice synthesis program such as the above-mentioned XIMERA. The voice recognition program is a program for the CPU 36 to recognize the contents of speech from a human being to the robot 12, which is input through the microphone 56, and the CPU 36 recognizes the contents of speech from the human being by, for example, DP matching or Hidden Markov Model (HMM).

[0058] The CPU 36 recognizes the speech input by the remote operator through the microphone 56 according to the speech recognition program. For example, the speech can be detected using an acoustic model such as the above-mentioned HMM method or a deep neural network (DNN).

[0059] In addition to directly outputting the voice of the remote operator from the speaker 54, the robot 12 can be made to speak in a synthetic voice. When an instruction to speak in a synthetic voice from the speaker 54 is input from the server 16, the CPU 36 outputs the synthetic voice to the speaker 54 in accordance with the synthetic voice data provided from the server 16.

[0060] In this embodiment, basically, all the operations of the robot 12 are given from the server 16 operated by an operator (not shown). To this end, the CPU 36 gives sensor data acquired through the sensor I / F 46 and voice data and image data acquired through the input / output I / F 52 to the CPU 16a of the server 16 via the communication module 40 and the network 14. The server 16 can obtain the data and know the state of the robot 12.

[0061] However, the server 16 may also autonomously control the movement of both the virtual agent 70 and the robot 12 using position and pressure information from the position sensing device 18a.

[0062] FIG. 4 shows an example of a state in which a user 60 uses the system 10, viewed from the side. As shown in FIG. 4, the user 60 wears the HMD 20 and stands directly in front of the robot 12. That is, the robot 12 and the user 60 face each other. The robot 12 and the user 60 can talk or converse with each other. When the user 60 approaches the robot 12 and the distance d between the robot 12 and the user 60 becomes equal to or smaller than a predetermined distance df, the robot 12 can touch the user 60 with the arms 122, 124, and 126. In this case, the user 60 can touch at least one of the arms 122, 124, and 126 of the robot 12. As an example, the predetermined distance df is 70 cm-1 m, and is determined by the length of the arms 122, 124, and 126.

[0063] As described above, a plurality of virtual agents 70 can be assigned to the robot 12. When two virtual agents 70 are assigned to the robot 12, one virtual agent 70 is placed at a position overlapping the arm 122 of the robot 12, and the other virtual agent 70 is placed at a position overlapping the arm 126 of the robot 12. In the example shown in Fig. 5, the virtual agents 70a and 70b are placed in the virtual space such that the left arm of the virtual agent 70a overlaps the arm 122, and the right arm of the virtual agent 70b overlaps the arm 126. In other words, when two virtual agents 70 are assigned to the robot 12, the arm 122 of the robot 12 corresponds to the left arm of the virtual agent 70a, and the arm 126 of the robot 12 corresponds to the right arm of the virtual agent 70b.

[0064] Furthermore, when three virtual agents 70 are assigned to the robot 12, one of the three virtual agents 70 is arranged at a position overlapping the arm 122 of the robot 12, one of the remaining two virtual agents 70b is arranged at a position overlapping the arm 126 of the robot 12, and the other of the remaining two virtual agents 70c is arranged at a position overlapping the arm 124 of the robot 12. In the example shown in FIG. 6, the right arm of the virtual agent 70a is arranged at a position overlapping the arm 122 of the robot 12, the right arm of the virtual agent 70b is arranged at a position overlapping the arm 126 of the robot 12, and the left arm of the virtual agent 70c is arranged at a position overlapping the arm 124 of the robot 12. In other words, when three virtual agents 70 are assigned to the robot 12, the arm 122 of the robot 12 corresponds to the right arm of the virtual agent 70a, the arm 124 of the robot 12 corresponds to the left arm of the virtual agent 70c, and the arm 126 of the robot 12 corresponds to the right arm of the virtual agent 70b.

[0065] Therefore, the CPU 16a of the server 16 can execute a process (hereinafter referred to as a "first pseudo contact process") in which each of the multiple virtual agents 70 individually performs a first predetermined action toward a user 60 (person or participant) approaching the robot 12, and also operates the arm 122, 124 or 126 to which the virtual agent 70 is assigned in synchronization with the virtual agent 70 performing the first predetermined action toward the user 60. However, the action of the arm 122, 124 or 126 is an action corresponding to the first predetermined action.

[0066] Furthermore, the CPU 16a of the server 16 can execute a process (hereinafter referred to as a "second pseudo contact process") of bringing a plurality of virtual agents 70 close to the user 60 in the virtual space, causing the plurality of virtual agents 70 to simultaneously perform a second predetermined action toward the user 60, and operating the arms 122 and 124, or the arms 122, 124 and 126, to which the plurality of virtual agents 70 are assigned. However, the actions of the arms 122, 124 and 126 are actions corresponding to the second predetermined action.

[0067] In this embodiment, the first predetermined action and the second predetermined action are handshakes. As an example, the first pseudo-contact process is executed in a situation where the user 60 individually shakes hands with a plurality of virtual agents 70 who are idols at a handshake event venue set in a virtual space. On the other hand, the second pseudo-contact process is executed in a situation where the user 60 is an idol, is surrounded by a plurality of virtual agents 70 who are fans, and is asked to shake hands by each virtual agent 70.

[0068] Therefore, which of the first pseudo-touch process and the second pseudo-touch process is to be executed is determined by the operator of the server 16 or a remote operator, or is determined by the scenario (context) of the application to which the system 10 is applied. However, the scenario of the application may be variably determined depending on which option the user 60 selects.

[0069] For example, in the first pseudo-contact process, the user 60 interacts with one, two or three virtual agents 70 assigned to the robot 12, and when the user 60 approaches the robot 12 to a predetermined distance df at which the user 60 can shake hands with the robot 12, the CPU 16a of the server 16 transmits a signal to both the virtual agent 70 and the robot 12, causes the HMD 20 to display an animation of the virtual agent 70 shaking hands with the robot 12, and causes the robot 12 to start shaking hands. That is, a contact action toward the user 60 is executed. Therefore, the user 60 can grasp the arm 122, 124 or 126 of the robot 12 to shake hands, and feels as if he or she is shaking hands with the virtual agent 70 assigned to the grasped arm 122, 124 or 126. That is, a pseudo-contact is presented to the user 60. However, among the multiple virtual agents 70, only the virtual agent 70 assigned to the arm 122, 124 or 126 that is closest to the user 60 displays the animation of the handshake.

[0070] For example, in a case where two virtual agents 70 are assigned to the robot 12, when the user 60 approaches the virtual agent 70a, the virtual agent 70a shakes hands as shown in Fig. 7, and in relation to (or in synchronization with) this, the arm 122 corresponding to the virtual agent 70a is operated. That is, the robot 12 uses the arm 122 to shake hands. Therefore, the user 60 grasps the arm 122 of the robot 12 to shake hands while watching the virtual agent 70a shaking hands on the HMD 20. That is, a visual stimulus in the form of an animation of the virtual agent 70 displayed on the HMD 20 is presented to the user 60, and a tactile stimulus in the form of the movement of the robot 12 is given to the user 60.

[0071] Although not shown in the figures, by making it possible to detect the user's hand movement from an image from the camera 18c, etc., when the user shakes hands, an image of the user's hand can also be displayed on the HMD 20. However, even if the user's hand movement is not detected, it is possible to know from the output of the tactile sensor 48 that the user's hand or the like has touched the arm of the robot 12, so an image of the user's hand may be displayed on the HMD 20 when it is detected that the user's hand or the like has touched the arm of the robot 12. The same applies below when users shake hands.

[0072] Also, although not shown in the figures, in the first pseudo-contact processing, when three virtual agents 70 are assigned to the robot 12, the virtual agent 70 that is closest to the user 60 (viewpoint) in the virtual space performs the handshake action, and the robot 12 performs the handshake action using the arm 122, 124 or 126 corresponding to that virtual agent 70.

[0073] In the first pseudo-contact processing, when one virtual agent 70 is assigned to the robot 12, when the user 60 (viewpoint) approaches the virtual agent 70 in the virtual space, the virtual agent 70 shakes hands, and the robot 12 performs the handshake action using the arm 122 or 124 corresponding to the virtual agent 70.

[0074] In this embodiment, the intimacy between the user 60 and each virtual agent 70 is set, and in the first pseudo-contact process, when the user 60 shakes hands with a virtual agent 70, the intimacy of the virtual agent 70 with respect to the user 60 is improved. For example, the intimacy of a virtual agent 70 with which the user has never shook hands is set to 0, and the intimacy is increased by 1 every time the user shakes hands. For example, the intimacy can be used as a condition for selecting a virtual agent 70, and can also be used to determine the content of a speech when greeting. When selecting a virtual agent 70, a virtual agent 70 with a high or low intimacy can be selected. Also, a user 60 with an intimacy of 0 can be greeted with "Nice to meet you," and a user 60 with an intimacy of 1 or more can be greeted with "Hello" or "It's been a while." These are just examples, and the intimacy may be used for other purposes.

[0075] In the first pseudo-contact process, at least one of the virtual agents 70 greets the user 60, and the virtual agents 70 converse individually with the user 60. When the virtual agent 70 greets the user 60, the user 60 is guided to approach the robot 12. When the user 60 converses individually with the virtual agents 70, the virtual agents 70 speak to the user 60 in turn, and when each of the virtual agents 70 finishes speaking, the user 60 speaks next, as an example.

[0076] In addition, since the purpose of the present invention is to allow the user 60 to virtually experience the movements of the virtual agent 70 through the movements of the robot 12, that is, to present a simulated contact, there is no need for interaction between the user 60 and the virtual agent 70.

[0077] The first pseudo-contact process is terminated when the user 60 performs a specific action such as moving away from the robot 12, waving at the robot 12, or removing the HMD 20, or when the user 60 utters a specific message such as "goodbye" or "bye bye." The same applies to the termination of the second pseudo-contact process. It is possible to detect that the user 60 has removed the HMD 20 based on the image from the camera 18c.

[0078] In the second pseudo-contact process, one, two, or three virtual agents 70 assigned to the robot 12 approach the user 60, and the user 60 shakes hands with the one, two, or three virtual agents 70 assigned to the robot 12. When the user 60 approaches the robot 12 to a predetermined distance df at which the user 60 can shake hands, the CPU 16a of the server 16 starts the second pseudo-contact process. This is because, in the second pseudo-contact process, the virtual agent 70 is brought close to the user 60 (viewpoint) in the virtual space to shake hands, and therefore the user 60 needs to be close enough to the robot 12 to shake hands.

[0079] In addition, since the second pseudo-contact process moves the virtual agent 70 closer to the user (point of view), in this embodiment, when the server 16 starts the second pseudo-contact process and draws one or more virtual agents 70, the server 16 places (draws) the one or more virtual agents 70 in a position farther away from the viewpoint than the positional relationship between the robot 12 and the user 60 in the real space.

[0080] When the second pseudo-contact process is started, one, two or three virtual agents 70 assigned to the robot 12 are gradually brought closer to the user 60 (virtual camera). A signal is sent to each of the one, two or three virtual agents 70 assigned to the robot 12 and the robot 12, and an animation of all the virtual agents 70 assigned to the robot 12 shaking hands is displayed on the HMD 20, and the robot 12 is made to start shaking hands. That is, a contact action toward the user 60 is executed. However, in the second pseudo-contact process, since all the virtual agents 70 assigned to the robot 12 shake hands at the same time, when the robot 12 shakes hands, the arms corresponding to all the virtual agents 70 assigned to the robot 12 are operated. That is, when one virtual agent 70 and two virtual agents 70 are assigned to the robot 12, the arms 122 and 126 are operated. Also, when three virtual agents 70 are assigned to the robot 12, the arms 122, 124 and 126 are operated. Therefore, the user 60 can grasp any one or two of the arms 122, 124, and 126 of the robot 12 to shake hands, and experience the sensation of shaking hands with one, two, or all three virtual agents 70 assigned to the robot 12. In other words, a pseudo-contact is presented to the user 60.

[0081] For example, in a case where three virtual agents 70 are assigned to the robot 12, when the three virtual agents 70 approach the user 60 (viewpoint) in the virtual space, as shown in FIG. 8, the three virtual agents 70 (70a, 70b, 70c) shake hands, and in association with (or in synchronization with) this, the arms 122, 124, and 126 corresponding to each of the three virtual agents 70 are operated. That is, the robot 12 shakes hands using the arms 122, 124, and 126. Therefore, the user 60 grasps one or two of the arms 122, 124, and 126 of the robot 12 and shakes hands while watching the three virtual agents 70 shaking hands on the HMD 20. That is, a visual stimulus due to the animation of the virtual agents 70 displayed on the HMD 20 is presented to the user 60, and a contact stimulus due to the movement of the robot 12 is given to the user 60.

[0082] Although not shown in the figure, in the second pseudo-contact processing, when two virtual agents 70 are assigned to the robot 12, similarly, when the two virtual agents 70 approach the user 60 (viewpoint) in the virtual space, the two virtual agents 70 shake hands, and the robot 12 performs the handshake action using the arms 122 and 126 corresponding to the two virtual agents 70.

[0083] In the second pseudo-contact processing, when one virtual agent 70 is assigned to the robot 12, when the one virtual agent 70 approaches the user 60 (viewpoint) in the virtual space, the one virtual agent 70 shakes hands, and the robot 12 performs the handshake action using the arm 122 corresponding to the one virtual agent 70.

[0084] Although detailed explanation is omitted, the robot 12 is equipped with tactile sensors 48 on the frames 122a, 124a, 126a of the arms 122, 124, 126, respectively. Therefore, in the first pseudo-contact process and the second pseudo-contact process, when the robot 12 is shaking hands, upon detecting contact with the hand or the like of the user 60, the arms 122, 124, 126 may be caused to move up and down repeatedly.

[0085] In addition, the movements of the arms 122, 124, and 126 change in relation to the movements of the virtual agent 70, and the server 16 is programmed with a program that can synchronize the movements (state transitions) of the robot 12 with the movements (state transitions) of the virtual agent 70 in the HMD 20 to further enhance such correlation.

[0086] In this program, the joint angles of the joints 122c, 122d, 124c, 124d, 126c, and 126d of the robot 12 (detected by the actuators A1-An that control each joint and fed back to the motor control) are controlled to match the angles of the arm and shoulder joints of the virtual agent 70.

[0087] However, it is not necessary to synchronize the movement of the robot 12 with the movement of the virtual agent 70. It has been confirmed that there is no problem even if the movement of the robot 12 and the movement of the virtual agent 70 displayed on the HMD 20 are slightly delayed in time, so long as the difference is within a certain degree.

[0088] When synchronizing the movements of the robot 12 and the virtual agent 70, the delay time in the movements of the robot 12 is detected in advance, and the animation of the virtual agent 70 is played back taking this delay time into account.

[0089] Fig. 9 shows an example of a memory map 300 of a memory (here, a RAM) 16b built into the server 16. As shown in Fig. 9, the memory 16b includes a program storage area 302 and a data storage area 304. The control program of this embodiment is stored in the program storage area 302.

[0090] The control programs include a communication program 302a, an allocation program 302b, a sensor value detection program 302c, an image generation program 302d, and a robot control program 302e.

[0091] The communication program 302a is a program for communicating with external devices, in this embodiment, the robot 12 and the HMD 20, either wired or wirelessly.

[0092] The allocation program 302b is a program for allocating one or more virtual agents 70 to the robot 12 in accordance with the number of virtual agents 70 set in accordance with a user's operation or the like.

[0093] The sensor value detection program 302c is a program for detecting outputs from various sensors (in this embodiment, the tactile sensor 48, the camera 50, and the microphone 56).

[0094] The image generation program 302d is a program for generating image data to be displayed on the HMD 20. In this embodiment, image data for still images and moving images (animations) of one or more virtual agents 70 assigned to the robot 12 is generated.

[0095] The robot control program 302e is a program for transmitting to the robot 12, when one or more virtual agents 70 assigned to the robot 12 perform a predetermined first motion and second motion (in this embodiment, the first motion and second motion are shaking hands), motion data for causing the arms (122 and 124, or 122, 124 and 126) corresponding to each virtual agent 70 to move in synchronization with the predetermined first motion and second motion, and, if necessary, voice data corresponding to the content of the speech of the robot 12.

[0096] Although not shown, the program storage area 302 also stores other programs for controlling the robot 12 and the virtual agent 70.

[0097] The data storage area 304 also stores sensor value data 304a, arm control data 304b, speech data 304c, and image data 304d.

[0098] The sensor value data 304a is data output from various sensors and is data detected according to the sensor value detection program 302c. The arm control data 304b is data for controlling the arms 122, 124, and 126 of the robot 12, and more specifically, data for driving actuators.

[0099] The speech data 304c is synthetic voice data of the contents spoken by the virtual agent 70 assigned to the robot 12. However, the speech (dialogue) contents can use contents set in advance in a database (for example, the memory 16b of the server 16) or can use an API (Application Programming Interface) on the web.

[0100] The image data 304d is image data for displaying the virtual agent 70 on the HMD 20, and is generated according to the image generation program 302d. However, the image data 304d includes still image data of the virtual agent 70 and animation data regarding the movements of the virtual agent 70.

[0101] Although not shown, the data storage area 304 stores other data necessary for controlling the robot 12 and the virtual agent 70, and is provided with a timer (counter) and a flag.

[0102] 10 and 11 are flow diagrams of the first pseudo-contact process of the CPU 16a built in the server 16 shown in FIG. 1. FIG. 12 and 13 are flow diagrams of the second pseudo-contact process of the CPU 16a built in the server 16 shown in FIG. 1. The first pseudo-contact process and the second pseudo-contact process will be described below, but the same processes as those already described will be described briefly. In this embodiment, the operations of the first pseudo-contact and the second pseudo-contact are executed by the CPU 16a of the server 16 according to a control program set in the memory 16b. That is, the CPU 16a of the server 16 sends a command signal to the CPU 36 (FIG. 3) of the robot 12, so that the server 16 indirectly controls the operation of the robot 12.

[0103] As shown in FIG. 10, when the CPU 16a starts the first pseudo-contact process, in step S1, it detects image data from the camera 18c, and in step S3, it identifies who the user 60 is based on the image data detected in step S1.

[0104] However, instead of image data from the camera 18c, other means such as an RFID (radio frequency identifier) ​​may be used to identify the individual of the user 60. In this case, the CPU 16a detects the RFID in step S1.

[0105] If the user 60 has not been registered beforehand and the identity of the user 60 cannot be identified in step S1, the process can be continued by provisionally assigning an anonymous ID.

[0106] In the next step S5, the number of virtual agents 70 is set. As described above, the number of virtual agents 70 can be set according to the user's operation within the number of arms provided to the robot 12, and is set to any one of 1 to 3 in this embodiment.

[0107] In the following step S7, it is determined whether or not the number of virtual agents 70 is 1. If "YES" in step S7, that is, if the number of virtual agents 70 is 1, one virtual agent 70 is displayed on the HMD 20 in step S9, and the process proceeds to step S17 shown in FIG.

[0108] However, in step S9, image data of one virtual agent 70 selected according to the user 60 identified in step S3, drawn in a position where its right arm overlaps with arm 122 and its left arm overlaps with arm 124, is transmitted to the HMD 20.

[0109] In this case, the server 16 may randomly select from candidates for virtual agent 70 of the same or opposite sex as the user 60, or the user 60 may manually select a virtual agent 70 that has been registered for himself or herself in advance.

[0110] However, a particular virtual agent 70 may be manually selected for a user 60 having the above-mentioned anonymous ID.

[0111] In this manner, when the virtual agent 70 is manually selected, for example, a selection switch (not shown) may be provided on the HMD 20, and in response to the operation of the selection switch, the server 16 may sequentially display images of candidate virtual agents 70 on the HMD 20, and when the desired virtual agent 70 is displayed, the operation of the selection switch may be stopped.

[0112] The distance d between the robot 12 and the user 60 is calculated based on the predetermined position of the robot 12 and position information from the position detection device 18a, and in the virtual space, the distance between the viewpoint (virtual camera) and the robot 12 is set to a distance equivalent to the distance d in the real space, and the virtual agent 70 is rendered. As described above, the gaze point of the virtual camera is the center of the robot 12 in the virtual space. The same applies to the following cases in which the virtual agent 70 is rendered and image data is generated.

[0113] On the other hand, if "NO" in step S7, that is, if the number of virtual agents 70 is not one, it is determined in step S11 whether the number of virtual agents 70 is two.

[0114] If "YES" in step S11, that is, if the number of virtual agents 70 is two, then in step S13, two virtual agents 70 are displayed on the HMD 20, and the process proceeds to step S17.

[0115] However, in step S13, two virtual agents 70 selected according to the users 60 identified in step S3 are drawn side-by-side, with the left arm of one of them overlapping with arm 122 and the right arm of the other one overlapping with arm 124, and image data is transmitted to the HMD 20.

[0116] On the other hand, if "NO" in step S11, that is, if the number of virtual agents 70 is three, three virtual agents 70 are displayed on the HMD 20 in step S15, and the process proceeds to step S17.

[0117] However, in step S15, the three virtual agents 70 selected according to the users 60 identified in step S3 are drawn side-by-side, with the right arm of one of them overlapping with arm 122, the right arm of one of the remaining two overlapping with arm 124, and the left arm of the other of the remaining two overlapping with arm 126, and image data is transmitted to the HMD 20.

[0118] As shown in FIG. 11, in step S17, a topic is selected and a dialogue is started. At this time, the user 60 may select a theme of the dialogue, or a specific theme may be automatically selected based on the dialogue history of each user 60 accumulated in the server 16. In addition, in step S17, when starting the dialogue, synthetic voice data of the speech content regarding a greeting is transmitted to the robot 12 and the HMD 20. However, instead of a greeting, the speech content may be for bringing the user 60 closer to the robot 12 or for guiding the dialogue. Furthermore, when multiple virtual agents 70 are assigned to the robot 12, synthetic voice data of the speech content for each of the multiple virtual agents 70 is transmitted to the robot 12 and the HMD 20 in a predetermined order. However, this is only an example, and synthetic voice data of the speech content of only one of the virtual agents 70 may be transmitted. However, the pitch (frequency) of the synthetic voice is changed according to the type of the virtual agent 70.

[0119] In the next step S19, all of the arms 122, 124, and 126 of the robot 12 are returned to their initial positions. That is, the CPU 16a transmits to the robot 12 control data for returning the arms 122, 124, and 126 to their initial positions.

[0120] Next, in step S21, the robot 12 waits for a certain time (for example, 2 to 3 seconds). Here, the movement and operation of the robot 12 and all virtual agents 70 are stopped for a certain time. Then, in step S23, it is determined whether the user 60 (viewpoint) has approached a distance where the user can shake hands with the virtual agents 70. However, when a plurality of virtual agents 70 are assigned to the robot 12, it is determined whether the user has approached a distance where the user can shake hands with any one of the virtual agents 70. In other words, it is determined whether the distance between the user 60 and the arms 122, 124, 126 of the robot 12 to which the virtual agents 70 are assigned in the real space is equal to or less than a predetermined distance df. Specifically, the CPU 16a determines whether the user 60 has approached a position where the user can shake hands with at least one of the arms 122, 124, and 126 based on the image data from the camera 50 received from the robot 12 through the network 14.

[0121] However, image data from the camera 50 may be viewed on a monitor (not shown) and the operator may make the decision.

[0122] In order to make the judgment in step S23, in the embodiment, the image data from the camera 50 is analyzed, but it is also possible to use a camera 18c installed in the environment, such as the floor or ceiling on which the robot 12 is placed, or a user 60 position detection system such as the laser range finder (Oculus Sensors) described above or a mat sensor.

[0123] If step S23 is "NO", that is, if the user 60 has not approached close enough to shake hands with the virtual agent 70, the process returns to step S21. On the other hand, if step S23 is "YES", that is, if the user 60 has approached close enough to shake hands with the virtual agent 70, then in step S25, the arm of the virtual agent 70 closest to the user 60 is moved closer to the user 60 (the viewpoint) to shake hands. Here, the CPU 16a plays an animation in which the arm of the virtual agent 70 closest to the user 60 is moved closer to the viewpoint in the virtual space to shake hands.

[0124] In the next step S27, the arm 122, 124 or 126 of the robot 12 corresponding to the virtual agent 70 closest to the user 60 is brought close to the user 60 to shake hands. In step S27, the CPU 16a controls the actuators of the joints (122c and 122d, 124c and 124d, or 126c and 126d) through the actuator control circuit 44 to slightly raise (or move forward) the arm 122, 124 or 126 of the robot 12 corresponding to the virtual agent 70 closest to the user 60 in the direction in which the user 60 is present, with the angle of the forearm (frame 122a) relative to the upper arm (frame 122b), the angle of the forearm (frame 124a) relative to the upper arm (frame 124b), or the angle of the forearm (frame 126a) relative to the upper arm (frame 126b) set to 70°.

[0125] Next, in step S29, the intimacy level of the virtual agent 70 is improved, and in step S31, a dialogue according to the topic is executed. Here, the CPU 16a executes a dialogue according to the topic selected in step S17. However, by storing a dialogue history, if a dialogue has been held in the past about the selected topic, a non-overlapping statement content is selected.

[0126] Furthermore, when the virtual agent 70 performs an action in addition to speaking, the arm 122, 124 or 126 to which the virtual agent 70 is assigned is also operated in synchronization with the action of the virtual agent 70.

[0127] Then, in step S33, it is determined whether the user 60 wishes to end the dialogue. Here, when the CPU 16a detects a specific action of the user 60 as described above or detects an utterance of the user 60 regarding a specific content as described above, the CPU 16a determines that the user 60 wishes to end the dialogue. This also applies to the process of step S81 described later.

[0128] If "NO" in step S33, that is, if the user 60 does not wish to end the dialogue, the process returns to step S31 to continue the dialogue. On the other hand, if "YES" in step S33, that is, if the user 60 wishes to end the dialogue, the first pseudo-contact process is ended.

[0129] Also, as shown in FIG. 12, when the CPU 16a starts the second pseudo-contact processing, in step S51, it detects image data from the camera 18c, and in step S53, it identifies who the user 60 is based on the image data detected in step S51.

[0130] However, in the second pseudo-contact process, an image is displayed on the HMD 20 to make the virtual agent 70 approach the user 60, and therefore, as described above, this second pseudo-contact process is started on the premise that the user 60 has approached a position where they can shake hands with the robot 12.

[0131] In the following step S55, the number of virtual agents 70 is set, and in step S57, it is determined whether the number of virtual agents 70 is 1. If "YES" in step S57, in step S59, one virtual agent 70 is displayed on the HMD 20, and the process proceeds to step S67 shown in FIG.

[0132] On the other hand, if "NO" in step S57, in step S61, it is determined whether or not the number of virtual agents 70 is two. If "YES" in step S61, two virtual agents 70 are displayed on the HMD 20 in step S63, and the process proceeds to step S67. On the other hand, if "NO" in step S61, three virtual agents 70 are displayed on the HMD 20 in step S65, and the process proceeds to step S67.

[0133] In the second pseudo-contact processing, as described above, in order to move the virtual agent 70 closer to the user (viewpoint), when drawing one or more virtual agents 70 in steps S59, S63 and S65, the CPU 16a places (draws) the one or more virtual agents 70 at a position farther away from the viewpoint than the positional relationship between the robot 12 and the user 60 in the real space.

[0134] 13, in step S67, all the arms 122, 124, and 126 of the robot 12 are returned to their initial positions. Then, in step S69, the process waits for a certain period of time. In the next step S71, all the virtual agents 70 are moved closer to the user 60. For example, the CPU 16a generates image data in the case where all the virtual agents 70 are moved a predetermined distance (e.g., 70 cm) toward the viewpoint in the virtual space, and transmits the image data to the HMD 20.

[0135] Then, in step S73, it is determined whether or not all of the virtual agents 70 have come close enough to shake hands with the user 60. Here, it is determined whether or not the distance between each of the virtual agents 70 and the viewpoint in the virtual space has become equal to or less than a predetermined distance df.

[0136] If the result in step S73 is "NO", that is, if all the virtual agents 70 have not approached close enough to shake hands with the user 60, the process returns to step S69. On the other hand, if the result in step S73 is "YES", that is, if all the virtual agents 70 have approached close enough to shake hands, in step S75, the arms of all the virtual agents 70 approach the user 60 and shake hands. Here, the CPU 16a plays an animation in which the arms of all the virtual agents 70 approach the direction of the viewpoint in the virtual space and shake hands. However, if there is one virtual agent 70, the animation of that virtual agent 70 is played. If there are two virtual agents 70, the animation of each of the two virtual agents 70 is played. If there are three virtual agents 70, the animation of each of the three virtual agents 70 is played.

[0137] In the next step S77, the arms of the robots 12 corresponding to all the virtual agents 70 are brought close to the user 60 to shake hands. In step S77, the CPU 16a raises (or moves forward) the arms of the robots 12 corresponding to all the virtual agents 70 slightly in the direction of the user 60 with the angle of the forearm relative to the upper arm set to 70°. However, when there is one virtual agent 70, the movement of the arm 122 is controlled. When there are two virtual agents 70, the movements of the arms 122 and 124 are controlled. When there are three virtual agents 70, the movements of the arms 122, 124 and 126 are controlled.

[0138] Next, in step S79, the process waits for a certain period of time (for example, 2 to 3 seconds), and in step S81, it is determined whether or not the user 60 wishes to end the handshake. If the result in step S81 is "NO," that is, if the user 60 does not wish to end the handshake, the process returns to step S79. On the other hand, if the result in step S81 is "YES," that is, if the user 60 wishes to end the handshake, the second pseudo-contact process is ended.

[0139] According to this embodiment, an arbitrary visual stimulus using multiple virtual agents is presented using an HMD, while tactile stimuli are presented from multiple arms of a robot to which multiple virtual agents are assigned, so that the user can experience the sensation of touching the virtual agents. In other words, a new pseudo-tactile presentation system can be provided.

[0140] In addition, according to this embodiment, the visual stimulation presentation device, i.e., the HMD, displays the movements of multiple virtual agents, and in association with this movement, the movable part of the contact stimulation presentation device, i.e., the multiple arms of the robot, is moved to present contact stimulation to the user, thereby enhancing the sense of unity between the visual stimulation and the contact stimulation.

[0141] In the second pseudo-contact process of this embodiment, when three virtual agents are assigned to the robot, the three virtual agents are moved closer to the user (point of view) to perform the action, but two of the three virtual agents may be moved closer to the user to perform the action. In this case, the two of the three virtual agents are selected according to a predetermined rule such as random or by the user.

[0142] In this embodiment, the robot arm has joints corresponding to an elbow joint and a shoulder joint, but it may have only an elbow joint or a shoulder joint. If it has only an elbow joint, it is possible to shake hands by moving the upper arm, and if it has only a shoulder joint, it is possible to shake hands by moving the whole arm.

[0143] Furthermore, in this embodiment, the robot has three arms, but it is also possible to provide four or more arms, as long as the number of arms is two or more. In the case of providing four or more arms, it is possible to display even more virtual agents on the HMD and to interact with each virtual agent individually or simultaneously.

[0144] Furthermore, in this embodiment, a case has been described in which the user shakes hands with the virtual agent, but the predetermined action does not have to be limited to a handshake, and may be a touch.

[0145] In the above embodiment, a single server controls each element of the system, but the robot, server and HMD may each be controlled by a separate computer, or the system may be managed and controlled by any number of computers.

[0146] In addition, the virtual agent displayed on the HMD shown in the above embodiment may be any character as long as it is a 3D computer graphics agent. Therefore, the appearance, gender, clothing, species (human, animal, anthropomorphic character, etc.), and realism (animation style, live action style, etc.) may be changed according to the user.

[0147] It should be noted that the specific values ​​of angles and the like given in the above embodiments are merely examples and can be appropriately changed as necessary. [Explanation of symbols]

[0148] 10. Pseudo-touch presentation system 12...Communication robot 16 ... Server 20...HMD 60...users 70 … Virtual Agent 122, 124, 126 ... Robot arm

Claims

1. A contact stimulus presentation device including a plurality of stimulus presentation units that act on a user to provide the user with a contact stimulus; an allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units; and a head-mounted display device that is worn on the head of the user, displays a plurality of virtual agents assigned by the assignment means, and operates some or all of the plurality of virtual agents to present a visual stimulus to the user; The contact stimulation presentation device selectively executes a first process in which, in synchronization with one of the virtual agents performing a first predetermined action toward the user, the stimulation presentation unit to which the virtual agent is assigned executes an action corresponding to the first predetermined action, or a second process in which two or more of the plurality of virtual agents simultaneously perform a second predetermined action toward the user and each of the stimulation presentation units to which two or more of the plurality of virtual agents are assigned executes an action corresponding to the second predetermined action. This is a pseudo-contact presentation system.

2. the head-mounted display displays some or all of the plurality of virtual agents as animations showing actions for the user; The pseudo-touch presentation system of claim 1, wherein the stimulus presentation unit includes a movable part, and the tactile stimulus presentation device presents the tactile stimulus to the user by operating the movable part so that the movement of the movable part changes in association with animation of actions for some or all of the plurality of virtual agents.

3. A control program executed by a computer of a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, the control program comprising: a stimulus presentation unit control means for controlling the plurality of stimulus presentation units of the tactile stimulus presentation device to present a tactile stimulus to the user; an allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units; a display control means for displaying, on the head-mounted display device, a part or all of the plurality of virtual agents assigned by the assignment means; and a sensory stimulus presentation program that functions as an execution means for selectively executing a first process of causing the stimulus presentation unit to which the virtual agent is assigned to perform an action corresponding to the first predetermined action in synchronization with one of the virtual agents performing a first predetermined action toward the user, or a second process of causing two or more of the plurality of virtual agents to simultaneously perform a second predetermined action toward the user and causing each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned to perform an action corresponding to the second predetermined action.

4. A method for controlling a pseudo-touch presentation system including a touch stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide the user with touch stimuli, and a head-mounted display device that is mounted on the user's head, comprising: A step of controlling the plurality of stimulus presentation units of the contact stimulus presentation device to present a contact stimulus to the user; assigning a virtual agent to each of two or more of the stimulus presentation units; displaying some or all of the assigned virtual agents on the head-mounted display; and A control method comprising a step of selectively executing a first process in which, in synchronization with one of the virtual agents performing a first predetermined action toward the user, the stimulus presentation unit to which the virtual agent is assigned performs an action corresponding to the first predetermined action, or a second process in which two or more of the plurality of virtual agents simultaneously perform a second predetermined action toward the user and each of the stimulus presentation units to which two or more of the plurality of virtual agents are assigned performs an action corresponding to the second predetermined action.

Images