System for controlling stimulus to be presented to sense of user in movement of object by user in virtual space

Abstract

To allow a user to feel a sense when holding or moving an object with strength matched with a situation according to the restriction of an actual operation.SOLUTION: The system includes a display device that presents a video of a virtual space including an object to a user, a measurement device that measures a motion of a hand of the user, a vibration presentation device that presents a vibration to the hand of the user, an audio device that presents a sound to the user, and a control device. The control device controls a plurality of parameters among a visual parameter of a video of the object, a tactile parameter of vibration of the vibration presentation device, and an auditory parameter of sound of the audio device in movement of the object in the virtual space on the basis of an attribute and / or a state of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space. [Background technology]

[0002] Trials are underway for education and training that combines virtual reality (VR) and haptic technology in maintenance and manufacturing operations. Training content using this technology allows for education that feels close to the actual work, without using actual on-site items.

[0003] On the other hand, there are also examples of tactile gloves that use air pressure in addition to vibration to provide stronger tactile stimulation (see Non-Patent Document 1). However, due to the characteristics of air pressure, the size of the device becomes large, which poses challenges such as an increased physical burden on the user and limitations on training locations.

[0004] As a technique for presenting vibration stimuli, a technique for imparting a change in tactile sensation by adjusting the amplitude, displacement, and deformation of the vibration stimulus (see Patent Document 1) has been proposed.

[0005] In addition, technology has been proposed that uses visual CD (Control-Display) ratios and auditory stimuli to create pseudo-tactile sensations, providing an effect similar to that of real tactile sensations.

[0006] Furthermore, a technology has been proposed that presents a highly immersive haptic sensation suited to the situation by changing the vibration frequency of a vibration device according to the sound source and images in the virtual environment (see Patent Document 3). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2020-098649 [Patent Document 2] Japanese Patent Publication No. 2023-068248 [Patent Document 3] International Publication No. 2019 / 038888 [Non-patent literature]

[0008] [Non-Patent Document 1] A. Lecuyer, "Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback," in Presence, vol. 18, no. 1, pp. 39-53, 1 Feb. 2009 Summary of the Invention [Problem to be solved by the invention]

[0009] VR training using the vibration-based tactile presentation technology described above is certainly useful in increasing the sense of reality during worker education and training.

[0010] However, it does not take into account the need for stronger tactile sensations when the object being worked on is heavy enough to restrict the worker's movements or when there are restrictions on the direction of movement. As a result, there may be a discrepancy in which movements that are impossible in reality are possible in VR.

[0011] The object of this invention is to combine the presentation of force sensations through vibration with pseudo-tactile sensations that utilize vision and hearing, in accordance with the constraints of real-world movements, to allow a user to feel the sensations of holding or moving an object with an intensity that is more suited to the situation, without the need for large-scale equipment. [Means for solving the problem]

[0012] One aspect of the present invention is a system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space, the system including a display device that presents an image of the virtual space including an object to the user, a measurement device that measures the user's hand movements, a vibration device that presents vibrations to the user's hand, an audio device that presents sound to the user, and a control device, wherein the control device controls multiple parameters, including visual parameters of the image of the object, tactile parameters of the vibration of the vibration device, and auditory parameters of the sound of the audio device, when the object moves in the virtual space based on the attributes and / or state of the object. [Effects of the Invention]

[0013] According to one aspect of the present invention, it is possible to make a user feel the sensation of moving an object with an intensity that is more suited to the situation, in accordance with the constraints of actual movement. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating a network configuration including a sensory information management device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the sensory information management device according to the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of the vibration presentation device according to the present embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of a head-mounted device according to the present embodiment. [Figure 5] 3 shows an example of a table in an educational virtual space information management database according to the present embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a flow of processing in this embodiment in which a user comes into contact with an object in a virtual space and parameters for visual and tactile reproduction are updated based on the weight of the object. [Figure 7]This figure shows an example flow chart of the process of updating the parameters for visual and tactile reproduction based on the number of hands with which a user holds an object and a comparison of the hand trajectory with the angle of the server rack access port in this embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a flow of processing for updating parameters for audio-visual and haptic reproduction based on the speed of the user's hand and the friction coefficient of the target object in this embodiment. [Figure 9] This figure shows an example flow of a process in this embodiment in which it is determined whether the module held by the user has come off the server rack, and if so, the parameters for visual and tactile reproduction are updated based on the height of the module, and visual and tactile reproduction is performed based on the visual and tactile parameter information updated up to this point. [Figure 10] FIG. 3 is a diagram illustrating an example of a table in an object information management database according to the present embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a table in a sensory information management database according to the present embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a situation in which a user is wearing an HMD and carrying a server rack in this embodiment. [Figure 13A] 10A and 10B are diagrams illustrating an example in which the amount of movement of the server module changes in reality and in VR by manipulating the visual CD ratio in this embodiment. [Figure 13B] 10A and 10B are diagrams illustrating an example in which the amount of movement of the server module changes in reality and in VR by manipulating the visual CD ratio in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] <Network configuration> DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 is a diagram showing a network configuration including a sensory information management device 100 according to the present embodiment.

[0016] 1, the sensory information management device 100 is a control device connected to be able to communicate with the vibration presentation device 200 and the head-mounted device 300 via an appropriate network 600 such as the Internet or a LAN (Local Area Network). Therefore, these may be collectively referred to as the sensory augmentation system 10.

[0017] The sensory information management device 100 is a device that stores content information and a sensory presentation device, and is a computer that adjusts parameters of visual information, auditory information, and vibration information according to the content information and user operation information, and transmits them to the vibration presentation device 200 and the head-mounted device 300. Specifically, it is assumed that the device is a general PC, tablet terminal, motion capture device, etc., either singly or in combination.

[0018] On the other hand, the vibration presentation device 200 is a terminal that presents the vibration stimulus in the above-mentioned content. This vibration presentation device 200 is a playback device for the vibration stimulus presentation information transmitted by the sensory information management device 100. Specifically, a tactile glove that can present vibration, a wristband-type vibration device worn on the wrist, etc. are envisioned.

[0019] Next, the head-mounted device 300 is a terminal that presents visual information and reproduces auditory information in the above-mentioned content. Specifically, it is assumed to be a device such as an HMD (Head Mounted Display) that the user wears on their head to view a scene different from reality. Note that the effect remains the same even if the auditory presentation function is configured separately in a separate device. Furthermore, one or more devices of a type different from the head-mounted device may be used to present visual and auditory information.

[0020] <Hardware configuration> The hardware configuration of the sensory information management device 100 of this embodiment includes an auxiliary storage device 101, a main storage device 113, a calculation device 114, a communication device 115, an input device 116, and an output device 117, as shown in FIG.

[0021] The auxiliary storage device 101 is a storage device configured with an appropriate nonvolatile storage element such as an SSD (Solid State Drive) or a hard disk drive.

[0022] The main memory device 113 is a memory device that is configured with a volatile memory element such as a RAM (Random Access Memory). One or a combination of the auxiliary memory device and the main memory device is a memory device.

[0023] The arithmetic unit 114 is, for example, a CPU (Central Processing Unit). The arithmetic unit 114 executes the program 102 stored in the auxiliary storage device 101 by reading it into the main storage device 113, and performs overall control of the sensory information management device 100 as well as various determinations, calculations, and control processes.

[0024] The communication device 115 is assumed to be a network interface card or the like that is connected to the above-mentioned network 400 and is responsible for communication processing with each of the sensory presentation devices (200, 300).

[0025] The input device 116 is assumed to be an operation panel of a tablet terminal that processes external input to the sensory information management device 100, a keyboard or mouse of a general PC, a controller, a motion capture device, etc. The input device 116 includes a sensor device that monitors actual user actions (mouse operations and actions that mimic the gripping and movement of actual objects) corresponding to the user's hand actions (including hand movement and gripping actions) in the virtual space, and provides information on the monitored user actions to the calculation device 114.

[0026] The output device 117 is responsible for processing to output the results of the program 102, which change in response to input processing from the input device 116 and communication processing from the communication device 115, as a 2D screen or 3D screen, sound, or vibration stimulation.

[0027] The auxiliary storage device 101 also stores an object information management database 110 and a sensory information management database 111 in addition to a program 102 for implementing the functions required for the sensory information management device 100 of this embodiment. The object information management database 110 stores the shape, material, weight, and movement direction parameters of objects in the virtual environment used for user training. The sensory information management database 111 stores weighting coefficients used to adjust each piece of sensory information in response to the user's actions and object parameters.

[0028] In addition, program 102 is a program for implementing the functions of the sensory information management device 100, and is pre-equipped with the following programs: a virtual space contact / grasp detection processing unit 1021, a real space motion measurement processing unit 1022, a virtual space CD ratio operation processing unit 1023, a vibration information adjustment processing unit 1024, an auditory information adjustment processing unit 1025, and a sensory coefficient calculation processing unit 1026.

[0029] 3, the hardware configuration of the vibration presentation device 200 includes an auxiliary storage device 201, a main storage device 213, a calculation device 214, a communication device 215, an input device 216, and an output device 217. The main storage device 213, the calculation device 214, and the communication device 215 may be similar to the main storage device 113, the calculation device 114, and the communication device 115 of the sensory information management device 100, and therefore a description thereof will be omitted.

[0030] The input device 216 is assumed to be a keyboard, mouse, or the like of a general PC that processes input from the outside to the vibration presentation device 200 .

[0031] The output device 217 is responsible for processing to output the results of the program 202, which change in response to input processing from the input device 216 and communication processing from the communication device 215, as vibrations.

[0032] The auxiliary storage device 201, like the auxiliary storage device 101 of the sensory information management device 100, is a storage device configured with an appropriate nonvolatile storage element such as an SSD or a hard disk drive.

[0033] The auxiliary storage device 201 stores a program 202 for implementing the functions required for the vibration presentation device 200 .

[0034] The program 202 also includes a program called a vibration information reproduction processing unit 2021 in advance.

[0035] 4, the hardware configuration of the head-mounted device 300 includes an auxiliary memory device 301, a main memory device 313, an arithmetic device 314, a communication device 315, an input device 316, and an output device 317. The main memory device 313, the arithmetic device 314, and the communication device 315 may be similar to the main memory device 113, the arithmetic device 114, and the communication device 115 of the sensory information management device 100, and therefore a description thereof will be omitted.

[0036] The input device 316 is assumed to be a typical PC keyboard, mouse, or the like that handles input processing from the outside to the head-mounted device 300 .

[0037] The output device 317 is responsible for processing to output the results of the program 302, which change depending on the input processing from the input device 316 and the communication processing from the communication device 315, as a 2D screen or a 3D screen and as sound. The output device 317 includes a display device that presents an image including an object to the user, and an audio device (including headphones, speakers, earphones, etc.) that presents sound.

[0038] The auxiliary storage device 301, like the auxiliary storage device 101 of the sensory information management device 100, is a storage device configured with an appropriate nonvolatile storage element such as an SSD or a hard disk drive.

[0039] The auxiliary storage device 301 stores a program 302 for implementing the functions required for the head-mounted device 300, as well as an educational virtual space information management database 310. The educational virtual space information management database 310 stores a 3D virtual space to be played back on the head-mounted device 300. The 3D virtual space is a space expressed in three-dimensional shapes, created using 3DCG (Three-Dimensional Computer Graphics) or CAD (Computer Aided Design).

[0040] The program 302 also includes a program called an educational virtual space display processing unit 3021 and an audio information presentation processing unit 3022 .

[0041] The arithmetic unit of each device realizes a predetermined function by executing a program stored in a main memory. The arithmetic unit may include multiple chips and multiple packages. The program is executed by the arithmetic unit to perform a predetermined process using a memory device and a communication device. Therefore, in this embodiment and other embodiments, a description using the program as the subject may also be a description using the arithmetic unit as the subject. Alternatively, the process executed by the program is a process performed by the computer and computer system on which the program runs.

[0042] The computing device operates as a functional unit (means) that realizes a predetermined function by operating in accordance with a program. Furthermore, the computing device also operates as a functional unit (means) that realizes each of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units (means). Note that at least some of the functions of the program may be realized by dedicated hardware. The program can be installed on each computer from a program distribution server or a computer-readable non-transitory storage medium, and can be stored in the non-volatile storage device of each computer.

[0043] <Flow example> The procedure for implementing the sensory information presentation method according to this embodiment will be described below with reference to Figures 5 to 11. Various operations corresponding to the sensory information presentation method described below are realized by a program that is read into the main memory device 113 or the like and executed by the arithmetic unit 114 of the sensory information management device 100.

[0044] First, to play back the virtual space, the head-mounted device 300 reads the educational virtual space from the educational virtual space information management database 310, performs sensory parameter processing (described below), and then plays back the information through the output device 317. An example table in the educational virtual space information management database 310 is shown in FIG. 5. The ID 3100 in FIG. 5 is a management label assigned to each educational virtual space, and is an arbitrary alphanumeric character that does not overlap with other educational virtual spaces. The educational virtual space storage file 3101 is educational virtual space information, and is a file saved with an extension such as .glb.

[0045] An example flow of the sensory information presentation method according to this embodiment is shown in Figures 6 to 9. Figures 6 to 9 show a series of steps of the sensory information presentation method when a user removes a module from a server rack. Note that the order of the sensory parameter update process shown in Figures 6 to 9 is not limited to the following example.

[0046] Figure 6 shows the process of updating the parameters for visual and tactile reproduction based on the weight of an object when the user comes into contact with it in virtual space.

[0047] The real-space motion measurement processing unit 1022 reads the user's motion through the input device 116 (a user motion measurement device such as a motion capture device) (S100). From the user's motion read here, the virtual-space contact / grasp detection processing unit 1021 determines whether the user's hand has come into contact with an object in the virtual space (S101). Contact between the user and an object in the virtual space can be determined from the positions of each part of the user and the position and shape of the object, that is, the position on the surface of the object. Here, contact between the user's hand and the object is determined. Note that until the user comes into contact with the object, various parameters for providing sensations are set to values ​​when the hand is in free space.

[0048] Next, when the user touches an object in the virtual space (S101: YES), the sensory coefficient calculation processing unit 1026 acquires the weight w, movable direction s, and material m of the object through the object information management database 110 (S102). In this way, when the user's hand touches the object, parameter control processing for providing a sensory presentation when the object moves is initiated. This allows for more appropriate sensory parameter control when the object moves.

[0049] An example of a table in the object information management database 110 is shown in Fig. 10. The ID 1100 in Fig. 10 is a management label assigned to each object in the educational virtual space, and is an arbitrary alphanumeric character that does not overlap with other objects. The weight 1101, movable direction 1102, material 1103, and shape 1104 are values ​​that are read into the sensory coefficient calculation processor 1026 and used to calculate sensory parameters, and are saved in the form of numerical data.

[0050] The movable direction 1102 indicates the movable direction of the object. In the example of FIG. 10, the movable direction is represented by a vector. The vector is defined by values ​​in the X-axis, Y-axis, and Z-axis directions, which are perpendicular to each other. For example, the Z-axis is the vertical direction. The movable direction is not limited to one direction or two directions consisting of one direction and an opposite direction, but can be specified within the range of each axis. For example, in free space, an object can be moved in any direction. For example, in the user's action of removing a module (object) from a server rack described below, the movable direction of the object is the direction in which the module is pulled out of the server rack and the direction in which it is pushed in (both directions).

[0051] Material 1103 indicates the coefficient of friction of the object surface. Shape 1104 indicates the length, width, and height of the object. Here, the object is assumed to be a rectangular parallelepiped (including a cube). The current position of the object is managed in management information (not shown).

[0052] As described above, the object information management database 110 manages the attributes and state of each object in the virtual space. For example, weight, material (friction coefficient), and shape are included in the attributes of an object, and the movable direction is included in the state of the object. Furthermore, the height position of an object is included in the state of the object. Note that only some of these may be managed, and other types of values ​​of attributes or states may be managed and sensory parameters may be controlled based on them.

[0053] 6, the virtual space contact / grasp detection processing unit 1021 then determines whether the user has grasped an object in the virtual space from the user action read in S100 (S103). This determination can be made from the positions of the user's hands and fingers and the position of the object.

[0054] If it is determined that the user has grasped an object in the virtual space, the sensory coefficient calculation processing unit 1026 acquires default visual and tactile parameters from the sensory information management database 111 and sets them as sensory parameters (S104). An example table of the sensory information management database 111 is shown in FIG.

[0055] 11 is a management label assigned to each action or event in the educational content, and is an arbitrary character string that does not overlap with other actions or events. The threshold value 1111, tactile coefficient 1112, auditory coefficient 1113, and visual coefficient 1114 are values ​​that are read into the sensory coefficient calculation processing unit 1026 and used to calculate sensory parameters, and are saved in the form of numerical data.

[0056] Each cell of the haptic coefficients 1112 indicates, for example, a value for vibration amplitude and frequency. Each cell of the auditory coefficients 1113 indicates, for example, a value for sound delay, volume, and frequency. Each cell of the visual coefficients 1114 indicates, for example, a value for CD ratio. The "default" record in the weighting elements 1110 indicates the default value for each coefficient. When the value of any weighting element other than the default exceeds the value indicated by the threshold 1111, the haptic, auditory, and visual parameters are updated using the values ​​of the haptic coefficients 1112, auditory coefficients 1113, and visual coefficients 1114. Details will be described later.

[0057] Returning to Figure 6, the sensory coefficient calculation processing unit 1026 further compares the weight w of the object acquired in S102 with the "weight" value in the threshold 1111 of the sensory information management database 111, and determines whether the weight of the object exceeds the threshold (S105).

[0058] If it is determined that the weight of the object is equal to or less than the threshold value (S105: YES), the current value of the sensory parameter is maintained. If it is determined that the weight of the object is greater than the threshold value (S105: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding visual and haptic parameters from the sensory information management database 111 and updates each sensory parameter using the weight w and / or the visual and haptic parameters (S106).

[0059] 6, the parameters of the vibration to be presented to the haptic sense are updated by adding or subtracting the value indicated by the haptic coefficient 1112 from the current values ​​of the amplitude and frequency. In other words, if the object is heavy, a vibration of larger amplitude and smaller frequency is given.

[0060] The parameter update for the sound presented to the auditory sense is performed by adding the product of the weight w and the value indicated by the auditory coefficient 1113 to the current delay time. Also, the value indicated by the auditory coefficient 1113 is added or subtracted from the current values ​​of volume and frequency. In other words, if the object is heavy, a larger delay and a sound with a larger volume and lower frequency are presented. The parameter update for the image presented to the visual sense is performed by subtracting the product of the weight and the value indicated by the visual coefficient 1114 from the current value of the CD ratio. In other words, if the object is heavy, an image with a smaller CD ratio is presented.

[0061] Next, the sensory information management device 100 performs a process of updating the parameters of the visual and tactile reproduction based on the number of hands with which the user grasps the object and a comparison of the hand trajectory and the angle of the server rack access port, as shown in Figure 7.

[0062] The virtual space contact / grasp detection processing unit 1021 determines whether the user is holding an object with both hands based on the user's movements acquired in S100 (S107). This determination can be made based on information on the positions of the user's hands and their fingers, as well as the position of the object.

[0063] If it is determined that the user is holding the object with one hand (S107: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding audio-visual and haptic parameters from the sensory information management database 111 and updates the sensory parameters (S108). This allows the user to be appropriately informed of the difference in sensation when moving an object between holding it with one hand and holding it with both hands.

[0064] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding or subtracting the value indicated by the "Number of gripping hands" cell in the haptic coefficient 1112 from the current values ​​of amplitude and frequency. In other words, when an object is gripped with one hand, a vibration of a larger amplitude and a smaller frequency is given.

[0065] The parameters of the sound presented to the auditory sense are updated by adding the value indicated by the "Number of gripping hands" cell in the auditory coefficient 1113 to the current delay time. Also, the values ​​indicated by the "Number of gripping hands" cell in the auditory coefficient 1113 are added to or subtracted from the current values ​​of volume and frequency. In other words, in the case of one-handed gripping, a sound with a longer delay, higher volume, and lower frequency is presented. The parameters of the image presented to the visual sense are updated by subtracting the value indicated by the "Number of gripping hands" cell in the visual coefficient 1114 from the current value of the CD ratio. In other words, in the case of one-handed gripping, an image with a smaller CD ratio is presented. In the case of two-handed gripping (S107: YES), S108 is omitted, and the current values ​​of the sensory parameters are maintained.

[0066] Next, the sensory coefficient calculation processing unit 1026 acquires the trajectory of the user's hand when the user pulls out the object from the user's movement acquired in S100, compares it with the movable direction s of the object acquired in S102, calculates the angular deviation d of the hand trajectory, and determines whether the deviation d exceeds a threshold (S109). In the case of a two-handed grasp, the angle obtained from the trajectories of both hands or the angle of the trajectory of one hand can be used.

[0067] If it is determined that the deviation d exceeds the threshold value (S109: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the deviation d and / or the visual and tactile parameters (S110).

[0068] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated by the cell for "deviation between moving direction and hand trajectory" in the haptic coefficient 1112 to the current value of amplitude, and subtracting the value indicated by the cell for "deviation between moving direction and hand trajectory" in the haptic coefficient 1112 from the current value of frequency. In other words, if the deviation in angle is large, a vibration with a larger amplitude and a smaller frequency is given.

[0069] The parameters of the sound presented to the ear are updated by adding the product of the deviation d and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113 to the current delay time. Also, the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113 is added to the current value of the volume, and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113 is subtracted from the current value of the frequency. In other words, if the deviation is large, a sound with a larger delay, larger volume, and lower frequency is presented.

[0070] The parameters of the image presented to the eyes are updated by subtracting the product of the deviation d and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the visual coefficient 1114 from the current value of the CD ratio. In other words, if the deviation is large, an image with a smaller CD ratio is presented. If the deviation d is equal to or smaller than the threshold (S109: YES), S110 is skipped and the current values ​​of the sensory parameters are maintained.

[0071] Next, the sensory information management device 100 executes the process shown in FIG. 8 to update the parameters for audiovisual and tactile reproduction based on the speed of the user's hand and the friction coefficient of the target object.

[0072] The sensory coefficient calculation processing unit 1026 obtains the hand speed v when the user pulls out the object from the user's movement obtained in S100, and determines whether the speed v exceeds the threshold indicated by the sensory information management database 111 (S111).

[0073] If it is determined that the speed v exceeds the threshold value (S111: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the speed v and / or the visual and tactile parameters (S112).

[0074] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated by the "hand speed" cell of the haptic coefficients 1112 to the current value of the amplitude, and subtracting the value indicated by the "hand speed" cell of the haptic coefficients 1112 from the current value of the frequency. In other words, if the speed is fast, a vibration of larger amplitude and smaller frequency is given.

[0075] The parameters of the sound presented to the ear are updated by adding the product of the speed v and the value indicated by the "hand speed" cell of the hearing coefficients 1113 to the current delay time. Also, the values ​​indicated by the "hand speed" cell of the hearing coefficients 1113 are added to or subtracted from the current values ​​of the volume and frequency. In other words, if the speed is fast, a sound with a larger delay, a higher volume, and a lower frequency is presented.

[0076] The parameters of the image presented to the eyes are updated by subtracting the product of the speed v and the value indicated by the "hand speed" cell of the visual coefficients 1114 from the current value of the CD ratio. In other words, if the speed is fast, an image with a smaller CD ratio is presented. If the speed v is equal to or less than the threshold (S111: YES), S112 is skipped and the current values ​​of the sensory parameters are maintained.

[0077] Next, the sensory coefficient calculation processing unit 1026 determines whether the friction coefficient m of the object exceeds a threshold value based on the material (friction coefficient) m of the object acquired in S102 and the sensory information management database (S113).

[0078] If it is determined that the friction coefficient m of the object exceeds the threshold value (S113: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database and updates each sensory parameter using the friction coefficient m and / or the visual and tactile parameters (S114).

[0079] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated by the "Material" cell of the haptic coefficient 1112 to the current value of the amplitude, and subtracting the value indicated by the "Material" cell of the haptic coefficient 1112 from the current value of the frequency. In other words, if the coefficient of friction is large, vibrations of larger amplitude and smaller frequency are given.

[0080] The parameters of the sound presented to the ear are updated by adding or subtracting the product of the friction coefficient m and the value indicated in the "Material" cell of the auditory coefficient 1113 to the current delay time. Also, the value indicated in the "Material" cell of the auditory coefficient 1113 is added to the current value of the volume, and the value indicated in the "Material" cell of the auditory coefficient 1113 is subtracted from the current value of the frequency. In other words, when the friction coefficient is large, a larger delay, a larger volume, and a smaller frequency sound are presented.

[0081] The parameters of the image presented to the eyes are updated by subtracting the product of the friction coefficient m and the value indicated by the "Material" cell of the visual coefficient 1114 from the current value of the CD ratio. In other words, if the friction coefficient is large, an image with a smaller CD ratio is presented. If the friction coefficient m is equal to or smaller than the threshold (S113: YES), S114 is skipped and the current value of the sensory parameter is maintained.

[0082] Next, the sensory information management device 100 determines whether the module being held by the user has come off the server rack, as shown in Figure 9, and if it has, updates the parameters for visual and tactile reproduction based on the height of the module, and executes the process of visual and tactile reproduction based on the visual and tactile parameter information updated up to this point.

[0083] The sensory coefficient calculation processing unit 1026 obtains the amount of movement by which the user pulled out the module (object) from the user's movement obtained in S100, and determines whether the module has come out of the rack it is in based on the object shape 1104 and amount of movement obtained in S102 (S115).

[0084] If it is determined that the module has come off the rack (S115: YES), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the weight v and / or the visual and tactile parameters, in which case the CD ratio is applied only to movement in the upward direction of height, and not to movement downward (S116).

[0085] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the current value of the amplitude to the product of the weight w of the module and the value indicated by the cell "When module is removed" of the haptic coefficient 1112. Also, the frequency is set to the value indicated by the cell "When module is removed" of the haptic coefficient 1112, which in this case is the default value of "medium."

[0086] The parameter update for the sound presented to the ear is performed by adding the value indicated by the "When module is disconnected" cell of the hearing coefficient 1113 to the current delay time. Here, the added value is 0, and the current value is maintained. Also, the volume and frequency are set to the value indicated by the "When module is disconnected" cell of the hearing coefficient 1113. Here, they are set to "medium", the same as the default value.

[0087] To update the parameters of the image presented to the eye, the CD ratio is set to a value obtained by subtracting from the default value "1" the product of the weight w and the value indicated in the "When module is removed" cell of the visual coefficient 1114. If the module is not removed from the rack (S115: NO), S116 to S118 are omitted.

[0088] Next, the sensory coefficient calculation processing unit 1026 obtains the height at which the user is holding the module from the user's movement obtained in S100, and compares it with the threshold value for the module height indicated by the threshold value 1111 in the sensory information management database 111 to determine whether the height h of the object being held exceeds the threshold value (S117).

[0089] If it is determined that the height h of the module exceeds the threshold value (S117: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the height h (height position h) and / or the visual and tactile parameters (S118). At this time, the CD ratio is applied only to movement in the upward direction of height, and is not applied to movement downward.

[0090] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the product of the height position h of the module and the value indicated by the "Module Height" cell of the haptic coefficient 1112 to the current value of the amplitude. Also, the product of the height position h of the module and a predetermined sine wave is added to the current value of .

[0091] The parameters of the sound presented to the ear are updated by adding the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 to the current delay time. Also, the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 is added to the current value of the volume. Also, the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 is added to the current value of the frequency.

[0092] The parameters of the image presented to the eye are updated by subtracting from the current value of the CD ratio the product of the height position h of the module and the value indicated by the "Module Height" cell of the visual coefficient 1114. If the height position of the module is equal to or less than h (S115: YES), S118 is omitted.

[0093] The virtual space CD ratio manipulation processing unit 1023 uses the latest updated audio-visual haptic parameters and the educational virtual space to transmit visual information to the head-mounted device 300 through the communication device 115, and plays back video on the head-mounted device 300. At the same time, the vibration information adjustment processing unit 1024 uses the latest updated audio-visual haptic parameters to transmit vibration information to the vibration presentation device 200 through the communication device 115, and presents vibration on the vibration presentation device 200. At the same time, the audio information adjustment processing unit 1025 uses the latest updated audio-visual haptic parameters to transmit audio information to the head-mounted device 300 through the communication device 115, and presents audio on the head-mounted device 300 (S119).

[0094] The virtual space contact / grasp detection processing unit 1021 determines, based on the user action read in S100, whether the user has released the object in the virtual space (S120). If it is determined that the user has released the object in the virtual space (S120: YES), the virtual space CD ratio operation processing unit 1023, the vibration information adjustment processing unit 1024, and the auditory information adjustment processing unit 1025 end the tactile presentation processing (S121). If it is determined that the user continues to grasp the object in the virtual space (S120: NO), the flow returns to S115.

[0095] The process described with reference to Figures 6 to 9 controls a combination of multiple sensory parameters when moving an object. Some of the sensory parameters described may be maintained unchanged, and other sensory parameters may be additionally controlled. Furthermore, the method for adjusting each sensory parameter is merely an example, and the sensory parameters may be controlled in a different manner.

[0096] Figure 12 shows a user 510 wearing an HMD 501 removing a module 503 from a server rack 502 in a virtual space 500 and lifting it up and down, while receiving vibration stimuli from a tactile glove 504 and auditory stimuli from an HMD 1201.

[0097] Figure 13A shows an example of visual CD ratio manipulation, in which if the movement distance of a lifting action in reality is 15 cm, when lifting a heavy object in virtual space, the movement distance is reduced to 5 cm. Figure 13B shows an example of visual CD ratio manipulation, in which if the movement distance of a lifting action in reality is 15 cm, when lifting a light object in virtual space, the movement distance is reduced to 10 cm.

[0098] The best mode for carrying out the present invention has been specifically described above, but the present invention is not limited to this and can be modified in various ways without departing from the spirit of the present invention. According to this embodiment, the sensation when holding or moving an object can be felt at an intensity that is more suited to the situation, in accordance with the constraints of actual movements.

[0099] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0100] Furthermore, the above-mentioned components, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned components, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0101] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0102] 10 Sensory Augmentation System 100 Sensory information management device 101 Auxiliary storage 102 Programs 1021 Virtual space contact and grip detection processing unit 1022 Real-space motion measurement processing unit 1023 Virtual space CD ratio operation processing unit 1024 vibration information adjustment processing unit 1025 Auditory information adjustment processing unit 1026 Sensory coefficient calculation processing unit 110 Object Information Management Database 1100 ID 1101 Weight 1102 Direction of movement 1103 Material 1104 Shape 111 Sensory Information Management Database 1110 Weight Factor 1111 threshold 1112 Tactile Coefficient 1113 Hearing Coefficient 1114 Visual Coefficient 113 Main storage 114 Arithmetic equipment 115 Communication equipment 116 Input Device 117 Output Device 200 Vibration presentation device 201 Auxiliary storage device 202 Program 2021 Vibration information reproduction processing unit 213 Main storage 214 Arithmetic equipment 215 Communication equipment 216 Input Device 217 Output Device 300 Head-Mounted Device 301 Auxiliary storage device 302 Program 3021 Educational Virtual Space Display Processing Unit 3022 Auditory information presentation processing unit 310 Educational Virtual Space Information Management Database 3100 ID 3101 Educational Virtual Space Storage File 313 Main memory 314 Arithmetic equipment 315 Communication Equipment 316 Input Device 317 Output Device 400 Network

Claims

1. A system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space, comprising: a display device that presents an image of a virtual space including the object to a user; a measurement device that measures the user's hand movement; a vibration presentation device that presents vibration to the user's hand; an audio device for presenting audio to the user; a control device; The control device controls multiple parameters, including visual parameters of the image of the object, tactile parameters of the vibration of the vibration presentation device, and auditory parameters of the sound of the audio device, when the object moves in the virtual space based on the attributes and / or state of the object.

2. 10. The system of claim 1, the object attributes include at least one of a weight and a coefficient of friction of the object; The state of the object includes at least one of a movable direction and a height position of the object.

3. 10. The system of claim 1, The control device determining the number of hands grasping the object in the virtual space; and controlling the plurality of parameters based on the number of hands grasping the object.

4. 10. The system of claim 1, the state of the object includes a movable direction of the object; The control device controls the plurality of parameters based on an angular deviation between the user's hand trajectory and the movement direction.

5. 10. The system of claim 1, The control device The system controls the plurality of parameters based on a speed of the user's hand grasping the object in the virtual space.

6. 10. The system of claim 1, The control device A system that controls the plurality of parameters based on a height position of the object when the object exists in free space in the virtual space.

7. 10. The system of claim 1, The control device The system starts a parameter control process for moving the object when the object is held by the user's hand in the virtual space.

8. A method for controlling a stimulus presented to a user's senses when the user moves an object in a virtual space, comprising: The system is a display device that presents an image of a virtual space including the object to a user; a measurement device that measures the user's hand movement; a vibration presentation device that presents vibration to the user's hand; an audio device for presenting sounds to the user; The method controls multiple parameters, including visual parameters of the image of the object, tactile parameters of the vibration of the vibration presentation device, and auditory parameters of the sound of the audio device, when the object moves in the virtual space based on the attributes and / or state of the object.

Images