System for controlling stimuli presented to senses of user with respect to movement of object by user in virtual space

Abstract

A system according to the present invention includes a display device that presents a user with video of a virtual space that includes an object, a measurement device that measures the motion of a hand of the user, a vibration presentation device that presents the hand of the user with vibration, an audio device that presents the user with sound, and a control device. On the basis of an attribute and / or the state of the object, the control device controls a plurality of parameters with respect to movement of the object in the virtual space from among visual parameters for video of the object, tactile parameters for vibration from the vibration presentation device, and auditory parameters for sound from the audio device.

Description

A system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2024-134361, filed on August 9, 2024, the contents of which are incorporated herein by reference.

[0002] The present invention relates to a system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space.

[0003] Trials are underway for education and training that combines virtual reality (VR) and tactile presentation technology in maintenance and manufacturing operations. Training content using this technology allows for training that feels close to the actual work, without using actual on-site items.

[0004] On the other hand, there are examples of tactile gloves that use air pressure in addition to vibration to provide stronger tactile stimulation (see Non-Patent Document 1). However, due to the characteristics of air pressure, the size of the device becomes large, which poses challenges such as an increased physical burden on the user and limitations on training locations.

[0005] As a technology for presenting vibration stimuli, a technology for imparting changes in tactile sensation by adjusting the amplitude, displacement, and deformation of the vibration stimuli (see Patent Document 1) has been proposed.

[0006] Furthermore, a technology has been proposed that provides an effect similar to that of a real tactile sensation by using a visual CD (Control-Display) ratio or an auditory stimulus to provide a pseudo-tactile sensation.

[0007] Furthermore, a technology has been proposed (see Patent Document 3) that presents a highly immersive tactile sensation appropriate to the situation by changing the vibration frequency of a vibration device according to the sound source and image in the virtual environment.

[0008] JP 2020-098649 A JP 2023-068248 A International Publication No. 2019 / 038888

[0009] A. Lecuyer, "Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback," in Presence, vol. 18, no. 1, pp. 39-53, 1 Feb. 2009

[0010] VR training using the vibration-based tactile presentation technology described above is certainly useful in increasing the sense of realism during worker education and training.

[0011] However, this does not take into account the need for stronger tactile sensations when the object being worked on is heavy enough to restrict the worker's movements or when there are restrictions on the direction of movement, which can lead to a discrepancy in which movements that are impossible in reality are possible in VR.

[0012] The object of this invention is to combine the presentation of force sensations through vibration with pseudo-tactile sensations that utilize vision and hearing, in accordance with the constraints of real-world movements, to allow a user to feel the sensations of holding or moving an object with an intensity that is more suited to the situation, without the need for large-scale equipment.

[0013] One aspect of the present invention is a system for controlling stimuli presented to a user's senses when the user moves an object in a virtual space, the system including a display device that presents an image of the virtual space including an object to the user, a measurement device that measures the user's hand movements, a vibration device that presents vibrations to the user's hand, an audio device that presents sound to the user, and a control device, wherein the control device controls multiple parameters, including visual parameters of the image of the object, tactile parameters of the vibration of the vibration device, and auditory parameters of the sound of the audio device, when the object moves in the virtual space based on the attributes and / or state of the object.

[0014] According to one aspect of the present invention, it is possible to make a user feel the sensation of moving an object with an intensity that is more suited to the situation, in accordance with the constraints of actual movement.

[0015] 1 is a diagram illustrating a network configuration including a sensory information management device according to this embodiment. FIG. 2 is a diagram illustrating an example of the hardware configuration of a sensory information management device according to this embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of a vibration presentation device according to this embodiment. FIG. 4 is a diagram illustrating an example of the hardware configuration of a head-mounted device according to this embodiment. FIG. 5 is a diagram illustrating an example of a table in an educational virtual space information management database according to this embodiment. FIG. 6 is a diagram illustrating an example of a process flow in which a user comes into contact with an object in a virtual space and updates parameters for audio-visual tactile reproduction based on the weight of the object. FIG. 7 is a diagram illustrating an example of a process flow in which parameters for audio-visual tactile reproduction are updated based on the number of hands with which the user holds an object and a comparison between the trajectory of the hands and the angle of the access port of the server rack according to this embodiment. FIG. 8 is a diagram illustrating an example of a process flow in which parameters for audio-visual tactile reproduction are updated based on the speed of the user's hands and the friction coefficient of the target object according to this embodiment. FIG. 9 is a diagram illustrating an example of a process flow in which it is determined whether a module held by a user has come off the server rack, and if so, parameters for audio-visual tactile reproduction are updated based on the height of the module, and audio-visual tactile parameter information updated up to this point is used to perform audio-visual tactile reproduction. FIG. 10 is a diagram illustrating an example of a table in an object information management database according to this embodiment. FIG. 11 is a diagram illustrating an example of a table in a sensory information management database according to this embodiment. 1 is a diagram showing an example of a situation in which a user is wearing an HMD and carrying a server rack in this embodiment. 2 is a diagram showing an example of a change in the amount of movement of a server module in reality and in VR due to an operation of the visual CD ratio in this embodiment. 3 is a diagram showing an example of a change in the amount of movement of a server module in reality and in VR due to an operation of the visual CD ratio in this embodiment.

[0016] <Network Configuration> An embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a diagram showing the configuration of a network including a sensory information management device 100 according to this embodiment.

[0017] 1, the sensory information management device 100 is a control device connected to be able to communicate with the vibration presentation device 200 and the head-mounted device 300 via an appropriate network 600 such as the Internet or a local area network (LAN). Therefore, these may be collectively referred to as a sensory augmentation system 10.

[0018] The sensory information management device 100 is a device that stores content information and a sensory presentation device, and is a computer that adjusts parameters of visual information, auditory information, and vibration information according to the content information and user operation information, and transmits them to the vibration presentation device 200 and the head-mounted device 300. Specifically, it is assumed that the device is a general PC, a tablet terminal, a motion capture device, or the like, either singly or in combination.

[0019] On the other hand, the vibration presentation device 200 is a terminal that presents the vibration stimulus in the above-mentioned content. This vibration presentation device 200 is a playback device for the vibration stimulus presentation information transmitted by the sensory information management device 100. Specifically, the vibration presentation device 200 is assumed to be a tactile glove that can present vibration, a wristband-type vibration device that is worn on the wrist, or the like.

[0020] Next, the head-mounted device 300 is a terminal that presents visual information and reproduces auditory information in the above-mentioned content. Specifically, a device such as an HMD (Head Mounted Display) is assumed, which the user wears on their head to view a scene different from reality. Note that the effect remains the same even if the auditory presentation function is configured separately in a separate device. Furthermore, one or more devices of a type different from the head-mounted device may be used to present visual and auditory information.

[0021] <Hardware Configuration> The hardware configuration of the sensory information management device 100 of this embodiment includes an auxiliary storage device 101, a main storage device 113, a calculation device 114, a communication device 115, an input device 116, and an output device 117, as shown in FIG.

[0022] The auxiliary storage device 101 is a storage device configured with an appropriate nonvolatile storage element, such as an SSD (Solid State Drive) or a hard disk drive.

[0023] The main memory device 113 is a memory device configured with a volatile memory element such as a RAM (Random Access Memory). One or a combination of the auxiliary memory device and the main memory device is a memory device.

[0024] The arithmetic unit 114 is, for example, a CPU (Central Processing Unit). The arithmetic unit 114 reads the program 102 stored in the auxiliary storage device 101 into the main storage device 113 and executes the program, thereby performing overall control of the sensory information management device 100 and various determination, calculation, and control processes.

[0025] The communication device 115 is assumed to be a network interface card or the like that is connected to the above-mentioned network 400 and handles communication processing with each sensory presentation device (200, 300).

[0026] The input device 116 is assumed to be an operation panel of a tablet terminal that processes external input to the sensory information management device 100, a keyboard or mouse of a general PC, a controller, a motion capture device, etc. The input device 116 includes a sensor device that monitors actual user actions (mouse operations and actions that simulate the grasping and movement of actual objects) corresponding to the user's hand actions (including hand movement and grasping actions) in the virtual space, and provides information on the monitored user actions to the calculation device 114.

[0027] The output device 117 is responsible for processing to output the results of the program 102, which change depending on the input processing from the input device 116 and the communication processing from the communication device 115, as a 2D screen or 3D screen, sound, or vibration stimulation.

[0028] The auxiliary storage device 101 also stores an object information management database 110 and a sensory information management database 111 in addition to a program 102 for implementing the functions required for the sensory information management device 100 of this embodiment. The object information management database 110 stores the shape, material, weight, and movable direction parameters of objects in the virtual environment used for user training. The sensory information management database 111 stores weighting coefficients used to adjust each piece of sensory information in response to the user's actions and object parameters.

[0029] In addition, program 102 is a program for implementing the functions of the sensory information management device 100, and is pre-equipped with the following programs: a virtual space contact / grasp detection processing unit 1021, a real space motion measurement processing unit 1022, a virtual space CD ratio operation processing unit 1023, a vibration information adjustment processing unit 1024, an auditory information adjustment processing unit 1025, and a sensory coefficient calculation processing unit 1026.

[0030] 3, the hardware configuration of the vibration presentation device 200 includes an auxiliary storage device 201, a main storage device 213, a calculation device 214, a communication device 215, an input device 216, and an output device 217. The main storage device 213, the calculation device 214, and the communication device 215 may be similar to the main storage device 113, the calculation device 114, and the communication device 115 of the sensory information management device 100, and therefore a description thereof will be omitted.

[0031] The input device 216 is assumed to be a keyboard, mouse, or the like of a general PC that processes input from the outside to the vibration presentation device 200 .

[0032] The output device 217 is responsible for processing to output, as vibration, the results of the program 202 that change in response to input processing from the input device 216 and communication processing from the communication device 215 .

[0033] The auxiliary storage device 201, like the auxiliary storage device 101 of the sensory information management device 100, is a storage device configured with an appropriate non-volatile storage element such as an SSD or a hard disk drive.

[0034] The auxiliary storage device 201 stores a program 202 for implementing the functions required for the vibration presentation device 200 .

[0035] The program 202 also includes a program called a vibration information reproduction processing unit 2021 in advance.

[0036] 4, the hardware configuration of the head-mounted device 300 includes an auxiliary storage device 301, a main storage device 313, an arithmetic device 314, a communication device 315, an input device 316, and an output device 317. The main storage device 313, the arithmetic device 314, and the communication device 315 may be similar to the main storage device 113, the arithmetic device 114, and the communication device 115 of the sensory information management device 100, and therefore a description thereof will be omitted.

[0037] The input device 316 is assumed to be a typical PC keyboard, mouse, or the like that handles input processing from the outside to the head-mounted device 300 .

[0038] The output device 317 is responsible for processing to output, as a 2D screen or a 3D screen, and as sound, the results of the program 302, which change depending on the input processing from the input device 316 and the communication processing from the communication device 315. The output device 317 includes a display device that presents an image including an object to the user, and an audio device (including headphones, speakers, earphones, etc.) that presents sound.

[0039] The auxiliary storage device 301, like the auxiliary storage device 101 of the sensory information management device 100, is a storage device configured with an appropriate non-volatile storage element such as an SSD or a hard disk drive.

[0040] The auxiliary storage device 301 stores a program 302 for implementing functions required for the head-mounted device 300, as well as an educational virtual space information management database 310. The educational virtual space information management database 310 stores a 3D virtual space to be reproduced by the head-mounted device 300. The 3D virtual space is a space expressed in three-dimensional form using 3DCG (Three-Dimensional Computer Graphics) or CAD (Computer Aided Design).

[0041] The program 302 also includes a program called an educational virtual space display processing unit 3021 and an audio information presentation processing unit 3022 .

[0042] The arithmetic unit of each device realizes a predetermined function by executing a program stored in a main memory. The arithmetic unit may include multiple chips and multiple packages. The program is executed by the arithmetic unit to perform a predetermined process using a memory device and a communication device. Therefore, in this embodiment and other embodiments, a description using the program as the subject may also be a description using the arithmetic unit as the subject. Alternatively, the process executed by the program is a process performed by the computer and computer system on which the program runs.

[0043] The computing device operates as a functional unit (means) that realizes a predetermined function by operating in accordance with a program. Furthermore, the computing device also operates as a functional unit (means) that realizes each of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units (means). Note that at least some of the functions of the program may be realized by dedicated hardware. The program can be installed on each computer from a program distribution server or a computer-readable non-transitory storage medium, and can be stored in the non-volatile storage device of each computer.

[0044] <Flow Example> The procedure for implementing the sensory information presentation method according to this embodiment will be described below with reference to Figures 5 to 11. Various operations corresponding to the sensory information presentation method described below are realized by a program that is read into the main memory device 113 or the like and executed by the arithmetic unit 114 of the sensory information management device 100.

[0045] First, to play back the virtual space, the head-mounted device 300 reads the educational virtual space from the educational virtual space information management database 310, performs sensory parameter processing (described below), and then plays back the educational virtual space through the output device 317. An example table of the educational virtual space information management database 310 is shown in FIG. 5. The ID 3100 in FIG. 5 is a management label assigned to each educational virtual space, and is an arbitrary alphanumeric character that does not overlap with other educational virtual spaces. The educational virtual space storage file 3101 is educational virtual space information, and is a file saved with an extension such as .glb.

[0046] An example flow of the sensory information presentation method according to this embodiment is shown in Figures 6 to 9. Figures 6 to 9 show a series of steps of the sensory information presentation method when a user removes a module from a server rack. Note that the order of the sensory parameter update process shown in Figures 6 to 9 is not limited to the following example.

[0047] FIG. 6 shows the process of updating the parameters for visual and tactile reproduction based on the weight of an object when the user comes into contact with the object in the virtual space.

[0048] The real-space motion measurement processing unit 1022 reads the user's motion through the input device 116 (a user motion measurement device such as a motion capture device) (S100). From the user's motion read here, the virtual-space contact / grasp detection processing unit 1021 determines whether the user's hand has contacted an object in the virtual space (S101). Contact between the user and an object in the virtual space can be determined from the positions of each part of the user and the position and shape of the object, i.e., the position on the surface of the object. Here, contact between the user's hand and the object is determined. Note that until the user contacts the object, various parameters for providing sensations are set to values ​​that are used when the hand is in free space.

[0049] Next, when the user touches an object in the virtual space (S101: YES), the sensory coefficient calculation processing unit 1026 acquires the weight w, movable direction s, and material m of the object through the object information management database 110 (S102). In this way, when the user's hand touches the object, parameter control processing for providing a sensory presentation when the object moves is initiated. This allows for more appropriate sensory parameter control when the object moves.

[0050] An example of a table in the object information management database 110 is shown in Figure 10. The ID 1100 in Figure 10 is a management label assigned to each object in the educational virtual space, and is an arbitrary alphanumeric character that is unique to other objects. The weight 1101, movable direction 1102, material 1103, and shape 1104 are values ​​that are read into the sensory coefficient calculation processing unit 1026 and used to calculate sensory parameters, and are saved in the form of numerical data.

[0051] The movable direction 1102 indicates the movable direction of the object. In the example of FIG. 10 , the movable direction is represented by a vector. The vector is defined by values ​​in the X-axis direction, Y-axis direction, and Z-axis direction, which are perpendicular to each other. For example, the Z-axis direction is the vertical direction. The movable direction is not limited to one direction or two directions consisting of one direction and an opposite direction, but can be specified within the range of each axis. For example, in free space, an object can be moved in any direction. For example, in the user's action of removing a module (object) from a server rack described below, the movable direction of the object is the direction in which the module is pulled out of the server rack and the direction in which it is pushed in (both directions).

[0052] Material 1103 indicates the coefficient of friction of the object surface. Shape 1104 indicates the length, width, and height of the object. Here, the object is assumed to be a rectangular parallelepiped (including a cube). The current position of the object is managed in management information (not shown).

[0053] As described above, the object information management database 110 manages the attributes and state of each object in the virtual space. For example, weight, material (friction coefficient), and shape are included in the attributes of an object, and the movable direction is included in the state of the object. Furthermore, the height position of an object is included in the state of the object. Note that only some of these may be managed, and other types of values ​​of attributes or states may be managed and sensory parameters may be controlled based on them.

[0054] 6 , the virtual space contact / grasp detection processing unit 1021 then determines whether the user has grasped an object in the virtual space from the user action read in S100 (S103). This determination can be made from the positions of the user's hands and fingers and the position of the object.

[0055] If it is determined that the user has grasped an object in the virtual space, the sensory coefficient calculation processing unit 1026 acquires default visual and tactile parameters from the sensory information management database 111 and sets them as sensory parameters (S104). An example table of the sensory information management database 111 is shown in FIG. 11.

[0056] 11 is a management label assigned to each action or event in the educational content, and is an arbitrary character string that does not overlap with other actions or events. The threshold value 1111, tactile coefficient 1112, auditory coefficient 1113, and visual coefficient 1114 are values ​​that are read into the sensory coefficient calculation processing unit 1026 and used to calculate sensory parameters, and are saved in the form of numerical data.

[0057] Each cell of the haptic coefficients 1112 indicates, for example, a value for vibration amplitude and frequency. Each cell of the auditory coefficients 1113 indicates, for example, a value for sound delay, volume, and frequency. Each cell of the visual coefficients 1114 indicates, for example, a value for CD ratio. The "default" record in the weighting elements 1110 indicates the default value for each coefficient. When the value of any weighting element other than the default exceeds the value indicated by the threshold 1111, the haptic, auditory, and visual parameters are updated using the values ​​of the haptic coefficients 1112, auditory coefficients 1113, and visual coefficients 1114. Details will be described later.

[0058] Returning to Figure 6, the sensory coefficient calculation processing unit 1026 further compares the weight w of the object obtained in S102 with the "weight" value in the threshold 1111 of the sensory information management database 111, and determines whether the weight of the object exceeds the threshold (S105).

[0059] If it is determined that the weight of the object is equal to or less than the threshold value (S105: YES), the current value of the sensory parameter is maintained. If it is determined that the weight of the object is greater than the threshold value (S105: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding visual and haptic parameters from the sensory information management database 111 and updates each sensory parameter using the weight w and / or the visual and haptic parameters (S106).

[0060] 6, the parameters of the vibration to be presented to the haptic sense are updated by adding or subtracting the value indicated by the haptic coefficient 1112 from the current values ​​of the amplitude and frequency, respectively. In other words, if the object is heavy, a vibration of a larger amplitude and a smaller frequency is given.

[0061] The parameter update for the sound presented to the auditory sense is performed by adding the product of the weight w and the value indicated by the auditory coefficient 1113 to the current delay time. Also, the values ​​indicated by the auditory coefficient 1113 are added or subtracted from the current values ​​of the volume and frequency. In other words, if the object is heavy, a larger delay and a sound with a larger volume and lower frequency are presented. The parameter update for the image presented to the visual sense is performed by subtracting the product of the weight and the value indicated by the visual coefficient 1114 from the current value of the CD ratio. In other words, if the object is heavy, an image with a smaller CD ratio is presented.

[0062] Next, the sensory information management device 100 performs a process of updating the parameters of the visual and tactile reproduction based on the number of hands with which the user grasps the object and a comparison of the hand trajectory and the angle of the server rack access port, as shown in Figure 7.

[0063] The virtual space contact / grasp detection processing unit 1021 determines whether the user is holding an object with both hands based on the user's movements acquired in S100 (S107). This determination can be made based on information on the positions of the user's hands and their fingers, as well as the position of the object.

[0064] If it is determined that the user is holding the object with one hand (S107: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding audio-visual and tactile parameters from the sensory information management database 111 and updates the sensory parameters (S108). This allows the user to be appropriately informed of the difference in sensation when moving an object between holding it with one hand and holding it with both hands.

[0065] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding or subtracting the value indicated by the "Number of gripping hands" cell in the haptic coefficient 1112 from the current values ​​of amplitude and frequency. In other words, when an object is gripped with one hand, vibrations of a larger amplitude and a smaller frequency are applied.

[0066] The parameter update for the sound presented to the auditory sense is performed by adding the value indicated by the "Number of gripping hands" cell in the auditory coefficient 1113 to the current delay time. Also, the value indicated by the "Number of gripping hands" cell in the auditory coefficient 1113 is added to or subtracted from the current values ​​of the volume and frequency. In other words, in the case of one-handed gripping, a sound with a longer delay, a higher volume, and a lower frequency is presented. The parameter update for the image presented to the visual sense is performed by subtracting the value indicated by the "Number of gripping hands" cell in the visual coefficient 1114 from the current value of the CD ratio. In other words, in the case of one-handed gripping, an image with a smaller CD ratio is presented. In the case of two-handed gripping (S107: YES), S108 is omitted, and the current values ​​of the sensory parameters are maintained.

[0067] Next, the sensation coefficient calculation processing unit 1026 acquires the hand trajectory when the user pulls out the object from the user's movement acquired in S100, compares it with the object's movable direction s acquired in S102, calculates the angle deviation d of the hand trajectory, and determines whether the deviation d exceeds a threshold (S109). In the case of a two-handed grasp, the angle obtained from the trajectories of both hands or the angle of the trajectory of one hand can be used.

[0068] If it is determined that the deviation d exceeds the threshold value (S109: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the deviation d and / or the visual and tactile parameters (S110).

[0069] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated by the cell for "deviation between moving direction and hand trajectory" in the haptic coefficient 1112 to the current value of amplitude, and subtracting the value indicated by the cell for "deviation between moving direction and hand trajectory" in the haptic coefficient 1112 from the current value of frequency. In other words, if the deviation in angle is large, a vibration with a larger amplitude and a smaller frequency is provided.

[0070] The parameters of the sound presented to the auditory sense are updated by adding to the current delay time the product of the deviation d and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113. Also, the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113 is added to the current value of the volume, and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the auditory coefficient 1113 is subtracted from the current value of the frequency. In other words, if the deviation is large, a sound with a larger delay, larger volume, and lower frequency is presented.

[0071] The parameters of the visually presented image are updated by subtracting the product of the deviation d and the value indicated by the cell for "deviation between moving direction and hand trajectory" in the visual coefficient 1114 from the current value of the CD ratio. In other words, if the deviation is large, an image with a smaller CD ratio is presented. If the deviation d is equal to or smaller than the threshold (S109: YES), S110 is skipped, and the current values ​​of the sensory parameters are maintained.

[0072] Next, the sensory information management device 100 executes the process shown in FIG. 8 to update the parameters for the audio-visual and tactile reproduction based on the speed of the user's hand and the friction coefficient of the target object.

[0073] The sensory coefficient calculation processing unit 1026 obtains the hand speed v when the user pulls out the object from the user's movement obtained in S100, and determines whether the speed v exceeds the threshold indicated by the sensory information management database 111 (S111).

[0074] If it is determined that the speed v exceeds the threshold value (S111: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the speed v and / or the visual and tactile parameters (S112).

[0075] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated by the "hand speed" cell of the haptic coefficients 1112 to the current value of the amplitude, and subtracting the value indicated by the "hand speed" cell of the haptic coefficients 1112 from the current value of the frequency. In other words, if the speed is fast, a vibration of larger amplitude and smaller frequency is given.

[0076] The parameters of the sound presented to the auditory sense are updated by adding the product of the speed v and the value indicated by the "hand speed" cell of the auditory coefficients 1113 to the current delay time. Also, the values ​​indicated by the "hand speed" cell of the auditory coefficients 1113 are added to or subtracted from the current values ​​of the volume and frequency. In other words, if the speed is fast, a sound with a larger delay, a larger volume, and a smaller frequency is presented.

[0077] The parameters of the image presented to the visual sense are updated by subtracting the product of the speed v and the value indicated by the "hand speed" cell in the visual coefficient 1114 from the current value of the CD ratio. In other words, if the speed is fast, an image with a smaller CD ratio is presented. If the speed v is equal to or less than the threshold (S111: YES), S112 is omitted and the current value of the sensory parameter is maintained.

[0078] Next, the sensory coefficient calculation processing unit 1026 determines whether the object's material (friction coefficient) m acquired in S102 and the sensory information management database has exceeded a threshold value (S113).

[0079] If it is determined that the friction coefficient m of the object exceeds the threshold value (S113: NO), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database and updates each sensory parameter using the friction coefficient m and / or the visual and tactile parameters (S114).

[0080] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the value indicated in the "Material" cell of the haptic coefficient 1112 to the current value of the amplitude, and subtracting the value indicated in the "Material" cell of the haptic coefficient 1112 from the current value of the frequency. In other words, if the coefficient of friction is large, vibrations of larger amplitude and smaller frequency are provided.

[0081] The parameters of the sound presented to the auditory sense are updated by adding or subtracting the product of the friction coefficient m and the value indicated in the "material" cell of the auditory coefficient 1113 to the current delay time. Also, the value indicated in the "material" cell of the auditory coefficient 1113 is added to the current value of the volume, and the value indicated in the "material" cell of the auditory coefficient 1113 is subtracted from the current value of the frequency. In other words, when the friction coefficient is large, a sound with a larger delay, a larger volume, and a smaller frequency is presented.

[0082] The parameters of the visually presented image are updated by subtracting the product of the friction coefficient m and the value indicated in the "Material" cell of the visual coefficient 1114 from the current value of the CD ratio. In other words, if the friction coefficient is large, an image with a smaller CD ratio is presented. If the friction coefficient m is equal to or smaller than the threshold value (S113: YES), S114 is omitted, and the current value of the sensory parameter is maintained.

[0083] Next, the sensory information management device 100 determines whether the module being held by the user has come off the server rack, as shown in Figure 9, and if it has, updates the parameters for visual and tactile reproduction based on the height of the module, and executes the process of visual and tactile reproduction based on the visual and tactile parameter information updated up to this point.

[0084] The sensory coefficient calculation processing unit 1026 obtains the amount of movement by which the user pulled out the module (object) from the user's movement obtained in S100, and determines whether the module has come out of the rack it is in based on the object shape 1104 and amount of movement obtained in S102 (S115).

[0085] If it is determined that the module has come off the rack (S115: YES), the sensory coefficient calculation processing unit 1026 obtains the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the weight v and / or the visual and tactile parameters, in which case the CD ratio is applied only to movement upward in height and not to movement downward (S116).

[0086] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the current value of the amplitude to the product of the weight w of the module and the value indicated in the "When module is removed" cell of the haptic coefficient 1112. In addition, the frequency is set to the value indicated in the "When module is removed" cell of the haptic coefficient 1112, which is the default value "Medium" in this case.

[0087] The parameters of the sound presented to the ear are updated by adding the value indicated by the "When module is disconnected" cell of the hearing coefficient 1113 to the current delay time. Here, the added value is 0, and the current values ​​are maintained. Also, the volume and frequency are set to the values ​​indicated by the "When module is disconnected" cell of the hearing coefficient 1113. Here, they are set to "medium," the same as the default value.

[0088] To update the parameters of the image presented to the eye, the CD ratio is set to a value obtained by subtracting from the default value "1" the product of the weight w and the value indicated in the "When module is removed" cell of the visual coefficient 1114. If the module is not removed from the rack (S115: NO), steps S116 to S118 are omitted.

[0089] Next, the sensory coefficient calculation processing unit 1026 obtains the height at which the user is holding the module from the user's movement obtained in S100, and compares it with the threshold value for the module height indicated by the threshold value 1111 in the sensory information management database 111 to determine whether the height h of the object being held exceeds the threshold value (S117).

[0090] If it is determined that the height h of the module exceeds the threshold value (S117: NO), the sensory coefficient calculation processing unit 1026 acquires the corresponding visual and tactile parameters from the sensory information management database 111 and updates each sensory parameter using the height h (height position h) and / or the visual and tactile parameters (S118). At this time, the CD ratio is applied only to movement in the upward direction of the height, and is not applied to movement downward.

[0091] Specifically, the parameters of the vibration presented to the haptic sense are updated by adding the product of the height position h of the module and the value indicated by the "Module Height" cell of the haptic coefficient 1112 to the current value of the amplitude. Also, the product of the height position h of the module and a predetermined sine wave is added to the current value of .

[0092] The parameters of the sound presented to the ear are updated by adding the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 to the current delay time. Also, the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 is added to the current value of the volume. Furthermore, the product of the height position h of the module and the value indicated by the "module height" cell of the hearing coefficient 1113 is added to the current value of the frequency.

[0093] The parameters of the image presented to the eye are updated by subtracting from the current value of the CD ratio the product of the height position h of the module and the value indicated by the "Module Height" cell of the visual coefficient 1114. If the height position of the module is equal to or less than h (S115: YES), S118 is omitted.

[0094] The virtual space CD ratio manipulation processing unit 1023 uses the latest updated audio-visual haptic parameters and the educational virtual space to transmit visual information to the head-mounted device 300 through the communication device 115, and plays back video on the head-mounted device 300. At the same time, the vibration information adjustment processing unit 1024 uses the latest updated audio-visual haptic parameters to transmit vibration information to the vibration presentation device 200 through the communication device 115, and presents vibration on the vibration presentation device 200. At the same time, the audio information adjustment processing unit 1025 uses the latest updated audio-visual haptic parameters to transmit audio information to the head-mounted device 300 through the communication device 115, and presents audio on the head-mounted device 300 (S119).

[0095] The virtual space contact / grasp detection processing unit 1021 determines whether the user has released the object in the virtual space based on the user action read in S100 (S120). If it is determined that the user has released the object in the virtual space (S120: YES), the virtual space CD ratio operation processing unit 1023, the vibration information adjustment processing unit 1024, and the auditory information adjustment processing unit 1025 end the tactile presentation processing (S121). If it is determined that the user is continuing to grasp the object in the virtual space (S120: NO), the flow returns to S115.

[0096] 6 to 9 controls a combination of multiple sensory parameters when moving an object. Some of the sensory parameters described may be maintained unchanged, and other sensory parameters may be additionally controlled. Furthermore, the adjustment method for each sensory parameter is merely an example, and control may be performed using a method different from the above method.

[0097] Figure 12 shows a user 510 wearing an HMD 501 removing a module 503 from a server rack 502 in a virtual space 500 and lifting it up and down, while receiving vibration stimulation from a tactile glove 504 and auditory stimulation from an HMD 1201.

[0098] 13A shows an example of visual CD ratio manipulation in which, when a movement amount of a lifting action in reality is 15 cm, the movement amount is reduced to 5 cm when lifting a heavy object in virtual space. FIG. 13B shows an example of visual CD ratio manipulation in which, when a movement amount of a lifting action in reality is 15 cm, the movement amount is reduced to 10 cm when lifting a light object in virtual space.

[0099] While the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this and various modifications are possible without departing from the spirit of the present invention. According to this embodiment, the sensation when holding or moving an object can be felt with an intensity that is more suited to the situation, in accordance with the constraints of actual movements.

[0100] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0101] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0102] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A system for controlling stimuli presented to the senses of a user when the user moves an object in a virtual space, the system comprising: a display device that presents an image of the virtual space including an object to the user; a measurement device that measures the movement of the user's hand; a vibration presentation device that presents vibrations to the user's hand; an audio device that presents sound to the user; and a control device, wherein the control device controls multiple parameters, based on the attributes and / or state of the object, among the visual parameters of the image of the object, the tactile parameters of the vibration of the vibration presentation device, and the auditory parameters of the sound of the audio device, when the object moves in the virtual space.

2. A system according to claim 1, wherein the object attributes include at least one of the weight and coefficient of friction of the object, and the object state includes at least one of the movable direction and height position of the object.

3. A system according to claim 1, wherein the control device determines the number of hands grasping the object in the virtual space, and controls the plurality of parameters based on the number of hands grasping the object.

4. The system of claim 1, wherein the state of the object includes a moving direction of the object, and the control device controls the plurality of parameters based on an angular deviation between the trajectory of the user's hand and the moving direction.

5. A system according to claim 1, wherein the control device controls the plurality of parameters based on the speed of the user's hand gripping the object in the virtual space.

6. A system according to claim 1, wherein the control device controls the plurality of parameters based on the height position of the object when the object exists in free space in the virtual space.

7. A system according to claim 1, wherein the control device initiates parameter control processing for moving the object when the object is held by the user's hand in the virtual space.

8. A method for controlling stimuli presented to a user's senses when the user moves an object in a virtual space, the system including: a display device that presents an image of the virtual space including the object to the user; a measurement device that measures the user's hand movements; a vibration presentation device that presents vibrations to the user's hand; and an audio device that presents sound to the user, the system controlling multiple parameters among the visual parameters of the image of the object, the tactile parameters of the vibration of the vibration presentation device, and the auditory parameters of the sound of the audio device when the object moves in the virtual space, based on the attributes and / or state of the object.

Images