Virtual reality terminal device and operation support method

Abstract

The problem this invention aims to solve is to enable the recognition of the orientation of objects in a virtual reality space in three dimensions. [Solution] One aspect of this invention, when assisting a user's operation on an object placed in a virtual reality space, obtains the coordinates of the object set as the target of operation in the virtual reality space, and sets sound source coordinates in the virtual reality space according to the coordinates of this object. Then, obtains the pointing coordinates in the virtual reality space instructed by the user's operation, and controls different types of auditory stimulus parameters set corresponding to each of the three-dimensional axes of the virtual reality space according to the three-dimensional positional relationship between these pointing coordinates and the sound source coordinates, and generates an auditory stimulus signal that reflects the controlled auditory stimulus parameters to present an auditory stimulus to the user.

Description

Technical Field

[0001] One aspect of this invention relates to a virtual reality terminal device having a function of assisting a user's operation on an object in a virtual reality space, and an operation assistance method executed by this device.

Background Art

[0002] In a virtual reality (VR) space, virtual objects can be freely arranged in a three-dimensional space and the user can operate these objects. For example, buttons and panels can be arranged in the VR space and the user can operate them using a finger or the like, or a virtual object can be placed in the VR space and the object can be grasped and operated. At this time, by tracking the position of the user's hand and the position of the controller held in the hand with a camera or an inertial sensor, it is possible to touch an object outside the field of view. As a result, the user can operate an object just by stretching out a hand without visually recognizing his or her own hand or the controller, so that work efficiency can be expected. In addition, since there is no need to move the viewpoint, there is also an advantage that it is difficult to cause VR sickness.

[0003] By the way, when trying to touch an object outside the field of view, in the VR space, since the tactile sensation when touching the object is not fed back to the user, the user needs to identify the position of the object relying only on proprioception.

[0004] However, relying only on proprioception, even when trying to stretch out a hand to a target position, the position of the hand may deviate by about 10 cm, for example. In order to solve this problem, Non-Patent Document 1 describes a method of making it easier to identify the position of an object by using an auditory stimulus by sound and changing the frequency of the auditory stimulus according to the distance between the position of the object and the position of the controller.

Prior Art Documents

Non-Patent Documents

[0005] [Non-Patent Document 1] Gao, BoYu, et al. "Spherical layout with proximity-based multimodal feedback for eyes-free target acquisition in virtual reality." Virtual, Augmented and Mixed Reality. Multimodal Interaction: 11th International Conference, VAMR 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26-31, 2019, Proceedings, Part I 21. Springer International Publishing, 2019. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, the method described in Non-Patent Document 1 has the problem that while the user can recognize the distance to an object from the difference in sound frequency, they cannot recognize the direction in which it is shifted in three-dimensional space.

[0007] This invention was made in view of the above circumstances and aims to provide a technology that enables the distinction and recognition of the orientation of objects in a virtual reality space in three dimensions. [Means for solving the problem]

[0008] To solve the above problems, one embodiment of the virtual reality terminal device or operation support method according to the present invention, when supporting a user's operation on an object placed in a virtual reality space, acquires the coordinates of the object set as the user's operation target in the virtual reality space, and sets sound source coordinates in the virtual reality space according to the coordinates of the object. In this state, acquires the pointing coordinates in the virtual reality space instructed by the user's operation, controls different types of auditory stimulus parameters set corresponding to each three-dimensional axis in the virtual reality space according to the three-dimensional positional relationship between the acquired pointing coordinates and the sound source coordinates, generates an auditory stimulus signal that reflects the controlled auditory stimulus parameters, and presents an auditory stimulus to the user.

[0009] According to one aspect of this invention, the user is presented with auditory stimuli that reflect three types of auditory stimulus parameters set corresponding to each axis of the three-dimensional virtual reality space. As a result, the user can recognize the direction and distance of the operation on the object to be manipulated in three-dimensional space through these auditory stimuli, thereby enabling smooth and accurate identification of the target's position. [Effects of the Invention]

[0010] In other words, according to one aspect of this invention, it is possible to provide a technology that enables the recognition of the orientation of objects in a virtual reality space in three dimensions. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a block diagram showing an example of the hardware configuration of a virtual reality terminal device according to one embodiment of the present invention. [Figure 2] Figure 2 is a block diagram showing an example of the software configuration of a virtual reality terminal device according to one embodiment of the present invention. [Figure 3] Figure 3 is a flowchart showing an example of the processing procedure and content of auditory stimulus presentation control performed by the control unit of the virtual reality terminal device shown in Figure 2. [Figure 4] Figure 4 shows an example of how a virtual reality image is displayed. [Figure 5] Figure 5 is a diagram illustrating a first example of auditory stimulus presentation control. [Figure 6] Figure 6 is a diagram illustrating a second example of auditory stimulus presentation control. [Modes for carrying out the invention]

[0012] Embodiments of this invention will be described below with reference to the drawings.

[0013] [One embodiment] (overview) One embodiment of this invention assists a user in manipulating an object placed in a VR space by moving their hand outside their field of view while viewing a VR space image through the HMD. This is achieved by setting sound source coordinates based on the coordinate position of the object in the VR space, and then individually controlling three types of pre-set auditory stimulus parameters corresponding to each of the three-dimensional axes based on the three-dimensional positional relationship between these sound source coordinates and the pointing coordinates in the VR space indicated by the user's hand movements. This enables the presentation of the three-dimensional positional relationship of the user's hand to the object using auditory stimuli.

[0014] As a result, the user is presented with auditory stimuli that reflect three types of auditory stimulus parameters set to correspond to each axis of the three-dimensional virtual reality space. Therefore, the user can recognize the direction and distance of the operation on the object being manipulated in three-dimensional space through these auditory stimuli, thereby enabling smooth and accurate identification of the target's position.

[0015] (Example configuration) FIG. 1 and FIG. 2 are block diagrams showing an example of the hardware configuration and software configuration of a virtual reality (VR) terminal device TM according to an embodiment of the present invention.

[0016] The VR terminal device TM is constituted by, for example, a head-mounted display (HMD), and includes a control unit 1 using a hardware processor such as a central processing unit (CPU). The control unit 1 is connected via a bus 6 to a storage medium including a program storage unit 2 and a data storage unit 3, a communication interface (hereinafter, the interface is referred to as I / F) unit 4, and an input / output I / F unit 5.

[0017] The communication I / F unit 4 receives image data representing a VR space transmitted from a server device that operates and manages a VR space such as a metaverse, or transmits coordinate data obtained by tracking the movement of a user's hand to the server device in accordance with a communication protocol defined by a network such as the Internet (not shown).

[0018] The input / output I / F unit 5 is connected to a display device DP, a camera CM, and an acoustic generation device SG, respectively. The display device DP is used to display a VR space image. The camera CM is used to track the movement of a user's hand, and captures a real space including the user's hand and outputs the captured image data. Note that an acceleration sensor may be used as a device for tracking the movement of a user's hand.

[0019] The acoustic generation device SG generates an acoustic signal in accordance with an auditory stimulus control signal output from the control unit 1, and supplies the generated acoustic signal to a plurality of speakers SP to output it as sound. The acoustic generation device SG and the speakers SP constitute an auditory stimulus presentation unit.

[0020] The program memory unit 2 is configured by combining, for example, a non-volatile memory such as an SSD (Solid State Drive) that can be written and read at any time as a storage medium and a non-volatile memory such as a ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores application programs necessary for executing various controls according to one embodiment. Hereinafter, the OS and each application program are collectively referred to as a program.

[0021] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written and read at any time as a storage medium and a volatile memory such as a RAM (Random Access Memory). In its storage area, as the main storage unit according to one embodiment of this invention, a target coordinate storage unit 31, a sound source coordinate storage unit 32, and a pointing coordinate storage unit 33 are provided.

[0022] The target coordinate storage unit 31 stores coordinate data of an object (hereinafter also referred to as a target) set as an operation target in the VR space. The sound source coordinate storage unit 32 stores coordinate data of a sound source set in the VR space. The sound source is for allowing the user to recognize the three-dimensional position of the target using an auditory stimulus. The pointing coordinate storage unit 33 stores coordinate data indicating a pointing position on the VR space indicated by the user's hand.

[0023] The control unit 1 includes, as processing functions necessary for realizing one embodiment of this invention, a VR space information display processing unit 11, a target setting processing unit 12, a target coordinate acquisition processing unit 13, a sound source coordinate setting processing unit 14, a pointing coordinate acquisition processing unit 15, and an auditory stimulus parameter control processing unit 16.

[0024] Each of the above-mentioned processing units 11 to 16 is implemented by having the hardware processor of the control unit 1 execute the application program stored in the program storage unit 2. Note that some or all of the above-mentioned processing units 11 to 16 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0025] The VR spatial information display processing unit 11 receives VR spatial information transmitted from a server device (not shown) via the communication I / F unit 4, and outputs the VR image data contained in the received VR spatial information to the display device DP via the input / output I / F unit 5 for display.

[0026] The target setting processing unit 12 sets, for example, an object that the user has set as the object to be manipulated in the VR space as the target. Note that other users or administrators can also set the target.

[0027] The target coordinate acquisition processing unit 13 acquires the coordinate data of the target in the VR space set by the target setting processing unit 12, and stores the acquired target coordinate data in the target coordinate storage unit 31.

[0028] The sound source coordinate setting processing unit 14 sets sound source coordinate data in the VR space according to the coordinate data of the target, and stores the set sound source coordinate data in the sound source coordinate storage unit 32. An example of the sound source coordinate setting process is described in the operation example.

[0029] The pointing coordinate acquisition processing unit 15 acquires real-space image data captured by the camera CM via the input / output interface unit 5, and detects the position of the user's hand from the acquired real-space image data. Then, in order to control the position of the hand avatar in the VR space, the pointing coordinate acquisition processing unit 15 transmits the detected position coordinates of the user's hand from the communication interface unit 4 to the server device.

[0030] Furthermore, the pointing coordinate acquisition processing unit 15 acquires the coordinate data of the pointing position from the VR space information in which the position of the hand avatar is controlled, and stores the acquired pointing coordinate data in the pointing coordinate storage unit 33.

[0031] The auditory stimulus parameter control processing unit 16 individually controls three types of pre-set auditory stimulus parameters corresponding to the x, y, and z axes representing the three-dimensional space, based on the three-dimensional positional relationship between the sound source coordinate data stored in the sound source coordinate storage unit 32 and the pointing coordinate data stored in the pointing coordinate storage unit 33. An example of this control operation of auditory stimulus parameters in three-dimensional space will be described in detail in the operation example.

[0032] (Example of operation) Next, we will explain an example of the operation of the VR terminal device TM configured as described above.

[0033] Figure 3 is a flowchart showing an example of the control procedure and processing content executed by the control unit 1 of the VR terminal device TM to support the user's operation on an object using auditory stimuli.

[0034] For example, suppose a user wearing a head-mounted display (HMD) inputs a request to display VR space information. When the control unit 1 of the VR terminal device TM detects the display request in step S10, in step S11, under the control of the VR space information display processing unit 11, it receives the VR space information transmitted from the server device that manages the VR space via the communication I / F unit 4. The received VR space information is then output from the input / output I / F unit 5 to the display device DP, which displays it as a VR space image VD.

[0035] Figure 4 shows an example of a displayed VR space image VD. In this example, the VR space displays multiple objects T1-T4, which consist of three-dimensional images, and an avatar HD representing the user's hands. Objects T1-T4 are operation buttons in this example.

[0036] (1) Setting a target for an object and obtaining the target coordinates With the above VR space image displayed, if, for example, the user specifies a target to be operated on for the objects T1 to T4, the control unit 1 of the VR terminal device TM, under the control of the target setting processing unit 12, sets the specified object as a target in step S12. This example shows the case where all of objects T1 to T4 are set as targets.

[0037] Once the above target is set, the control unit 1 of the VR terminal device TM, under the control of the target coordinate acquisition processing unit 13, acquires the three-dimensional coordinate data of the objects T1 to T4 set as targets in the VR space in step S13. Then, the acquired three-dimensional coordinate data of the objects T1 to T4 is stored in the target coordinate storage unit 31.

[0038] (2) Setting the sound source coordinates Once the above target is set, the control unit 1 of the VR terminal device TM then performs the process of setting sound source coordinates in the VR space under the control of the sound source coordinate setting processing unit 14 as follows.

[0039] In other words, the sound source coordinate setting processing unit 14 first calculates the distance between objects T1 to T4 in the VR space based on the three-dimensional coordinates of the objects T1 to T4 set as targets in step S14, and determines whether the calculated distance is shorter than a preset threshold. For example, if the threshold is set to 10 cm, it determines whether the distance between objects T1 to T4 is shorter than 10 cm.

[0040] Based on the determination result of the distance between the objects T1 to T4, the sound source coordinate setting processing unit 14, if the distance between the objects T1 to T4 is 10 cm or more, sets the sound source coordinates at the coordinate positions of each object T1 to T4 in step S15 and stores the set sound source coordinate data in the sound source coordinate storage unit 32.

[0041] Figure 5 shows an example where sound source coordinates are set at the coordinate positions of objects T1 to T4. In this example, sound source coordinates are set at coordinate positions O1 to O4 of objects T1 to T4, respectively.

[0042] In contrast, suppose the distance between objects T1 to T4 is less than 10 cm. In this case, the sound source coordinate setting processing unit 14 sets the sound source coordinates at the center coordinate position between objects T1 to T4 in step S16, and stores the set sound source coordinate data in the sound source coordinate storage unit 32.

[0043] Figure 6 shows an example where the sound source coordinates are set at the central coordinate position between objects T1 to T4. In this example, the sound source coordinates are set at the central coordinate position O0 between each of the objects T1 to T4.

[0044] (3) Control of auditory stimulus presentation Next, we will explain the control of auditory stimulus presentation in response to pointing operations, divided into two cases: one where the sound source coordinates are set to the coordinate positions O1 to O4 of the objects T1 to T4, and another where the sound source coordinates are set to the central coordinate position O0 between the objects T1 to T4.

[0045] (3-1) When the sound source coordinates are set to the coordinate positions O1 to O4 of objects T1 to T4 (3-1-1) Obtaining pointing coordinates With the above sound source coordinates set to the coordinate positions O1 to O4 of objects T1 to T4, the control unit 1 of the VR terminal device TM, under the control of the pointing coordinate acquisition processing unit 15, executes the process of acquiring the pointing coordinates indicated by the user's hand movements as follows.

[0046] In other words, when a user moves their hand to point to objects T1 to T4, the pointing coordinate acquisition processing unit 15 detects the hand movement from the image data captured by camera CM in step S17. Once the hand movement is detected, the pointing coordinate acquisition processing unit 15 detects the real-world position coordinates that the hand is pointing to from the image data and transmits the detected position coordinate data from the communication I / F unit 4 to the server device operating the VR space. As a result, the server device reflects the hand's position coordinates in the position of the hand avatar in the VR space. That is, the position of the avatar in the VR space is controlled to follow the hand movement.

[0047] In step S18, the pointing coordinate acquisition processing unit 15 acquires the pointing coordinates indicated by the avatar from the VR space information in which the avatar's position is controlled. The pointing coordinate acquisition processing unit 15 then stores the acquired pointing coordinate data in the pointing coordinate storage unit 33. Figure 5 shows an example of the hand avatar HD after control and its pointing position P.

[0048] Alternatively, the pointing coordinate acquisition processing unit 15 may convert the position coordinates of the user's hand in real space detected from the captured image data into pointing coordinates in the VR space, and store this pointing coordinate data in the pointing coordinate storage unit 33.

[0049] (3-1-2) Control of auditory stimulus parameters When pointing coordinate data is detected by the user's hand movements, the control unit of the VR terminal device™ controls the auditory stimulus parameters in step S19 under the control of the auditory stimulus parameter control processing unit 16 as follows.

[0050] In other words, the auditory stimulus parameter control processing unit 16 first defines the x, y, and z axes that represent three dimensions in the VR space. For example, as shown in Figure 5, the x-axis is set in the horizontal scanning direction of the VR space image VD, the y-axis in the vertical scanning direction, and the z-axis in the depth direction. Note that the z-axis is not shown in Figure 5.

[0051] Furthermore, the auditory stimulus parameter control processing unit 16 associates three types of auditory stimulus parameters with each of the x, y, and z axes. For example, the x-axis is associated with sound localization, the y-axis with sound frequency, and the z-axis with sound amplitude.

[0052] The auditory stimulus parameter control processing unit 16 then reads sound source coordinate data from the sound source coordinate storage unit 32 and pointing coordinate data from the pointing coordinate storage unit 33. It then calculates the three-dimensional relationship between the read sound source coordinate data and pointing coordinate data, that is, the three-dimensional direction of the pointing coordinates as seen from the sound source coordinates and the distance between the sound source coordinates and the pointing coordinate P.

[0053] For example, in the VR spatial image VD shown in Figure 5, the three-dimensional directions represented by the x, y, and z axes of the pointing coordinate P as seen from the sound source coordinate O1, and the three-dimensional distance represented by the x, y, and z axes between the sound source coordinate O1 and the pointing coordinate P are calculated.

[0054] Next, the auditory stimulus parameter control processing unit 16 controls the sound localization along the x-axis, the sound frequency along the y-axis, and the sound amplitude along the z-axis, based on the calculated three-dimensional directions.

[0055] For example, if the pointing coordinate P is located to the right of the sound source coordinate O1 in the VR space, the sound localization is placed to the right side of the VR space; conversely, if it is located to the left, the sound localization is placed to the left side of the VR space. Also, if the pointing coordinate P is located above the sound source coordinate O1 in the VR space, the sound frequency is set to a frequency higher than the reference frequency of the sound source coordinate O1; if it is located below, the sound frequency is set to a frequency lower than the reference frequency of the sound source coordinate O1. Furthermore, if the pointing coordinate P is located in front of the sound source coordinate O1 in the VR space, the sound amplitude is set to a value greater than the reference value at the sound source coordinate O1; if it is located behind, the sound amplitude is set to a value smaller than the reference value at the sound source coordinate O1.

[0056] Furthermore, the auditory stimulus parameter control processing unit 16 controls the generation of sound controlled by the three types of auditory stimulus parameters according to the distance from the sound source coordinate O1 to the pointing coordinate P.

[0057] For example, as shown in Figure 5, the auditory stimulus parameter control processing unit 16 sets up multiple concentric auditory stimulus presentation areas around targets T1 to T4, in this example, three areas D1, D2, and D3. Then, when the position of the pointing coordinate P enters one of the auditory stimulus presentation areas D1, D2, or D3, in step S20, the auditory stimulus parameter control processing unit 16 generates an auditory stimulus control signal for each of the areas D1, D2, and D3 according to the auditory stimulus parameters set for each of the x, y, and z axes as described above, and outputs this auditory stimulus control signal from the input / output I / F unit 5 to the sound generation device SG.

[0058] As a result, the sound generation device SG generates an acoustic signal in which the direction of the pointing coordinate as seen from the sound source coordinate O1 and the distance from the sound source coordinate O1 to the auditory stimulus presentation areas D1, D2, and D3 are reflected in the sound localization, frequency, and amplitude, according to the auditory stimulus control signal. Sound is then generated from multiple speakers SP according to this acoustic signal.

[0059] In other words, for example, the three-dimensional positional relationship of the pointing position relative to target T1 is reflected in the sound's localization, frequency, and amplitude, and this is presented to the user as an auditory stimulus.

[0060] As a result, the user can recognize the current three-dimensional spatial positional relationship of their hand relative to target T1 through the auditory stimuli described above, that is, the discrepancy in direction and distance, which enables them to identify the target's position more smoothly and accurately.

[0061] The method for changing the auditory stimulus parameters described above can be, for example, a method of continuously changing them linearly at a constant rate for each pixel of the VR spatial image, or a method of discretely changing them exponentially, for example, by doubling the amount of change every 1 cm.

[0062] The series of auditory stimulus presentation controls described in steps S17 to S20 are repeatedly executed in response to the user's hand movements until a command to end the display of the VR space is detected in step S21.

[0063] (3-2) When the sound source coordinates are set to the center coordinate position O0 between objects T1 and T4 (3-2-1) Obtaining pointing coordinates The process for obtaining pointing coordinates in response to the user's hand movements is the same as the process described in (3-1-1), so the explanation is omitted here.

[0064] (3-2-2) Control of auditory stimulus parameters When pointing coordinate data is detected by the user's hand movements, the control unit of the VR terminal device™ controls the auditory stimulus parameters in step S19 under the control of the auditory stimulus parameter control processing unit 16 as follows.

[0065] In other words, the auditory stimulus parameter control processing unit 16 first defines the x, y, and z axes that represent three dimensions in the VR space. For example, as shown in Figure 6, the x-axis is set in the horizontal scanning direction of the VR space image VD, the y-axis in the vertical scanning direction, and the z-axis in the depth direction. Note that the z-axis is not shown in Figure 6.

[0066] Furthermore, the auditory stimulus parameter control processing unit 16 associates three types of auditory stimulus parameters with each of the x, y, and z axes. For example, the x-axis is associated with sound localization, the y-axis with sound frequency, and the z-axis with sound amplitude.

[0067] The auditory stimulus parameter control processing unit 16 then reads sound source coordinate data from the sound source coordinate storage unit 32 and pointing coordinate data from the pointing coordinate storage unit 33. It then calculates the three-dimensional relationship between the read sound source coordinate data and pointing coordinate data, that is, the three-dimensional direction of the pointing coordinates as seen from the sound source coordinates and the distance between the sound source coordinates and the pointing coordinate P.

[0068] For example, in the VR spatial image VD shown in Figure 6, the directions represented by the x, y, and z axes of the pointing coordinate P as seen from the sound source coordinate O0, and the distances represented by the x, y, and z axes between the sound source coordinate O0 and the pointing coordinate P are calculated, respectively.

[0069] Next, the auditory stimulus parameter control processing unit 16 controls the sound localization along the x-axis, the sound frequency along the y-axis, and the sound amplitude along the z-axis, with respect to the calculated three-dimensional directions.

[0070] For example, if the pointing coordinate P is located to the right of the sound source coordinate O0 in the VR space, the sound localization is placed to the right side of the VR space; conversely, if it is located to the left, the sound localization is placed to the left side of the VR space. Also, if the pointing coordinate P is located above the sound source coordinate O0 in the VR space, the sound frequency is set to a higher frequency than the reference frequency of the sound source coordinate O0; if it is located below, the sound frequency is set to a lower frequency than the reference frequency of the sound source coordinate O0. Furthermore, if the pointing coordinate P is located in front of the sound source coordinate O0 in the VR space, the sound amplitude is set to a value greater than the reference value at the sound source coordinate O0; if it is located behind, the sound amplitude is set to a value smaller than the reference value at the sound source coordinate O0.

[0071] Furthermore, the auditory stimulus parameter control processing unit 16 controls the generation of sound controlled by the three types of auditory stimulus parameters according to the distance from the sound source coordinate O0 to the pointing coordinate P.

[0072] For example, as shown in Figure 6, the auditory stimulus parameter control processing unit 16 sets up multiple concentric auditory stimulus presentation areas around the sound source coordinate O0, in this example, three areas D1, D2, and D3. Then, when the position of the pointing coordinate P enters one of the auditory stimulus presentation areas D1, D2, or D3, in step S20, the auditory stimulus parameter control processing unit 16 generates an auditory stimulus control signal for each of the areas D1, D2, and D3 according to the auditory stimulus parameters set for each of the x, y, and z axes as described above, and outputs this auditory stimulus control signal from the input / output I / F unit 5 to the sound generation device SG.

[0073] As a result, the sound generation device SG generates an acoustic signal that reflects the directions represented by the x, y, and z axes of the pointing coordinate P as seen from the sound source coordinate O0, and the distance represented by the x, y, and z axes between the sound source coordinate O0 and the pointing coordinate P, in accordance with the auditory stimulus control signal. Sound is then generated from multiple speakers SP according to this acoustic signal.

[0074] In other words, for example, the three-dimensional positional relationship of the pointing position relative to targets T1 to T4 is reflected in the sound localization, frequency, and amplitude, and the resulting sound is presented to the user as an auditory stimulus.

[0075] As a result, the user can recognize the current positional relationship of their hand in three-dimensional space relative to targets T1-T4, that is, the discrepancy in distance and direction, through the auditory stimuli described above. Then, while viewing the current pointing position of the avatar HD displayed in the VR space image, the user can move the pointing position to a desired target among targets T1-T4, for example, T1.

[0076] In this case as well, the method of changing the auditory stimulus parameters may be, for example, a method of continuously changing them linearly at a constant rate for each pixel of the VR spatial image, or a method of discretely changing them exponentially, for example, by doubling the amount of change every 1 cm.

[0077] (effect) As described above, in one embodiment, auditory stimulus presentation control is performed as follows.

[0078] In other words, the sound source coordinates are set according to the coordinates of the target object among the objects placed in the VR space. In this state, when the pointing position changes according to the user's hand movements, three types of auditory stimulus parameters, which are set corresponding to the x, y, and z axes in three-dimensional space, are controlled according to the positional relationship between the sound source coordinates and the pointing coordinates in three-dimensional space, and auditory stimuli with sounds that reflect these three types of auditory stimulus parameters are presented to the user.

[0079] In this way, the user can recognize the direction and distance in three-dimensional space of the pointing operation to the target through auditory stimuli generated using three types of auditory stimulus parameters, thereby enabling smooth and accurate identification of the target's position.

[0080] Furthermore, when setting the sound source coordinates, if there is a single target, or if there are multiple targets but the distance between them exceeds a threshold, the sound source coordinates are set to the coordinate position of the target. On the other hand, if there are multiple targets and the distance between them is less than a threshold, the sound source coordinates are set to the center between the targets.

[0081] Therefore, when multiple targets are sufficiently spaced apart, the user can identify each target individually while receiving auditory stimuli for each target. Furthermore, when multiple targets are clustered together, the central position is presented as a temporary target for auditory stimuli, thus preventing the problem of sounds corresponding to each target becoming mixed and difficult to distinguish, which can occur when auditory stimuli are presented for each target.

[0082] [Other embodiments] (1) In one embodiment, the case in which "localization," "frequency," and "amplitude" are used as three types of auditory stimulus parameters was explained as an example, but it is also possible to use other parameters such as "sound intermittent pattern" or "sound intermittent period."

[0083] (2) In one embodiment, the case in which the user moves their own hand to perform pointing operations was described as an example, but this invention is also applicable when the user operates a controller to perform pointing operations. In this case, it is possible to obtain the pointing coordinates by obtaining the operation information of the controller.

[0084] (3) In addition, the types and configurations of VR terminal devices, the configurations and processing procedures of each processing function performed by the control unit, the types of objects, etc., can be modified in various ways without departing from the spirit of this invention.

[0085] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0086] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of Symbols]

[0087] TM...VR terminal device DP…Display device CM...Camera SG... Acoustic generation device SP...Speaker HD... Avatar 1…Control Unit 2…Program memory 3…Data storage unit 4…Communication I / F section 5…Input / Output I / F section 6... Bus 11…VR Spatial Information Display Processing Unit 12…Target setting processing unit 13…Target coordinate acquisition processing unit 14…Sound source coordinate setting processing unit 15…Pointing coordinate acquisition processing unit 16…Auditory stimulus parameter control processing unit 31...Target coordinate storage unit 32…Sound source coordinate storage unit 33... Pointing coordinate storage unit

Claims

1. A virtual reality terminal device having a function to assist user interaction with objects placed in a virtual reality space, A first processing unit that acquires the coordinates in the virtual reality space of the object set as the target of the user's operation, A second processing unit sets sound source coordinates in the virtual reality space according to the coordinates of the aforementioned object, A third processing unit that acquires the pointing coordinates in the virtual reality space as instructed by the user's operation, A fourth processing unit sets different types of auditory stimulus parameters corresponding to each of the three-dimensional axes set in the virtual reality space, and controls each of the auditory stimulus parameters corresponding to each of the three-dimensional axes according to the three-dimensional positional relationship of the pointing coordinates with respect to the sound source coordinates. A fifth processing unit generates an auditory stimulus signal that reflects the aforementioned auditory stimulus parameters and presents the auditory stimulus to the user. A virtual reality terminal device equipped with [a specific feature / feature].

2. The second processing unit is, The system determines whether the distance between the multiple objects set as targets for the user's operation is greater than or less than a predetermined threshold. If the distance between the objects is determined to be greater than or equal to the threshold, the sound source coordinates are set to the coordinates of each of the multiple objects. If the distance between the objects is determined to be less than the threshold, the sound source coordinates are set to the center position between the coordinates of the multiple objects. The virtual reality terminal device according to claim 1.

3. The fourth processing unit sets the sound localization, frequency, and amplitude as auditory stimulus parameters for each of the three-dimensional axes set in the virtual reality space, and controls the sound localization, frequency, and amplitude corresponding to each of the three-dimensional axes according to the three-dimensional direction and distance of the pointing coordinates relative to the sound source coordinates. The virtual reality terminal device according to claim 1.

4. An operation support method that uses an information processing device to perform processing to support user operations on objects placed in a virtual reality space, The process of obtaining the coordinates in the virtual reality space of the object set as the target of the user's operation, The process of setting sound source coordinates in the virtual reality space according to the coordinates of the aforementioned object, The process of obtaining the pointing coordinates in the virtual reality space as instructed by the user's operation, A process of setting different types of auditory stimulus parameters corresponding to each of the three-dimensional axes set in the virtual reality space, and controlling each of the auditory stimulus parameters corresponding to each of the three-dimensional axes according to the three-dimensional positional relationship of the pointing coordinates with respect to the sound source coordinates, A process of generating an auditory stimulus signal that reflects the aforementioned auditory stimulus parameters and presenting the auditory stimulus to the user. An operation support method comprising the following:

Images