Program and acoustic control device

Abstract

To improve user experience in a service provided in a virtual space.SOLUTION: A display 61 is worn by a user, and provides a video of a virtual space to the user. A detection device 63 outputs a first signal to a computer system in response to the instruction of the user. The detection device 63 outputs a second signal in response to change of a position of a part in a body of the user. A conversion part 562 converts the second signal into a position in the virtual space. When a predetermined first signal is output, a reproduction part 564 reproduces voice with a position obtained by conversion regarded as a position of a virtual sound source in the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a program and an audio control device. [Background technology]

[0002] Among game programs, there are those that reproduce sound effects in a virtual space (see, for example, Patent Document 1). In the example of Patent Document 1, gunshot sounds and the like are reproduced during a shooting game. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-156589 A Summary of the Invention [Problem to be solved by the invention]

[0004] When various services (e.g., games) are provided in virtual space, it is required that audio appropriate for the situation be reproduced from the perspective of user experience.

[0005] An object of the present disclosure is to improve the user experience in services provided in virtual space. [Means for solving the problem]

[0006] A first aspect is a program for causing a computer system having one computer or a plurality of computers working together to function as a conversion unit and a reproduction unit, comprising: the computer system is connected to a display and a detector; the display is worn by a user and provides an image of a virtual space to the user; the detection device outputs a first signal to the computer system in response to an instruction from the user; The detection device outputs a second signal corresponding to a change in a position of a part on the user's body; The conversion unit converts the second signal into a position in the virtual space; When a predetermined first signal is output, the reproduction unit reproduces sound by assuming the position obtained by the conversion as the position of a virtual sound source in the virtual space. It is a program.

[0007] In a first aspect, The conversion unit performs the conversion based on an orientation of the display; The reproduction unit may reproduce the sound without localizing a sound image when the position obtained by the conversion is included in a predetermined range (hereinafter referred to as a first range).

[0008] In the above aspect, The reproduction section may set the first range according to a type of the audio.

[0009] In the above aspect, The computer system is caused to function as a display unit; The display unit displays an image of a virtual medium on the display in correspondence with the position obtained by the conversion; The reproduction unit may select an audio to be reproduced depending on a type of the virtual medium.

[0010] A second aspect of the present invention is a method for controlling a computer-implemented program for a computer-implemented system, comprising: A control unit that executes the program; It is an acoustic control device equipped with the above. Effect of the Invention

[0011] According to the present disclosure, it is possible to improve the user experience. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing the configuration of the game device. [Diagram 2] FIG. 1 is a perspective view showing an example of VR goggles. [Diagram 3] FIG. 2 is a schematic diagram showing a state in which a user is wearing VR goggles. [Figure 4] 11 is a flowchart illustrating a process in a playback unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [Embodiment] An example in which the program is implemented as a game program and the sound control device is realized as a game device will be described below. The game device is an example of a computer system having one computer or multiple computers working together.

[0014] Game Overview The game played by this game program is played in a virtual space (three-dimensional). This game program is implemented using Virtual Reality (abbreviated as VR) technology. VR technology creates a virtual space on a computer that resembles reality, allowing the user to experience the sensation of being in that virtual space.

[0015] There is no limitation on the type of game, etc. In this embodiment, a battle-type action game in which a player character attacks and defeats an enemy character will be described as an example of the game.

[0016] This game program is implemented to execute a game from a first-person perspective. In this game program, a character such as a player character may possess a virtual medium within the game screen. Possession here means, for example, that the character and the virtual medium are displayed integrally on the screen. In some games, possession may be expressed as "equipping."

[0017] An example of a virtual medium held by a character is a weapon. In the following description, it is assumed that the player character holds a firearm (e.g., a handgun) as a virtual medium in his / her hand. The player character can attack enemy characters with the firearm.

[0018] In this game, various sounds are played as sounds emitted by virtual sound sources in the virtual space. A firearm held by a character can be a virtual sound source in the virtual space. That is, in this game, sounds are played as sounds emitted by a held virtual medium. For example, a firearm (virtual medium) may emit a gunshot sound (or more accurately, a sound that simulates a gunshot sound may be played).

[0019] Configuration of the Game Device Hardware Configuration 1 is a block diagram showing the configuration of a game device 5 (an example of an audio control device). The game device 5 executes a predetermined game based on a user's operation. A display 61, a speaker 62, and a controller 63 are connected to the game device 5.

[0020] The game device 5 may be, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), Nintendo Switch (registered trademark), or other commercially available device.

[0021] In the game device 5, the game progresses based on the installed game program and game data. The game devices 5 can communicate data with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0022] The game device 5 has a network interface 51, a graphics processing unit 52, an audio processing unit 53, an operation unit 54, a storage unit 55, and a control unit 56. The network interface 51, the graphics processing unit 52, the audio processing unit 53, the operation unit 54, and the storage unit 55 are electrically connected to the control unit 56 via a bus 59.

[0023] The network interface 51 is communicatively connected to the communication network in order to transmit and receive various data to and from, for example, other game devices 5 and external server devices (not shown).

[0024] The graphics processing unit 52 renders game images including the player character and various objects in the virtual space in a moving image format according to the game image information output from the control unit 56. This game image is an image of the virtual space based on VR technology. Hereinafter, an image displaying the virtual space based on VR technology is called a VR image.

[0025] The graphic processing unit 52 is connected to a display 61. The VR image is displayed on the display 61.

[0026] For example, a liquid crystal display is used for the display 61. The display 61 is housed in a housing that can be worn by a user (hereinafter, referred to as user P). The display 61 of this embodiment is a display device incorporated in a head-mounted display (hereinafter, referred to as VR goggles 70).

[0027] 2 is a perspective view showing an example of the VR goggles 70. As shown in FIG. 2, the VR goggles 70 include a main body 71 and a headband 72.

[0028] The headband 72 is for mounting the VR goggles 70 on the head of a user P (a person who is a player of the game). Fig. 3 is a schematic diagram of a state in which the user P is wearing the VR goggles 70. The VR goggles 70 are mounted on the head of the user P with the main body 71 covering both of the user P's eyes.

[0029] 2 (hereinafter, referred to as vector Vd) indicates the direction that is in front of the user P when wearing the VR goggles 70. In this game program, vector Vd is the normal to the display surface of the display 61.

[0030] The display 61 is fixed inside the main body 71. When the VR goggles 70 are worn on the head of the user P, the display surface of the display 61 faces both of the user P's eyes.

[0031] The VR goggles 70 are equipped with an acceleration sensor and a gyro sensor (not shown). Some VR goggles 70 may also be equipped with a camera. The output of the sensor of the VR goggles 70 is output to the operation unit 54. By analyzing the output of the sensor of the VR goggles 70, information on the position and orientation of the head of the user P (in other words, the position and orientation of the display 61) can be obtained.

[0032] The audio processing unit 53 is connected to the speaker 62. The audio processing unit 53 reproduces digital audio data in accordance with an instruction from the control unit 56. Specifically, the audio processing unit 53 converts the audio data output by the control unit 56 into an analog signal and outputs it to the speaker 62.

[0033] The audio processing unit 53 is configured to be capable of outputting multi-channel audio in order to reproduce three-dimensional sound. Accordingly, a plurality of speakers 62 are connected to the game device 5. These speakers 62 may be arranged on the left and right, front and rear, above, etc. of a listener (e.g., a game player) in order to reproduce three-dimensional sound.

[0034] In this embodiment, a front left speaker (L speaker), a front right speaker (R speaker), a front center speaker (C speaker), a subwoofer (LFE speaker), a left surround (SL speaker) and a right surround (SR speaker) are connected to the game device 5 as speakers 62.

[0035] The operation unit 54 is connected to the controller 63. The operation unit 54 transmits and receives signals to and from the controller 63. The user P operates various operators such as buttons of the controller 63 to input signals (such as commands and data) to the game device 5.

[0036] In this embodiment, two controllers 63 (an example of a detection device) are provided. One controller 63 is held in the right hand of the user P. The other controller 63 is held in the left hand of the user P. When the user P moves his / her arm or hand, the position of the held controller 63 changes. Depending on the form of the controller 63, the user P may wear the controller 63.

[0037] The controller 63 has a plurality of operators (e.g., buttons). The user P can input a signal (hereinafter, a first signal) to the operation unit 54 by operating various operators such as the buttons of the controller 63. The user P can give various instructions in the game by inputting the first signal to the operation unit 54. The operation unit 54 sends the first signal to the control unit 56.

[0038] The controller 63 includes an acceleration sensor and a gyro sensor (not shown). When the user P moves his / her arm or hand, the output of the sensor in the controller 63 (hereinafter, referred to as a second signal) changes. By analyzing the output of the sensor in the controller 63, information on the position and orientation of the user P's hand (arm) can be obtained. The second signal is input to the operation unit 54. The operation unit 54 sends the second signal to the control unit 56.

[0039] The storage unit 55 is composed of an HDD, an SSD, a RAM, a ROM, etc. Game data, various programs including a game program, etc. are stored in the storage unit 55. Examples of game data include a virtual medium and user account information.

[0040] The control unit 56 controls the operation of the game device 5. The control unit 56 includes a CPU (microcomputer) and a semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[0041] <Functional configuration of the control unit 56> The control unit 56 executes a game program, thereby functioning as a game progression unit 561, a conversion unit 562, a display unit 563, and a playback unit 564 (see FIG. 1).

[0042] (Game Progression Section 561) The game progression unit 561 progresses the game by moving player characters, non-player characters, and predetermined objects in a virtual space. The game progression unit 561 uses, for example, AI (artificial intelligence) in controlling the movements of non-player characters.

[0043] The game progression unit 561 controls the movements of the player character and the like in response to instructions from a user P (player). The instructions from the user P are input to the operation unit 54 via the controller 63.

[0044] For example, the user P can instruct the game progression unit 561 to move the player character or attack an enemy by operating the controller 63. For example, when the user P operates a predetermined button on the controller 63, a first signal (hereinafter referred to as a trigger signal) is output, which means that the trigger of a firearm held by the player character in his or her hand has been pulled.

[0045] (Conversion unit 562) The conversion unit 562 converts the second signal into a position and orientation in the virtual space based on the orientation of the display 61 (the direction of the vector Vd). Here, "based on the orientation of the display 61 (the direction of the vector Vd)" means that the vector Vd is treated as one of the coordinate axes.

[0046] In the following, for convenience of explanation, a position in the virtual space that is associated with a position in the real world is called a “position in the virtual space”, and a direction in the virtual space that is associated with a direction in the real world is called a “direction in the virtual space”. The display unit 563 and the playback unit 564 can read information on the position in the virtual space and the direction in the virtual space from the conversion unit 562.

[0047] (Display section 563) The display unit 563 generates a VR image and outputs it to the graphics processing unit 52. For example, the display unit 563 displays an image of the player character's hand by linking the position and direction of the hand with the position and direction of the controller 63 in the virtual space, respectively. The user P can change the position and direction of the player character's hand by changing the position and direction of his / her own hand (in other words, the position and direction of the controller 63).

[0048] The display unit 563 displays the virtual medium held in the hand of the player character on the display 61. The display unit 563 displays an image of the virtual medium on the display 61 in conjunction with the position and direction in the virtual space of the controller 63. In this game program, a firearm held in the hand of the player character is displayed on the display 61 in conjunction with the position and direction in the virtual space of the controller 63.

[0049] (Playback section 564) In order to reproduce sounds, various sound data are prepared in advance in this game program. For example, sound data for the sounds of gun fire, bullet impact, etc. are prepared in this game program.

[0050] The playback unit 564 selects the audio data to be played back, and instructs the audio processing unit 53 to play back the audio data. For example, when the playback unit 564 is to play back the sound of a gunshot of a firearm possessed by the user P, the playback unit 564 identifies the type of the firearm, and selects the audio data for the sound of the gunshot of the identified firearm.

[0051] The playback unit 564 instructs playback of the sound of a virtual sound source existing in a virtual space in accordance with the progress of the game. For example, when a trigger signal (first signal) is input, the playback unit 564 plays back the sound by treating the position obtained by the conversion of the conversion unit 562 as the position of a virtual medium (firearm) in the virtual space.

[0052] The playback unit 564 supports two playback modes. The first playback mode is a playback mode (hereinafter, referred to as a first playback mode) in which sound is played back without performing sound image localization. Note that sound image localization refers to localizing the position of a sound source felt by a human (user P) at a predetermined position in a virtual space.

[0053] In the first playback mode, the playback unit 564 plays back audio from the fixed speaker 62. In the first playback mode, audio is played back from the L speaker, the R speaker, the C speaker, and the LFE speaker. In this way, the sound image is not localized, but a sound that gives a sense of spaciousness is played back.

[0054] The second playback mode is a playback mode that performs sound image localization (hereinafter referred to as the second playback mode). In the second playback mode, an acoustic presentation is performed in which the sound of a virtual sound source existing in a virtual space is produced as if it were emitted from the position of the virtual sound source in the virtual space.

[0055] For example, if a virtual sound source on the left of the player character virtually emits a sound, the sound is reproduced from a predetermined speaker 62 (for example, the L speaker) so that the user P can recognize the sound as coming from the left.

[0056] In order to perform sound image localization, the reproduction unit 564 sets a virtual microphone (hereinafter, virtual microphone) in the virtual space as a virtual sound receiving point where sound is heard. The virtual microphone is not displayed on the screen (display 61). The virtual microphone is a virtual listener, and the position of the virtual microphone is a reference position for sound image localization by the reproduction unit 564. In the second reproduction mode, the sound image may be localized taking into account the direction in the virtual space.

[0057] In this game program, the virtual microphone is set at the position of the head of the player character, and moves in conjunction with the movement of the player character.

[0058] The reproducing unit 564 selectively executes one of a first process, a second process, and a third process, which will be described later, in reproducing the sound in response to the trigger signal. The reproducing unit 564 selects the first process, the second process, and the third process in consideration of the type and the position of the virtual sound source.

[0059] When selecting a process, the playback unit 564 specifies whether the sound generation position of the virtual sound source corresponds to the forward range, left / right range, or rear range of the player character. The forward range (first range) is the range in the virtual space where the sound of the virtual sound source is reproduced as a sound in front of the player character. The left / right range is the range in the virtual space where the sound of the virtual sound source is reproduced as a sound to the right or left of the player character. The rear range is the range in the virtual space where the sound of the virtual sound source is reproduced as a sound behind the player character. Hereinafter, the forward range, left / right range, and rear range are collectively referred to as sound generation divisions.

[0060] Here, the vector Vd projected onto the horizontal plane of the virtual space (hereinafter simply referred to as the horizontal plane) is named vector Vdp. In this game program, each sound division is defined by an angle range relative to vector Vdp on the horizontal plane.

[0061] The numerical setting (angle range) of the sound generation category differs depending on the type of firearm. In this game program, the setting values ​​of the forward range, left and right range, and rear range are listed for each firearm. The playback unit 564 uses this list to specify the numerical setting of the sound generation category.

[0062] In this game program, for example, when the firearm is a virtual handgun, the following sound categories are set:

[0063] Forward range: The range of angles from 0 degrees to the vector Vdp on the horizontal plane ±20 degrees Left / right range: The angle range from 20 degrees to 120 degrees and -20 degrees to 120 degrees with respect to the vector Vdp on the horizontal plane. Rear range: The range of angles from 120 degrees to 180 degrees and -120 degrees to -180 degrees relative to the vector Vdp on the horizontal plane. The numerical values ​​set for the pronunciation divisions are merely an example, and may be arbitrarily determined by the developer of the program.

[0064] The reproduction unit 564 determines whether the sound generation position of the virtual sound source is included in the forward range, the left / right range, or the rear range, based on the position of the controller 63 in the virtual space. Here, the vector obtained by projecting the position vector of the controller 63 onto the horizontal plane is defined as vector Vc. The reproduction unit 564 determines which sound generation section the position of the controller 63 in the virtual space corresponds to by calculating the angle formed by vector Vdp and vector Vc on the horizontal plane.

[0065] The playback unit 564 determines which process to execute from among a first process, a second process, or a third process (all of which are described below) depending on the position of the virtual medium in the virtual space (in other words, the position of the controller 63 in the virtual space), as follows. (1) When the position of the virtual medium corresponds to the forward range: First process (2) When the position of the virtual medium corresponds to the left and right range: second process (3) When the position of the virtual medium corresponds to the rear range: third process The contents of the first process, the second process and the third process are as follows.

[0066] (1) First Processing In the first process, sound is reproduced as sound in front of the user P. Specifically, the reproduction unit 564 performs reproduction in the first reproduction mode. That is, the reproduction unit 564 reproduces sound without localizing a sound image. For example, when the user P (player character) fires a handgun (virtual sound source) in a forward range, sound that gives a sense of spaciousness is reproduced without localizing the sound image.

[0067] (2) Second Processing In the second process, the sound is reproduced as a sound coming from the left or right of the user P. Specifically, in the second process, the sound in the first reproduction mode and the sound in the second reproduction mode are mixed and reproduced. The sound in the second reproduction mode is localized as a sound image with a virtual sound source located at the position in the virtual space of the controller 63. The second process allows the user P to hear both a wide sound and a sound with a sense of localization.

[0068] In the second process, the playback unit 564 changes the ratio of mixing these sounds depending on the position in the virtual space of the controller 63. The closer the position in the virtual space of the controller 63 is to the forward range, the more the playback unit 564 increases the ratio of the sound in the first playback mode.

[0069] (3)Third Processing In the third process, the sound is reproduced as a sound behind the user P. Specifically, in the third process, the reproduction unit 564 performs reproduction in the second reproduction mode. In the second reproduction mode, the sound is reproduced by localizing a sound image, with a virtual sound source being located at the position of the controller 63 in the virtual space. In this way, the user P hears sound with a sense of localization.

[0070] Example of operation When the game starts, the game progression unit 561 moves the player character in the virtual space in response to the operation of the controller 63 by the user P (the game player). The game progression unit 561 also moves objects such as non-player characters in the virtual space as necessary. Here, too, it is assumed that the player character holds a firearm (e.g., a handgun) in his hand.

[0071] Assume that during a game, user P moves his / her arm to aim a firearm held by a player character at an enemy character (non-player character), and further assumes that user P presses a button on controller 63 that corresponds to the trigger of the firearm to instruct the firearm to be fired.

[0072] When the button corresponding to the trigger is pressed, a trigger signal (first signal) is output from the controller 63. When the trigger signal is output, a process of reproducing the sound of a gun being fired by a firearm held by the user is started.

[0073] 4 is a flowchart explaining the processing in the reproducing unit 564. When a command to fire is given by the user P, the reproducing unit 564 identifies the type of sound source (step S01). In detail, the reproducing unit 564 identifies the type of firearm possessed by the player character. Furthermore, the reproducing unit 564 acquires sound data corresponding to the identified firearm.

[0074] The playback unit 564 identifies the numerical setting (angle range) of the sound generation category according to the firearm held by the player character (step S02). The playback unit 564 determines which angle range the vector Vc corresponds to (step S03). That is, the playback unit 564 determines whether the virtual space position of the firearm corresponds to the forward range, the left / right range, or the rear range. Furthermore, the playback unit 564 determines the playback process to be performed (step S03). The playback unit 564 performs the playback process according to the determination result in step S03 (steps S04 to S06).

[0075] For example, assume that user P fires a handgun (firearm) in front (in a direction in which the angle with respect to vector Vdp is 0 degrees on the horizontal plane). In this case, in step S03, it is determined that the position of the handgun in the virtual space corresponds to the forward range. Depending on the result of this determination, step S04 is selected as the next step.

[0076] In step S04, a first process (audio reproduction without sound image localization) is performed. In the first process, the gunfire sound is reproduced from the L speaker, the R speaker, the C speaker, and the LFE speaker.

[0077] Also, assume that user P fires a handgun (firearm) to the right (for example, in a direction at an angle of 90 degrees with respect to vector Vdp). In this case, in step S03, it is determined that the position of the handgun in the virtual space corresponds to the left-right range.

[0078] Depending on the result of the determination, step S05 is selected as the next step, and the second process is performed. In the second process, the sound in the first playback mode and the sound in the second playback mode are mixed and played back. The second process allows the user P to hear both a wide sound and a sound with a sense of localization (here, sound from the right direction).

[0079] Also, assume that user P fires a handgun (firearm) backward (for example, in a direction at an angle of 180 degrees with respect to vector Vdp). In this case, in step S03, it is determined that the position of the handgun in the virtual space corresponds to the rear range.

[0080] Depending on the result of the determination, step S06 is selected as the next step, and the third process is performed. In the third process, playback is performed in the second playback mode. This allows the user P to hear a sound with a sense of localization. Specifically, the user P hears the sound of a gunshot as a sound from behind.

[0081] To summarize the above, the present embodiment is a program that causes a computer system (game device 5) having one computer or multiple computers working together to function as a conversion unit 562 and a playback unit 564, wherein the computer system is connected to a display 61 and a detection device (controller 63), the display 61 is worn by a user P and provides the user P with an image of a virtual space, the detection device outputs a first signal to the computer system in response to an instruction from the user, and the detection device outputs a second signal in response to a change in the position of a part of the user's body, the conversion unit 562 converts the second signal into a position in the virtual space, and the playback unit 564, when a specific first signal is output, plays sound by treating the position obtained by the conversion as the position of a virtual sound source in the virtual space.

[0082] Effects of this embodiment As described above, in this embodiment, the way in which sound is heard (for example, the sense of localization of a sound image) is controlled according to the relationship between the direction of the user's face (the direction of the display 61) and the direction of the user's hands. This makes it possible to improve the user experience in this embodiment.

[0083] [Other embodiments] The game is not limited to a first-person perspective game, and may be a third-person perspective game.

[0084] Services provided in the virtual space are not limited to games. Various services can be provided in the virtual space, such as a place for commercial transactions and a place for communication. In other words, this program can be used to provide services in the so-called metaverse.

[0085] The part for detecting the user's movement is not limited to the hand.

[0086] There is no limitation on the type of sensor that detects the user's movements. In addition, means other than a sensor (for example, a camera) may be used to detect the position and orientation of a body part.

[0087] The program may be implemented so as to be executed by multiple computers working together (for example, a computer acting as a server and a computer acting as a client). For example, when the game device 5 operates as a client connected to a server, the processing performed by the control unit 56 may be performed by the server, or the processing may be shared between the server and the client.

[0088] The voice data may be prepared in advance or may be synthesized in real time.

[0089] The device that outputs the sound is not limited to the speaker 62. For example, instead of the speaker 62, earphones, headphones, etc. that support 3D audio may be used.

[0090] Even when these other embodiments are adopted, the effects of the present aspect are exhibited. In addition, this embodiment can be appropriately combined with other embodiments, and other embodiments can be appropriately combined with each other. [Explanation of symbols]

[0091] 55 Storage section 56 Control Unit (Computer) 61 Display 63 Controller (detection device) 562 Conversion unit 563 Display section 564 Playback Department P User

Claims

1. A program for causing a computer system having one computer or a plurality of computers working together to function as a converter and a player, comprising: the computer system is connected to a display and a detector; the display is worn by a user and provides an image of a virtual space to the user; the detection device outputs a first signal to the computer system in response to an instruction from the user; The detection device outputs a second signal in response to a change in a position of a part on the user's body; The conversion unit converts the second signal into a position in the virtual space, When a predetermined first signal is output, the reproduction unit reproduces sound by assuming that the position obtained by the conversion is a position of a virtual sound source in the virtual space. program.

2. In the program of claim 1, The conversion unit performs the conversion based on an orientation of the display; The reproduction unit reproduces the sound without localizing a sound image when the position obtained by the conversion is within a predetermined range (hereinafter referred to as a first range). program.

3. In the program of claim 2, The reproduction unit sets the first range in accordance with the type of the audio. program.

4. In the program of claim 1 or 2, The computer system is caused to function as a display unit; The display unit displays an image of a virtual medium on the display in correspondence with the position obtained by the conversion; The reproduction unit selects an audio to be reproduced according to the type of the virtual medium. program.

5. A storage unit storing the program according to claim 1 or 2; A control unit that executes the program; An acoustic control device comprising:

Images