Program and sound control device

Abstract

To improve user experience.SOLUTION: A sound control device includes: a virtual-space control section that controls a virtual space such that a virtual portion moves in the virtual space in response to a movement of a predetermined portion of a user; an image reproducing section that allows a display to reproduce an image in the virtual space; a sound generating section that, if a condition for reproducing effect sound responsive to a predetermined movement of the virtual portion is established, generates effect sound in response to a moving speed of the predetermined portion; and a sound reproducing section that allows a speaker to reproduce the effect sound generated by the sound generating section.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This disclosure relates to a program and an acoustic control device.

Background Art

[0002] Among game programs, there are those that reproduce sound effects in a virtual space. In the example of Patent Document 1, gunshots and the like are reproduced in a shooting game.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When various services (such as games) are provided in a virtual space, from the perspective of the user experience, it is desirable to reproduce sound suitable for the scene.

[0005] An object of this disclosure is to improve the user experience in services provided in a virtual space.

Means for Solving the Problems

[0006] A first aspect is a program that causes a computer system having one computer or a plurality of cooperating computers to function as a virtual space control unit, an image reproduction unit, a voice generation unit, and a voice reproduction unit, The computer system is used together with a display, a speaker, and a sensor, The sensor outputs a signal indicating the movement of a predetermined part of the user's body, The virtual space control unit controls the virtual space so that the virtual part moves within the virtual space in accordance with the movement of the predetermined part indicated by the signal. The image playback unit plays the image of the virtual space on the display. When the conditions for playing a sound effect corresponding to a predetermined movement of the virtual part are met, the sound generation unit generates the sound effect according to the movement speed of the predetermined part. The audio playback unit plays the sound effects generated by the audio generation unit through the speaker. It is a program.

[0007] In the first embodiment, The computer system may be used in conjunction with a controller that is attached to or held and operated by the user. The sensor may be included in the controller. The virtual space control unit may control the virtual space based on the signals output from the sensors and the operations input to the controller.

[0008] In the first embodiment, The aforementioned designated part may be a hand. The aforementioned virtual part may be a virtual hand.

[0009] In the first embodiment, Virtual objects that can be grasped by the virtual hand may be placed within the virtual space. The virtual space control unit may control the virtual space so that when a gripping condition is met, which is a condition for performing a gripping operation in which the virtual hand grips the virtual object within the virtual space, the gripping operation is performed. When the gripping condition is met, the sound generation unit may generate a gripping sound, which is a sound effect corresponding to the gripping operation, according to the speed of movement of the predetermined part during the gripping start period, starting from the time the gripping condition is met.

[0010] In the first embodiment, The gripping condition may also be that the virtual hand enters a predetermined range based on the position of the virtual object in the virtual space, and an operation for performing the gripping operation is input to the controller.

[0011] In the first embodiment, The virtual object may be portable while being held by the virtual hand. The virtual space control unit may control the virtual space so that, after the gripping condition is met within the virtual space, when the carrying condition is met, which is a condition for performing a carrying operation in which the virtual hand grasps and carries the virtual object, the carrying operation is performed. The sound generation unit may, when the carrying condition is met after the gripping condition is met, generate a carrying sound, which is a sound effect corresponding to the carrying operation, according to the speed of movement of the predetermined part during the carrying start period from the time the gripping condition is met to the time the carrying condition is met.

[0012] In the first embodiment, The carrying condition may also be that, while the virtual hand continues to grasp the virtual object within the virtual space, the distance traveled by the predetermined part after the grasping condition is met reaches a predetermined distance.

[0013] The second aspect is, A storage unit that stores a program according to the first embodiment, The system comprises a control unit that executes the aforementioned program. It is an acoustic control device. [Effects of the Invention]

[0014] According to this disclosure, the user experience can be improved. [Brief explanation of the drawing]

[0015] [Figure 1]It is a block diagram illustrating the configuration of the game device according to the embodiment. [Figure 2] It is a diagram illustrating the appearance of the head-mounted display. [Figure 3] It is a schematic diagram illustrating a player wearing a head-mounted display and a controller. [Figure 4] It is a schematic diagram for explaining the gripping operation. [Figure 5] It is a flowchart illustrating the flow of acoustic processing according to the gripping operation. [Figure 6] It is a schematic diagram for explaining the carrying operation. [Figure 7] It is a flowchart illustrating the flow of acoustic processing according to the carrying operation.

Mode for Carrying Out the Invention

[0016] Hereinafter, embodiments will be described in detail with reference to the drawings. In the figures, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated. Further, hereinafter, an example in which a program is implemented as a game program and an acoustic control device is implemented as a game device will be described. The game device is an example of a computer system having one computer or a plurality of cooperating computers.

[0017] (Explanation of the game) The game in the following description is a game (video game) performed in a virtual space. The virtual space is a three-dimensional game space and is displayed on a display as an image. Objects are arranged in the virtual space. Examples of objects include player characters operated by a player (a user who plays the game), non-player characters operated by a computer, and the like.

[0018] Furthermore, objects include sound source objects, which act as virtual sound sources within the virtual space. Examples of sound source objects include objects that operate within the virtual space and objects that only output sound within the virtual space. Player characters and non-player characters can be sound source objects. In the game described below, sound is played. The sounds played in the game include sounds output from sound source objects, BGM (Background Music), and sound effects.

[0019] Furthermore, in the game described below, the player character is a virtual player (a virtual human) placed in a virtual space, and moves within the virtual space in accordance with the player's actions and movements in the real world. Specifically, virtual body parts move within the virtual space in accordance with the movements of predetermined body parts of the player. These virtual body parts are virtual parts that correspond to predetermined body parts of the player (predetermined body parts of the player character). For example, the virtual hand of the player character moves within the virtual space in accordance with the player's hand movements in the real world.

[0020] Furthermore, within the virtual space described below, virtual objects that can be grasped by the virtual hand (the player character's hand) are placed. These virtual objects can be carried while being grasped by the virtual hand. Examples of virtual objects include weapons equipped to the player character. Examples of weapons include firearms such as handguns and knives. The player character performs various actions such as grasping and carrying. Grasping is the action of the virtual hand grasping a virtual object within the virtual space. Carrying is the action of the virtual hand carrying the virtual object while it is being grasped.

[0021] For the sake of explanation, the following example of a game will be a competitive action game in which the player character attacks and defeats enemy characters. There are no limitations on the type of game. Furthermore, the example of a virtual space will be a virtual reality space where one can experience virtual reality (VR). Additionally, an example will be given where the game progresses with images of the virtual space displayed on a screen from a first-person perspective.

[0022] Furthermore, in the following example, we will consider a case where a designated part of the user's body is the "user's right hand," the virtual part corresponding to that designated part is the "virtual right hand," and the virtual object that can be grasped by the virtual hand is a "handgun." The player character can equip the handgun on the virtual right hand and attack enemy characters with that handgun.

[0023] (Game device) Figure 1 illustrates the configuration of a game device 10 according to an embodiment. The game device 10 executes a game in response to user input. In this example, the game device 10 is used together with a head-mounted display 20, a speaker 30, and a controller 40, and is connected to these devices by wire or wireless. The game device 10 is an example of an audio control device.

[0024] The game device 10 can utilize commercially available devices such as personal computers, PlayStation®, Xbox®, PlayStation Vita®, and Nintendo Switch®.

[0025] In the game device 10, the game progresses based on the installed game program and game data. The game devices 10 can communicate with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0026] [Head-mounted display] The head-mounted display 20 includes a display 21 and a motion sensor 22.

[0027] Display 21 plays (displays) images. For example, display 21 is a liquid crystal display.

[0028] The motion sensor 22 outputs motion signals that indicate the player's head movements. For example, the motion sensor 22 includes an accelerometer that detects acceleration, a gyroscope that detects angular velocity (rotation and change of direction), and so on. The motion signals output from the motion sensor 22 are transmitted to the game device 10.

[0029] As shown in Figure 2, the head-mounted display 20 has a main body 25 and a headband 26. As shown in Figure 3, the head-mounted display 20 is attached to the head of the player 50 such that the main body 25 covers both of the player 50's eyes. A display 21 is fixed to the inside of the main body 25 (the side facing the player 50's eyes). The display surface of the display 21 faces the player 50's eyes when the head-mounted display 20 is attached to the player 50's head.

[0030] Furthermore, the head-mounted display 20 attached to the player 50's head moves in conjunction with the player 50's head movements. Therefore, the position and orientation of the player 50's head and the position and orientation of the display 21 included in the head-mounted display 20 can be derived based on the motion signals output from the motion sensor 22 included in the head-mounted display 20.

[0031] Furthermore, vector Vd in the head-mounted display 20 is the reference line for the image of the virtual space represented from a first-person perspective. Vector Vd extends in the direction of the normal to the display surface of the display 21. Vector Vd can be derived from the position and orientation of the display 21 included in the head-mounted display 20.

[0032] [Speaker] Speaker 30 plays sound (game sounds in this example). In this example, multiple speakers 30 are provided to enable three-dimensional sound reproduction.

[0033] 〔controller〕 The controller 40 is operated by being held or attached to the player 50. In this example, the controller 40 is operated by being held in the hand of the player 50. As shown in Figure 1, the controller 40 includes an operator 41 and an action sensor 42.

[0034] The control element 41 is composed of various buttons and the like. The controller 50 outputs an operation signal to the game device 10 corresponding to the operation input to the control element 41 (for example, pressing a button). With this configuration, the player 50 inputs an operation signal (a signal indicating a command or data, etc.) to the game device 10 by operating the control element 41 of the controller 40.

[0035] The motion sensor 42 outputs motion signals that indicate the player's hand movements. For example, the motion sensor 22 includes an accelerometer that detects acceleration, a gyroscope that detects angular velocity (rotation and change of direction), and so on. The motion signals output from the motion sensor 42 are transmitted to the game device 10.

[0036] Furthermore, the controller 40, which is held (or attached) to the player 50's hand, moves in conjunction with the player 50's hand movements. Therefore, the position and orientation of the player 50's hand can be derived based on the motion signal output from the motion sensor 42 included in the controller 40. The motion sensor 42 is an example of a sensor that outputs a signal indicating the movement of a predetermined part of the user's body.

[0037] As shown in Figure 3, in this example, the controller 40 consists of a first controller 40a and a second controller 40b. The configurations of the first controller 40a and the second controller 40b are the same as those of the controller 40.

[0038] The first controller 40a is held and operated by the player 50's right hand 55 and moves in conjunction with the player 50's right hand 55. The second controller 40b is held and operated by the player 50's left hand 56 and moves in conjunction with the player 50's left hand 56. Therefore, the position and orientation of the player 50's right hand 55 can be derived based on the motion signal output from the motion sensor 42 included in the first controller 40a. Similarly, the position and orientation of the player 50's left hand 56 can be derived based on the motion signal output from the motion sensor 42 included in the second controller 40b.

[0039] [Hardware configuration of the game device] As shown in Figure 1, the game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, graphics processing unit 12, audio processing unit 13, operation processing unit 14, and storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0040] The network interface 11 is connected to a communication network (not shown) in a communicative manner for sending and receiving various data with, for example, other game devices 10 or external server devices (not shown).

[0041] The graphics processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wireless connection. The graphics processing unit 12 renders game images, including various objects in the virtual space, in video format according to the game image information (image data) output from the control unit 16. The game images (images of the virtual space) rendered in video format are displayed on the display 21 as the game screen.

[0042] The audio processing unit 13 is connected to the speaker 30 by wire or wireless connection. The audio processing unit 13 plays digital game sounds according to the instructions of the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the speaker 30.

[0043] In this example, the audio processing unit 13 is configured to provide multi-channel audio output in order to reproduce three-dimensional sound. Multiple speakers 30 are connected to the game device 10. The multiple speakers 30 are positioned to the left and right, front and back, and above the listener (player) to enable three-dimensional sound reproduction.

[0044] The operation processing unit 14 is connected to the head-mounted display 20 and the controller 40 by wire or wireless connection. The operation processing unit 14 sends and receives signals to and from the head-mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits the operation signal output from the motion sensor 22 of the head-mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to the operation input to the operator 41), and the operation signal output from the motion sensor 42 of the controller 40 to the control unit 16.

[0045] The storage unit 15 stores various types of information and data. The storage unit 15 is composed of HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores various programs, including game data and game programs. Examples of game data include game media and user account information.

[0046] Furthermore, the memory unit 15 stores various audio data that represent various pre-prepared sounds. These various audio data include audio data that represents sounds output from sound source objects, audio data that represents music (BGM) played in the virtual space, and audio data that represents sound effects.

[0047] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[0048] [Functional configuration of the control unit] The control unit 16 of the game device 10 includes a game progression unit 100, an image playback unit 101, an audio generation unit 102, and an audio playback unit 103. Specifically, the control unit 16 functions as the game progression unit 100, the image playback unit 101, the audio generation unit 102, and the audio playback unit 103 by executing a game program.

[0049] <Game Management Department (Virtual Space Control Department)> The game progress unit 100 manages the game played within the virtual space. The game progress unit 100 is an example of a virtual space control unit that controls the virtual space.

[0050] In this example, the game progression unit 100 controls (generates or updates) image data representing the virtual space so that these movements and operations are reflected in the virtual space, based on the movements indicated by the motion signal output from the motion sensor 22 (the player's head movements), the movements indicated by the motion signal output from the motion sensor 42 (the player's hand movements), and the operations indicated by the operation signal output from the controller 40 (operations input to the controller's control element 41). This controls the game progression unit 100.

[0051] Specifically, the game progress unit 100 controls the movement of the player character in the virtual space in accordance with the above movements and operations. Furthermore, the game progress unit 100 controls the movement of other objects (such as non-player characters) in the virtual space according to the progress of the game. For example, AI (artificial intelligence) is used to control the movement of other objects by the game progress unit 100.

[0052] Furthermore, as shown in Figures 4 and 6, the game progress unit 100 controls the virtual space so that virtual parts 61 (predetermined parts of the player character 60) move in the virtual space in accordance with the movement of a predetermined part 51 of the player's body, as indicated by the motion signal output from the motion sensor 42.

[0053] In this example, the game progress unit 100 controls the virtual space so that the virtual right hand 65 (the right hand of the player character 60) moves in accordance with the movement of the player's right hand 55, as indicated by the motion signal output from the motion sensor 42 included in the first controller 40a. The game progress unit 100 also controls the virtual space so that the virtual left hand 66 (the left hand of the player character 60) moves in accordance with the movement of the player's left hand 56, as indicated by the motion signal output from the motion sensor 42 included in the second controller 40b.

[0054] <Image Playback Section> The image playback unit 101 plays images of the virtual space controlled by the game progress unit 100 on the display 21. In this example, the image playback unit 101 supplies image data (image data showing images of the virtual space) controlled by the game progress unit 100 to the graphics processing unit 12. As a result, the images of the virtual space shown in the image data are played (displayed) on the display 21 of the head-mounted display 20.

[0055] <Speech generation unit> The sound generation unit 102 generates sounds (game sounds) for the virtual space controlled by the game progress unit 100. In this example, the sound generation unit 102 generates sounds that correspond to the situation in the virtual space (such as the movement of objects). Specifically, the sound generation unit 102 selects the sound data to be played from the sound data stored in the memory unit 15 (various pre-prepared sound data) according to the situation in the virtual space (in this example, the game progress), and generates sound data that indicates sounds corresponding to the situation in the virtual space based on the selected sound data (or new sound data obtained by synthesizing multiple selected sound data).

[0056] In this example, the sound generation unit 102 also performs sound localization processing. Sound localization processing is the process of localizing the position of the sound source as perceived by a human (player) to a predetermined virtual position in the virtual space (for example, the position of the sound source object).

[0057] In sound localization processing, the sound generation unit 102 places a virtual microphone (not shown) in the virtual space and generates sound data including the sound obtained by the virtual microphone. The virtual microphone is a virtual listener (sound collection point) and is the reference point for sound localization in sound localization processing. For example, the virtual microphone is placed near the player character and moves in accordance with the movement of the player character.

[0058] Specifically, in sound localization processing, the sound generation unit 102 generates sound data according to the sound collection conditions of the virtual microphone (for example, the position and direction of the virtual microphone relative to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, etc.) so that the sound reproduced (output) from the speaker 30 includes the sound obtained by the virtual microphone.

[0059] The audio data generated by the sound image localization process is processed by the audio processing unit 13, making it possible to output the sound from the speaker 30 with an acoustic representation as if it were picked up by a virtual microphone, even though the sound was output from a virtual sound source (sound source object) in the virtual space.

[0060] Furthermore, when the conditions for playing sound effects corresponding to predetermined movements of virtual body parts (predetermined body parts of the player character) in the virtual space are met, the sound generation unit 102 generates sound effects according to the movement speed of predetermined body parts of the user in the real world. Specifically, when the above conditions are met, the sound generation unit 102 generates sound effects according to the movement speed of predetermined body parts within a predetermined period immediately before or after the fulfillment of the above conditions (a predetermined period starting or ending at the time when the above conditions are fulfilled). The generation of sound effects (sound effects corresponding to predetermined movements of virtual body parts) by the sound generation unit 102 will be explained in detail later.

[0061] <Audio Playback Unit> The audio playback unit 103 plays the audio generated by the audio generation unit 102 (including sound effects corresponding to predetermined movements of virtual body parts) through the speaker 30. In this example, the audio playback unit 103 supplies the audio data generated by the audio generation unit 102 to the audio processing unit 13. As a result, the audio of the virtual space indicated in the audio data (including sound effects corresponding to predetermined movements of virtual body parts) is played (output) from the speaker 30.

[0062] [Processing related to gripping] Next, referring to Figure 4, the processes related to the grasping action will be explained. The grasping action is an action in which the virtual right hand 65 (virtual hand) grasps the handgun 71 (virtual object 70) in the virtual space. The grasping action is an example of a predetermined movement of the virtual body part 61.

[0063] The game progress unit 100 controls the virtual space so that a gripping action is performed when the gripping condition, which is a condition for performing a gripping action, is met. Specifically, when the gripping condition is met, the game progress unit 100 causes the player character 60's virtual right hand 65 to grip the handgun 71 in the virtual space.

[0064] When the gripping condition is met, the sound generation unit 102 generates a gripping sound, which is a sound effect corresponding to the gripping motion, according to the speed of movement of the player's right hand 55 (a predetermined part 51) during the gripping start period, which begins from the time the gripping condition is met. The sound processing corresponding to the gripping motion will be explained in detail later.

[0065] The gripping condition is that the virtual right hand 65 (virtual hand) enters a predetermined range R1 based on the position Q1 of the handgun 71 (virtual object 70) in the virtual space, and an operation for performing a gripping action is input to the operator 41 of the controller 40. The gripping condition is an example of a condition for playing a sound effect corresponding to a predetermined movement of the virtual body part 61.

[0066] [Gripping sound] In this example, the gripping sound is the sound of the player character 60 grasping the handgun 71, which is removably housed in a holster attached to the player character 60, with their virtual right hand 65 (for example, a "clack" sound). The gripping sound is a sound effect that mimics the sound made when grasping an object (a handgun in this example) with one's hand.

[0067] In the real world, the sound produced when grasping an object changes depending on the speed at which the hand moves towards the object. For example, the sound produced when grasping an object tends to be louder and more intense as the speed at which the hand moves towards the object increases. Also, the faster the hand moves towards the object, the faster the hand moves immediately after grasping the object, due to the remaining momentum.

[0068] The relationship between the "sound emitted when grasping an object with the hand" and the "speed of the hand moving toward the object to grasp it (speed of movement over a predetermined period)" can be derived through experimentation and simulation. Similarly, the relationship between the "sound emitted when grasping an object with the hand" and the "speed of the hand moving immediately after grasping the object (speed of movement over a predetermined period)" can be derived through experimentation and simulation.

[0069] [Acoustic processing corresponding to gripping motion] Next, with reference to Figure 5, the acoustic processing (processing to generate sound effects) corresponding to the gripping operation will be explained. In this example, for each type of virtual object 70 and for each range of movement speed of a predetermined part 51 during the gripping start period described later, a gripping sound (multiple gripping sounds) corresponding to the gripping operation of that virtual object 70 is prepared. The audio data (multiple audio data) representing these gripping sounds is stored in the storage unit 15. In this example, the range of movement speed of the virtual part 61 is divided into three ranges: low speed range (range of movement speed higher than zero), medium speed range (range of movement speed higher than the low speed range), and high speed range (range of movement speed higher than the medium speed range).

[0070] <Step S10> The voice generation unit 102 determines whether the virtual part 61 has entered a predetermined range R1 based on the position Q1 of the virtual object 70. If the virtual part 61 has entered the predetermined range R1, the process in step S11 is performed.

[0071] As shown in Figure 4, in this example, the predetermined range R1 is a spherical range centered on the position of the handgun 71, which is the position Q1 of the virtual object 70, and is the range in which the virtual right hand 65, which is the virtual body part 61, is considered to be in a position to grasp the handgun 71. The voice generation unit 102 acquires the position (e.g., coordinates) of the virtual right hand 65 in the virtual space, and determines that the virtual right hand 65 has entered the predetermined range R1 if the position of the virtual right hand 65 is included within the predetermined range R1.

[0072] <Step S11> The voice generation unit 102 determines whether an operation to grasp the virtual object 70 has been input to the controller 40. If an operation to grasp the virtual object 70 has been input to the controller 40, the process in step S12 is performed; otherwise, the process in step S10 is performed.

[0073] In this example, the operation to grasp the virtual object 70, which is the handgun 71, is the operation to press a predetermined button included in the control element 41 of the controller 40. Based on the operation signal output from the controller 40 (a signal corresponding to the operation input to the control element 41), the voice generation unit 102 determines whether or not the operation to grasp the handgun 71 (the operation to press a predetermined button) has been input to the control element 41 of the controller 40.

[0074] <Step S12> When the virtual part 61 enters a predetermined range R1, the gripping condition is met when an operation to grip the virtual object 70 is input to the controller 40. The determination of whether or not the gripping condition is met (processing in steps S10 and S11) may be performed by the game progression unit 100. When the gripping condition is met, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the gripping operation. In this example, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the gripping operation as "handgun 71".

[0075] <Step S13> Next, the voice generation unit 102 derives the "movement speed of a predetermined part 51 on the player 50's body in the real world" during the gripping start period, which begins from the point when the gripping condition is met.

[0076] In this example, the voice generation unit 102 derives the "movement speed of the player's right hand 55 in the real world" during the gripping start period based on the motion signal output from the motion sensor 42 of the first controller 40a. For example, the voice generation unit 102 derives the movement speed of the right hand 55 during the gripping start period by integrating the "acceleration indicated by the motion signal output from the motion sensor 42" during the gripping start period.

[0077] <Step S14> Next, the voice generation unit 102 determines whether the movement speed of the predetermined part 51 derived in step S13 (in this example, the movement speed of the right hand 55) falls into one of the predetermined "low speed range," "medium speed range," or "high speed range." If the movement speed of the predetermined part 51 falls into the low speed range, the process in step S15 is performed. If the movement speed of the predetermined part 51 falls into the medium speed range, the process in step S16 is performed. If the movement speed of the predetermined part 51 falls into the high speed range, the process in step S17 is performed.

[0078] <Step S15> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the low-speed range, the sound generation unit 102 generates a gripping sound corresponding to the low speed. The sound playback unit 103 plays the "gripping sound corresponding to the low speed" generated by the sound generation unit 102 through the speaker 30.

[0079] In this example, the voice generation unit 102 selects multiple voice data (voice data indicating gripping sounds) stored in the memory unit 15 that correspond to the type of virtual object 70 identified in step S12 (in this example, a handgun 71). Next, the voice generation unit 102 selects voice data indicating a "gripping sound corresponding to low speed" corresponding to the low speed range from among the multiple voice data (voice data prepared for each range of movement speed of the virtual part 61 during the gripping start period). Then, the voice playback unit 103 supplies the voice data indicating a "gripping sound corresponding to low speed" selected by the voice generation unit 102 to the audio processing unit 13, causing the speaker 30 to play a gripping sound corresponding to low speed.

[0080] <Step S16> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the medium speed range, the sound generation unit 102 generates a gripping sound corresponding to the medium speed. The sound playback unit 103 plays the "gripping sound corresponding to the medium speed" generated by the sound generation unit 102 through the speaker 30. For example, the gripping sound corresponding to the medium speed is louder and more intense than the gripping sound corresponding to the low speed.

[0081] The voice generation unit 102 selects from among multiple voice data (voice data indicating gripping sounds) stored in the memory unit 15 voice data that corresponds to the type of virtual object 70 identified in step S12 (handgun 71 in this example) and corresponds to the medium speed range, indicating a "gripping sound corresponding to medium speed". The selection of voice data indicating a "gripping sound corresponding to medium speed" by the voice generation unit 102 is the same as the selection of voice data indicating a "gripping sound corresponding to low speed". The voice playback unit 103 supplies the voice data indicating a "gripping sound corresponding to medium speed" selected by the voice generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to play a gripping sound corresponding to medium speed.

[0082] <Step S17> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the high-speed range, the sound generation unit 102 generates a gripping sound corresponding to the high speed. The sound playback unit 103 plays the "high-speed gripping sound" generated by the sound generation unit 102 through the speaker 30. For example, the high-speed gripping sound is louder and more intense than the medium-speed gripping sound.

[0083] The voice generation unit 102 selects from among multiple voice data (voice data indicating gripping sounds) stored in the memory unit 15 voice data that corresponds to the type of virtual object 70 identified in step S12 (handgun 71 in this example) and corresponds to the high-speed range, indicating a "gripping sound corresponding to high speed". The selection of voice data indicating a "gripping sound corresponding to high speed" by the voice generation unit 102 is the same as the selection of voice data indicating a "gripping sound corresponding to low speed". The voice playback unit 103 supplies the voice data indicating a "gripping sound corresponding to high speed" selected by the voice generation unit 102 to the audio processing unit 13, thereby playing the gripping sound corresponding to high speed through the speaker 30.

[0084] [Processing related to carrying] Next, with reference to Figure 6, the carrying motion will be explained. The carrying motion is the action of the virtual right hand 65 (virtual hand) holding and carrying the handgun 71 (virtual object 70). The carrying motion is an example of a predetermined movement of the virtual body part 61.

[0085] The game progress unit 100 controls the virtual space so that when the carrying condition, which is a condition for performing a carrying action, is met after the gripping condition is met in the virtual space, the carrying action is performed. Specifically, when the carrying condition is met, the game progress unit 100 controls the movement of the virtual right hand 65 in conjunction with the movement of the player's right hand 55 so that the virtual right hand 65 grips the handgun 71 and carries it.

[0086] When the carrying condition is met after the gripping condition is met, the sound generation unit 102 generates a carrying sound, which is a sound effect corresponding to the carrying action, according to the speed of movement of the right hand 55 (a predetermined part 51) during the carrying start period from the time the gripping condition is met to the time the carrying condition is met. The sound processing corresponding to the carrying action will be explained in detail later.

[0087] The carrying condition is that, within the virtual space, the virtual right hand 65 (virtual hand) continues to grip the handgun 71 (virtual object 70), and after the gripping condition is met, the distance traveled by the right hand 55 (predetermined part 51) reaches a predetermined distance. The carrying condition is an example of a condition for playing a sound effect corresponding to a predetermined movement of the virtual part 61.

[0088] [Portable sound] In this example, the carrying sound is the sound of the handgun 71, which is housed in a holster attached to the player character 60, being drawn out of the holster while rubbing against the inside of the holster (for example, a "whoosh" sound). The carrying sound is a sound effect that mimics the sound made when carrying (moving) an object (in this example, a handgun) while rubbing it with your hands.

[0089] In the real world, the sound produced when carrying an object by rubbing it between your hands changes depending on the speed at which the hand carrying the object moves. For example, the sound produced when carrying an object by rubbing it between your hands tends to become louder and more intense as the speed at which the hand carrying the object increases. The relationship between the "sound produced when carrying an object by rubbing it between your hands" and the "speed at which the hand carrying the object moves (speed at which it moves over a given period)" can be derived through experiments and simulations.

[0090] [Sound processing based on carrying motion] Next, with reference to Figure 7, the sound processing (processing to generate sound effects) corresponding to the carrying operation will be explained. In this example, for each type of virtual object 70 and for each range of movement speed of a predetermined part 51 during the carrying start period described later, a carrying sound (multiple carrying sounds) corresponding to the carrying operation of that virtual object 70 is prepared. The audio data (multiple audio data) representing these carrying sounds is stored in the storage unit 15.

[0091] <Step S20> The voice generation unit 102 determines whether the gripping condition is met. If the gripping condition is met, the process in step S21 is performed.

[0092] <Step S21> The voice generation unit 102 starts measuring the distance traveled by the virtual part 61 from the gripping position of the virtual object 70 (the position when the gripping operation started). Next, the process in step S22 is performed.

[0093] In this example, the voice generation unit 102 obtains the position (e.g., coordinates) of the virtual right hand 65, which is a virtual body part 61, when the gripping operation of the virtual right hand 65 to grasp the virtual object 70, which is a handgun 71, begins, and sets the position of the virtual right hand 65 as the "starting position for measuring the distance of movement". The voice generation unit 102 then tracks the position (e.g., coordinates) of the virtual right hand 65 in the virtual space and derives the distance of the trajectory of the movement of the virtual right hand 65 as the "distance traveled by the virtual right hand 65 from the gripping position of the handgun 71". Alternatively, the voice generation unit 102 may derive the straight-line distance between the current position of the virtual right hand 65 in the virtual space and the measurement start position as the "distance traveled by the virtual right hand 65 from the gripping position of the handgun 71".

[0094] <Step S22> The voice generation unit 102 determines whether or not an operation to release the virtual object 70 has been input to the controller 40. If no operation to release the virtual object 70 has been input to the controller 40, the process in step S23 is performed. On the other hand, if an operation to release the virtual object 70 has been input to the controller 40, the process ends.

[0095] In this example, the operation to release the grip on the virtual object 70, the handgun 71, is the operation to release the predetermined button (the button pressed to grip the handgun 71) included in the operator 41 of the controller 40. The voice generation unit 102 determines whether the operation to grip the handgun 71 (the operation of pressing the predetermined button) has been released based on the operation signal output from the controller 40 (a signal corresponding to the operation input to the operator 41). If the operation to grip the handgun 71 is not released and continues, the process in step S23 is performed. On the other hand, if the operation to grip the handgun 71 is released, the process ends.

[0096] <Step S23> Next, the voice generation unit 102 determines whether the distance traveled by the virtual part 61 (virtual right hand 65 in this example) from the gripping position of the virtual object 70 (handgun 71 in this example) has reached a predetermined distance. If the distance traveled by the virtual part 61 from the gripping position of the virtual object 70 has reached the predetermined distance, the process in step S24 is performed; otherwise, the process in step S22 is performed.

[0097] <Step S24> The carrying condition is met when the distance traveled by the virtual part 61 from the gripping position of the virtual object 70 reaches a predetermined distance without any operation to release the gripping of the virtual object 70 being input to the controller 40. The determination of whether or not the carrying condition is met (processing in steps S21 to S23) may be performed by the game progress unit 100. When the gripping condition is met, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the carrying operation. In this example, the sound generation unit 102 identifies the type of virtual object 70 that is the target of the carrying operation as "handgun 71".

[0098] <Step S25> Next, the voice generation unit 102 derives the "movement speed of a predetermined part 51 on the player 50's body in the real world" during the carrying start period, from the time the gripping condition is met to the time the carrying condition is met.

[0099] In this example, the voice generation unit 102 derives the "movement speed of the player's right hand 55 in the real world" during the carrying start period based on the motion signal output from the motion sensor 42 of the first controller 40a. For example, the voice generation unit 102 derives the movement speed of the right hand 55 during the carrying start period by integrating the "acceleration indicated by the motion signal output from the motion sensor 42" during the carrying start period.

[0100] <Step S26> Next, the voice generation unit 102 determines whether the movement speed of the predetermined part 51 derived in step S25 (in this example, the movement speed of the right hand 55) falls into one of the predetermined "low speed range," "medium speed range," or "high speed range." If the movement speed of the predetermined part 51 falls into the low speed range, the process in step S27 is performed. If the movement speed of the predetermined part 51 falls into the medium speed range, the process in step S28 is performed. If the movement speed of the predetermined part 51 falls into the high speed range, the process in step S29 is performed.

[0101] <Step S27> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the low-speed range, the sound generation unit 102 generates a carrying sound corresponding to the low speed. The sound playback unit 103 plays the "carrying sound corresponding to the low speed" generated by the sound generation unit 102 through the speaker 30.

[0102] In this example, the voice generation unit 102 selects multiple voice data (voice data indicating carrying sounds) stored in the memory unit 15 that correspond to the type of virtual object 70 identified in step S24 (in this example, a handgun 71). Next, the voice generation unit 102 selects voice data indicating "carrying sounds corresponding to low speed" corresponding to the low-speed range from among the multiple voice data (voice data prepared for each range of movement speed of the virtual part 61 during the carrying start period). Then, the voice playback unit 103 supplies the voice data indicating "carrying sounds corresponding to low speed" selected by the voice generation unit 102 to the audio processing unit 13, thereby playing the carrying sounds corresponding to low speed through the speaker 30.

[0103] <Step S28> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the medium speed range, the sound generation unit 102 generates a carrying sound corresponding to the medium speed. The sound playback unit 103 plays the "carrying sound corresponding to the medium speed" generated by the sound generation unit 102 through the speaker 30. For example, the carrying sound corresponding to the medium speed is louder and more intense than the carrying sound corresponding to the low speed.

[0104] The voice generation unit 102 selects from among multiple voice data (voice data indicating carrying sounds) stored in the memory unit 15 voice data that corresponds to the type of virtual object 70 identified in step S24 (handgun 71 in this example) and corresponds to the medium speed range, and that indicates a "carrying sound corresponding to medium speed". The selection of voice data indicating a "carrying sound corresponding to medium speed" by the voice generation unit 102 is the same as the selection of voice data indicating a "carrying sound corresponding to low speed" by the voice generation unit 102. The voice playback unit 103 supplies the voice data indicating a "carrying sound corresponding to medium speed" selected by the voice generation unit 102 to the audio processing unit 13, thereby playing the carrying sound corresponding to medium speed through the speaker 30.

[0105] <Step S29> If the movement speed of a predetermined part 51 (in this example, the movement speed of the right hand 55) falls within the high-speed range, the sound generation unit 102 generates a carrying sound corresponding to the high speed. The sound playback unit 103 plays the "carrying sound corresponding to high speed" generated by the sound generation unit 102 through the speaker 30. For example, the carrying sound corresponding to high speed is louder and more intense than the carrying sound corresponding to medium speed.

[0106] The voice generation unit 102 selects from among multiple voice data (voice data indicating carrying sounds) stored in the memory unit 15 voice data that corresponds to the type of virtual object 70 identified in step S24 (handgun 71 in this example) and corresponds to the high-speed range, and indicates a "carrying sound corresponding to high speed". The selection of voice data indicating a "carrying sound corresponding to high speed" by the voice generation unit 102 is the same as the selection of voice data indicating a "carrying sound corresponding to low speed". The voice playback unit 103 supplies the voice data indicating a "carrying sound corresponding to high speed" selected by the voice generation unit 102 to the audio processing unit 13, thereby causing the speaker 30 to play a gripping sound corresponding to high speed.

[0107] [Summary of Embodiments] In summary, the program (game program) of the embodiment is a program that causes a computer system (game device 10) having one computer or multiple cooperating computers to function as a virtual space control unit (game progress unit 100), an image playback unit 101, a sound generation unit 102, and a sound playback unit 103. The computer system is used together with a display 21, a speaker 30, and a sensor (motion sensor 42). The sensor outputs a signal indicating the movement of a predetermined part 51 of the user's (player's 50) body. The virtual space control unit controls the virtual space so that a virtual part 61 moves in the virtual space in accordance with the movement of the predetermined part 51 indicated by the signal output from the sensor. The image playback unit 101 plays images of the virtual space on the display 21. When the conditions for playing sound effects corresponding to the predetermined movement of the virtual part 61 are met, the sound generation unit 102 generates sound effects according to the movement speed of the predetermined part 51. The sound playback unit 103 plays the sound effects generated by the sound generation unit 102 on the speaker 30.

[0108] The sound control device (game device 10) of this embodiment includes a storage unit 15 for storing the above-mentioned program and a control unit 16 for executing the above-mentioned program.

[0109] [Effects of the Embodiment] In the game device 10 of this embodiment, when the conditions for playing sound effects corresponding to predetermined movements of virtual body parts in the virtual space (virtual body parts that move in conjunction with the movements of predetermined body parts of the player) are met, the above-mentioned sound effects are generated according to the movement speed of the predetermined body parts of the player in the real world. This can improve the user experience. For example, the realism of the sound effects can be improved, thereby enhancing the level of enjoyment.

[0110] Furthermore, in the game device 10 of this embodiment, a gripping sound corresponding to the gripping action in which a virtual hand grasps a virtual object in the virtual space, and a carrying sound corresponding to the carrying action in which the virtual hand carries the virtual object while gripping it are generated separately. This can further improve the user experience. For example, the realism of the sound effects corresponding to a series of actions, such as a gripping action followed by a carrying action, can be improved, thereby enhancing the level of enjoyment.

[0111] (Other embodiments) In the above description, the following configuration or settings may be used.

[0112] The designated body part of the player (user) is not limited to the hands. For example, the designated body part of the player may be the player's torso, legs, or other body parts.

[0113] The sensor that outputs a signal indicating the movement of a predetermined part of the player's (user's) body is not limited to the motion sensor 42 of the controller 40. For example, such a sensor may be a combination of a camera (image sensor) that captures an image of a predetermined part of the player's body and a recognition processing unit that recognizes the movement of the predetermined part based on the "image of the predetermined part" obtained by the camera.

[0114] Virtual objects that can be grasped by a virtual hand are not limited to firearms such as handguns. For example, such virtual objects may be other weapons such as knives, armor worn by the player character, or other objects (virtual objects) that are neither weapons nor armor.

[0115] Audio data does not necessarily need to be prepared in advance. For example, the audio generation unit 102 may synthesize or generate audio data each time based on some data (for example, audio data stored in the memory unit 15) or information.

[0116] Speaker 30 may be a stationary speaker or a wearable speaker such as earphones or headphones.

[0117] The player character may or may not be displayed on the screen. In other words, the player character may or may not be included in the image of the virtual space displayed (played) on the screen. For example, if the image of the virtual space is displayed on the screen in a first-person perspective, only the virtual hand (virtual body part), which is part of the player character, may be displayed, or the virtual hand may not be displayed, and only the handgun (virtual object) held in the virtual hand may be displayed.

[0118] The images in the virtual space may be first-person or third-person perspective images.

[0119] Furthermore, in the above description, the game program may be implemented as a game program for so-called online games. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in the control unit of the server device (not shown), or it may be shared between the control unit of the game device 10 and the control unit of the server device. The game device 10 may be a portable information terminal such as a smartphone or tablet.

[0120] Furthermore, while the above explanation uses the virtual space where a game takes place (game space) as an example of a virtual space, it is not limited to this. For example, a virtual space could be a virtual space where avatars communicate with each other (metaverse space). In other words, the services provided in a virtual space are not limited to games. For example, the services provided in a virtual space could be the provision of a platform for commercial transactions, a platform for communication, etc.

[0121] The effects and benefits of the present invention will also be achieved when these other embodiments are adopted. Furthermore, it is possible to combine these embodiments with other embodiments, and with other embodiments as appropriate. The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses. [Explanation of Symbols]

[0122] 10. Game device (sound control device) 11 Network Interfaces 12 Graphics Processing Unit 13 Audio Processing Unit 14 Operation Processing Unit 20 Head-mounted displays 21 displays 22 Motion Sensor 25 Main body 26 Headbands 30 speakers 40 controllers 40a First Controller 40b Second Controller 41 Operator 42 Motion Sensor (Sensor) 50 players (users) 51 Designated area 55 Right hand (hand) 56 Left hand (hand) 60 Player Characters (Virtual Users) 61 Virtual body part 65. Virtual Right Hand (Virtual Hand) 66. Virtual left hand (virtual hand) 70 Virtual Objects 71 Handgun 100 Game Progression Unit (Virtual Space Control Unit) 101 Image Playback Unit 102 Voice generation unit 103 Audio playback unit

Claims

1. A program that causes a computer system having one computer or multiple cooperating computers to function as a virtual space control unit, an image playback unit, an audio generation unit, and an audio playback unit, The aforementioned computer system is used together with a display, speaker, and sensor. The sensor outputs a signal indicating the movement of a predetermined part of the user's body. The virtual space control unit controls the virtual space so that the virtual part moves within the virtual space in accordance with the movement of the predetermined part indicated by the signal. The image playback unit plays the image of the virtual space on the display. When the conditions for playing a sound effect corresponding to a predetermined movement of the virtual part are met, the sound generation unit generates the sound effect according to the movement speed of the predetermined part. The sound playback unit plays the sound effect generated by the sound generation unit through the speaker. The computer system is used in conjunction with a controller that is worn or held and operated by the user. The sensor is included in the controller, The virtual space control unit controls the virtual space based on the signal output from the sensor and the operation input to the controller. The aforementioned designated part is the hand, The aforementioned virtual part is a virtual hand, Within the virtual space, virtual objects that can be grasped by the virtual hand are placed, The virtual space control unit controls the virtual space so that the grasping operation is performed when the grasping condition, which is the condition for the grasping operation in which the virtual hand grasps the virtual object within the virtual space, is met. The virtual object is portable while being held by the virtual hand. The virtual space control unit controls the virtual space so that the carrying operation is performed when the carrying condition is met, which is a condition for performing a carrying operation in which the virtual hand carries the virtual object while grasping it, after the gripping condition has been met within the virtual space. When the carrying condition is met after the gripping condition is met, the sound generation unit generates a carrying sound, which is a sound effect corresponding to the carrying action, according to the speed of movement of the predetermined part during the carrying start period from the time the gripping condition is met to the time the carrying condition is met. program.

2. In the program of claim 1, The carrying condition is that, while the virtual hand continues to grasp the virtual object within the virtual space, the distance traveled by the predetermined part after the grasping condition is met reaches a predetermined distance. program.

3. A storage unit for storing the program according to claim 1 or 2, The system comprises a control unit that executes the aforementioned program. Acoustic control device.

Images