Program, acoustic control device, and system

Abstract

To enhance user experience.SOLUTION: A detection unit detects the orientation of a user P1 on the basis of signals output from a sensor. A display processing unit changes the orientation of a virtual camera 61 arranged in a virtual space so as to follow the orientation of the user P1 detected by the detection unit, and causes a display to display the image of the virtual space obtained by the virtual camera 61. An audio processing unit causes a plurality of speakers 30 to output audio in the virtual space, including a predetermined specific sound A5, in such a manner that the specific sound A5 is output from a specific direction toward the user P1 in accordance with the orientation of the user P1 detected by the detection unit.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a program, an acoustic control device, and a system.

Background Art

[0002] Some game programs reproduce voices in a virtual space. For example, in Patent Document 1, voices such as gunshots are reproduced in a shooting game.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to provide a user with images and voices in a virtual space in which various services (such as games) are provided using a system including a display (such as a head-mounted display) that follows the user's orientation and a plurality of speakers arranged around the user. In such a system, from the perspective of the user experience, it is desirable that a specific voice (such as an operation sound) be reproduced from a specific direction (such as in front of the user) toward the user with respect to the user's orientation.

[0005] An object of the present disclosure is to improve the user experience in services provided in a virtual space.

Means for Solving the Problems

[0006] A first aspect is a program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the orientation of the user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user. The detection unit detects the orientation of the user based on the signal output from the sensor. The display processing unit changes the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection unit, and causes the display to display an image of the virtual space obtained by the virtual camera. The acoustic processing unit causes the plurality of speakers to output the sound in the virtual space including the specific voice so that the specific voice is output from a specific direction toward the user with respect to the orientation of the user detected by the detection unit. It is a program.

[0007] In a first aspect, The computer system may be caused to function as a virtual space control unit. A specific virtual sound source for outputting the specific voice may be arranged in the virtual space. The acoustic processing unit may move a virtual microphone arranged in the virtual space so as to follow the movement of the virtual camera, and cause the plurality of speakers to output the sound in the virtual space obtained by the virtual microphone. The orientation of the virtual microphone is a direction serving as a reference for the orientation of the virtual microphone in the virtual space, and may be maintained in a virtual reference direction that does not change in accordance with a change in the orientation of the user detected by the detection unit. The virtual space control unit may move the specific virtual sound source following a change in the orientation of the virtual camera so that the specific virtual sound source is arranged in the specific direction with respect to the orientation of the virtual camera.

[0008] In a first aspect, The virtual space control unit may move the specific virtual sound source following a change in the orientation of the virtual camera and movement of the virtual microphone so that the specific virtual sound source is arranged in the specific direction with respect to the orientation of the virtual camera and the distance between the virtual microphone and the specific virtual sound source is maintained at a predetermined distance.

[0009] In a first aspect, The specific direction may be the front direction.

[0010] In a first aspect, The specific sound may include a sound output in response to an operation of the user on the user interface.

[0011] A second aspect is a storage unit that stores the program of the first aspect, and a control unit that executes the program. It is an acoustic control device.

[0012] A third aspect is a display that follows the orientation of the user, a plurality of speakers arranged around the user, a sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user, and a control unit, wherein the control unit performs a detection process of detecting the orientation of the user based on a signal output from the sensor, a display process of changing the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection process and displaying an image of the virtual space obtained by the virtual camera on the display, and an acoustic process of outputting sound in the virtual space including the specific sound so that the specific sound is output from a specific direction toward the user with respect to the orientation of the user detected by the detection process to the plurality of speakers. It is a system.

Advantages of the Invention

[0013] According to the present disclosure, the user experience can be improved.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments will be described in detail with reference to the drawings. In the figures, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated. Further, hereinafter, an example in which a program is implemented as a game program, an acoustic control device is implemented as a game device, and a system is implemented as a game system will be described. The game device is an example of a computer system having one computer or a plurality of computers operating together.

[0016] (Description of the game) The game in the following description is a game (video game) performed in a virtual space. The virtual space is a three-dimensional game space and is displayed on a display as an image. Objects are arranged in the virtual space. Examples of objects include player characters operated by a player (a user who plays the game), non-player characters operated by a computer, and the like. A player is an example of a user.

[0017] In addition, the objects include sound source objects that serve as virtual sound sources in the virtual space. Examples of sound source objects include objects that operate in the virtual space, objects that only output sound in the virtual space, and the like. Player characters and non-player characters can become sound source objects. And in the game in the following description, sound is reproduced. The sound reproduced in the game includes sound output from the sound source object, BGM (Background Music), sound effects, and the like.

[0018] Also, in the game described below, the player character is a virtual player (virtual human) placed in a virtual space and moves within the virtual space according to the operations and movements of the player in the real world. Specifically, the virtual part moves within the virtual space according to the movement of a predetermined part of the player's body. The virtual part is a virtual part corresponding to a predetermined part of the player (a predetermined part of the player character's body). For example, according to the movement of the player's head in the real world, the virtual head (virtual head) of the player character moves in the virtual space.

[0019] Hereinafter, for the sake of convenience of explanation, as an example of the game, a "fighting action game in which the player character attacks and defeats enemy characters" will be given. Note that there is no limitation on the type of the game or the like. Also, as an example of the virtual space, a "virtual reality space where virtual reality (VR: Virtual Reality) can be experienced" will be given. Also, a case where the game progresses with the image of the virtual space being displayed on the display from the first-person perspective will be taken as an example.

[0020] (Game System) FIG. 1 illustrates the configuration of the game system 1 of the embodiment. The game system 1 includes a game device 10, a head-mounted display 20, a plurality of speakers 30, and a controller 40.

[0021] 〔Game Device〕 The game device 10 executes a game according to an operation by the user. In this example, the game device 10 is used together with the head-mounted display 20, the plurality of speakers 30, and the controller 40 and is connected to these devices by wire or wirelessly. The game device 10 is an example of an acoustic control device.

[0022] Commercially available devices such as, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), and Nintendo Switch (registered trademark) can be used as the game device 10.

[0023] In the game device 10, the game progresses based on the installed game program and game data. The game devices 10 can perform data communication with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0024] 〔Head-Mounted Display〕 The head-mounted display 20 includes a display 21 and a motion sensor 22.

[0025] The display 21 reproduces (displays) an image. For example, the display 21 is a liquid crystal display. The display 21 is an example of a display that follows the player's orientation.

[0026] The motion sensor 22 outputs a motion signal indicating the movement of the player's head. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (rotation and change in orientation), and the like. The motion signal output from the motion sensor 22 is transmitted to the game device 10.

[0027] As shown in FIG. 2, the head-mounted display 20 has a main body portion 25 and a headband 26. As shown in FIG. 3, the head-mounted display 20 is worn on the player P1's head in a state where the main body portion 25 covers both eyes of the player P1. Also, the display 21 is fixed inside the main body portion 25 (the side facing both eyes of the player P1). The display surface of the display 21 faces both eyes of the player P1 when the head-mounted display 20 is worn on the player P1's head.

[0028] Note that in conjunction with the movement of the head of player P1, the head-mounted display 20 worn on the head of player P1 moves. Therefore, based on the motion signal output from the motion sensor 22 included in the head-mounted display 20, the "position and orientation of the head of player P1" and the "position and orientation of the display 21 included in the head-mounted display 20" can be derived. The motion sensor 22 is an example of a sensor that outputs a signal indicating the movement of player P1, including changes in the orientation of player P1.

[0029] Also, the vector Vd in the head-mounted display 20 is the reference line of the image in the virtual space represented from the first-person perspective. The vector Vd extends in the normal direction of the display surface of the display 21. The vector Vd can be derived from the position and orientation of the display 21 included in the head-mounted display 20.

[0030] 〔Speaker〕 A plurality of speakers 30 are arranged around the player and reproduce sound (game sound in this example). In this example, a plurality of speakers 30 are provided so that three-dimensional sound can be reproduced.

[0031] As shown in FIG. 4, in this example, the plurality of speakers 30 are composed of a front center speaker 50, a front right speaker 51, a front left speaker 52, a surround right speaker 53, a surround left speaker 54, a surround back right speaker 55, and a surround back left speaker 56.

[0032] The front center speaker 50 is arranged in front of the reference point Q. The front right speaker 51 is arranged in the front right of the reference point Q. The front left speaker 52 is arranged in the front left of the reference point Q. The surround right speaker 53 is arranged on the right of the reference point Q. The surround left speaker 54 is arranged on the left of the reference point Q. The surround back right speaker 55 is arranged in the rear right of the reference point Q. The surround back left speaker 56 is arranged in the rear left of the reference point Q.

[0033] The reference point Q is a point (position) serving as a reference for determining the directions (front, rear, left, and right) in the plurality of speakers 30. For example, the reference point Q is the initial position (reference position) of the player. The directions in the plurality of speakers 30 will be described in detail later.

[0034] 〔Controller〕 The controller 40 is held or worn by the player and operated. In this example, the controller 40 is held and operated by the player's hand. As shown in FIG. 1, the controller 40 includes an operator 41 and a motion sensor 42.

[0035] The operator 41 is composed of various buttons and the like. The controller 40 outputs an operation signal corresponding to the operation input to the operator 41 (for example, the operation of pressing a button) to the game device 10. With such a configuration, the player inputs an operation signal (a signal indicating commands, data, etc.) to the game device 10 by operating the operator 41 of the controller 40.

[0036] The motion sensor 42 outputs a motion signal indicating the movement of the player's hand. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (rotation and change in direction), and the like. The motion signal output from the motion sensor 42 is transmitted to the game device 10.

[0037] Note that in conjunction with the movement of the player's hand, the controller 40 held (or worn) by the player's hand moves. Therefore, based on the motion signal output from the motion sensor 42 included in the controller 40, the position and orientation of the player's hand can be derived.

[0038] As shown in FIG. 3, in this example, a first controller 40a and a second controller 40b are provided as the controller 40. The configuration of each of the first controller 40a and the second controller 40b is the same as that of the controller 40.

[0039] The first controller 40a is held and operated by the right hand of player P1 and moves in conjunction with the movement of the right hand of player P1. The second controller 40b is held and operated by the left hand of player P1 and moves in conjunction with the movement of the left hand of player P1. Therefore, based on the motion signal output from the motion sensor 42 included in the first controller 40a, the position and orientation of the right hand of player P1 can be derived. Also, based on the motion signal output from the motion sensor 42 included in the second controller 40b, the position and orientation of the left hand of player P1 can be derived.

[0040] 〔Hardware Configuration of Game Device〕 As shown in FIG. 1, the game device 10 includes a network interface 11, a graphic processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, the graphic processing unit 12, the audio processing unit 13, the operation processing unit 14, and the storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0041] The network interface 11 is communicably connected to a communication network (not shown), for example, to transmit and receive various data to and from other game devices 10 and external server devices (not shown).

[0042] The graphic processing unit 12 is connected to the display 21 of the head-mounted display 20 either wired or wirelessly. The graphic processing unit 12 draws a game image including various objects in the virtual space in video format according to the game image information (image data) output from the control unit 16. The game image (image of the virtual space) drawn in video format is displayed on the display 21 as a game screen.

[0043] The audio processing unit 13 is connected to a plurality of speakers 30 either wired or wirelessly. The audio processing unit 13 reproduces digital game sounds according to the instructions of the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the plurality of speakers 30. In this example, the audio processing unit 13 is configured to be able to output multi-channel audio in order to perform three-dimensional sound reproduction.

[0044] The operation processing unit 14 is connected to each of the head-mounted display 20 and the controller 40 either wired or wirelessly. The operation processing unit 14 transmits and receives signals to and from each of the head-mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits the operation signal output from the motion sensor 22 of the head-mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to the operation input to the operation element 41), and the motion signal output from the motion sensor 42 of the controller 40 to the control unit 16.

[0045] The storage unit 15 stores various kinds of information and data. The storage unit 15 is composed of an HDD, an SSD, a RAM, a ROM, etc. For example, the storage unit 15 stores game data, various programs including game programs, etc. Examples of game data include game media, user account information, etc.

[0046] Also, the storage unit 15 stores various audio data indicating various pre-prepared sounds. The various audio data includes audio data indicating sounds output from sound source objects, audio data indicating music (BGM) reproduced in the virtual space, audio data indicating sound effects, etc.

[0047] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various kinds of information and data, and processes various kinds of information and data. The control unit 16 has a CPU (microcomputer) and a semiconductor memory. Programs and data for operating the CPU are stored in the semiconductor memory.

[0048] 〔Functional Configuration of Control Unit〕 The control unit 16 of the game device 10 includes a game progress unit 100, a detection unit 101, a display processing unit 102, and an audio processing unit 103. Specifically, by executing a game program, the control unit 16 functions as the game progress unit 100, the detection unit 101, the display processing unit 102, and the audio processing unit 103.

[0049] In other words, the control unit 16 performs game progress processing (virtual space control processing), detection processing, display processing, and audio processing. The detection unit 101 performs detection processing. The game progress processing is the processing by the game progress unit 100. The detection processing is the processing by the detection unit 101. The display processing is the processing by the display processing unit 102. The audio processing is the processing by the audio processing unit 103.

[0050] 〈Game Progress Unit (Virtual Space Control Unit)〉 The game progress unit 100 progresses the game performed in the virtual space. The game progress unit 100 is an example of a virtual space control unit that controls the virtual space. The game progress processing is an example of virtual space control processing (processing by the virtual space control unit).

[0051] In this example, based on the movement shown in the motion signal output from the motion sensor 22 (the movement of the player's head), the movement shown in the motion signal output from the motion sensor 42 (the movement of the player's hand), and the operation shown in the operation signal output from the controller 40 (the operation input to the operator 41 of the controller 40), the game progress unit 100 controls (generates or updates) the three-dimensional data indicating the virtual space so that these movements and operations are reflected in the virtual space, thereby progressing (controlling) the game performed in the virtual space.

[0052] The three-dimensional data indicating the virtual space includes well-known data for constructing the virtual space, such as data regarding the coordinate system (e.g., world coordinate system) serving as a reference for directions within the virtual space, data regarding the coordinate systems (e.g., local coordinate systems) of various objects arranged within the virtual space, and data regarding the shapes of various objects. By controlling the three-dimensional data indicating the virtual space, the movement of objects within the virtual space can be controlled.

[0053] For example, the game progress unit 100 controls the movement of the player character within the virtual space according to the above-mentioned movement and operations. Also, the game progress unit 100 controls the movement of other objects (such as non-player characters) within the virtual space according to the progress of the game. For example, AI (artificial intelligence) is utilized for the game progress unit 100 to control the movement of other objects.

[0054] Specifically, the game progress unit 100 controls the virtual space so that a virtual part (a predetermined part of the player character) moves within the virtual space according to the "movement of a predetermined part (head, right hand, left hand) of the player's body" indicated by the movement signal output from the motion sensor (motion sensor 22 or motion sensor 42). For example, the game progress unit 100 controls the virtual space so that the virtual head (the head of the player character) moves within the virtual space according to the "movement of the player's head" indicated by the movement signal output from the motion sensor 22. By moving the virtual head, the direction of the player character can be changed.

[0055] <Detection Unit> The detection unit 101 detects the orientation of the player based on the signal output from the motion sensor 22. The orientation of the player is the direction in which the front (front face) of the player faces. In this example, the orientation of the player is the vector Vd extending in the normal direction of the display surface of the display 21. For example, the detection unit 101 derives the position and orientation of the display 21 included in the head-mounted display 20 based on the motion signals (acceleration and angular velocity) output from the motion sensor 22 included in the head-mounted display 20, and derives the vector Vd based on the position and orientation of the display 21.

[0056] 〈Display processing unit〉 The display processing unit 102 generates an image of a virtual space (game image in this example) controlled by the game progress unit 100. Specifically, the display processing unit 102 generates image data indicating an image of the virtual space corresponding to the vector Vd based on the three-dimensional data (three-dimensional data indicating the virtual space) controlled by the game progress unit 100 and the vector Vd (vector Vd indicating the orientation of the player) detected by the detection unit 101. The process of generating an image corresponding to the vector Vd from the three-dimensional data is a well-known process (such as coordinate transformation and projection transformation).

[0057] Then, the display processing unit 102 reproduces (displays) the image generated as described above on the display 21. In this example, the display processing unit 102 supplies the image data generated as described above to the graphic processing unit 12. As a result, the image of the virtual space shown in the image data is reproduced (displayed) on the display 21 of the head-mounted display 20.

[0058] As shown in FIG. 5, a virtual camera 61 is arranged in the virtual space. The virtual camera 61 is arranged in the vicinity of the player character C1. The virtual camera 61 is a virtual viewer (viewpoint), for example, a reference point for reproducing the viewpoint (first-person viewpoint) of the player character. The display processing unit 102 changes the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101. Further, the display processing unit 102 moves the virtual camera 61 so as to follow the movement of the player character C1. Then, the display processing unit 102 causes the display 21 to display an image of the virtual space obtained by the virtual camera 61.

[0059] Note that the process of deriving the orientation of the virtual camera 61 from the orientation of the player (for example, vector Vd) is a well-known process (such as coordinate transformation). The movement of the player character C1 is performed according to the movement or operation of the player (the operation input to the controller 40).

[0060] Specifically, the display processing unit 102 generates image data indicating the above image according to the shooting situation of the virtual camera 61 (for example, the position and orientation of the virtual camera 61 with respect to the object in the virtual space, the orientation of the virtual camera 61, the orientation of the object, etc.) so that the image displayed on the display 21 is an image including the image obtained by the virtual camera 61. The image (image data) obtained by the virtual camera 61 is an image (image data) generated according to the shooting situation of the virtual camera 61.

[0061] <Acoustic Processing Unit> The audio processing unit 103 generates the sound (game sound in this example) of the virtual space controlled by the game progress unit 100. The sound of the virtual space is sound corresponding to the situation (such as the movement of objects) in the virtual space. Specifically, the audio processing unit 103 selects the audio data to be played from the audio data (various types of pre-prepared audio data) stored in the storage unit 15 according to the situation in the virtual space (the progress situation of the game in this example), and based on the selected audio data (or new audio data obtained by synthesizing the selected multiple audio data), generates audio data indicating the sound corresponding to the situation in the virtual space.

[0062] Then, the audio processing unit 103 causes the sound generated as described above to be reproduced (output) by a plurality of speakers 30. In this example, the audio processing unit 103 supplies the audio data generated as described above to the audio processing unit 13. As a result, the sound of the virtual space indicated by the audio data is reproduced (output) from the plurality of speakers 30.

[0063] 《Sound field maintenance processing》 The audio processing unit 103 performs sound field maintenance processing. In the sound field maintenance processing, the audio processing unit 103 causes the sound in the virtual space to be output to the plurality of speakers 30 so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 does not change even when the direction of the virtual camera 61 changes. The sound field maintenance processing is included in the audio processing.

[0064] 《Sound image localization processing》 In this example, the audio processing unit 103 performs sound image localization processing. The sound image localization processing is the processing of localizing the position of the sound source felt by a human (player) to a predetermined virtual position (for example, the position of the sound source object) in the virtual space. The sound image localization processing is included in the audio processing.

[0065] As shown in FIG. 5, a virtual microphone 62 is arranged in the virtual space. The virtual microphone 62 is arranged in the vicinity of the player character C1 together with the virtual camera 61. The virtual microphone 62 is a virtual listener (sound collection point) and is a reference point for sound image localization in the sound image localization process. The acoustic processing unit 103 moves the virtual microphone 62 so as to follow the movement of the player character C1 and the virtual camera 61. Then, the acoustic processing unit 103 reproduces (outputs) the sound obtained by the virtual microphone 62 to the plurality of speakers 30.

[0066] Specifically, in the sound image localization process, the acoustic processing unit 103 generates audio data indicating the above-mentioned sound so that the sound reproduced (output) from the plurality of speakers 30 becomes a sound including the sound obtained by the virtual microphone 62, according to the sound collection situation of the virtual microphone 62 (for example, the position and direction of the virtual microphone 62 with respect to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, the orientation of the virtual microphone 62, the orientation of the sound source object, etc.). The sound (audio data) obtained by the virtual microphone 62 is the sound (audio data) generated according to the sound collection situation of the virtual microphone 62. The sound output from the sound source object is the sound (pre-prepared sound) associated with the sound source object.

[0067] By processing the audio data generated by the sound image localization process by the audio processing unit 13, it becomes possible to output from the plurality of speakers 30 the sound output from the virtual sound source (sound source object) in the virtual space in an acoustic expression as if it were collected by the virtual microphone 62.

[0068] And in this example, the acoustic processing unit 103 performs sound field maintenance processing in the sound image localization processing. The sound field maintenance processing is a process of maintaining (fixing) the direction of the virtual microphone 62 in the sound image localization processing to the virtual reference direction. Specifically, the acoustic processing unit 103 moves the virtual microphone 62 arranged in the virtual space so as to follow the movement of the virtual camera 61, and outputs the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30. Note that the direction of the virtual microphone 62 is maintained (fixed) to a predetermined virtual reference direction. In other words, the acoustic processing unit 103 moves the virtual microphone 62 so as to follow the movement of the virtual camera 61 while maintaining the direction of the virtual microphone 62 to the virtual reference direction.

[0069] The virtual reference direction is a direction that serves as a reference for the direction of the virtual microphone 62 in the virtual space and does not change according to the change in the direction of the player detected by the detection unit 101. The virtual reference direction will be described in detail later.

[0070] 《Specific Acoustic Processing》 Also, the acoustic processing unit 103 performs specific acoustic processing. In the specific acoustic processing, the acoustic processing unit 103 outputs the sound in the virtual space including the specific sound to a plurality of speakers 30 so that the specific sound predetermined from a specific direction toward the player is output with respect to the direction of the player detected by the detection unit 101.

[0071] As shown in FIG. 5, in this example, a specific virtual sound source 65 is arranged in the virtual space. The specific virtual sound source 65 is a sound source object (an object not displayed in the image in this example) for outputting the specific sound A5. The game progress unit 100 moves the specific virtual sound source 65 following the change in the direction of the virtual camera 61 so that the specific virtual sound source 65 is arranged in a specific direction with respect to the direction of the virtual camera 61. The direction of the virtual camera 61 corresponds to the direction of the player detected by the detection unit 101. The acoustic processing unit 103 performs sound image localization processing. Thereby, the sound in the virtual space including the specific sound A5 is output from the plurality of speakers 30.

[0072] In this example, the specific direction is the front direction. In the specific acoustic processing, the acoustic processing unit 103 causes the speakers 30 to output the sound in the virtual space including the specific voice so that the specific voice is output from the front direction (front) of the player detected by the detection unit 101 toward the player. The game progress unit 100 moves the specific virtual sound source 65 following the change in the direction of the virtual camera 61 so that the specific virtual sound source 65 is arranged in the front direction (front) of the virtual camera 61.

[0073] Also, in this example, the specific voice includes the sound (operation sound) output according to the operation of the player on the user interface. For example, the user interface is the controller 40. Other examples of the specific voice include guidance voices such as narration voices. The specific voice is the voice that should be (should continue to be) output from a specific direction (a certain direction) toward the player P1 with respect to the direction of the player P1.

[0074] Note that, in this example, the game progress unit 100 moves the specific virtual sound source 65 following the change in the direction of the virtual camera 61 and the movement of the virtual microphone 62 so that the specific virtual sound source 65 is arranged in a specific direction with respect to the direction of the virtual camera 61 and the distance between the virtual microphone 62 and the specific virtual sound source 65 is maintained at a predetermined distance. For example, the game progress unit 100 moves the specific virtual sound source 65 on a horizontal plane circumference centered on the virtual microphone 62 (a circumference with a radius set to a predetermined distance).

[0075] 〔Various Directions〕 As shown in FIG. 4, the directions (front, back, left, and right) of the plurality of speakers 30 are defined based on a reference point Q surrounded by the plurality of speakers 30. For example, the "front" of the plurality of speakers 30 corresponds to the front of the reference point Q, and the speaker 30 located in the "front" among the plurality of speakers 30 is the "front center speaker 50" arranged in front of the reference point Q.

[0076] The direction of player P1 is the direction in which the front of player P1 faces. In the example of FIG. 4, the direction of player P1 is the direction facing "front" among the plurality of speakers 30.

[0077] As shown in FIG. 5, the directions (front-back, left-right, up-down) in the virtual space are predetermined. For example, "front-back, left-right" in the virtual space corresponds to "north-south, west-east" predetermined in the virtual space, and "up-down" in the virtual space corresponds to "heaven-earth" predetermined in the virtual space. The directions (front-back, left-right, up-down) in the virtual space are expressed in a three-dimensional coordinate system (for example, the world coordinate system).

[0078] The direction of the virtual camera 61 is the direction in which the front of the virtual camera 61 faces. In this example, the direction of the virtual camera 61 is the direction corresponding to the direction of the player (specifically, the vector Vd) derived based on the signal output from the motion sensor 42 of the head-mounted display 20.

[0079] The direction of the virtual microphone 62 is the direction in which the front of the virtual microphone 62 faces. The directions (front-back, left-right) of the virtual microphone 62 correspond to the directions (front-back, left-right) of the plurality of speakers 30. For example, the sound (sound in the virtual space) output from the "front" of the virtual microphone 62 toward the virtual microphone 62 is output from the front center speaker 50 (the speaker 30 located at the "front") corresponding to the "front" among the plurality of speakers 30.

[0080] In the embodiment, the direction of the virtual microphone 62 is maintained (fixed) in the virtual reference direction. The virtual reference direction (the direction serving as a reference for the direction of the virtual microphone 62 in the virtual space) is set to the direction of the virtual microphone 62 when the relationship between the direction of the virtual microphone 62 and the direction in the virtual space becomes a predetermined relationship. Specifically, the virtual reference direction is set to the direction of the virtual microphone 62 (the direction in which the front of the virtual microphone 62 faces) when the direction (front-back, left-right) of the virtual microphone 62 and the direction (front-back, left-right) in the virtual space coincide.

[0081] In the example of FIG. 5, the virtual reference direction is the direction facing "front (north)" in the virtual space. When the front of the virtual microphone 62 faces "front (north)" in the virtual space, the directions (front, rear, left, right) in the virtual microphone 62 and the directions (front, rear, left, right) in the virtual space coincide.

[0082] The direction of the specific virtual sound source 65 is the direction that the front of the specific virtual sound source 65 faces. The direction of the specific virtual sound source 65 is the direction facing the virtual microphone 62.

[0083] 〔Details of Processing by the Control Unit〕 Next, with reference to FIGS. 6 and 7, the processing by the control unit 16 will be described in detail. FIG. 6 illustrates the case where the player P1 faces the "front" of the plurality of speakers 30, and FIG. 7 illustrates the case where the player P1 faces the "right side" of the plurality of speakers 30.

[0084] As shown in FIG. 6, when the player P1 faces the "front" of the plurality of speakers 30, the player character C1 faces "front (north)" in the virtual space. Similarly, the virtual camera 61 faces "front (north)" in the virtual space. As a result, an image including the enemy character C2 located in the "front (north)" of the player character C1 in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by the player P1. The enemy character C2 is an example of a sound source object.

[0085] The virtual microphone 62 is maintained in a state of facing "front (north)" in the virtual space. As a result, the character voice A2 output from the enemy character C2 located in the "front (north)" of the player character C1 in the virtual space is output from the front center speaker 50 arranged in the "front" of the player P1.

[0086] The specific virtual sound source 65 is arranged in the front direction (forward) with respect to the orientation of the virtual camera 61. As a result, the specific voice A5 output from the specific virtual sound source 65 located in the "front (north)" of the player character C1 in the virtual space is output from the front center speaker 50 arranged in the "front" of the player P1.

[0087] As shown in FIG. 7, when the player P1 faces the "right side" among the plurality of speakers 30, the player character C1 faces the "right side (east)" in the virtual space. Similarly, the virtual camera 61 faces the "right side (east)" in the virtual space. As a result, an image of the "right side (east)" of the player character C1 in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by the player P1. The enemy character C2 is located on the "left side (north)" of the player character C1.

[0088] The virtual microphone 62 is maintained in a state of facing the "front (north)" in the virtual space. Therefore, the character voice A2 output from the enemy character C2 located on the "left side (north)" of the player character C1 in the virtual space is output from the front center speaker 50 arranged on the "left side" of the player P1.

[0089] The specific virtual sound source 65 is arranged in the front direction (forward) with respect to the orientation of the virtual camera 61. As a result, the specific voice A5 output from the specific virtual sound source 65 located in the "front (east)" of the player character C1 in the virtual space is output from the surround light speaker 53 arranged in the "front" of the player P1.

[0090] As described above, an image in the virtual space corresponding to the orientation of player P1 is displayed on the display 21 of the head-mounted display 20. Also, the direction of the sound heading toward the virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) coincides with the direction of the sound heading from the plurality of speakers 30 toward player P1 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the plurality of speakers 30 with appropriate sound localization. Also, a specific sound A5 is output toward player P1 from a specific direction (the front direction in this example) with respect to the orientation of player P1.

[0091] 〔Summary of Embodiment〕 In summary, the program (game program) of the embodiment is a program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit 101, a display processing unit 102, and an acoustic processing unit 103. The computer system is used together with a display 21 that follows the orientation of the user (player), a plurality of speakers 30 arranged around the user, and a sensor (motion sensor 22) that outputs a signal indicating the motion of the user including a change in the orientation of the user. The detection unit 101 detects the orientation of the user based on the signal output from the sensor. The display processing unit 102 changes the orientation of the virtual camera 61 arranged in the virtual space so as to follow the orientation of the user detected by the detection unit 101, and causes the image of the virtual space obtained by the virtual camera 61 to be displayed on the display 21. The acoustic processing unit 103 causes the sound in the virtual space including the specific sound to be output from the plurality of speakers 30 so that the specific sound is output toward the user from a specific direction with respect to the orientation of the user detected by the detection unit 101.

[0092] The acoustic control device (game device 10) of the embodiment includes a storage unit 15 that stores the above program and a control unit 16 that executes the above program.

[0093] The system of the embodiment (game system 1) includes a display 21 that follows the direction of a user (player), a plurality of speakers 30 arranged around the user, a sensor (motion sensor 22) that outputs a signal indicating the motion of the user including a change in the direction of the user, and a control unit 16. The control unit 16 performs detection processing, display processing, and acoustic processing. In the detection processing, the control unit (detection unit 101) detects the direction of the user based on the signal output from the sensor. In the display processing, the control unit (display processing unit 102) changes the direction of a virtual camera 61 arranged in the virtual space so as to follow the direction of the user detected by the detection processing, and causes the display 21 to display an image of the virtual space obtained by the virtual camera 61. In the acoustic processing, the control unit (acoustic processing unit 103) causes the plurality of speakers to output the sound in the virtual space including the specific sound so that the specific sound is output from a specific direction toward the user with respect to the direction of the user detected by the detection processing.

[0094] [Effect of the Embodiment] As described above, in the embodiment, the sound in the virtual space including the specific sound is output to the plurality of speakers 30 so that the specific sound A5 is output from a specific direction (front direction in this example) toward the player P1 with respect to the direction of the player P1 detected by the detection unit 101.

[0095] According to the above control, since the specific sound A5 can be output from a specific direction toward the player P1 with respect to the direction of the player P1, the user experience can be improved. For example, the state in which the specific sound A5 is output from a specific direction toward the player P1 with respect to the direction of the player P1 (the state in which the specific sound A5 can be heard from the same direction) can be continued. Thereby, it is possible to prevent an uncomfortable feeling from being caused by the frequent change in the direction from which the specific sound A5 can be heard. Also, the specific sound A5 can be made easier to hear than in the case where the direction from which the specific sound A5 can be heard frequently changes.

[0096] (Modification Example of the Embodiment) In the game system 1 of the modification example of the embodiment, the processing by the control unit 16 is different from that of the game system 1 of the embodiment. Other configurations and processes of the game system 1 of the modification example of the embodiment are the same as those of the game system 1 of the embodiment. The control unit 16 (game progress unit 100, display processing unit 102, and audio processing unit 103) in the modification example of the embodiment performs the following processing in addition to the processing by the control unit 16 in the embodiment.

[0097] As shown in FIG. 8, in the modification example of the embodiment, the virtual space includes a first virtual space S1 and a second virtual space S2. The first virtual space S1 is the main space where the game mainly progresses. The second virtual space S2 is a sub - space where a predetermined event is held. In this example, the predetermined event is an event of purchasing a commodity (such as a weapon) from the merchant character C3.

[0098] When the predetermined event occurrence condition is satisfied, the game progress unit 100 transfers (warps) the player character C1 from the first virtual space S1 to the second virtual space S2, and sets the direction of the player character C1 to a predetermined direction within the second virtual space S2. In this example, the predetermined event occurrence condition is a condition that the player character C1 contacts (for example, talks to) a merchant character (not shown) arranged within the first virtual space S1. The predetermined direction is a direction facing the right (east) within the second virtual space S2. Within the second virtual space S2, the player character C1 is arranged to the left (west) of the merchant character C3.

[0099] When the predetermined event occurrence condition is satisfied, the display processing unit 102 transfers the virtual camera 61 from the first virtual space S1 to the second virtual space S2 so as to follow the transfer of the player character C1, and sets the direction of the virtual camera 61 to a predetermined direction within the second virtual space S2. In this example, the predetermined direction of the virtual camera 61 is the same as the predetermined direction of the player character C1. Within the second virtual space S2, the virtual camera 61 is arranged to the left (west) of the merchant character C3.

[0100] The audio processing unit 103 maintains the position of the virtual microphone 62 within the first virtual space S1 without following the transfer of the player character C1 and the virtual camera 61.

[0101] Also, when a predetermined event occurrence condition is satisfied, the game progress unit 100 arranges a specific virtual sound source 65 (a specific virtual sound source 65 related to a predetermined event) within the first virtual space S1 such that the direction of the virtual microphone 62 with respect to the specific virtual sound source 65 within the first virtual space S1 coincides with the direction of the player with respect to the plurality of speakers 30.

[0102] 〔Details of Processing by Control Unit〕 Next, with reference to FIG. 8, the processing by the control unit 16 in a modification of the embodiment will be described in detail. Hereinafter, the case where the direction of the player P1 with respect to the plurality of speakers 30 is "forward" (when the player P1 is facing "forward" among the plurality of speakers 30) will be taken as an example.

[0103] When a predetermined event occurrence condition is satisfied, the player character C1 and the virtual camera 61 transfer from the first virtual space S1 to the second virtual space S2 and are arranged "to the left (west)" of the merchant character C3 within the second virtual space S2. The direction of each of the player character C1 and the virtual camera 61 is set to face "to the right (east)" within the second virtual space S2. Thereby, an image (not shown) including the merchant character C3 located in front (west) of the player character C1 within the second virtual space S2 is displayed on the display 21 of the head-mounted display 20 worn by the player P1.

[0104] Also, the virtual microphone 62 is maintained in a state of facing "forward (north)" within the first virtual space S1. And in front of the virtual microphone 62, a specific virtual sound source 65 that outputs a character voice A3 (an example of a specific voice A5) related to the merchant character C3 is arranged. As a result, the character voice A3 output from the specific virtual sound source 65 located "forward (north)" of the virtual microphone 62 within the first virtual space S1 is output from the front center speaker 50 arranged in the "front" of the player P1.

[0105] Through the above processing, an image including the merchant character C3 is displayed on the display 21 of the head-mounted display 20, and for the player P1 facing the "front" (front center speaker 50) among the plurality of speakers 30, the character voice A3 is output from the front (specific direction) of the player P1 toward the player P1.

[0106] [Effect of the Modification of the Embodiment] In the modification of the embodiment, an effect similar to that of the embodiment can be obtained.

[0107] Also, in the modification of the embodiment, even when a predetermined event occurrence condition is satisfied, the specific voice A5 can be output from a specific direction toward the player P1 with respect to the orientation of the player P1, so the user experience can be improved.

[0108] Note that in the modification of the embodiment, when a predetermined event occurrence condition is satisfied, the position of the player character C1 may be maintained within the first virtual space S1 without the player character C1 transferring from the first virtual space S1 to the second virtual space S2.

[0109] Also, the first virtual space S1 and the second virtual space S2 may be virtual spaces belonging to different coordinate systems. Or, the first virtual space S1 and the second virtual space S2 may be virtual spaces belonging to a common coordinate system but not continuous.

[0110] (Other Embodiments) In the above description, it may be configured or set as follows.

[0111] The voice data does not necessarily have to be prepared in advance. For example, the acoustic processing unit 103 may synthesize or generate voice data each time based on some data (such as the voice data stored in the storage unit 15), information, etc.

[0112] The player character may or may not be displayed on the display. In other words, in the image of the virtual space displayed (played) on the display, the player character may or may not be included. For example, when the image of the virtual space is displayed on the display from the first-person perspective, only a virtual hand (virtual part), which is a part of the player character, may be displayed, or only a handgun (virtual object) held by the virtual hand without the virtual hand being displayed may be displayed.

[0113] The image of the virtual space may be an image from the first-person perspective or an image from the third-person perspective (for example, an image when looking at the player character from behind the player character).

[0114] The plurality of speakers 30 may be arranged on the left and right, front and back, above, etc. of the listener (player) so as to be able to reproduce three-dimensional sound.

[0115] The sensor that outputs a signal indicating the movement of the player including the change in the direction of the player is not limited to the motion sensor 22 included in the head-mounted display 20. For example, such a sensor may be a combination of a camera (image sensor) that images a predetermined part (for example, the head) on the player's body and a recognition processing unit that recognizes the movement of the player including the change in the direction of the player based on the "image of the predetermined part" obtained by the camera.

[0116] The game program may be implemented as a game program for so-called online games. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in the control unit (not shown) of the server device, or may be shared between the control unit of the game device 10 and the control unit of the server device. The game device 10 may be a portable information terminal such as a smartphone or a tablet.

[0117] Also, in the above description, as an example of the virtual space, a virtual space (game space) in which the game progresses is given, but it is not limited to this. For example, the virtual space may be a virtual space (metaverse space) in which avatars communicate with each other. In other words, the services provided in the virtual space are not limited to games. For example, the services provided in the virtual space may be the provision of a place for business transactions, the provision of a place for communication, etc.

[0118] Even when these other embodiments are adopted, the effects of the present invention are exhibited. Also, it is possible to appropriately combine this embodiment with other embodiments and between other embodiments. The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses.

Explanation of Reference Numerals

[0119] 1 Game system (system) 10 Game device (acoustic control device) 11 Network interface 12 Graphics processing unit 13 Audio processing unit 14 Operation processing unit 20 Head-mounted display 21 Display 22 Motion sensor 25 Main body unit 26 Headband 30 Speaker 40 Controller 40a First controller 40b Second Controller 41 Operator 42 Motion Sensor (Sensor) 50 Front Center Speaker 51 Front Right Speaker 52 Front Left Speaker 53 Surround Right Speaker 54 Surround Left Speaker 55 Surround Back Right Speaker 56 Surround Back Left Speaker 61 Virtual Camera 62 Virtual Microphone 65 Specific Virtual Sound Source 100 Game Progression Unit (Virtual Space Control Unit) 101 Detection Unit 102 Display Processing Unit 103 Acoustic Processing Unit P1 Player (User) C1 Player Character (User Character) C2 Enemy Character (Sound Source Object) A2 Character Voice (Voice in Virtual Space) A5 Specific Voice

Claims

1. A program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the orientation of the user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user, The detection unit detects the orientation of the user based on the signal output from the sensor, The display processing unit changes the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection unit, and displays an image of the virtual space obtained by the virtual camera on the display, The acoustic processing unit outputs the sound in the virtual space including the specific sound so that the specific sound is output from a specific direction toward the user with respect to the orientation of the user detected by the detection unit to the plurality of speakers Program.

2. In the program of Claim 1, The computer system is caused to function as a virtual space control unit, A specific virtual sound source for outputting the specific sound is arranged in the virtual space, The acoustic processing unit moves a virtual microphone arranged in the virtual space so as to follow the movement of the virtual camera, and outputs the sound in the virtual space obtained by the virtual microphone to the plurality of speakers, The orientation of the virtual microphone is a reference direction for the orientation of the virtual microphone in the virtual space, and is maintained in a virtual reference direction that does not change according to the change in the orientation of the user detected by the detection unit, The virtual space control unit moves the specific virtual sound source following the change in the orientation of the virtual camera so that the specific virtual sound source is arranged in the specific direction with respect to the orientation of the virtual camera, Program.

3. In the program of claim 2, the virtual space control unit moves the specific virtual sound source following the change in the orientation of the virtual camera and the movement of the virtual microphone so that the specific virtual sound source is arranged in the specific direction with respect to the orientation of the virtual camera and the distance between the virtual microphone and the specific virtual sound source is maintained at a predetermined distance. Program.

4. In the program of claim 1, the specific direction is the front direction Program.

5. In the program of claim 1, the specific sound includes the sound output in response to the user's operation on the user interface Program.

6. A storage unit that stores any one of the programs of claims 1 to 5, and a control unit that executes the program. Acoustic control device.

7. A display that follows the user's orientation, a plurality of speakers arranged around the user, a sensor that outputs a signal indicating the movement of the user including the change in the orientation of the user, and a control unit, the control unit performs a detection process of detecting the orientation of the user based on the signal output from the sensor, a display process of changing the orientation of a virtual camera arranged in the virtual space so as to follow the orientation of the user detected by the detection process and displaying an image of the virtual space obtained by the virtual camera on the display, and an acoustic process of outputting the sound in the virtual space including the specific sound to the plurality of speakers so that the specific sound is output from a specific direction toward the user with respect to the orientation of the user detected by the detection process. System.

Images