Program, acoustic control device, and system

Abstract

To enhance user experience.SOLUTION: A detection unit detects the orientation of a user P1 on the basis of signals output from a sensor. A display processing unit changes the orientation of a virtual camera 61 arranged in a virtual space so as to follow the orientation of the user P1 detected by the detection unit, and causes a display to display the image of the virtual space obtained by the virtual camera 61. An audio processing unit causes a plurality of speakers 30 to output audio in the virtual space, in such a manner that the orientation of a sound field formed by the audio in the virtual space output from the speakers 30 does not change even if the orientation of the virtual camera 61 changes.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present disclosure relates to a program, an acoustic control device, and a system.

Background Art

[0002] Some game programs reproduce voices in a virtual space. For example, in Patent Document 1, voices such as gunshots are reproduced in a shooting game.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to provide a user with images and voices in a virtual space in which various services (for example, games) are provided using a system including a display (for example, a head-mounted display) that follows the user's orientation and a plurality of speakers arranged around the user. In such a system, from the viewpoint of the user experience, it is desirable that the voices in the virtual space corresponding to the images in the virtual space displayed on the display be reproduced from the plurality of speakers with appropriate localization.

[0005] An object of the present disclosure is to improve the user experience in services provided in a virtual space.

Means for Solving the Problems

[0006] A first aspect is a program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the orientation of the user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user. The detection unit detects the orientation of the user based on the signal output from the sensor. The display processing unit changes the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection unit, and causes the display to display an image of the virtual space obtained by the virtual camera. The acoustic processing unit causes the plurality of speakers to output the sound in the virtual space so that the orientation of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even when the orientation of the virtual camera changes. It is a program.

[0007] In a first aspect, The acoustic processing unit may move a virtual microphone arranged in the virtual space so as to follow the movement of the virtual camera, and cause the plurality of speakers to output the sound in the virtual space obtained by the virtual microphone. The orientation of the virtual microphone may be maintained in a virtual reference direction that is a reference direction for the orientation of the virtual microphone in the virtual space and does not change in response to a change in the orientation of the user detected by the detection unit.

[0008] In a first aspect, The computer system may be used together with a controller operated by the user. When a direction change operation for changing the orientation of the virtual camera from the front direction corresponding to the orientation of the user detected by the detection unit is input to the controller, the display processing unit may change the orientation of the virtual camera from the front direction in response to the direction change operation. When the direction of the virtual camera changes from the front direction in response to the direction change operation, the audio processing unit may change the direction of the virtual microphone so as to follow the direction of the virtual camera.

[0009] In the first aspect, When a reset operation for resetting the direction of the virtual camera to the front direction is input to the controller, the display processing unit may reset the direction of the virtual camera to the front direction in response to the reset operation. When the direction of the virtual camera is reset to the front direction in response to the reset operation, the audio processing unit may reset the direction of the virtual microphone to the virtual reference direction.

[0010] The second aspect includes a storage unit that stores the program of the first aspect, and a control unit that executes the program. It is an acoustic control device.

[0011] The third aspect is a display that follows the user's direction, a plurality of speakers arranged around the user, a sensor that outputs a signal indicating the movement of the user including a change in the user's direction, and a control unit, wherein the control unit performs a detection process of detecting the user's direction based on the signal output from the sensor, a display process of changing the direction of a virtual camera arranged in a virtual space so as to follow the user's direction detected by the detection process and displaying an image of the virtual space obtained by the virtual camera on the display, and an acoustic process of outputting the sound in the virtual space to the plurality of speakers so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even when the direction of the virtual camera changes. It is a system.

Advantages of the Invention

[0012] According to the present disclosure, the user experience can be improved.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated. Further, hereinafter, an example will be described in which a program is implemented as a game program, an acoustic control device is implemented as a game device, and a system is implemented as a game system. The game device is an example of a computer system having one computer or a plurality of cooperating computers.

[0015] (Description of the game) The game in the following description is a game (video game) that is played in a virtual space. The virtual space is a three-dimensional game space and is displayed on a display as an image. Objects are arranged in the virtual space. Examples of objects include player characters operated by a player (a user who plays the game), non-player characters operated by a computer, and the like. A player is an example of a user.

[0016] In addition, the objects include sound source objects that serve as virtual sound sources in the virtual space. Examples of sound source objects include objects that operate in the virtual space, objects that only output sound in the virtual space, and the like. Player characters and non-player characters can become sound source objects. And, in the game in the following description, sound is reproduced. The sound reproduced in the game includes sound output from sound source objects, BGM (Background Music), sound effects, and the like.

[0017] Also, in the game described below, the player character is a virtual player (virtual human) placed in a virtual space and moves within the virtual space according to the operations and movements of the player in the real world. Specifically, the virtual part moves within the virtual space according to the movement of a predetermined part of the player's body. The virtual part is a virtual part corresponding to a predetermined part of the player (a predetermined part of the player character's body). For example, according to the movement of the player's head in the real world, the virtual head (virtual head) of the player character moves in the virtual space.

[0018] Hereinafter, for the sake of convenience of explanation, as an example of the game, a "fighting action game in which the player character attacks and defeats enemy characters" will be given. Note that there is no limitation on the type of the game. Also, as an example of the virtual space, a "virtual reality space where virtual reality (VR) can be experienced" will be given. Also, a case will be given where the game progresses with the image of the virtual space being displayed on the display from the first-person perspective.

[0019] (Game System) FIG. 1 illustrates the configuration of the game system 1 of the embodiment. The game system 1 includes a game device 10, a head-mounted display 20, a plurality of speakers 30, and a controller 40.

[0020] 〔Game Device〕 The game device 10 executes a game according to an operation by the user. In this example, the game device 10 is used together with the head-mounted display 20, the plurality of speakers 30, and the controller 40 and is connected to these devices by wire or wirelessly. The game device 10 is an example of an acoustic control device.

[0021] Commercially available devices such as, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), Nintendo Switch (registered trademark) can be used as the game device 10.

[0022] In the game device 10, the game progresses based on the installed game program and game data. The game devices 10 can perform data communication with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0023] 〔Head-Mounted Display〕 The head-mounted display 20 includes a display 21 and a motion sensor 22.

[0024] The display 21 reproduces (displays) an image. For example, the display 21 is a liquid crystal display. The display 21 is an example of a display that follows the player's orientation.

[0025] The motion sensor 22 outputs a motion signal indicating the movement of the player's head. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (rotation and change in orientation), and the like. The motion signal output from the motion sensor 22 is transmitted to the game device 10.

[0026] As shown in FIG. 2, the head-mounted display 20 has a main body portion 25 and a headband 26. As shown in FIG. 3, the head-mounted display 20 is worn on the player P1's head in a state where the main body portion 25 covers both eyes of the player P1. Also, the display 21 is fixed inside the main body portion 25 (the side facing both eyes of the player P1). The display surface of the display 21 faces both eyes of the player P1 when the head-mounted display 20 is worn on the player P1's head.

[0027] Note that in conjunction with the movement of the head of player P1, the head-mounted display 20 worn on the head of player P1 moves. Therefore, based on the motion signal output from the motion sensor 22 included in the head-mounted display 20, the "position and orientation of the head of player P1" and the "position and orientation of the display 21 included in the head-mounted display 20" can be derived. The motion sensor 22 is an example of a sensor that outputs a signal indicating the movement of player P1 including a change in the orientation of player P1.

[0028] Also, the vector Vd in the head-mounted display 20 is a reference line of an image in a virtual space represented from a first-person perspective. The vector Vd extends in the normal direction of the display surface of the display 21. The vector Vd can be derived from the position and orientation of the display 21 included in the head-mounted display 20.

[0029] 〔Speaker〕 A plurality of speakers 30 are arranged around the player and reproduce sound (game sound in this example). In this example, a plurality of speakers 30 are provided so that three-dimensional sound can be reproduced.

[0030] As shown in FIG. 4, in this example, the plurality of speakers 30 are composed of a front center speaker 50, a front right speaker 51, a front left speaker 52, a surround right speaker 53, a surround left speaker 54, a surround back right speaker 55, and a surround back left speaker 56.

[0031] The front center speaker 50 is arranged in front of the reference point Q. The front right speaker 51 is arranged in the front right of the reference point Q. The front left speaker 52 is arranged in the front left of the reference point Q. The surround right speaker 53 is arranged on the right of the reference point Q. The surround left speaker 54 is arranged on the left of the reference point Q. The surround back right speaker 55 is arranged in the right rear of the reference point Q. The surround back left speaker 56 is arranged in the left rear of the reference point Q.

[0032] The reference point Q is a point (position) serving as a reference for determining directions (front, back, left, and right) in a plurality of speakers 30. For example, the reference point Q is the initial position (reference position) of the player. The directions in the plurality of speakers 30 will be described in detail later.

[0033] 〔Controller〕 The controller 40 is held or worn by the player and operated. In this example, the controller 40 is held and operated by the player's hand. As shown in FIG. 1, the controller 40 includes an operator 41 and a motion sensor 42.

[0034] The operator 41 is composed of various buttons and the like. The controller 40 outputs an operation signal corresponding to an operation input to the operator 41 (for example, an operation of pressing a button) to the game device 10. With such a configuration, the player inputs an operation signal (a signal indicating commands, data, etc.) to the game device 10 by operating the operator 41 of the controller 40.

[0035] The motion sensor 42 outputs a motion signal indicating the movement of the player's hand. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (rotation and change in direction), and the like. The motion signal output from the motion sensor 42 is transmitted to the game device 10.

[0036] Note that in conjunction with the movement of the player's hand, the controller 40 held (or worn) by the player's hand moves. Therefore, based on the motion signal output from the motion sensor 42 included in the controller 40, the position and orientation of the player's hand can be derived.

[0037] As shown in FIG. 3, in this example, a first controller 40a and a second controller 40b are provided as the controller 40. The configuration of each of the first controller 40a and the second controller 40b is the same as that of the controller 40.

[0038] The first controller 40a is held and operated by the right hand of player P1 and moves in conjunction with the movement of the right hand of player P1. The second controller 40b is held and operated by the left hand of player P1 and moves in conjunction with the movement of the left hand of player P1. Therefore, based on the motion signal output from the motion sensor 42 included in the first controller 40a, the position and orientation of the right hand of player P1 can be derived. Also, based on the motion signal output from the motion sensor 42 included in the second controller 40b, the position and orientation of the left hand of player P1 can be derived.

[0039] 〔Hardware Configuration of Game Device〕 As shown in FIG. 1, the game device 10 includes a network interface 11, a graphic processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, the graphic processing unit 12, the audio processing unit 13, the operation processing unit 14, and the storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0040] The network interface 11 is communicably connected to a communication network (not shown), for example, to transmit and receive various data to and from other game devices 10 and external server devices (not shown).

[0041] The graphic processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wirelessly. The graphic processing unit 12 draws a game image including various objects in the virtual space in video format according to the game image information (image data) output from the control unit 16. The game image (image of the virtual space) drawn in video format is displayed on the display 21 as a game screen.

[0042] The audio processing unit 13 is connected to a plurality of speakers 30 either wired or wirelessly. The audio processing unit 13 reproduces digital game sounds according to the instructions of the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the plurality of speakers 30. In this example, the audio processing unit 13 is configured to be able to output multi-channel audio in order to perform three-dimensional sound reproduction.

[0043] The operation processing unit 14 is connected to each of the head-mounted display 20 and the controller 40 either wired or wirelessly. The operation processing unit 14 transmits and receives signals to and from each of the head-mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits the operation signal output from the motion sensor 22 of the head-mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to the operation input to the operation element 41), and the motion signal output from the motion sensor 42 of the controller 40 to the control unit 16.

[0044] The storage unit 15 stores various kinds of information and data. The storage unit 15 is composed of an HDD, an SSD, a RAM, a ROM, etc. For example, the storage unit 15 stores game data, various programs including game programs, etc. Examples of game data include game media, user account information, etc.

[0045] In addition, the storage unit 15 stores various audio data indicating various pre-prepared sounds. The various audio data includes audio data indicating sounds output from sound source objects, audio data indicating music (BGM) reproduced in the virtual space, audio data indicating sound effects, etc.

[0046] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various kinds of information and data and processes various kinds of information and data. The control unit 16 has a CPU (microcomputer) and a semiconductor memory. Programs and data for operating the CPU are stored in the semiconductor memory.

[0047] 〔Functional Configuration of Control Unit〕 The control unit 16 of the game device 10 includes a game progress unit 100, a detection unit 101, a display processing unit 102, and an audio processing unit 103. Specifically, by executing a game program, the control unit 16 functions as the game progress unit 100, the detection unit 101, the display processing unit 102, and the audio processing unit 103.

[0048] In other words, the control unit 16 performs game progress processing (virtual space control processing), detection processing, display processing, and audio processing. The detection unit 101 performs detection processing. The game progress processing is the processing by the game progress unit 100. The detection processing is the processing by the detection unit 101. The display processing is the processing by the display processing unit 102. The audio processing is the processing by the audio processing unit 103.

[0049] 〈Game Progress Unit (Virtual Space Control Unit)〉 The game progress unit 100 progresses the game that is carried out in the virtual space. The game progress unit 100 is an example of a virtual space control unit that controls the virtual space. The game progress processing is an example of virtual space control processing (processing by the virtual space control unit).

[0050] In this example, based on the movement indicated by the movement signal output from the motion sensor 22 (the movement of the player's head), the movement indicated by the movement signal output from the motion sensor 42 (the movement of the player's hand), and the operation indicated by the operation signal output from the controller 40 (the operation input to the operator 41 of the controller 40), the game progress unit 100 controls (generates or updates) the three-dimensional data indicating the virtual space so that these movements and operations are reflected in the virtual space, and thereby progresses (controls) the game carried out in the virtual space.

[0051] The three-dimensional data indicating the virtual space includes well-known data for constructing the virtual space, such as data regarding the coordinate system (e.g., world coordinate system) serving as the reference for directions within the virtual space, data regarding the coordinate systems (e.g., local coordinate systems) of various objects arranged within the virtual space, and data regarding the shapes of various objects. By controlling the three-dimensional data indicating the virtual space, the movement of objects within the virtual space can be controlled.

[0052] For example, the game progress unit 100 controls the movement of the player character within the virtual space according to the above-mentioned movement and operations. Also, the game progress unit 100 controls the movement of other objects (such as non-player characters) within the virtual space according to the progress of the game. For example, AI (artificial intelligence) is utilized for the control of the movement of other objects by the game progress unit 100.

[0053] Specifically, the game progress unit 100 controls the virtual space so that a virtual part (a predetermined part of the player character) moves within the virtual space according to the "movement of a predetermined part (head, right hand, left hand) of the player's body" indicated by the movement signal output from the motion sensor (motion sensor 22 or motion sensor 42). For example, the game progress unit 100 controls the virtual space so that the virtual head (the head of the player character) moves within the virtual space according to the "movement of the player's head" indicated by the movement signal output from the motion sensor 22. By moving the virtual head, the direction of the player character can be changed.

[0054] <Detection Unit> The detection unit 101 detects the orientation of the player based on the signal output from the motion sensor 22. The orientation of the player is the direction in which the front (front face) of the player faces. In this example, the orientation of the player is the vector Vd extending in the normal direction of the display surface of the display 21. For example, the detection unit 101 derives the position and orientation of the display 21 included in the head-mounted display 20 based on the motion signals (acceleration and angular velocity) output from the motion sensor 22 included in the head-mounted display 20, and derives the vector Vd based on the position and orientation of the display 21.

[0055] 〈Display processing unit〉 The display processing unit 102 generates an image of the virtual space (game image in this example) controlled by the game progress unit 100. Specifically, the display processing unit 102 generates image data showing an image of the virtual space corresponding to the vector Vd based on the three-dimensional data (three-dimensional data indicating the virtual space) controlled by the game progress unit 100 and the vector Vd (vector Vd indicating the orientation of the player) detected by the detection unit 101. The process of generating an image corresponding to the vector Vd from the three-dimensional data is a well-known process (such as coordinate transformation and projection transformation).

[0056] Then, the display processing unit 102 reproduces (displays) the image generated as described above on the display 21. In this example, the display processing unit 102 supplies the image data generated as described above to the graphic processing unit 12. As a result, the image of the virtual space shown in the image data is reproduced (displayed) on the display 21 of the head-mounted display 20.

[0057] As shown in FIG. 5, a virtual camera 61 is arranged in the virtual space. The virtual camera 61 is arranged in the vicinity of the player character C1. The virtual camera 61 is a virtual viewer (viewpoint), and is, for example, a reference point for reproducing the viewpoint (first-person viewpoint) of the player character. The display processing unit 102 changes the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101. Further, the display processing unit 102 moves the virtual camera 61 so as to follow the movement of the player character C1. Then, the display processing unit 102 causes the display 21 to display an image of the virtual space obtained by the virtual camera 61.

[0058] Note that the process of deriving the orientation of the virtual camera 61 from the orientation of the player (for example, vector Vd) is a well-known process (such as coordinate conversion). The movement of the player character C1 is performed according to the movement or operation of the player (the operation input to the controller 40).

[0059] Specifically, the display processing unit 102 generates image data indicating the above image according to the shooting situation of the virtual camera 61 (for example, the position and orientation of the virtual camera 61 with respect to the object in the virtual space, the orientation of the virtual camera 61, the orientation of the object, etc.) so that the image displayed on the display 21 is an image including the image obtained by the virtual camera 61. The image (image data) obtained by the virtual camera 61 is the image (image data) generated according to the shooting situation of the virtual camera 61.

[0060] <Acoustic Processing Unit> The audio processing unit 103 generates the sound (game sound in this example) in the virtual space controlled by the game progress unit 100. The sound in the virtual space is the sound corresponding to the situation (such as the movement of objects) in the virtual space. Specifically, the audio processing unit 103 selects the audio data to be played from the audio data (various types of pre-prepared audio data) stored in the storage unit 15 according to the situation in the virtual space (the progress situation of the game in this example), and based on the selected audio data (or new audio data obtained by synthesizing the selected multiple audio data), generates audio data indicating the sound corresponding to the situation in the virtual space.

[0061] Then, the audio processing unit 103 causes the sound generated as described above to be reproduced (output) by a plurality of speakers 30. In this example, the audio processing unit 103 supplies the audio data generated as described above to the audio processing unit 13. As a result, the sound in the virtual space indicated by the audio data is reproduced (output) from the plurality of speakers 30.

[0062] 《Sound field maintenance processing》 The audio processing unit 103 performs sound field maintenance processing. In the sound field maintenance processing, the audio processing unit 103 causes the sound in the virtual space to be output to the plurality of speakers 30 so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 does not change even when the direction of the virtual camera 61 changes. The sound field maintenance processing is included in the audio processing.

[0063] 《Sound image localization processing》 In this example, the audio processing unit 103 performs sound image localization processing. The sound image localization processing is the processing of localizing the position of the sound source felt by a human (player) to a predetermined virtual position (for example, the position of the sound source object) in the virtual space. The sound image localization processing is included in the audio processing.

[0064] As shown in FIG. 5, a virtual microphone 62 is arranged in the virtual space. The virtual microphone 62 is arranged near the player character C1 together with the virtual camera 61. The virtual microphone 62 is a virtual listener (sound collection point) and is a reference point for sound image localization in the sound image localization process. The acoustic processing unit 103 moves the virtual microphone 62 so as to follow the movement of the player character C1 and the virtual camera 61. Then, the acoustic processing unit 103 reproduces (outputs) the sound obtained by the virtual microphone 62 to a plurality of speakers 30.

[0065] Specifically, in the sound image localization process, the acoustic processing unit 103 generates audio data indicating the above-mentioned sound so that the sound reproduced (output) from the plurality of speakers 30 becomes a sound including the sound obtained by the virtual microphone 62, according to the sound collection situation of the virtual microphone 62 (for example, the position and direction of the virtual microphone 62 with respect to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, the orientation of the virtual microphone 62, the orientation of the sound source object, etc.). The sound (audio data) obtained by the virtual microphone 62 is sound (audio data) generated according to the sound collection situation of the virtual microphone 62. The sound output from the sound source object is sound (pre-prepared sound) associated with the sound source object.

[0066] By processing the audio data generated by the sound image localization process by the audio processing unit 13, it becomes possible to output from the plurality of speakers 30 the sound output from the virtual sound source (sound source object) in the virtual space in an acoustic expression as if it were collected by the virtual microphone 62.

[0067] And in this example, the acoustic processing unit 103 performs a sound field maintenance process in the sound image localization process. The sound field maintenance process is a process of maintaining (fixing) the direction of the virtual microphone 62 in the sound image localization process to the virtual reference direction. Specifically, the acoustic processing unit 103 moves the virtual microphone 62 arranged in the virtual space so as to follow the movement of the virtual camera 61, and outputs the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30. Note that the direction of the virtual microphone 62 is maintained (fixed) to a predetermined virtual reference direction. In other words, the acoustic processing unit 103 moves the virtual microphone 62 so as to follow the movement of the virtual camera 61 while maintaining the direction of the virtual microphone 62 to the virtual reference direction.

[0068] The virtual reference direction is a direction that serves as a reference for the direction of the virtual microphone 62 in the virtual space and does not change in accordance with the change in the direction of the player detected by the detection unit 101. The virtual reference direction will be described in detail later.

[0069] 〔Various Directions〕 As shown in FIG. 4, the directions (front, rear, left, and right) of the plurality of speakers 30 are defined based on a reference point Q surrounded by the plurality of speakers 30. For example, the "front" of the plurality of speakers 30 corresponds to the front of the reference point Q, and the speaker 30 located in the "front" among the plurality of speakers 30 is the "front center speaker 50" arranged in front of the reference point Q.

[0070] The direction of the player P1 is the direction in which the front of the player P1 faces. In the example of FIG. 4, the direction of the player P1 is the direction facing the "front" of the plurality of speakers 30.

[0071] As shown in FIG. 5, the directions (front, rear, left, right, up, and down) in the virtual space are predetermined. For example, the "front, rear, left, and right" in the virtual space correspond to the "north, south, west, and east" predetermined in the virtual space, and the "up and down" in the virtual space correspond to the "heaven and earth" predetermined in the virtual space. The directions (front, rear, left, right, up, and down) in the virtual space are expressed in a three-dimensional coordinate system (for example, a world coordinate system).

[0072] The orientation of the virtual camera 61 is the direction in which the front of the virtual camera 61 faces. In this example, the orientation of the virtual camera 61 is the orientation (specifically, the vector Vd) of the player derived based on the signal output from the motion sensor 42 of the head-mounted display 20, which is the corresponding orientation.

[0073] The orientation of the virtual microphone 62 is the direction in which the front of the virtual microphone 62 faces. The directions (front, back, left, right) of the virtual microphone 62 correspond to the directions (front, back, left, right) of the plurality of speakers 30. For example, the sound (sound in the virtual space) output toward the virtual microphone 62 from the "front" of the virtual microphone 62 is output from the front center speaker 50 (the speaker 30 located in the "front") corresponding to the "front" of the plurality of speakers 30.

[0074] In the embodiment, the orientation of the virtual microphone 62 is maintained (fixed) in the virtual reference direction. The virtual reference direction (the direction serving as a reference for the orientation of the virtual microphone 62 in the virtual space) is set to the orientation of the virtual microphone 62 when the relationship between the direction of the virtual microphone 62 and the direction in the virtual space is a predetermined relationship. Specifically, the virtual reference direction is set to the orientation of the virtual microphone 62 (the direction in which the front of the virtual microphone 62 faces) when the direction (front, back, left, right) of the virtual microphone 62 and the direction (front, back, left, right) in the virtual space match.

[0075] In the example of FIG. 5, the virtual reference direction is the direction facing "front (north)" in the virtual space. When the front of the virtual microphone 62 faces "front (north)" in the virtual space, the direction (front, back, left, right) of the virtual microphone 62 and the direction (front, back, left, right) in the virtual space match.

[0076] 〔Details of Processing by the Control Unit〕 Next, with reference to FIG. 6, the processing by the control unit 16 will be described in detail. Hereinafter, the case where the player P1 is facing the "front" of the speaker 30 will be described as an example.

[0077] When player P1 faces "forward" among the plurality of speakers 30, player character C1 faces "forward (north)" in the virtual space. Similarly, virtual camera 61 faces "forward (north)" in the virtual space. As a result, an image including enemy character C2 located "forward (north)" of player character C1 in the virtual space is displayed on display 21 of head-mounted display 20 worn by player P1. Enemy character C2 is an example of a sound source object.

[0078] Also, virtual microphone 62 is maintained in a state of facing "forward (north)" in the virtual space. As a result, character voice A2 output from enemy character C2 located "forward (north)" of player character C1 in the virtual space is output from front center speaker 50 arranged in "front" of player P1.

[0079] Through the above processing, an image in the virtual space corresponding to the orientation of player P1 is displayed on display 21 of head-mounted display 20. Also, the direction of the sound heading towards virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) coincides with the direction of the sound heading from the plurality of speakers 30 towards player P1 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on display 21 is output from the plurality of speakers 30 with appropriate sound localization.

[0080] 〔Comparison between the Embodiment and the Comparative Example〕 Next, the embodiment and the comparative example will be described in comparison. Hereinafter, for convenience of explanation, in the description of the components in the comparative example, the reference numerals of the components in the embodiment are used. Also, the comparative example in the following description is different from the embodiment in that the orientation of virtual microphone 62 is changed so as to follow the orientation of virtual camera 61. Other processing in the comparative example is the same as the processing in the embodiment. Note that in a conventional game system (for example, a VR system), it is common to change the orientation of virtual microphone 62 so as to follow the orientation of virtual camera 61.

[0081] As shown in FIG. 7, in the comparative example, when player P1 faces "right" among the plurality of speakers 30, player character C1 faces "right (east)" in the virtual space. Similarly, virtual camera 61 faces "right (east)" in the virtual space. As a result, an image of "right (east)" of player character C1 is displayed on display 21 of head-mounted display 20 worn by player P1 in the virtual space. Enemy character C2 is located "left (north)" of player character C1.

[0082] Also, virtual microphone 62 faces "right (east)" in the virtual space, similar to virtual camera 61. Therefore, character voice A2 output from enemy character C2 located "left (north)" of player character C1 in the virtual space is output from surround left speaker 54 arranged "behind" player P1.

[0083] Thus, in the comparative example, the direction of the sound toward virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) does not match the direction of the sound from the plurality of speakers 30 toward player P1 (the direction of the sound field in the real world). Therefore, the sound in the virtual space corresponding to the image in the virtual space displayed on display 21 cannot be output from the plurality of speakers 30 with appropriate localization. In the example of FIG. 7, character voice A2 should originally be output from "left" of player P1 toward player P1, but is output from "behind" player P1 toward player P1.

[0084] On the other hand, as shown in FIG. 8, in the embodiment, virtual microphone 62 is maintained in a state of facing "front (north)" in the virtual space. Therefore, character voice A2 output from enemy character C2 located "left (north)" of player character C1 in the virtual space is output from front center speaker 50 arranged "left" of player P1.

[0085] In this way, by maintaining the orientation of the virtual microphone 62 in the virtual reference direction (in the example of FIG. 8, "front (north)"), it is possible to maintain a state where the direction of the sound heading towards the virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) coincides with the direction of the sound heading from the plurality of speakers 30 towards the player P1 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the plurality of speakers 30 with appropriate sound localization.

[0086] 〔Summary of Embodiment〕 In summary, the program (game program) of the embodiment is a program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit 101, a display processing unit 102, and an acoustic processing unit 103. The computer system is used together with a display 21 that follows the orientation of the user (player), a plurality of speakers 30 arranged around the user, and a sensor (motion sensor 22) that outputs a signal indicating the motion of the user including a change in the orientation of the user. The detection unit 101 detects the orientation of the user based on the signal output from the sensor. The display processing unit 102 changes the orientation of the virtual camera 61 arranged in the virtual space so as to follow the orientation of the user detected by the detection unit 101, and causes the display 21 to display an image of the virtual space obtained by the virtual camera 61. The acoustic processing unit 103 causes the plurality of speakers 30 to output the sound in the virtual space so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 does not change even when the orientation of the virtual camera 61 changes.

[0087] The acoustic control device (game device 10) of the embodiment includes a storage unit 15 that stores the above program and a control unit 16 that executes the above program.

[0088] The system of the embodiment (game system 1) includes a display 21 that follows the orientation of the user (player), a plurality of speakers 30 arranged around the user, a sensor (motion sensor 22) that outputs a signal indicating the motion of the user including a change in the orientation of the user, and a control unit 16. The control unit 16 performs detection processing, display processing, and acoustic processing. In the detection processing, the control unit (detection unit 101) detects the orientation of the user based on the signal output from the sensor. In the display processing, the control unit (display processing unit 102) changes the orientation of a virtual camera 61 arranged in the virtual space so as to follow the orientation of the user detected by the detection processing, and displays an image of the virtual space obtained by the virtual camera 61 on the display 21. In the acoustic processing, the control unit (acoustic processing unit 103) outputs the sound in the virtual space to the plurality of speakers 30 so that the orientation of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 does not change even when the orientation of the virtual camera 61 changes.

[0089] 〔Effects of the Embodiment〕 As described above, in the embodiment, the sound in the virtual space is output to the plurality of speakers 30 so that the orientation of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 (hereinafter referred to as "the orientation of the sound field in the real world") does not change even when the orientation of the virtual camera 61 changes.

[0090] According to the above control, it is possible to prevent the orientation of the sound field in the real world from changing due to a change in the orientation of the player P1. Therefore, the "orientation of the sound field in the real world" can be prevented from deviating from the "orientation of the sound field in the virtual space (the orientation of the sound field formed by the sound output from the sound source object in the virtual space)". As a result, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 can be output from the plurality of speakers 30 with appropriate localization, and the user experience can be improved. For example, the reality in the way the sound in the virtual space is heard can be improved, so the interestingness can be improved.

[0091] (Modification of the Embodiment) In the game system 1 of the modification of the embodiment, the processing by the display processing unit 102 and the audio processing unit 103 is different from that of the game system 1 of the embodiment. Other configurations and processes of the game system 1 of the modification of the embodiment are the same as those of the game system 1 according to the embodiment.

[0092] In the modification of the embodiment, a direction change operation and a reset operation are input to the controller 40. The direction change operation is an operation for changing the direction of the virtual camera 61 from the front direction corresponding to the direction of the player detected by the detection unit 101 to another direction. In the direction change operation, a direction different from the front direction (for example, the direction for a third-person view) is specified. The reset operation is an operation for resetting the direction of the virtual camera 61 to the front direction.

[0093] The front direction is a direction that coincides with the direction of the player character C1 corresponding to the direction of the player detected by the detection unit 101. For example, when the player P1 faces the "front" among the plurality of speakers 30, the player character C1 faces the "front (north)" in the virtual space, and the front direction is the "front (north)" in the virtual space.

[0094] When a direction change operation is input to the controller 40, the display processing unit 102 changes the direction of the virtual camera 61 from the front direction according to the direction change operation. When the direction of the virtual camera 61 changes from the front direction according to the direction change operation, the audio processing unit 103 changes the direction of the virtual microphone 62 so as to follow the direction of the virtual camera 61.

[0095] When a reset operation is input to the controller 40, the display processing unit 102 resets the direction of the virtual camera 61 to the front direction according to the reset operation. When the direction of the virtual camera 61 is reset to the front direction according to the reset operation, the audio processing unit 103 resets the direction of the virtual microphone 62 to the virtual reference direction.

[0096] In this example, the display processing unit 102 selectively performs a first display process and a second display process. The first display process is a process of changing the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101, and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21. The second display process is a process of setting the orientation of the virtual camera 61 to a direction different from the front direction (the direction specified by the direction change operation), and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21.

[0097] When a direction change operation is input to the controller 40, the display processing unit 102 ends the first display process and starts the second display process. Also, when a reset operation is input to the controller 40, the display processing unit 102 ends the second display process and starts the first display process.

[0098] Also, in this example, the acoustic processing unit 103 selectively performs a first audio localization process and a second audio localization process. The first audio localization process is a process of moving the virtual microphone 62 so as to follow the movement of the virtual camera 61 while maintaining the orientation of the virtual microphone 62 in the virtual reference direction, and outputting the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30. The second audio localization process is a process of changing the orientation of the virtual microphone 62 so as to follow the orientation of the virtual camera 61 and moving the virtual microphone 62 so as to follow the movement of the virtual camera 61, and outputting the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30.

[0099] When the orientation of the virtual camera 61 changes from the front direction in response to a direction change operation, the acoustic processing unit 103 ends the first audio localization process and starts the second audio localization process. Also, when the orientation of the virtual camera 61 is reset to the front direction in response to a reset operation, the acoustic processing unit 103 ends the second audio localization process and starts the first audio localization process.

[0100] 〔Details of Processing by the Control Unit〕 Next, with reference to FIG. 9, the processing by the control unit 16 in the modification of the embodiment will be described in detail. Hereinafter, the case where the player P1 is facing "forward" in the speaker 30 will be described as an example. In this example, when the direction of the virtual camera 61 changes from the front direction to another direction in response to a direction change operation, the image displayed on the display 21 changes from the "first-person perspective image" to the "third-person perspective image".

[0101] When a direction change operation for changing the direction of the virtual camera 61 from "forward (north)" (front direction) in the virtual space to "left (west)" in the virtual space is input to the controller 40 when the player P1 faces "forward" in a plurality of speakers 30, the virtual camera 61 faces "left (west)" in the virtual space. As a result, an image of the player character C1 as seen from the "right (east)" in the virtual space is displayed on the display 21 of the head-mounted display 20 worn by the player P1.

[0102] The left-right direction in the image displayed on the display 21 corresponds to the front-back direction (north-south direction) in the virtual space. In the image displayed on the display 21, the enemy character C2 is located "right" of the player character C1. In the example of FIG. 9, the image displayed on the display 21 is a third-person perspective image including the player character C1 and the enemy character C2. The display processing unit 102 changes the direction of the virtual camera 61 and moves the virtual camera 61 so that a third-person perspective image can be obtained by the virtual camera 61.

[0103] Also, the direction of the virtual microphone 62 changes from "forward (north)" (front direction) in the virtual space to "left (west)" in the virtual space so as to follow the direction of the virtual camera 61. As a result, the character voice A2 output from the enemy character C2 located on the right of the player character C1 in the image displayed on the display 21 is output from the surround light speaker 53 arranged on the "right" of the player P1.

[0104] Through the above processing, an image (third-person perspective image) in the virtual space obtained by the virtual camera 61 facing a direction different from the front direction corresponding to the orientation of the player P1 is displayed on the display 21 of the head-mounted display 20. Also, in the image displayed on the display 21, the direction of the sound toward the player character C1 (the direction of the sound field in the virtual space) coincides with the direction of the sound from the plurality of speakers 30 toward the player P1 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the plurality of speakers 30 with appropriate localization.

[0105] 〔Effect of the Modification of the Embodiment〕 In the modification of the embodiment, the same effects as those of the embodiment can be obtained.

[0106] Also, in the modification of the embodiment, even when a direction change operation for changing the direction of the virtual camera 61 from the front direction corresponding to the orientation of the player detected by the detection unit 101 to another direction is input to the controller 40, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 can be output from the plurality of speakers 30 with appropriate localization, and the user experience can be improved.

[0107] (Other Embodiments) In the above description, it may be configured or set as follows.

[0108] The voice data does not necessarily have to be prepared in advance. For example, the acoustic processing unit 103 may synthesize or generate voice data each time based on some data (for example, voice data stored in the storage unit 15) or information.

[0109] The player character may or may not be displayed on the display. In other words, in the image of the virtual space displayed (played) on the display, the player character may or may not be included. For example, when the image of the virtual space is displayed on the display from the first-person perspective, only the virtual hand (virtual part), which is a part of the player character, may be displayed, or only the handgun (virtual object) held by the virtual hand without the virtual hand being displayed may be displayed.

[0110] The image of the virtual space may be an image from the first-person perspective or an image from the third-person perspective (for example, an image when looking at the player character from behind the player character).

[0111] The plurality of speakers 30 may be arranged on the left and right, front and back, above, etc. of the listener (player) so as to be able to reproduce three-dimensional sounds.

[0112] The sensor that outputs a signal indicating the movement of the player including the change in the direction of the player is not limited to the motion sensor 22 included in the head-mounted display 20. For example, such a sensor may be a combination of a camera (image sensor) that images a predetermined part (for example, the head) on the player's body and a recognition processing unit that recognizes the movement of the player including the change in the direction of the player based on the "image of the predetermined part" obtained by the camera.

[0113] The game program may be implemented as a game program for a so-called online game. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in the control unit of the server device (not shown) or may be shared between the control unit of the game device 10 and the control unit of the server device. The game device 10 may be a portable information terminal such as a smartphone or a tablet.

[0114] In the above description, as an example of the virtual space, a virtual space (game space) in which a game progresses is given, but it is not limited to this. For example, the virtual space may be a virtual space (metaverse space) in which avatars communicate with each other. In other words, the service provided in the virtual space is not limited to games. For example, the service provided in the virtual space may be the provision of a business transaction venue, the provision of a communication venue, or the like.

[0115] Even when these other embodiments are adopted, the effects of the present invention are exhibited. Also, it is possible to appropriately combine this embodiment with other embodiments and between other embodiments. The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses.

Explanation of Reference Numerals

[0116] 1 Game system (system) 10 Game device (acoustic control device) 11 Network interface 12 Graphics processing unit 13 Audio processing unit 14 Operation processing unit 20 Head-mounted display 21 Display 22 Motion sensor 25 Main body unit 26 Headband 30 Speaker 40 Controller 40a First controller 40b Second controller 41 Operator 42 Motion sensor (sensor) 50 Front center speaker 51 Front right speaker 52 Front left speaker 53 Surround right speaker 54 Surround left speaker 55 Surround Backlight Speaker 56 Surround Back Left Speaker 61 Virtual Camera 62 Virtual Microphone 100 Game Progress Section (Virtual Space Control Section) 101 Detection Section 102 Display Processing Section 103 Acoustic Processing Section P1 Player (User) C1 Player Character (User Character) C2 Enemy Character (Sound Source Object) A2 Character Voice (Voice in Virtual Space)

Claims

1. A program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the orientation of the user, a plurality of speakers arranged around the user, and a sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user, The detection unit detects the orientation of the user based on the signal output from the sensor, The display processing unit changes the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection unit, and displays an image of the virtual space obtained by the virtual camera on the display, The acoustic processing unit causes the plurality of speakers to output sound in the virtual space so that the orientation of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even when the orientation of the virtual camera changes. Program.

2. In the program of Claim 1, The acoustic processing unit moves a virtual microphone arranged in the virtual space so as to follow the movement of the virtual camera, and outputs the sound in the virtual space obtained by the virtual microphone to the plurality of speakers, The orientation of the virtual microphone is a reference direction for the orientation of the virtual microphone in the virtual space, and is maintained in a virtual reference direction that does not change in accordance with a change in the orientation of the user detected by the detection unit. Program.

3. In the program of Claim 2, The computer system is used together with a controller operated by the user, When a direction change operation for changing the orientation of the virtual camera from the front direction corresponding to the orientation of the user detected by the detection unit is input to the controller, the display processing unit changes the orientation of the virtual camera from the front direction according to the direction change operation. When the orientation of the virtual camera changes from the front direction according to the direction change operation, the acoustic processing unit changes the orientation of the virtual microphone so as to follow the orientation of the virtual camera. Program.

4. In the program of claim 3, When a reset operation for resetting the orientation of the virtual camera to the front direction is input to the controller, the display processing unit resets the orientation of the virtual camera to the front direction according to the reset operation. When the orientation of the virtual camera is reset to the front direction according to the reset operation, the acoustic processing unit resets the orientation of the virtual microphone to the virtual reference direction. Program.

5. A storage unit that stores any one of the programs of claims 1 to 4, And a control unit that executes the program. Acoustic control device.

6. A display that follows the orientation of the user, A plurality of speakers arranged around the user, A sensor that outputs a signal indicating the movement of the user including a change in the orientation of the user, And a control unit, The control unit, A detection process for detecting the orientation of the user based on the signal output from the sensor, A display process for changing the orientation of a virtual camera arranged in a virtual space so as to follow the orientation of the user detected by the detection process and displaying an image of the virtual space obtained by the virtual camera on the display. Perform acoustic processing to output the sound in the virtual space to the plurality of speakers so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even when the direction of the virtual camera changes. System.

Images