Programs, sound control devices, systems

Abstract

To enhance user experience.SOLUTION: A detection unit detects the orientation of a user P1 on the basis of signals output from a sensor. A display processing unit changes the orientation of a virtual camera 61 arranged in a virtual space so as to follow the orientation of the user P1 detected by the detection unit, and causes a display to display the image of the virtual space obtained by the virtual camera 61. An audio processing unit causes a plurality of speakers 30 to output audio in the virtual space, in such a manner that the orientation of a sound field formed by the audio in the virtual space output from the speakers 30 does not change even if the orientation of the virtual camera 61 changes.SELECTED DRAWING: Figure 8

Description

Technical Field

[0006] , ,

[0001] This disclosure relates to a program, an acoustic control device, and a system.

Background Art

[0002] Some game programs play voices in a virtual space. For example, in Patent Document 1, voices such as gunshots are played in a shooting game.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to provide a user with images and voices in a virtual space in which various services (such as games) are provided using a system including a display (such as a head-mounted display) that follows the user's orientation and a plurality of speakers arranged around the user. In such a system, from the perspective of the user experience, it is desirable that voices in the virtual space corresponding to the images in the virtual space displayed on the display be reproduced from the plurality of speakers with appropriate localization.

[0005] An object of this disclosure is to improve the user experience in services provided in a virtual space.

Means for Solving the Problems

[0006] A first aspect is a program that causes a computer system having one or a plurality of cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the user's orientation, a plurality of speakers arranged around the user, and sensors that output signals indicating the user's movement, including changes in the user's orientation. The detection unit detects the user's orientation based on the signal output from the sensor. The display processing unit changes the orientation of a virtual camera placed in the virtual space to follow the orientation of the user detected by the detection unit, and displays the image of the virtual space obtained by the virtual camera on the display. The sound processing unit causes the plurality of speakers to output sound from the virtual space so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even if the orientation of the virtual camera changes. It is a program.

[0007] In the first embodiment, The sound processing unit may move a virtual microphone placed in the virtual space to follow the movement of the virtual camera, and output the sound from the virtual space obtained by the virtual microphone to the plurality of speakers. The orientation of the virtual microphone is a reference direction for the orientation of the virtual microphone within the virtual space, and may be maintained in a virtual reference direction that does not change in response to changes in the user's orientation detected by the detection unit.

[0008] In the first embodiment, The computer system may be used in conjunction with a controller operated by the user. When a direction change operation is input to the controller for changing the orientation of the virtual camera from the front direction corresponding to the user's orientation detected by the detection unit, the display processing unit may change the orientation of the virtual camera from the front direction in response to the direction change operation. The sound processing unit may change the orientation of the virtual microphone to follow the orientation of the virtual camera when the orientation of the virtual camera changes from the front direction in response to the orientation change operation.

[0009] In the first embodiment, When a reset operation for resetting the orientation of the virtual camera to the front direction is input to the controller, the display processing unit may reset the orientation of the virtual camera to the front direction in response to the reset operation. The sound processing unit may reset the orientation of the virtual microphone to the virtual reference direction when the orientation of the virtual camera is reset to the front direction in response to the reset operation.

[0010] The second aspect includes a storage unit that stores the program of the first aspect, The system comprises a control unit that executes the aforementioned program. It is an acoustic control device.

[0011] The third aspect is, A display that follows the user's orientation, Multiple speakers arranged around the user, A sensor that outputs a signal indicating the user's movement, including a change in the user's orientation, It includes a control unit, The control unit, A detection process that detects the orientation of the user based on the signal output from the sensor, A display process that changes the orientation of a virtual camera placed in the virtual space to follow the orientation of the user detected by the detection process, and displays the image of the virtual space obtained by the virtual camera on the display, The system performs acoustic processing to cause the speakers to output sound from the virtual space so that the direction of the sound field formed by the sound in the virtual space output from the speakers does not change even if the orientation of the virtual camera changes. It is a system. [Effects of the Invention]

[0012] According to the present disclosure, the user experience can be improved.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram illustrating the configuration of the game system of the embodiment. [Figure 2] It is a diagram illustrating the appearance of the head-mounted display. [Figure 3] It is a schematic diagram illustrating a player wearing a head-mounted display and a controller. [Figure 4] It is a schematic diagram illustrating the directions in a plurality of speakers and the orientation of the player. [Figure 5] It is a schematic diagram illustrating the directions in the virtual space, the orientation of the player character, the orientation of the virtual camera, and the orientation of the virtual microphone. [[ID=XX]] [Figure 6] It is a schematic diagram illustrating the orientation of the virtual camera, the orientation of the virtual microphone, and the direction of the speaker sound when the player is facing forward in the embodiment. [Figure 7] It is a schematic diagram illustrating the orientation of the virtual camera, the orientation of the virtual microphone, and the direction of the speaker sound when the player is facing right in the comparative example. [Figure 8] It is a schematic diagram illustrating the orientation of the virtual camera, the orientation of the virtual microphone, and the direction of the speaker sound when the player is facing right in the embodiment. [Figure 9] It is a schematic diagram illustrating the orientation of the virtual camera, the orientation of the virtual microphone, the direction of the speaker sound, and the image displayed on the display after the input of the direction change operation. )

Modes for Carrying Out the Invention

[0014] The embodiments will be described in detail below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and their descriptions will not be repeated. Furthermore, the following describes an example in which the program is implemented as a game program, the sound control device is implemented as a game device, and the system is implemented as a game system. The game device is an example of a computer system having one computer or multiple cooperating computers.

[0015] (Game description) In the following description, "game" refers to a game played in a virtual space (video game). The virtual space is a three-dimensional game space, displayed as an image on a screen. Objects are placed within the virtual space. Examples of objects include player characters controlled by the player (the user playing the game) and non-player characters controlled by the computer. A player is an example of a user.

[0016] Furthermore, objects include sound source objects, which act as virtual sound sources within the virtual space. Examples of sound source objects include objects that operate within the virtual space and objects that only output sound within the virtual space. Player characters and non-player characters can be sound source objects. In the game described below, sound is played. The sounds played in the game include sounds output from sound source objects, BGM (Background Music), and sound effects.

[0017] Furthermore, in the game described below, the player character is a virtual player (a virtual human) placed in a virtual space, and moves within the virtual space in accordance with the player's actions and movements in the real world. Specifically, virtual body parts move within the virtual space in accordance with the movement of predetermined body parts of the player. These virtual body parts are virtual parts (predetermined body parts of the player character) that correspond to predetermined body parts of the player. For example, the virtual head of the player character moves within the virtual space in accordance with the movement of the player's head in the real world.

[0018] For the sake of explanation, the following example of a game will be a competitive action game in which the player character attacks and defeats enemy characters. There are no limitations on the type of game. Furthermore, the example of a virtual space will be a virtual reality space where one can experience virtual reality (VR). Additionally, an example will be given where the game progresses with images of the virtual space displayed on a screen from a first-person perspective.

[0019] (Game System) Figure 1 illustrates the configuration of the game system 1 of the embodiment. The game system 1 comprises a game device 10, a head-mounted display 20, a plurality of speakers 30, and a controller 40.

[0020] [Game device] The game device 10 executes the game in response to user input. In this example, the game device 10 is used with a head-mounted display 20, multiple speakers 30, and a controller 40, and is connected to these devices by wire or wireless. The game device 10 is an example of an audio control device.

[0021] The game device 10 can utilize commercially available devices such as personal computers, PlayStation®, Xbox®, PlayStation Vita®, and Nintendo Switch®.

[0022] In the game device 10, the game progresses based on the installed game program and game data. The game devices 10 can communicate with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0023] [Head-mounted display] The head-mounted display 20 includes a display 21 and a motion sensor 22.

[0024] Display 21 plays (displays) images. For example, display 21 is a liquid crystal display. Display 21 is an example of a display that follows the orientation of the player.

[0025] The motion sensor 22 outputs motion signals that indicate the player's head movements. For example, the motion sensor 22 includes an accelerometer that detects acceleration, a gyroscope that detects angular velocity (rotation and change of direction), and so on. The motion signals output from the motion sensor 22 are transmitted to the game device 10.

[0026] As shown in Figure 2, the head-mounted display 20 has a main body 25 and a headband 26. As shown in Figure 3, the head-mounted display 20 is attached to the head of player P1 such that the main body 25 covers both of player P1's eyes. A display 21 is fixed to the inside of the main body 25 (the side facing player P1's eyes). The display surface of the display 21 faces both of player P1's eyes when the head-mounted display 20 is attached to player P1's head.

[0027] Furthermore, the head-mounted display 20 attached to the head of player P1 moves in conjunction with the movement of player P1's head. Therefore, the position and orientation of player P1's head and the position and orientation of the display 21 included in the head-mounted display 20 can be derived based on the motion signal output from the motion sensor 22 included in the head-mounted display 20. The motion sensor 22 is an example of a sensor that outputs a signal indicating the movement of player P1, including changes in player P1's orientation.

[0028] Furthermore, vector Vd in the head-mounted display 20 is the reference line for the image of the virtual space represented from a first-person perspective. Vector Vd extends in the direction of the normal to the display surface of the display 21. Vector Vd can be derived from the position and orientation of the display 21 included in the head-mounted display 20.

[0029] [Speaker] Multiple speakers 30 are arranged around the player to play sound (game sounds in this example). In this example, multiple speakers 30 are provided to enable three-dimensional sound reproduction.

[0030] As shown in Figure 4, in this example, the multiple speakers 30 consist of a front center speaker 50, a front right speaker 51, a front left speaker 52, a surround right speaker 53, a surround left speaker 54, a surround back right speaker 55, and a surround back left speaker 56.

[0031] The front center speaker 50 is positioned in front of the reference point Q. The front right speaker 51 is positioned to the right front of the reference point Q. The front left speaker 52 is positioned to the left front of the reference point Q. The surround right speaker 53 is positioned to the right of the reference point Q. The surround left speaker 54 is positioned to the left of the reference point Q. The surround back right speaker 55 is positioned to the right rear of the reference point Q. The surround back left speaker 56 is positioned to the left rear of the reference point Q.

[0032] Reference point Q is a reference point (position) for determining the direction (front, back, left, and right) of the multiple speakers 30. For example, reference point Q is the initial position (reference position) of the player. The direction of the multiple speakers 30 will be explained in detail later.

[0033] 〔controller〕 The controller 40 is operated by being held or attached to the player. In this example, the controller 40 is operated by being held in the player's hand. As shown in Figure 1, the controller 40 includes an operator 41 and an motion sensor 42.

[0034] The control element 41 is composed of various buttons and the like. The controller 40 outputs an operation signal to the game device 10 corresponding to the operation input to the control element 41 (for example, pressing a button). With this configuration, the player inputs an operation signal (a signal indicating a command or data, etc.) to the game device 10 by operating the control element 41 of the controller 40.

[0035] The motion sensor 42 outputs motion signals that indicate the player's hand movements. For example, the motion sensor 22 includes an accelerometer that detects acceleration, a gyroscope that detects angular velocity (rotation and change of direction), and so on. The motion signals output from the motion sensor 42 are transmitted to the game device 10.

[0036] Furthermore, the controller 40, which is held (or attached) to the player's hand, moves in conjunction with the player's hand movements. Therefore, the position and orientation of the player's hand can be derived based on the motion signal output from the motion sensor 42 included in the controller 40.

[0037] As shown in Figure 3, in this example, the controller 40 consists of a first controller 40a and a second controller 40b. The configurations of the first controller 40a and the second controller 40b are the same as those of the controller 40.

[0038] The first controller 40a is held in the right hand of player P1 and operated, and moves in conjunction with the movement of player P1's right hand. The second controller 40b is held in the left hand of player P1 and operated, and moves in conjunction with the movement of player P1's left hand. Therefore, the position and orientation of player P1's right hand can be derived based on the motion signal output from the motion sensor 42 included in the first controller 40a. Similarly, the position and orientation of player P1's left hand can be derived based on the motion signal output from the motion sensor 42 included in the second controller 40b.

[0039] [Hardware configuration of the game device] As shown in Figure 1, the game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, graphics processing unit 12, audio processing unit 13, operation processing unit 14, and storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0040] The network interface 11 is connected to a communication network (not shown) in a communicative manner for sending and receiving various data with, for example, other game devices 10 or external server devices (not shown).

[0041] The graphics processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wireless connection. The graphics processing unit 12 renders game images, including various objects in the virtual space, in video format according to the game image information (image data) output from the control unit 16. The game images (images of the virtual space) rendered in video format are displayed on the display 21 as the game screen.

[0042] The audio processing unit 13 is connected to multiple speakers 30 by wire or wirelessly. The audio processing unit 13 plays digital game sounds according to the instructions of the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the multiple speakers 30. In this example, the audio processing unit 13 is configured to provide multi-channel audio output in order to reproduce three-dimensional sound.

[0043] The operation processing unit 14 is connected to the head-mounted display 20 and the controller 40 by wire or wireless connection. The operation processing unit 14 sends and receives signals to and from the head-mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits the operation signal output from the motion sensor 22 of the head-mounted display 20, the operation signal transmitted from the controller 40 (a signal corresponding to the operation input to the operator 41), and the operation signal output from the motion sensor 42 of the controller 40 to the control unit 16.

[0044] The storage unit 15 stores various types of information and data. The storage unit 15 is composed of HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores various programs, including game data and game programs. Examples of game data include game media and user account information.

[0045] Furthermore, the memory unit 15 stores various audio data that represent various pre-prepared sounds. These various audio data include audio data that represents sounds output from sound source objects, audio data that represents music (BGM) played in the virtual space, and audio data that represents sound effects.

[0046] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[0047] [Functional configuration of the control unit] The control unit 16 of the game device 10 includes a game progress unit 100, a detection unit 101, a display processing unit 102, and an audio processing unit 103. Specifically, the control unit 16 functions as the game progress unit 100, the detection unit 101, the display processing unit 102, and the audio processing unit 103 by executing a game program.

[0048] In other words, the control unit 16 performs game progression processing (virtual space control processing), detection processing, display processing, and sound processing. The detection unit 101 performs detection processing. Game progression processing is performed by the game progression unit 100. Detection processing is performed by the detection unit 101. Display processing is performed by the display processing unit 102. Sound processing is performed by the sound processing unit 103.

[0049] <Game Management Department (Virtual Space Control Department)> The game progress unit 100 manages the game played within the virtual space. The game progress unit 100 is an example of a virtual space control unit that controls the virtual space. The game progress processing is an example of virtual space control processing (processing by the virtual space control unit).

[0050] In this example, the game progression unit 100 controls (generates or updates) three-dimensional data representing the virtual space so that these movements and operations are reflected in the virtual space, based on the movements indicated by the motion signals output from the motion sensor 22 (player's head movements), the movements indicated by the motion signals output from the motion sensor 42 (player's hand movements), and the operations indicated by the operation signals output from the controller 40 (operations input to the controller 41 of the controller 40).

[0051] The three-dimensional data representing a virtual space includes well-known data for constructing that virtual space, such as data on the coordinate system that serves as the reference for direction within the virtual space (e.g., the world coordinate system), data on the coordinate systems of various objects placed within the virtual space (e.g., the local coordinate system), and data on the shapes of various objects. By controlling the three-dimensional data representing the virtual space, the movement of objects within the virtual space can be controlled.

[0052] For example, the game progress unit 100 controls the movement of the player character in the virtual space in accordance with the above movements and operations. The game progress unit 100 also controls the movement of other objects (such as non-player characters) in the virtual space in accordance with the progress of the game. For example, AI (artificial intelligence) is used to control the movement of other objects by the game progress unit 100.

[0053] Specifically, the game progress unit 100 controls the virtual space so that virtual parts (predetermined parts of the player character) move in the virtual space in accordance with the movement of predetermined parts of the player's body (head, right hand, left hand) indicated by the motion signals output from the motion sensors (motion sensors 22 or 42). For example, the game progress unit 100 controls the virtual space so that the virtual head (the player character's head) moves in the virtual space in accordance with the movement of the player's head indicated by the motion signals output from the motion sensors 22. By moving the virtual head, the orientation of the player character can be changed.

[0054] <Detection Unit> The detection unit 101 detects the orientation of the player based on the signal output from the motion sensor 22. The orientation of the player is the direction in which the front of the player faces. In this example, the orientation of the player is the vector Vd extending in the direction of the normal to the display surface of the display 21. For example, the detection unit 101 derives the position and orientation of the display 21 included in the head-mounted display 20 based on the motion signals (acceleration and angular velocity) output from the motion sensor 22 included in the head-mounted display 20, and derives the vector Vd based on the position and orientation of the display 21.

[0055] <Display Processing Section> The display processing unit 102 generates an image of the virtual space (in this example, a game image) controlled by the game progress unit 100. Specifically, the display processing unit 102 generates image data representing the virtual space image corresponding to vector Vd, based on the three-dimensional data (three-dimensional data representing the virtual space) controlled by the game progress unit 100 and the vector Vd (vector Vd indicating the player's orientation) detected by the detection unit 101. The process of generating an image corresponding to vector Vd from three-dimensional data is a well-known process (such as coordinate transformation and projection transformation).

[0056] The display processing unit 102 then plays (displays) the image generated as described above on the display 21. In this example, the display processing unit 102 supplies the image data generated as described above to the graphics processing unit 12. As a result, the image of the virtual space indicated in the image data is played (displayed) on the display 21 of the head-mounted display 20.

[0057] As shown in Figure 5, a virtual camera 61 is placed in the virtual space. The virtual camera 61 is placed near the player character C1. The virtual camera 61 is a virtual viewer (viewpoint), and for example, it is a reference point for reproducing the viewpoint (first-person view) of the player character. The display processing unit 102 changes the orientation of the virtual camera 61 to follow the orientation of the player detected by the detection unit 101. The display processing unit 102 also moves the virtual camera 61 to follow the movement of the player character C1. Then, the display processing unit 102 displays the image of the virtual space obtained by the virtual camera 61 on the display 21.

[0058] The process of deriving the orientation of the virtual camera 61 from the player's orientation (e.g., vector Vd) is a well-known process (such as coordinate transformation). The movement of the player character C1 is performed according to the player's movement or operation (operation input to the controller 40).

[0059] Specifically, the display processing unit 102 generates image data representing the above image according to the shooting conditions of the virtual camera 61 (for example, the position and direction of the virtual camera 61 relative to an object in the virtual space, the orientation of the virtual camera 61, the orientation of the object, etc.) so that the image displayed on the display 21 includes the image obtained by the virtual camera 61. The image obtained by the virtual camera 61 (image data) is an image (image data) generated according to the shooting conditions of the virtual camera 61.

[0060] <Acoustic Processing Section> The sound processing unit 103 generates sounds for the virtual space controlled by the game progress unit 100 (in this example, game sounds). The sounds for the virtual space are sounds that correspond to the situation in the virtual space (such as the movement of objects). Specifically, the sound processing unit 103 selects the sound data to be played from the sound data stored in the memory unit 15 (various pre-prepared sound data) according to the situation in the virtual space (in this example, the game progress), and generates sound data that indicates sounds corresponding to the situation in the virtual space based on the selected sound data (or new sound data obtained by synthesizing multiple selected sound data).

[0061] The sound processing unit 103 then plays (outputs) the generated sound to the multiple speakers 30. In this example, the sound processing unit 103 supplies the generated sound data to the audio processing unit 13. As a result, the sound of the virtual space indicated in the sound data is played (output) from the multiple speakers 30.

[0062] Sound field preservation processing The sound processing unit 103 performs sound field maintenance processing. In sound field maintenance processing, the sound processing unit 103 causes the multiple speakers 30 to output sound from the virtual space so that even if the orientation of the virtual camera 61 changes, the orientation of the sound field formed by the sound in the virtual space output from the multiple speakers 30 does not change. Sound field maintenance processing is included in sound processing.

[0063] Sound localization processing In this example, the sound processing unit 103 performs sound localization processing. Sound localization processing is the process of localizing the position of a sound source as perceived by a human (player) to a predetermined virtual position in the virtual space (for example, the position of the sound source object). Sound localization processing is included in sound processing.

[0064] As shown in Figure 5, a virtual microphone 62 is placed in the virtual space. The virtual microphone 62 is placed near the player character C1, together with the virtual camera 61. The virtual microphone 62 is a virtual listener (sound collection point) and is the reference point for sound localization in sound localization processing. The sound processing unit 103 moves the virtual microphone 62 to follow the movement of the player character C1 and the virtual camera 61. The sound processing unit 103 then plays (outputs) the sound obtained by the virtual microphone 62 to multiple speakers 30.

[0065] Specifically, in sound localization processing, the sound processing unit 103 generates audio data that represents the above-mentioned audio according to the sound pickup conditions of the virtual microphone 62 (for example, the position and direction of the virtual microphone 62 relative to the sound source object in the virtual space, the volume and direction of the audio output from the sound source object, the orientation of the virtual microphone 62, the orientation of the sound source object, etc.) so that the audio played back (output) from the multiple speakers 30 includes the audio obtained by the virtual microphone 62. The audio obtained by the virtual microphone 62 (audio data) is audio (audio data) generated according to the sound pickup conditions of the virtual microphone 62. The audio output from the sound source object is audio associated with the sound source object (pre-prepared audio).

[0066] The audio data generated by the sound image localization process is processed by the audio processing unit 13, making it possible to output audio from multiple speakers 30 in a way that makes it appear as if the audio output from a virtual sound source (sound source object) in the virtual space was collected by a virtual microphone 62.

[0067] In this example, the sound processing unit 103 performs sound field maintenance processing during sound image localization processing. Sound field maintenance processing is the process of maintaining (fixing) the orientation of the virtual microphone 62 in the virtual reference direction during sound image localization processing. Specifically, the sound processing unit 103 moves the virtual microphone 62, which is placed in the virtual space, to follow the movement of the virtual camera 61, and outputs the sound obtained from the virtual microphone 62 in the virtual space to multiple speakers 30. The orientation of the virtual microphone 62 is maintained (fixed) in a predetermined virtual reference direction. In other words, the sound processing unit 103 moves the virtual microphone 62 to follow the movement of the virtual camera 61 while maintaining the orientation of the virtual microphone 62 in the virtual reference direction.

[0068] The virtual reference direction is the direction that serves as the reference for the orientation of the virtual microphone 62 within the virtual space, and it is a direction that does not change in response to changes in the orientation of the player detected by the detection unit 101. The virtual reference direction will be explained in more detail later.

[0069] [Various directions] As shown in Figure 4, the direction (front, back, left, and right) of the multiple speakers 30 is determined with reference to a reference point Q surrounded by the multiple speakers 30. For example, "front" of the multiple speakers 30 corresponds to the front of the reference point Q, and the speaker 30 located "front" among the multiple speakers 30 is the "front center speaker 50" positioned in front of the reference point Q.

[0070] The orientation of player P1 is the direction in which the front of player P1 faces. In the example in Figure 4, the orientation of player P1 is the direction in which the multiple speakers 30 face "forward".

[0071] As shown in Figure 5, directions (forward, backward, left, right, up, and down) within the virtual space are predetermined. For example, "forward, backward, left, and right" within the virtual space correspond to the predetermined "north, south, west, and east" within the virtual space, and "up and down" within the virtual space correspond to the predetermined "height and earth" within the virtual space. Directions (forward, backward, left, right, up, and down) within the virtual space are represented in a three-dimensional coordinate system (e.g., the world coordinate system).

[0072] The orientation of the virtual camera 61 is the direction that the front of the virtual camera 61 faces. In this example, the orientation of the virtual camera 61 corresponds to the orientation of the player (specifically, vector Vd) derived based on the signal output from the motion sensor 42 of the head-mounted display 20.

[0073] The orientation of the virtual microphone 62 is the direction that the front of the virtual microphone 62 faces. The orientation (front, back, left, right) of the virtual microphone 62 corresponds to the orientation (front, back, left, right) of the multiple speakers 30. For example, sound output from the "front" of the virtual microphone 62 toward the virtual microphone 62 (sound in the virtual space) is output from the front center speaker 50 (speaker 30 located "front") which corresponds to the "front" of the multiple speakers 30.

[0074] In this embodiment, the orientation of the virtual microphone 62 is maintained (fixed) in the virtual reference direction. The virtual reference direction (the direction that serves as the reference for the orientation of the virtual microphone 62 in the virtual space) is set to the orientation of the virtual microphone 62 when the relationship between the orientation of the virtual microphone 62 and the orientation in the virtual space is a predetermined relationship. Specifically, the virtual reference direction is set to the orientation of the virtual microphone 62 (the direction in which the front of the virtual microphone 62 faces) when the orientation of the virtual microphone 62 (front, back, left, right) and the orientation in the virtual space (front, back, left, right) coincide.

[0075] In the example in Figure 5, the virtual reference direction is the direction facing "forward (north)" in the virtual space. When the front of the virtual microphone 62 faces "forward (north)" in the virtual space, the direction of the virtual microphone 62 (front, back, left, right) coincides with the direction in the virtual space (front, back, left, right).

[0076] [Details of processing by the control unit] Next, with reference to Figure 6, the processing by the control unit 16 will be explained in detail. In the following explanation, we will use the case where the player P1 is facing "forward" towards the speaker 30 as an example.

[0077] When player P1 faces "forward" in relation to the multiple speakers 30, player character C1 faces "forward (north)" in the virtual space. Similarly, virtual camera 61 faces "forward (north)" in the virtual space. As a result, the display 21 of the head-mounted display 20 worn by player P1 displays an image including enemy character C2, which is located "forward (north)" of player character C1 in the virtual space. Enemy character C2 is an example of a sound source object.

[0078] Furthermore, the virtual microphone 62 is maintained facing "forward (north)" in the virtual space. As a result, the character voice A2 output from enemy character C2, which is located "forward (north)" of player character C1 in the virtual space, is output from the front center speaker 50, which is positioned "in front" of player P1.

[0079] Through the above process, an image in the virtual space corresponding to the orientation of player P1 is displayed on the display 21 of the head-mounted display 20. Furthermore, the direction of sound directed towards the virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) matches the direction of sound directed towards player P1 from the multiple speakers 30 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate localization.

[0080] [Comparison of the embodiment and the comparative example] Next, the embodiment and the comparative example will be described in comparison. For the sake of explanation, the reference numerals of the components in the comparative example are used in the description of the components in the embodiment. In addition, the comparative example in the following description differs from the embodiment in that the orientation of the virtual microphone 62 is changed to follow the orientation of the virtual camera 61. Other processes in the comparative example are the same as those in the embodiment. It should be noted that in conventional game systems (e.g., VR systems), it is common to change the orientation of the virtual microphone 62 to follow the orientation of the virtual camera 61.

[0081] As shown in Figure 7, in the comparative example, when player P1 faces "to the right" of the multiple speakers 30, player character C1 faces "to the right (east)" in the virtual space. Similarly, the virtual camera 61 faces "to the right (east)" in the virtual space. As a result, the display 21 of the head-mounted display 20 worn by player P1 displays an image of player character C1 "to the right (east)" in the virtual space. Enemy character C2 is located "to the left (north)" of player character C1.

[0082] Furthermore, the virtual microphone 62, like the virtual camera 61, faces "right (east)" within the virtual space. Therefore, the character voice A2 output from enemy character C2, which is located "to the left (north)" of player character C1 within the virtual space, is output from surround left speaker 54, which is positioned "behind" player P1.

[0083] Thus, in the comparative example, the direction of sound directed towards the virtual microphone 62 in the virtual space (the direction of the sound field in the virtual space) and the direction of sound directed from the multiple speakers 30 towards player P1 (the direction of the sound field in the real world) do not match. Therefore, it is not possible to output sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 from the multiple speakers 30 with appropriate localization. In the example in Figure 7, the character voice A2 should ideally be output from the "left" of player P1 toward player P1, but it is output from the "rear" of player P1 toward player P1.

[0084] On the other hand, as shown in Figure 8, in this embodiment, the virtual microphone 62 is maintained facing "forward (north)" in the virtual space. Therefore, the character voice A2 output from the enemy character C2 located to the "left (north)" of the player character C1 in the virtual space is output from the front center speaker 50 positioned to the "left" of the player P1.

[0085] In this way, by maintaining the orientation of the virtual microphone 62 in the virtual reference direction (forward (north) in the example of Figure 8), it is possible to maintain a state in which the direction of sound directed towards the virtual microphone 62 in the virtual space (direction of the sound field in the virtual space) and the direction of sound directed from the multiple speakers 30 towards the player P1 (direction of the sound field in the real world) coincide. In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate localization.

[0086] [Summary of Embodiments] In summary, the program (game program) of the embodiment is a program that causes a computer system having one or more cooperating computers to function as a detection unit 101, a display processing unit 102, and an audio processing unit 103. The computer system is used together with a display 21 that follows the orientation of the user (player), multiple speakers 30 arranged around the user, and a sensor (motion sensor 22) that outputs a signal indicating the user's movement, including changes in the user's orientation. The detection unit 101 detects the user's orientation based on the signal output from the sensor. The display processing unit 102 changes the orientation of a virtual camera 61 placed in the virtual space to follow the orientation of the user detected by the detection unit 101, and displays the image of the virtual space obtained by the virtual camera 61 on the display 21. The audio processing unit 103 causes the multiple speakers 30 to output sound from the virtual space so that the orientation of the sound field formed by the sound in the virtual space output from the multiple speakers 30 does not change even if the orientation of the virtual camera 61 changes.

[0087] The sound control device (game device 10) of this embodiment includes a storage unit 15 for storing the above-mentioned program and a control unit 16 for executing the above-mentioned program.

[0088] The system of this embodiment (game system 1) includes a display 21 that follows the orientation of the user (player), a plurality of speakers 30 arranged around the user, a sensor (motion sensor 22) that outputs a signal indicating the user's movement, including changes in the user's orientation, and a control unit 16. The control unit 16 performs detection processing, display processing, and sound processing. In the detection processing, the control unit (detection unit 101) detects the user's orientation based on the signal output from the sensor. In the display processing, the control unit (display processing unit 102) changes the orientation of a virtual camera 61 arranged in the virtual space to follow the orientation of the user detected by the detection processing, and displays the image of the virtual space obtained by the virtual camera 61 on the display 21. In the sound processing, the control unit (sound processing unit 103) causes the plurality of speakers 30 to output sound from the virtual space so that the orientation of the sound field formed by the sound in the virtual space output from the plurality of speakers 30 does not change even if the orientation of the virtual camera 61 changes.

[0089] [Effects of the Embodiment] As described above, in this embodiment, even if the orientation of the virtual camera 61 changes, the orientation of the sound field formed by the sound in the virtual space output from the multiple speakers 30 (hereinafter referred to as "the orientation of the sound field in the real world") does not change, so that the sound in the virtual space is output from the multiple speakers 30.

[0090] According to the above control, it is possible to prevent the orientation of the sound field in the real world from changing due to changes in the orientation of the player P1, so that the "orientation of the sound field in the real world" does not deviate from the "orientation of the sound field in the virtual space (the orientation of the sound field formed by the sound output from the sound source object in the virtual space)". As a result, it is possible to output sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 from multiple speakers 30 with appropriate localization, thereby improving the user experience. For example, it is possible to improve the realism of how sound is heard in the virtual space, thereby improving interest.

[0091] (Modified examples of the embodiment) The modified game system 1 of the embodiment differs from the game system 1 of the embodiment in that the processing by the display processing unit 102 and the sound processing unit 103 is different. The other configurations and processing of the modified game system 1 of the embodiment are the same as those of the game system 1 of the embodiment.

[0092] In a modified embodiment, the controller 40 receives input for a direction change operation and a reset operation. The direction change operation is an operation to change the orientation of the virtual camera 61 from the forward direction, which corresponds to the player's orientation detected by the detection unit 101, to a different direction. In the direction change operation, a direction different from the forward direction (for example, a direction that results in a third-person view) is specified. The reset operation is an operation to reset the orientation of the virtual camera 61 to the forward direction.

[0093] The forward direction is the direction that matches the orientation of the player character C1, which corresponds to the orientation of the player detected by the detection unit 101. For example, if player P1 faces "forward" in relation to the multiple speakers 30, then player character C1 will face "forward (north)" in the virtual space, and the forward direction will be "forward (north)" in the virtual space.

[0094] When a direction change operation is input to the controller 40, the display processing unit 102 changes the orientation of the virtual camera 61 from the front direction in accordance with the direction change operation. When the orientation of the virtual camera 61 changes from the front direction in accordance with the direction change operation, the sound processing unit 103 changes the orientation of the virtual microphone 62 to follow the orientation of the virtual camera 61.

[0095] When a reset operation is input to the controller 40, the display processing unit 102 resets the orientation of the virtual camera 61 to the front direction in response to the reset operation. When the orientation of the virtual camera 61 is reset to the front direction in response to the reset operation, the sound processing unit 103 resets the orientation of the virtual microphone 62 to the virtual reference direction.

[0096] In this example, the display processing unit 102 selectively performs the first display processing and the second display processing. The first display processing is a process of changing the orientation of the virtual camera 61 so as to follow the orientation of the player detected by the detection unit 101 and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21. The second display processing is a process of setting the orientation of the virtual camera 61 to a direction different from the front direction (the direction specified by the direction change operation) and displaying an image of the virtual space obtained by the virtual camera 61 on the display 21.

[0097] When a direction change operation is input to the controller 40, the display processing unit 102 ends the first display processing and starts the second display processing. Also, when a reset operation is input to the controller 40, the display processing unit 102 ends the second display processing and starts the first display processing.

[0098] Also, in this example, the acoustic processing unit 103 selectively performs the first audio localization processing and the second audio localization processing. The first audio localization processing is a process of moving the virtual microphone 62 so as to follow the movement of the virtual camera 61 while maintaining the orientation of the virtual microphone 62 in the virtual reference direction and outputting the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30. The second audio localization processing is a process of changing the orientation of the virtual microphone 62 so as to follow the orientation of the virtual camera 61 and moving the virtual microphone 62 so as to follow the movement of the virtual camera 61, and outputting the sound in the virtual space obtained by the virtual microphone 62 to a plurality of speakers 30.

[0099] When the orientation of the virtual camera 61 changes from the front direction in response to a direction change operation, the acoustic processing unit 103 ends the first audio localization processing and starts the second audio localization processing. Also, when the orientation of the virtual camera 61 is reset to the front direction in response to a reset operation, the acoustic processing unit 103 ends the second audio localization processing and starts the first audio localization processing.

[0100] 〔Details of Processing by Control Unit〕 Next, with reference to Figure 9, the processing by the control unit 16 in a modified embodiment will be described in detail. In the following, the case in which the player P1 is facing "forward" towards the speaker 30 will be used as an example. In this example, when the orientation of the virtual camera 61 changes from the front direction to another direction in response to a direction change operation, the image displayed on the display 21 changes from a "first-person view image" to a "third-person view image".

[0101] When player P1 faces "forward" in relation to the multiple speakers 30, if a direction change operation is input to the controller 40 to change the orientation of the virtual camera 61 from "forward (north)" (front direction) in the virtual space to "left (west)" in the virtual space, the virtual camera 61 will face "left (west)" in the virtual space. As a result, the display 21 of the head-mounted display 20 worn by player P1 will display an image of player character C1 as seen from "right (east)" in the virtual space.

[0102] The left-right direction in the image displayed on display 21 corresponds to the front-back direction (north-south direction) in the virtual space. In the image displayed on display 21, the enemy character C2 is located to the "right" of the player character C1. In the example in Figure 9, the image displayed on display 21 is a third-person view image including the player character C1 and the enemy character C2. The display processing unit 102 changes the orientation of the virtual camera 61 and moves the virtual camera 61 so that a third-person view image can be obtained using the virtual camera 61.

[0103] Furthermore, to follow the orientation of the virtual camera 61, the orientation of the virtual microphone 62 changes from "forward (north)" (front direction) in the virtual space to "left (west)" in the virtual space. As a result, the character voice A2 output from the enemy character C2, which is located to the right of the player character C1 in the image displayed on the display 21, is output from the surround light speaker 53, which is positioned to the "right" of the player P1.

[0104] As a result of the above processing, an image of the virtual space (a third-person view image) obtained by a virtual camera 61 facing a direction different from the forward direction corresponding to the orientation of player P1 is displayed on the display 21 of the head-mounted display 20. Furthermore, the direction of the sound directed towards player character C1 (the direction of the sound field in the virtual space) in the image displayed on the display 21 matches the direction of the sound directed towards player P1 from the multiple speakers 30 (the direction of the sound field in the real world). In this way, the sound in the virtual space corresponding to the image in the virtual space displayed on the display 21 is output from the multiple speakers 30 with appropriate localization.

[0105] [Effects of modified embodiments] In modified embodiments of the embodiment, the same effects as those of the embodiment can be obtained.

[0106] Furthermore, in a modified embodiment, even when a direction change operation is input to the controller 40 to change the orientation of the virtual camera 61 from the front direction corresponding to the orientation of the player detected by the detection unit 101 to a different direction, the audio in the virtual space corresponding to the image in the virtual space displayed on the display 21 can be output from multiple speakers 30 with appropriate localization, thereby improving the user experience.

[0107] (Other embodiments) In the above description, the following configuration or settings may be used.

[0108] Audio data does not necessarily need to be prepared in advance. For example, the sound processing unit 103 may synthesize or generate audio data each time based on some data (for example, audio data stored in the memory unit 15) or information.

[0109] The player character may or may not be displayed on the screen. In other words, the player character may or may not be included in the image of the virtual space displayed (played) on the screen. For example, if the image of the virtual space is displayed on the screen in a first-person perspective, only the virtual hand (virtual body part), which is part of the player character, may be displayed, or the virtual hand may not be displayed, and only the handgun (virtual object) held in the virtual hand may be displayed.

[0110] The image of the virtual space may be a first-person perspective image or a third-person perspective image (for example, an image of the player character viewed from behind the player character).

[0111] Multiple speakers 30 may be arranged to the left, right, front, back, and above the listener (player) to enable three-dimensional sound reproduction.

[0112] The sensor that outputs a signal indicating the player's movement, including changes in the player's orientation, is not limited to the motion sensor 22 included in the head-mounted display 20. For example, such a sensor may be a combination of a camera (image sensor) that captures an image of a predetermined part of the player's body (e.g., the head) and a recognition processing unit that recognizes the player's movement, including changes in the player's orientation, based on the "image of the predetermined part" obtained by the camera.

[0113] The game program may be implemented as a game program for so-called online games. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in the control unit of the server device (not shown), or it may be shared between the control unit of the game device 10 and the control unit of the server device. The game device 10 may be a mobile information terminal such as a smartphone or tablet.

[0114] Furthermore, while the above explanation uses the virtual space where a game takes place (game space) as an example of a virtual space, it is not limited to this. For example, a virtual space could be a virtual space where avatars communicate with each other (metaverse space). In other words, the services provided in a virtual space are not limited to games. For example, the services provided in a virtual space could be the provision of a platform for commercial transactions, a platform for communication, etc.

[0115] The effects and benefits of the present invention will also be achieved when these other embodiments are adopted. Furthermore, it is possible to combine these embodiments with other embodiments, and with other embodiments as appropriate. The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses. [Explanation of Symbols]

[0116] 1. Game System (System) 10. Game device (sound control device) 11 Network Interfaces 12 Graphics Processing Unit 13 Audio Processing Unit 14 Operation Processing Unit 20 Head-mounted displays 21 displays 22 Motion Sensor 25 Main body 26 Headbands 30 speakers 40 controllers 40a First Controller 40b Second Controller 41 Operator 42 Motion Sensor (Sensor) 50 Front Center Speaker 51 Front light speaker 52 Front left speaker 53 Surround Light Speakers 54 Surround Left Speaker 55 Surround Backlit Speakers 56 Surround Back Left Speaker 61 Virtual Camera 62 Virtual Microphone 100 Game Progression Unit (Virtual Space Control Unit) 101 Detection unit 102 Display Processing Unit 103 Acoustic Processing Unit P1 Player (User) C1 Player Character (User Character) C2 Enemy Character (Sound Source Object) A2 Character voice (voice within the virtual space)

Claims

1. A program for causing a computer system having one or more cooperating computers to function as a detection unit, a display processing unit, and an acoustic processing unit, The computer system is used together with a display that follows the user's orientation, a plurality of speakers arranged around the user, and sensors that output signals indicating the user's movement, including changes in the user's orientation. The detection unit detects the user's orientation based on the signal output from the sensor, The display processing unit changes the orientation of a virtual camera placed in the virtual space to follow the orientation of the user detected by the detection unit, and displays the image of the virtual space obtained by the virtual camera on the display. The sound processing unit causes the plurality of speakers to output sound from the virtual space so that the direction of the sound field formed by the sound in the virtual space output from the plurality of speakers does not change even if the orientation of the virtual camera changes. The sound processing unit moves a virtual microphone positioned in the virtual space to follow the movement of the virtual camera, and outputs the sound from the virtual space obtained by the virtual microphone to the plurality of speakers. The orientation of the virtual microphone is a reference direction for the orientation of the virtual microphone within the virtual space, and is maintained in a virtual reference direction that does not change in response to changes in the user's orientation detected by the detection unit. program.

2. In the program of claim 1, The aforementioned computer system is used in conjunction with a controller operated by the user. When a direction change operation is input to the controller for changing the orientation of the virtual camera from the front direction corresponding to the user's orientation detected by the detection unit, the display processing unit changes the orientation of the virtual camera from the front direction in accordance with the direction change operation. The sound processing unit, when the orientation of the virtual camera changes from the front direction in response to the direction change operation, changes the orientation of the virtual microphone to follow the orientation of the virtual camera. program.

3. In the program of claim 2, When a reset operation to reset the orientation of the virtual camera to the front direction is input to the controller, the display processing unit resets the orientation of the virtual camera to the front direction in response to the reset operation. When the orientation of the virtual camera is reset to the front direction in response to the reset operation, the sound processing unit resets the orientation of the virtual microphone to the virtual reference direction. program.

4. A storage unit that stores any one of the programs from claim 1 to 3, The system comprises a control unit that executes the aforementioned program. Acoustic control device.

5. A display that follows the user's orientation, Multiple speakers arranged around the user, A sensor that outputs a signal indicating the user's movement, including a change in the user's orientation, It includes a control unit, The control unit, A detection process that detects the orientation of the user based on the signal output from the sensor, A display process that changes the orientation of a virtual camera placed in the virtual space to follow the orientation of the user detected by the detection process, and displays the image of the virtual space obtained by the virtual camera on the display, Even if the orientation of the virtual camera changes, acoustic processing is performed to cause the multiple speakers to output sound from the virtual space so that the orientation of the sound field formed by the sound in the virtual space output from the multiple speakers does not change. The control unit, in the sound processing, moves a virtual microphone placed in the virtual space to follow the movement of the virtual camera, and outputs the sound from the virtual space obtained by the virtual microphone to the plurality of speakers. The orientation of the virtual microphone is the reference direction for the orientation of the virtual microphone within the virtual space, and is maintained in a virtual reference direction that does not change in response to changes in the user's orientation detected by the detection process. system.

Images