Programs, systems, and methods

Abstract

To allow a user to recognize a situation in a virtual space without using hearing or vision. [Solution] A detection unit 101 detects the orientation of a user in the real world. A provision processing unit 102 performs processing for providing a three-dimensional virtual space to the user. An auxiliary control unit 103 provides the user with guide information that enables the user to recognize the situation in the virtual space with or without using the user's hearing, in order to assist the user's experience of the virtual space, and changes the guide information in response to changes in the user's orientation in the virtual space that accompany changes in the user's orientation in the real world.

Description

[Technical Field]

[0001] The present invention relates to a program, a system, and a method relating to the provision of a virtual space. [Background technology]

[0002] Patent Document 1 discloses that images and sounds in a virtual space are provided to a user using a head-mounted display and headphones. In this way, by providing images and sounds in a virtual space to a user, it becomes possible for the user to experience the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-098257 Summary of the Invention [Problem to be solved by the invention]

[0004] From the viewpoint of universal design, it is desirable to enable users to recognize the situation in a virtual space without using their hearing or vision. However, Patent Document 1 does not disclose or suggest anything about such a desire.

[0005] An object of the present invention is to enable a user to recognize a situation in a virtual space without using either hearing or vision. [Means for solving the problem]

[0006] The first aspect is A computer system having one or more computers working together, a detection unit that detects the orientation of the user in the real world; a providing processing unit that performs processing to provide a three-dimensional virtual space to the user; In order to assist the user in experiencing the virtual space, the virtual space control unit provides the user with guide information that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, and functions as an auxiliary control unit that changes the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. It is a program.

[0007] In a first aspect, The providing processing unit may include a display processing unit, The display processing unit may perform processing to provide an image of the virtual space to the user, The assistance control unit may display the guide information on a display that follows the orientation of the user in the real world.

[0008] In a first aspect, An object may be placed in the virtual space, The guide information displayed on the display may include an object-related display associated with the object placed in the virtual space.

[0009] In a first aspect, The object-related display may include a notification display associated with an out-of-field object, which is an object among the objects arranged in the virtual space that is not included in an image of the virtual space corresponding to an orientation of the user in the virtual space, The notification display may change in response to a change in at least one of a distance and a direction from the user's position to the position of the out-of-field object in the virtual space.

[0010] In a first aspect, The notification display may include a notification icon; The notification icon may be placed on the screen of the display at a position according to a direction from the user's position in the virtual space to the position of the out-of-field object.

[0011] In a first aspect, The notification display may include a notification icon; The notification icon may be arranged on the screen of the display, and may change size depending on the distance from the user's position to the out-of-field object's position in the virtual space.

[0012] In a first aspect, The object-related display may include a text display associated with a field-of-view object, which is an object included in an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space; The text display may show text corresponding to the sound output from the object within the field of view.

[0013] In a first aspect, The text display may include a peripheral object text display associated with a peripheral object that is an in-field object included in a peripheral region of the image of the virtual space according to an orientation of the user in the virtual space, The peripheral object text display may be positioned on the screen of the display at a position corresponding to the position of the peripheral object in the virtual space, and may show text related to the text of the voice output from the peripheral object.

[0014] In a first aspect, The text display may include a central object text display associated with a central object, which is the in-field object included in a central region of an image of the virtual space according to an orientation of the user in the virtual space; The central object text display may be arranged in a text display window provided at a predetermined position on the screen of the display to display text related to the situation within the virtual space, and may show the text of the voice output from the central object.

[0015] In a first aspect, The provision processing unit may include an audio processing unit, the audio processing unit performs processing to provide audio of the virtual space to the user; The assistance control unit may output the guide information to a speaker that follows the orientation of the user in the real world.

[0016] In a first aspect, The guide information output from the speaker may include an environment description voice for explaining a situation in an image of the virtual space according to an orientation of the user in the virtual space.

[0017] In a first aspect, An object may be placed in the virtual space, The guide information output from the speaker may include a guidance sound associated with a peripheral object, which is an object included in a peripheral region of an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space, The guiding sound may be sound for guiding the user to change their orientation in the real world so that the peripheral objects in the image of the virtual space move toward a central area of ​​the image in accordance with the user's orientation in the virtual space.

[0018] In a first aspect, An object may be placed in the virtual space, The guide information output from the speaker may include an explanatory voice associated with a central object, which is an object included in a central region of an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space, The explanatory voice may be a voice for explaining the status of the central object.

[0019] In a first aspect, An object may be placed in the virtual space, The guide information output from the speaker may include a notification sound associated with the object arranged in the virtual space, The notification sound may change in accordance with a change in at least one of a distance and a direction from the user's position to the object's position in the virtual space.

[0020] The second aspect is a detection unit that detects the orientation of the user in the real world; a providing processing unit that performs processing to provide a three-dimensional virtual space to the user; and an auxiliary control unit that provides the user with guide information that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, in order to assist the user in experiencing the virtual space, and that changes the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. It is a system.

[0021] The third aspect is 1. A method for assisting a user in experiencing a three-dimensional virtual space by a computer system having one or more computers working together, comprising: a detecting step of detecting an orientation of the user in the real world; and an auxiliary step of providing the user with guide information that enables the user to recognize a situation in the virtual space with or without using the user's hearing, in order to assist the user in experiencing the virtual space, and changing the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. It is a method. [Effects of the Invention]

[0022] According to the present invention, by providing guide information, the user can recognize the situation in the virtual space without using hearing or vision. Furthermore, by changing the guide information in response to changes in the user's orientation in the virtual space, the user can appropriately recognize the situation in the virtual space according to the user's orientation in the virtual space. This improves the user experience. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a block diagram illustrating an example of a system configuration according to a first embodiment. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of the appearance of a head-mounted display. [Figure 3] FIG. 1 is a schematic diagram illustrating a player wearing a head-mounted display, a speaker, and a controller. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of the arrangement of objects in a virtual space. [Figure 5] 3A and 3B are schematic diagrams illustrating an example of an image displayed on a display and sound output from a speaker in a normal mode. [Figure 6] 10 is a schematic diagram illustrating an example of an environment description display and a notification icon (notification display) included in an image displayed on a display in hearing assistance mode. FIG. [Figure 7] FIG. 10 is a schematic diagram illustrating a peripheral object text display included in an image displayed on the display in hearing assistance mode. [Figure 8]10 is a schematic diagram illustrating a central object text display included in an image displayed on the display in hearing assistance mode. FIG. [Figure 9] 10 is a schematic diagram illustrating an example of an environment description sound and an object sound included in the sound output from the speaker in visual assistance mode. FIG. [Figure 10] 10 is a schematic diagram illustrating an example of a prompting voice included in the voice output from the speaker in the visual assistance mode. FIG. [Figure 11] 10 is a schematic diagram illustrating an example of explanatory audio and object audio included in audio output from a speaker in visual assistance mode. FIG. [Figure 12] 10 is a schematic diagram illustrating an example of object-related sounds included in sounds output from a speaker in visual assistance mode. FIG. [Figure 13] FIG. 10 is a block diagram illustrating an example of the configuration of a system according to a second embodiment. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of the arrangement of a display and a speaker in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, an embodiment will be described in detail with reference to the drawings. In the drawings, identical or corresponding parts are designated by the same reference numerals, and their description will not be repeated. In the following, an example will be described in which the program is implemented as a "game program" and the system is implemented as a "game system including a game device." The game device is an example of a virtual space control device, and the game system is an example of a computer system including one computer or multiple computers working together.

[0025] (Game Description) In the following description, a game refers to a game (video game) played in a virtual space. The virtual space is a three-dimensional game space, and is displayed as an image on a display. Objects are placed in the virtual space. Examples of objects include player characters operated by players (users playing the game) and non-player characters operated by a computer. A player is an example of a user.

[0026] The objects also include sound source objects that act as virtual sound sources within the virtual space. Examples of sound source objects include objects that operate within the virtual space and objects that only output sound within the virtual space. Player characters and non-player characters can be sound source objects. Sounds are played in the games described below. Sounds played in the games include sounds output from sound source objects, background music (BGM), sound effects, and the like.

[0027] In the game described below, the player character is a virtual player (virtual human) placed in a virtual space, and moves in the virtual space in response to the player's operations and movements in the real world. Specifically, a virtual part moves in the virtual space in response to the movement of a predetermined part of the player's body. The virtual part is a virtual part (a predetermined part of the player character's body) that corresponds to a predetermined part of the player. For example, the player character's virtual head (virtual head) moves in the virtual space in response to the movement of the player's head in the real world.

[0028] There are no restrictions on the type of game. An example of a virtual space is a "virtual reality space where you can experience virtual reality (VR)." Another example is a game in which an image of the virtual space is displayed on a display from a first-person perspective.

[0029] (Embodiment 1) 1 illustrates the configuration of a game system 1 according to embodiment 1. The game system 1 includes a game device 10, a head-mounted display 20, a speaker 30, and a controller 40.

[0030] [Game device] The game device 10 executes a game in response to operations by a player. In this example, the game device 10 is used together with a head-mounted display 20, a speaker 30, and a controller 40, and is connected to these devices by wire or wirelessly.

[0031] The game device 10 may be a commercially available device such as a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), or Nintendo Switch (registered trademark).

[0032] A game progresses on the game device 10 based on the installed game program and game data. Game devices 10 can communicate data with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0033] [Head-mounted display] The head-mounted display 20 is worn on the head of a player. In this example, the head-mounted display 20 includes a display 21 and a motion sensor 22.

[0034] The display 21 displays (plays back) an image toward the player. For example, the display 21 is a liquid crystal display. The display 21 is an example of a display that follows the orientation of the player in the real world.

[0035] The motion sensor 22 outputs a motion signal that indicates the movement of the player's head. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), etc. The motion signal output from the motion sensor 22 is transmitted to the game device 10.

[0036] As shown in Fig. 2, the head mounted display 20 has a main body 211 and a headband 212. As shown in Fig. 3, the head mounted display 20 is worn on the head of the player P1 with the main body 211 covering both eyes of the player P1. A display 21 is fixed to the inside of the main body 211 (the side facing the eyes of the player P1). The display surface of the display 21 faces both eyes of the player P1 when the head mounted display 20 is worn on the head of the player P1.

[0037] The head-mounted display 20 attached to the head of the player P1 moves in conjunction with the movement of the head of the player P1. Therefore, it is possible to derive the "position and orientation of the head of the player P1" and the "position and orientation of the display 21 included in the head-mounted display 20" based on the movement signal output from the movement sensor 22 included in the head-mounted display 20. The movement sensor 22 is an example of a sensor that outputs a signal indicating the movement of the player P1, including changes in the orientation of the player P1.

[0038] Furthermore, vector Vd in the head mounted display 20 is a reference line of the image of the virtual space expressed from a first-person perspective. Vector Vd extends in the normal direction of the display surface of display 21. Vector Vd can be derived from the position and orientation of display 21 included in the head mounted display 20.

[0039] [Speaker] The speaker 30 is attached to the head (specifically, the ear) of the player. For example, the speaker 30 is a wearable speaker such as a headphone or an earphone. The speaker 30 outputs (plays) sound toward the player. The speaker 30 is an example of a speaker that follows the direction of the player in the real world.

[0040] 〔controller〕 The controller 40 is operated by being held or worn by a player. In this example, the controller 40 is operated by being held in the player's hand. As shown in FIG. 1, the controller 40 includes an operator 41 and a motion sensor 42.

[0041] The operators 41 are composed of various buttons, etc. The controller 40 outputs an operation signal corresponding to an operation input to the operators 41 (for example, an operation of pressing a button) to the game device 10. With this configuration, the player operates the operators 41 of the controller 40 to input an operation signal (a signal indicating a command, data, etc.) to the game device 10.

[0042] The motion sensor 42 outputs a motion signal that indicates the movement of the player's hand. For example, the motion sensor 22 includes an acceleration sensor that detects acceleration, a gyro sensor that detects angular velocity (changes in rotation and orientation), etc. The motion signal output from the motion sensor 42 is transmitted to the game device 10.

[0043] The controller 40 held in (or worn by) the player's hand moves in conjunction with the movement of the player's hand. Therefore, the position and orientation of the player's hand can be derived based on the movement signal output from the movement sensor 42 included in the controller 40.

[0044] As shown in Fig. 3, in this example, a first controller 40a and a second controller 40b are provided as the controller 40. The first controller 40a and the second controller 40b each have the same configuration as the controller 40. The first controller 40a is held and operated in the right hand of the player P1, and moves in conjunction with the movement of the player P1's right hand. The second controller 40b is held and operated in the left hand of the player P1, and moves in conjunction with the movement of the player P1's left hand.

[0045] [Hardware configuration of game device] 1, the game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation processing unit 14, a storage unit 15, and a control unit 16. The network interface 11, the graphics processing unit 12, the audio processing unit 13, the operation processing unit 14, and the storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0046] The network interface 11 is communicably connected to a communication network (not shown) for transmitting and receiving various data to and from, for example, other game devices 10 and external server devices (not shown).

[0047] The graphics processing unit 12 is connected to the display 21 of the head-mounted display 20 by wire or wirelessly. The graphics processing unit 12 renders game images including various objects in the virtual space in a moving image format in accordance with game image information (image data) output from the control unit 16. The game images (images of the virtual space) rendered in a moving image format are displayed on the display 21 as a game screen.

[0048] The audio processing unit 13 is connected to the speaker 30 by wire or wirelessly. The audio processing unit 13 plays digital game sounds (audio in the virtual space) in accordance with instructions from the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the speaker 30. In this example, the audio processing unit 13 is configured to be able to output multi-channel audio in order to play back three-dimensional sound.

[0049] The operation processing unit 14 is connected to each of the head mounted display 20 and the controller 40 by wire or wirelessly. The operation processing unit 14 transmits and receives signals to and from each of the head mounted display 20 and the controller 40. In this example, the operation processing unit 14 transmits to the control unit 16 an operation signal output from the motion sensor 22 of the head mounted display 20, an operation signal transmitted from the controller 40 (a signal corresponding to an operation input to the operator 41), and an operation signal output from the motion sensor 42 of the controller 40.

[0050] The storage unit 15 stores various types of information and data. The storage unit 15 is configured with an HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores game data, various programs including game programs, etc. Examples of game data include game media, player account information, etc.

[0051] The storage unit 15 also stores various image data representing various images prepared in advance. The various image data include image data representing scenery within the virtual space, image data representing images of objects to be placed within the virtual space, image data representing images related to effects within the virtual space, etc. These image data are used to generate images of the virtual space.

[0052] The storage unit 15 also stores various types of audio data representing various types of sounds prepared in advance. The various types of audio data include audio data representing sounds output from sound source objects, audio data representing music (BGM) played in the virtual space, audio data representing sound effects, etc. These types of audio data are used to generate sounds in the virtual space.

[0053] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and a semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[0054] [Functional configuration of the control unit] The control unit 16 of the game device 10 has a game progression unit 100, a detection unit 101, a provision processing unit 102, and an auxiliary control unit 103. Specifically, the control unit 16 functions as the game progression unit 100, the detection unit 101, the provision processing unit 102, and the auxiliary control unit 103 by executing a game program.

[0055] In other words, the control unit 16 performs game progression processing, provision processing, detection processing, and auxiliary processing. The game progression processing is processing by the game progression unit 100. The provision processing is processing by the provision processing unit 102. The detection processing is processing by the detection unit 101. The auxiliary processing is processing by the auxiliary control unit 103.

[0056] <Game Progression Section (Virtual Space Control Section)> The game progression unit 100 controls the virtual space to progress a game that takes place in the virtual space. The game progression unit 100 is an example of a virtual space control unit that controls the virtual space. The game progression processing is an example of virtual space control processing (processing by the virtual space control unit).

[0057] In this example, the game progression unit 100 progresses (controls) the game taking place in the virtual space by controlling (generating or updating) three-dimensional data representing the virtual space based on the movements (movement of the player's head) indicated in the movement signals output from the movement sensor 22, the movements (movement of the player's hands) indicated in the movement signals output from the movement sensor 42, and the operations (operations input to the operators 41 of the controller 40) indicated in the operation signals output from the controller 40, so that these movements and operations are reflected in the virtual space.

[0058] The three-dimensional data representing the virtual space includes well-known data for configuring the virtual space, such as data on a coordinate system (e.g., a world coordinate system) that serves as a reference for directions within the virtual space, data on the coordinate systems (e.g., a local coordinate system) of various objects placed within the virtual space, data on the shapes of various objects, etc. By controlling the three-dimensional data representing the virtual space, it is possible to control the movement of objects within the virtual space.

[0059] For example, the game progression unit 100 controls the movement of the player character within the virtual space in accordance with the above movements and operations. The game progression unit 100 also controls the movement of other objects (such as non-player characters) within the virtual space in accordance with the progress of the game. For example, AI (artificial intelligence) is used to control the movement of other objects by the game progression unit 100.

[0060] Specifically, the game progression unit 100 controls the virtual space so that a virtual part (a predetermined part of the player character) moves within the virtual space in accordance with the "movement of a predetermined part of the player's body (head, right hand, left hand)" indicated in the movement signal output from the movement sensor (motion sensor 22 or movement sensor 42). For example, the game progression unit 100 controls the virtual space so that the virtual head (the head of the player character) moves within the virtual space in accordance with the "movement of the player's head" indicated in the movement signal output from the movement sensor 22. By moving the virtual head, the orientation of the player character can be changed.

[0061] <Detection unit> The detection unit 101 detects the orientation of the player in the real world based on the signal output from the motion sensor 22. The orientation of the player is the direction in which the front (front face) of the player faces. In this example, the orientation of the player is a vector Vd extending in the normal direction of the display surface of the display 21. For example, the detection unit 101 derives the position and orientation of the display 21 included in the head-mounted display 20 based on the motion signal (acceleration and angular velocity) output from the motion sensor 22 included in the head-mounted display 20, and derives the vector Vd based on the position and orientation of the display 21.

[0062] <Provision Processing Section> The provision processing unit 102 performs processing to provide a three-dimensional virtual space to the player. In this example, the provision processing unit 102 includes a display processing unit 105 that performs processing to provide an image of the virtual space to the player, and an audio processing unit 106 that performs processing to provide audio of the virtual space to the player.

[0063] <<Display Processing Section>> The display processing unit 105 generates an image of the virtual space (in this example, a game image) controlled by the game progression unit 100, and displays that image on the display 21. In this example, the display processing unit 105 displays on the display 21 an image of the virtual space that corresponds to the "orientation of the player in the real world" detected by the detection unit 101.

[0064] Specifically, the display processing unit 105 determines a virtual vector (the position and orientation of the player in the virtual space) based on the vector Vd (the position and orientation of the player in the real world) detected by the detection unit 101. Next, the display processing unit 105 generates image data that shows an image of the virtual space corresponding to the vector Vd, based on the three-dimensional data (three-dimensional data that indicates the virtual space) controlled by the game progression unit 100 and the virtual vector.

[0065] Then, the display processing unit 105 supplies the image data generated as described above to the graphics processing unit 12. As a result, an image of the virtual space represented by the image data is displayed on the display 21 of the head-mounted display 20.

[0066] The process of generating an image according to the vector Vd (the position and orientation of the player in the real world) is a well-known process (for example, coordinate transformation, projection transformation, etc.).

[0067] For example, a virtual camera (not shown) is placed in the virtual space. The virtual camera is placed near the position of the player character (the position of the player in the virtual space). The virtual camera is a virtual viewer (viewpoint) and is a reference point for reproducing the viewpoint of the player character (first-person viewpoint).

[0068] The display processing unit 105 changes the position and orientation of the virtual camera so as to follow the position and orientation of the player character. The position and orientation of the player character follow the vector Vd (the position and orientation of the player in the real world) detected by the detection unit 101. The display processing unit 105 then displays an image of the virtual space obtained by the virtual camera on the display 21.

[0069] Specifically, the display processing unit 105 generates image data showing the image according to the shooting conditions of the virtual camera (for example, the position and orientation of the virtual camera 61, the position and orientation of the object, etc.) so that the image displayed on the display 21 is an image including an image obtained by the virtual camera. The image (image data) obtained by the virtual camera is an image (image data) generated according to the shooting conditions of the virtual camera.

[0070] <<Audio Processing Unit>> The sound processing unit 106 generates sound for the virtual space (game sounds in this example) controlled by the game progression unit 100, and outputs the sound to the speaker 30. The sound for the virtual space is sound that corresponds to the situation in the virtual space (such as the movement of objects). In this example, the sound processing unit 106 outputs sound for the virtual space to the speaker 30 that corresponds to the "direction of the player in the real world" detected by the detection unit 101.

[0071] Specifically, the audio processing unit 106 determines a virtual vector (the position and orientation of the player in the virtual space) based on the vector Vd (the position and orientation of the player in the real world) detected by the detection unit 101. Next, the audio processing unit 106 selects audio data to be played from the audio data (various types of audio data prepared in advance) stored in the storage unit 15 according to the situation in the virtual space (the progress of the game in this example).

[0072] Next, the audio processing unit 106 generates audio data indicating the audio of the virtual space according to the vector Vd based on the three-dimensional data (three-dimensional data indicating the virtual space) controlled by the game progression unit 100, the virtual vector, and the above-mentioned selected audio data (or new audio data obtained by synthesizing the multiple selected audio data).

[0073] The audio processing unit 106 then supplies the audio data generated as described above to the audio processing unit 13. As a result, the audio of the virtual space represented by the audio data is output from the speaker 30.

[0074] The process of generating sound according to the vector Vd (the position and orientation of the player in the real world) is well-known (for example, voice synthesis or sound image localization). Sound image localization is a process of localizing the position of a sound source perceived by a human (player) to a predetermined virtual position in a virtual space (for example, the position of a sound source object).

[0075] For example, a virtual microphone (not shown) is placed in the virtual space. The virtual microphone, together with a virtual camera, is placed near the position of the player character (the position of the player in the virtual space). The virtual microphone is a virtual listener (sound collection point) and is a reference point for sound image localization in sound image localization processing.

[0076] The audio processing unit 106 changes the position and orientation of the virtual microphone so as to follow the position and orientation of the player character. The position and orientation of the player character follow the vector Vd (the position and orientation of the player in the real world) detected by the detection unit 101. The audio processing unit 106 then outputs the sound of the virtual space obtained by the virtual microphone to the speaker 30.

[0077] Specifically, in the sound image localization process, the sound processing unit 106 generates sound data indicating the sound according to the sound collection conditions of the virtual microphone (for example, the volume and direction of the sound output from the sound source object, the position and orientation of the virtual microphone 62, the position and orientation of the sound source object, etc.) so that the sound output from the speaker 30 includes the sound obtained by the virtual microphone. The sound (sound data) obtained by the virtual microphone is sound (sound data) generated according to the sound collection conditions of the virtual microphone. The sound output from the sound source object is sound (sound prepared in advance) associated with the sound source object.

[0078] By processing the audio data generated as described above by the audio processing unit 13, it becomes possible to output the sound output from the virtual sound source (sound source object) in the virtual space from the speaker 30 with an acoustic representation that sounds as if the sound was collected by a virtual microphone.

[0079] Auxiliary control unit The auxiliary control unit 103 provides the player with guide information to assist the player in experiencing the virtual space. The guide information is information that enables the player to recognize the situation in the virtual space without using the player's hearing or vision. The auxiliary control unit 103 also changes the guide information in response to a change in the player's orientation in the virtual space that accompanies a change in the player's orientation in the real world.

[0080] [Game system operation mode] In this example, the operation mode of the game system 1 can be switched among a “normal mode,” an “auditory assistance mode,” and a “visual assistance mode.” For example, the control unit 16 determines the operation mode of the game system 1 in response to an external instruction (e.g., an instruction from the player).

[0081] The normal mode is a mode in which the player experiences the virtual space without using either "auditory aid guide information" or "visual aid guide information." The auditory aid mode is a mode in which the player experiences the virtual space using "auditory aid guide information." The visual aid mode is a mode in which the player experiences the virtual space using "visual aid guide information."

[0082] The auditory guide information is guide information that allows the player to recognize the situation in the virtual space without using the sense of hearing, and the visual guide information is guide information that allows the player to recognize the situation in the virtual space without using the sense of sight.

[0083] [Operation in normal mode] In normal mode, the game progression unit 100, detection unit 101, and provision processing unit 102 operate, but the auxiliary control unit 103 does not operate. In normal mode, the game progression unit 100, detection unit 101, and provision processing unit 102 operate as follows.

[0084] The game progression unit 100 controls (generates or updates) three-dimensional data representing the virtual space so that the movements (movement of the player's head) represented in the movement signals output from the movement sensor 22 and the operations (operations input to the controls 41 of the controller 40) represented in the operation signals output from the controller 40 are reflected in the virtual space. The detection unit 101 detects the orientation of the player in the real world based on the signals output from the movement sensor 22.

[0085] The provision processing unit 102 performs processing to provide a three-dimensional virtual space to the player. Specifically, the display processing unit 105 causes the display 21 to display an image of the virtual space according to the "direction of the player in the real world" detected by the detection unit 101. The audio processing unit 106 causes the speaker 30 to output audio of the virtual space according to the "direction of the player in the real world" detected by the detection unit 101.

[0086] For example, as shown in FIG. 4, when three objects OB are arranged around the position of the player character P2 (the position of the player in the virtual space) in the virtual space and the three objects OB are arranged outside the field of view of the player character P2 (outside the field of view of the player in the virtual space), the three objects OB are not included in the image 70 (an image of the virtual space corresponding to the orientation of the player in the real world) displayed on the display 21 as shown in FIG. 5.

[0087] The above "surrounding the position of the player character P2" refers to a predetermined range (area) based on the position of the player character P2, for example, a circular range centered on the position of the player character P2. The above "field of view of the player character P2" refers to a field of view according to the orientation of the player character P2, for example, a quadrangular pyramid-shaped range (spatial area) extending from the position of the player character P2 along the orientation of the player character P2.

[0088] Furthermore, when three objects OB are arranged around the position of the player character P2 in the virtual space as shown in Fig. 4, the sound output from the speaker 30 (the sound in the virtual space according to the orientation of the player in the real world) includes three object sounds OV output from the three objects OB as shown in Fig. 5. Note that in Fig. 5, the object sounds OV output from the objects OB are illustrated as "speech bubbles of the object OB," but these speech bubbles are not displayed as images.

[0089] [Operation in hearing assistance mode] In hearing assistance mode, the game progression unit 100, detection unit 101, and provision processing unit 102 operate, and the assistance control unit 103 also operates. The operation of the game progression unit 100, detection unit 101, and provision processing unit 102 in hearing assistance mode is the same as their operation in normal mode.

[0090] In the hearing assistance mode, the assistance control unit 103 displays guide information on the display 21 that follows the orientation of the player in the real world. Hereinafter, the guide information displayed on the display 21 will be referred to as a "guide display." In this example, the guide display is displayed on the display 21 while being superimposed on the image 70 displayed on the display 21.

[0091] <Guide display> As shown in FIGS. 6 to 8, in this example, the guide display includes an environment explanation display 81 and an object-related display 82.

[0092] <Environmental Information Display> As shown in FIG. 6, the environment explanation display 81 is a display for explaining the situation in the image 70 of the virtual space according to the direction of the player in the virtual space.

[0093] In this example, the environment explanation display 81 is placed in the text display window 71. The text display window 71 is provided at a predetermined position on the screen of the display 21 (in this example, the image 70 displayed on the display 21) to display text relating to the situation in the virtual space.

[0094] 6, the environment explanation display 81 displays the explanation (text) "There are various living creatures around you." The text display window 71 is provided at the top center of the image .

[0095] Object-related display 6 to 8, the object-related display 82 is associated with an object OB placed in the virtual space. In this example, the object-related display 82 includes a notification display 83 and a text display 84.

[0096] <Notification display> As shown in FIG. 6, the notification display 83 is associated with an out-of-field object OB1. The out-of-field object OB1 is an object OB placed in the virtual space that is not included in the virtual space image 70 corresponding to the player's orientation in the virtual space. The notification display 83 changes in response to a change in at least one of the distance and direction from the player's position to the out-of-field object OB1 in the virtual space. In this example, the notification display 83 includes a notification icon 85.

[0097] Notification icon 6, the notification icon 85 is placed on the screen of the display 21 (in this example, the image 70 displayed on the display 21) at a position corresponding to the direction from the player's position in the virtual space to the position of the out-of-field object OB1. The notification icon 85 is placed on the screen of the display 21, and its size changes depending on the distance from the player's position in the virtual space to the position of the out-of-field object OB1.

[0098] In the example of Figure 6, the screen (image 70) of the display 21 has arranged thereon a first notification icon 85a associated with a first object OBa, a second notification icon 85b associated with a second object OBa, and a third notification icon 85c associated with a third object OBC.

[0099] In this example, the shape of the notification icon 85 is circular. The notification icon 85 is placed in the peripheral region of the screen of the display 21 (in this example, the image 70 displayed on the display 21). The direction from the center of the screen of the display 21 to the position of the notification icon 85 corresponds to the "direction from the position of the player to the position of the out-of-field object OB1 in the virtual space" when the player's orientation in the virtual space is set as the forward direction (reference). The longer the distance from the player's position to the position of the out-of-field object OB1 in the virtual space, the smaller the size of the notification icon 85.

[0100] For example, as shown in Figure 4, if the direction from the player's position in the virtual space to the position of the third object OBc is "leftward (left frontward)", then as shown in Figure 6, the direction from the center of the screen of the display 21 to the position of the third notification icon 85c will be "leftward".

[0101] Also, as shown in Figure 4, if the "distance from the player's position in the virtual space to the third object OBc" is longer than the "distance from the player's position in the virtual space to the first object OBa", the size of the third notification icon 85c will be smaller than the size of the first notification icon 85a, as shown in Figure 6.

[0102] <Text display> 7 and 8, the text display 84 is associated with the in-field object OB2. The in-field object OB2 is one of the objects OB arranged in the virtual space that is included in the image 70 of the virtual space according to the orientation of the player in the virtual space. The text display 84 shows text according to the sound output from the in-field object OB2. In this example, the text display 84 includes a peripheral object text display 86 and a central object text display 87.

[0103] <Displaying text on surrounding objects> 7, the peripheral object text display 86 is associated with the peripheral object OB3. The peripheral object OB3 is an in-field object OB2 included in the peripheral region R1 of the image 70 of the virtual space according to the orientation of the player in the virtual space. The peripheral object text display 86 is placed at a position on the screen of the display 21 (in this example, the image 70 displayed on the display 21) that corresponds to the position of the peripheral object OB3 in the virtual space, and displays text related to the text of the sound output from the peripheral object OB3.

[0104] In this example, the peripheral object OB3 is an object OB that is partly or entirely included in the peripheral region R1 of the image 70 of the virtual space, but not included in the central region R2 of the image 70 of the virtual space (located outside the central region R2). The text shown in the peripheral object text display 86 is text that indicates the main part of the audio output from the peripheral object OB3, and is shorter than the text of the audio output from the peripheral object OB3. For example, the peripheral object text display 86 shows text obtained by simplifying the text of the audio output from the peripheral object OB3.

[0105] In the example of Fig. 7, the peripheral object OB3 is a "first object OBa." As the first object OBa gradually approaches the position of the player in the virtual space, the first object OBa changes from an "out-of-field object OB1 (see Fig. 6)" to a "peripheral object OB3 (see Fig. 7)." The object-related display 82 associated with the first object OBa changes from a "first notification display 84a (see Fig. 6)" to a "peripheral object text display 86 (see Fig. 7)."

[0106] In the example of FIG. 7, the text displayed in the peripheral object text display 86 is the text "Myaa," which is the opening part of the voice "Play with Meow meow" (see FIG. 8) output from the first object OBa.

[0107] <Central object text display> 8, the central object text display 87 is associated with the central object OB4. The central object OB4 is the in-field object OB2 included in the central region R2 of the image 70 of the virtual space according to the orientation of the player in the virtual space. The central object text display 87 is arranged in a text display window 71 provided at a predetermined position on the screen of the display 21 (in this example, the image 70 displayed on the display 21), and displays the text of the voice output from the central object OB4.

[0108] In this example, the central object OB4 is an object OB that is partly or entirely included in the "central region R2 of the image 70 in the virtual space." Note that a part of the central object OB4 may also be included in the "peripheral region R1 of the image 70 in the virtual space." The text shown in the central object text display 87 is the entire text of the voice (specifically, voice in a predetermined coherent unit) output from the central object OB4.

[0109] In the example of Fig. 8, the central object OB4 is the "first object OBa." As the first object OBa gradually approaches the position of the player in the virtual space, the first object OBa changes from the "peripheral object OB3 (see Fig. 7)" to the "central object OB4 (see Fig. 8)." The object-related display 82 associated with the first object OBa changes from the "peripheral object text display 86 (see Fig. 7)" to the "central object text display 87 (see Fig. 8)."

[0110] <Generation of Various Displays in Hearing Assist Mode> In the hearing assistance mode, the assistance control unit 103 generates an environment explanation display 81 based on the situation of the virtual space controlled by the game progression unit 100 and the "direction of the player in the real world" detected by the detection unit 101, and displays the environment explanation display 81 on the display 21.

[0111] Specifically, the auxiliary control unit 103 generates an explanatory text indicating the status of the object OB within a predetermined range (for example, a predetermined range extending in front of the player character) based on the "status (placement, movement, etc.) of the object OB" within the predetermined range based on the position of the player character in the virtual space, and generates the environment explanatory display 81 indicating the explanatory text. The explanatory text may be selected from explanatory text candidates prepared in advance, or may be automatically generated by AI (artificial intelligence) or the like.

[0112] In addition, in the hearing assistance mode, the assistance control unit 103 determines, for each of the objects OB arranged around the player's position in the virtual space, whether the object OB corresponds to an "out-of-field object OB1," a "peripheral object OB3," or a "central object OB4."

[0113] For the out-of-field object OB1, the auxiliary control unit 103 generates a notification icon 85 associated with the out-of-field object OB1 and displays the notification icon 85 on the display 21. Specifically, the auxiliary control unit 103 determines the placement (placement within the screen of the display 21) and size of the notification icon 85 based on the direction and distance from the player's position to the position of the out-of-field object OB1 in the virtual space.

[0114] For the peripheral object OB3, the auxiliary control unit 103 generates a peripheral object text display 86 associated with the peripheral object OB3 and displays the peripheral object text display 86 on the display 21. Specifically, the auxiliary control unit 103 acquires the text (text data) of the object audio OV output from the peripheral object OB3, simplifies the text (e.g., extracts the opening part), thereby generating new text associated with the text of the object audio OV, and generates a peripheral object text display 86 showing the new text.

[0115] For the central object OB4, the auxiliary control unit 103 generates a central object text display 87 associated with the central object OB4 and displays the central object text display 87 on the display 21. Specifically, the auxiliary control unit 103 obtains the text (text data) of the object voice OV output from the central object OB4, and generates the central object text display 87 showing the text.

[0116] The text (text data) of the object audio OV may be pre-associated with the audio data indicating the object audio OV and stored in the storage unit 15, or may be automatically generated from the audio data indicating the object audio OV using AI (artificial intelligence) or the like.

[0117] [Operation in visual assistance mode] In visual assistance mode, the game progression unit 100, detection unit 101, and provision processing unit 102 operate, and the assistance control unit 103 also operates. The operation of the game progression unit 100, detection unit 101, and provision processing unit 102 in visual assistance mode is the same as their operation in normal mode.

[0118] In the visual assistance mode, the assistance control unit 103 outputs guide information to the speaker 30 that follows the direction of the player in the real world. Hereinafter, the guide information output from the speaker 30 will be referred to as a "guide voice." In this example, the guide voice is output from the speaker 30 while being superimposed on the voice output from the speaker 30 (voice in the virtual space).

[0119] <Guide audio> 9 to 12, in this example, the guide audio includes an environment explanation audio 91 and an object-related audio 92. Note that in Fig. 9 and Fig. 11, the object audio OV output from the object OB is illustrated as a "speech bubble of the object OB," but this speech bubble is not displayed as an image.

[0120] <Environmental explanation audio> As shown in Fig. 9, the environment explanation audio 91 is audio for explaining the situation in the virtual space image 70 according to the orientation of the player in the virtual space. In the example of Fig. 9, the environment explanation audio 91 is audio that explains, "There is a green grassland under a wide blue sky, and various creatures live in it."

[0121] Object-related audio 10 to 12, the object-related audio 92 is associated with an object OB placed in a virtual space. In this example, the object-related audio 92 includes a guidance audio 93, an explanation audio 94, and a notification audio 95.

[0122] <Guidance voice> 10, the guiding sound 93 is associated with a peripheral object OB3. The peripheral object OB3 is an object OB that is included in a peripheral region R1 of an image 70 of the virtual space according to the orientation of the player in the virtual space, among the objects OB placed in the virtual space.

[0123] The guiding sound 93 is a sound that guides the player to change his or her orientation in the real world so that the peripheral object OB3 in the virtual space image 70 moves toward the central region R2 of the image 70 in accordance with the player's orientation in the virtual space.

[0124] In the example of Fig. 10, the peripheral object OB3 is the "first object OBa." As the first object OBa gradually approaches the position of the player in the virtual space, the first object OBa changes from an "out-of-field object OB1 (see Fig. 9)" to a "peripheral object OB3 (see Fig. 10)," and a guiding voice 93 associated with the first object OBa is output from the speaker 30. The guiding voice 93 is a voice that explains, "Someone is talking to you from the right."

[0125] <Explanatory audio> 11, the explanatory audio 94 is associated with a central object OB4. The central object OB4 is an object OB that is included in the central region R2 of the image 70 of the virtual space according to the orientation of the player in the virtual space, among the objects OB placed in the virtual space. The explanatory audio 94 is audio for explaining the status of the central object OB4.

[0126] In the example of Fig. 11, the central object OB4 is the "first object OBa." As the first object OBa gradually approaches the position of the player in the virtual space, the first object OBa changes from the "peripheral object OB3 (see Fig. 10)" to the "central object OB4 (see Fig. 11)," and an explanatory audio 94 associated with the first object OBa is output from the speaker 30. The explanatory audio 94 is an audio that provides an explanation such as, "A cat standing on two legs is looking at you and saying hello."

[0127] <Notification voice> 12, the notification sound 95 is associated with an object OB (in this example, an out-of-field object) placed in the virtual space. Note that in FIG. 12, the notification sound 95 is illustrated as a "speech bubble for the object OB," but this speech bubble is not displayed as an image.

[0128] The notification sound 95 changes in accordance with changes in at least one of the distance and direction from the player's position to the position of the object OB in the virtual space.

[0129] In this example, the notification sound 95 is a sound (loop sound) that is generated repeatedly at a predetermined cycle. The direction from which the player recognizes the notification sound 95 as coming varies depending on the direction from the player's position to the object OB in the virtual space. The cycle of the notification sound 95 (the cycle of repeating the sound) varies depending on the distance from the player's position to the object OB in the virtual space.

[0130] In the example of Figure 12, a first notification sound 95a associated with the first object OBa, a second notification sound 95b associated with the second object OBi, and a third notification sound 95c associated with the third object OBi are output from the speaker 30.

[0131] In this example, the direction from which the player recognizes the notification sound 95 as coming corresponds to the direction from the player's position to the object OB in the virtual space. The longer the distance from the player's position to the object OB in the virtual space, the longer the period (sound repetition period) of the notification sound 95 becomes.

[0132] For example, as shown in Figure 4, if the direction from the player's position in the virtual space to the position of the third object OBc is "leftward (left forward)," then the direction that the player recognizes as the direction from which the third notification sound 95c is heard will be "leftward," as shown in Figure 12.

[0133] Furthermore, as shown in FIG. 4, when the "distance from the player's position in the virtual space to the third object OBc" is longer than the "distance from the player's position in the virtual space to the second object OBb" and the "distance from the player's position in the virtual space to the second object OBb" is longer than the "distance from the player's position in the virtual space to the first object OBa," as shown in FIG. 12, the period of the "beep beep beep" sound, which is the third notification sound 95c, is longer than the period of the "beep beep beep" sound, which is the second notification sound 95b, and the period of the second notification sound 95b is longer than the period of the "beep beep beep" sound, which is the third notification sound 95c.

[0134] <Generation of various sounds in visually assisted mode> In the visual assistance mode, the assistance control unit 103 generates an environment description audio 91 based on the situation of the virtual space controlled by the game progression unit 100 and the "direction of the player in the real world" detected by the detection unit 101, and outputs the environment description audio 91 to the speaker 30.

[0135] Specifically, the auxiliary control unit 103 generates an explanatory sentence indicating the scenery in the virtual space and the situation of the object OB based on the "scenery in the virtual space" and the "situation of the object OB (such as placement and movement)" within a predetermined range (for example, a predetermined range extending in front of the player character) based on the position of the player character in the virtual space, and generates environment explanatory audio 91 indicating the explanatory sentence. The explanatory sentence may be selected from explanatory sentence candidates prepared in advance, or may be automatically generated by AI (artificial intelligence) or the like.

[0136] In addition, in the visual assistance mode, the assistance control unit 103 determines, for each of the objects OB arranged around the player's position in the virtual space, whether the object OB corresponds to a "peripheral object OB3" or a "central object OB4."

[0137] For the peripheral object OB3, the auxiliary control unit 103 generates a guiding sound 93 associated with the peripheral object OB3 and outputs the guiding sound 93 to the speaker 30. Specifically, based on the position of the peripheral object OB3 and the position and orientation of the player in the virtual space, the auxiliary control unit 103 generates the guiding sound 93 for guiding the player to change the orientation of the player in the real world so that the peripheral object OB3 moves toward the central region R2 of the image 70.

[0138] For example, if the position of the peripheral object OB3 is "to the right (to the front right)" relative to the position of the player in the virtual space, the guiding sound 93 will be a sound that guides the player to change their orientation in the real world to "to the right."

[0139] For the central object OB4, the auxiliary control unit 103 generates explanatory audio 94 associated with the central object OB4 and outputs the explanatory audio 94 to the speaker 30. Specifically, the auxiliary control unit 103 generates explanatory text indicating the status of the central object OB4 based on the status (appearance, movement, etc.) of the central object OB4, and generates explanatory audio 94 indicating the explanatory text. The explanatory text may be selected from explanatory text candidates prepared in advance, or may be automatically generated by AI (artificial intelligence) or the like.

[0140] Furthermore, in the visual assistance mode, the assistance control unit 103 generates a notification sound 95 associated with each object OB (out-of-field object OB1 in this example) arranged around the player's position in the virtual space, and outputs the notification sound 95 to the speaker 30. Specifically, the assistance control unit 103 determines the "direction from which the player will recognize the notification sound 95 as coming" and the "period of the notification sound 95" based on the direction and distance from the player's position to the position of the object OB in the virtual space.

[0141] [Summary of the embodiment] To summarize the above, the program (game program) of the embodiment causes a computer system (control unit 16) having one or more cooperating computers to function as a detection unit 101, a provision processing unit 102, and an auxiliary control unit 103. The detection unit 101 detects the orientation of the user (player) in the real world. The provision processing unit 102 performs processing to provide a three-dimensional virtual space to the user. In order to assist the user's experience of the virtual space, the auxiliary control unit 103 provides the user with guide information that enables the user to recognize the situation in the virtual space without using the user's hearing or vision, and changes the guide information in response to changes in the user's orientation in the virtual space that occur as the user's orientation changes in the real world.

[0142] The system (game system 1) of the embodiment includes a detection unit 101, a provision processing unit 102, and an auxiliary control unit 103.

[0143] A method according to an embodiment is a method for assisting a user in experiencing a three-dimensional virtual space by a computer system having one or more computers working together, and includes a detection step and an assistance step. In the detection step, the user's orientation in the real world is detected. In the assistance step, in order to assist the user's experience of the virtual space, guide information is provided to the user that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, and the guide information is changed in response to a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world.

[0144] [Effects of the embodiment] As described above, in the embodiment, by providing guide information, the player can recognize the situation in the virtual space without using hearing or sight. Furthermore, by changing the guide information in response to a change in the player's orientation in the virtual space, the player can appropriately recognize the situation in the virtual space in response to the player's orientation in the virtual space. This can improve the user experience.

[0145] In the embodiment, the providing processing unit 102 includes a display processing unit 105 that performs processing to provide an image of the virtual space to the player. The auxiliary control unit 103 displays guide information on the display 21 that follows the orientation of the player in the real world.

[0146] According to the above configuration, by displaying guide information as an image on the display 21, the player can be made aware of the situation in the virtual space without using the player's sense of hearing.

[0147] In the embodiment, the guide information displayed on the display 21 includes an object-related display 82 associated with an object OB placed in the virtual space.

[0148] According to the above configuration, the object-related display 82 allows the player to recognize the object OB placed in the virtual space.

[0149] In the embodiment, the object-related display 82 also includes a notification display 83 associated with an out-of-field object OB1, which is an object OB arranged in the virtual space and is not included in the image 70 of the virtual space corresponding to the orientation of the player in the virtual space. The notification display 83 changes in response to a change in at least one of the distance and direction from the position of the player to the position of the out-of-field object OB1 in the virtual space.

[0150] According to the above configuration, the notification display 83 allows the player to recognize at least one of the distance and direction from the player's position in the virtual space to the position of the out-of-field object OB1.

[0151] In addition, in the embodiment, a notification icon 85, which is an example of a notification display 83, is placed on the screen of the display 21 (in this example, the image 70 displayed on the display 21) at a position corresponding to the direction from the player's position in the virtual space to the position of the out-of-field object OB1.

[0152] According to the above configuration, based on the position of the notification icon 85, the player can recognize the direction from the player's position in the virtual space to the position of the out-of-field object OB1.

[0153] In addition, in the embodiment, a notification icon 85, which is an example of a notification display 83, is placed on the screen of the display 21 (in this example, the image 70 displayed on the display 21), and its size changes depending on the distance from the player's position in the virtual space to the position of the out-of-field object OB1.

[0154] According to the above configuration, the size of the notification icon 85 can be used to allow the player to recognize the distance from the player's position in the virtual space to the position of the out-of-field object OB1.

[0155] In the embodiment, the object-related display 82 includes a text display 84 associated with a field-of-view object OB2, which is an object OB included in the image 70 of the virtual space according to the orientation of the player in the virtual space, among the objects OB arranged in the virtual space. The text display 84 displays text according to the sound output from the field-of-view object OB2.

[0156] According to the above configuration, the player can recognize the sound output from the in-field object OB2 based on the text display 84.

[0157] Furthermore, in the embodiment, the text display 84 includes a peripheral object text display 86 associated with a peripheral object OB3, which is a field-of-view object OB2 included in the peripheral region R1 of the image 70 of the virtual space according to the orientation of the player in the virtual space. The peripheral object text display 86 is arranged in a position on the screen of the display 21 (in this example, the image 70 displayed on the display 21) that corresponds to the position of the peripheral object OB3 in the virtual space, and displays text related to the text of the voice output from the peripheral object OB3.

[0158] According to the above configuration, the player can recognize the sound output from the peripheral object OB3 based on the peripheral object text display 86. For example, by making the text shown in the peripheral object text display 86 "shorter text (e.g., simplified text) than the text of the sound output from the peripheral object OB3," the player can quickly recognize the content (summary) of the sound output from the peripheral object OB3.

[0159] Furthermore, in the embodiment, the text display 84 includes a central object text display 87 associated with a central object OB4, which is a field-of-view object OB2 included in the central region R2 of the image 70 of the virtual space according to the orientation of the player in the virtual space. The central object text display 87 is arranged in a text display window 71 provided at a predetermined position on the screen of the display 21 (in this example, the image 70 displayed on the display 21) to display text relating to the situation in the virtual space, and shows the text of the voice output from the central object OB4.

[0160] According to the above configuration, the player can recognize the sound output from the central object OB4 based on the central object text display 87. Furthermore, by setting the text displayed in the central object text display 87 to "text of the sound output from the central object OB4," the player can more accurately recognize the content of the sound output from the central object OB4 than when the text displayed in the central object text display 87 is simplified text.

[0161] In the embodiment, the providing processing unit 102 includes an audio processing unit 106 that performs processing to provide audio in the virtual space to the player. The auxiliary control unit 103 outputs guide information to a speaker 30 that follows the orientation of the player in the real world.

[0162] According to the above configuration, by outputting guide information as sound from the speaker 30, the player can be made aware of the situation in the virtual space without using the player's vision.

[0163] In the embodiment, the guide information output from the speaker 30 includes an environment description voice 91 for describing the situation in the image 70 of the virtual space according to the orientation of the player in the virtual space.

[0164] According to the above configuration, it is possible to make the player aware of the situation of the virtual space according to the player's orientation in the virtual space, based on the environment-explanatory voice 91.

[0165] Furthermore, in the embodiment, the guide information output from the speaker 30 includes a guiding sound 93 associated with a peripheral object OB3, which is an object OB among the objects OB arranged in the virtual space and is included in a peripheral region R1 of the virtual space image 70 according to the orientation of the player in the virtual space. The guiding sound 93 is sound for guiding the player to change the orientation of the player in the real world so that the peripheral object OB3 moves toward the central region R2 of the virtual space image 70 according to the orientation of the player in the virtual space.

[0166] According to the above configuration, the player's orientation in the virtual space can be changed to "face the peripheral object OB3" by inducing the player to change his / her orientation in the real world so that the peripheral object OB3 moves toward the central region R2 of the image 70 in the virtual space according to the player's orientation in the virtual space. This makes it easier for the player to recognize the status of the peripheral object OB3.

[0167] In the embodiment, the guide information output from the speaker 30 includes an explanatory audio 94 associated with a central object OB4, which is an object OB included in the central region R2 of the image 70 of the virtual space according to the orientation of the player in the virtual space, among the objects OB arranged in the virtual space. The explanatory audio 94 is audio for explaining the status of the central object OB4.

[0168] According to the above configuration, the player can be made aware of the state of the central object OB4 based on the explanatory voice 94.

[0169] In the embodiment, the guide information output from the speaker 30 includes a notification sound 95 associated with an object OB (out-of-field object OB1 in this example) placed in the virtual space. The notification sound 95 changes in response to a change in at least one of the distance and direction from the player's position to the position of the object OB in the virtual space.

[0170] According to the above configuration, the notification sound 95 can allow the player to recognize at least one of the distance and direction from the player's position to the position of the object OB in the virtual space.

[0171] (Embodiment 2) 13 illustrates the configuration of a game system 1 according to embodiment 2. The game system 1 according to embodiment 2 includes a plurality of environmental displays 50 and a plurality of environmental speakers 60 in addition to the configuration of the game system 1 according to embodiment 1 shown in FIG. 1. Furthermore, the game system 1 according to embodiment 2 includes AR (Augmented Reality) glasses 25 instead of the head-mounted display 20 shown in FIG. 1.

[0172] [AR Glasses] The AR glasses 25 are worn on the head of the player. In this example, the AR glasses 25 have a transparent display 26 and a motion sensor 22. The motion sensor 22 of the second embodiment is the same as the motion sensor 22 of the first embodiment.

[0173] The transparent display 26 displays (plays) an image toward the player. The transparency of the transparent display 26 is set so that the player wearing the AR glasses 25 can see the scenery behind the transparent display 26 through the transparent display 26. The transparent display 26 is an example of a display (first display) that follows the orientation of the player in the real world.

[0174] [Environmental Display] As shown in Fig. 14, the multiple environmental displays 50 are arranged to surround the player in the real world and display (play) images toward the player. For example, the multiple environmental displays 50 are liquid crystal displays. The multiple environmental displays 50 are an example of displays (second displays) that are arranged to surround the player without following the player's orientation in the real world.

[0175] [Environmental Speaker] As shown in FIG. 14, the multiple environmental speakers 60 are arranged so as to surround the player in the real world, and output (play) sounds toward the player. For example, the multiple environmental speakers 60 are stationary speakers. The multiple environmental speakers 60 are an example of speakers (second speakers) that are arranged so as to surround the player without following the player's orientation in the real world. Note that the speaker 30 is an example of a speaker (first speaker) that follows the player's orientation in the real world.

[0176] [Functional configuration of the control unit] The processing by the control unit 16 of the second embodiment differs from the processing by the control unit 16 of the second embodiment in the processing by the provision processing unit 102 (the display processing unit 105 and the audio processing unit 106) and the auxiliary control unit 103. The other processing by the control unit 16 of the second embodiment is the same as the processing by the control unit 16 of the second embodiment.

[0177] The display processing unit 105 in embodiment 2 generates an image of the virtual space controlled by the game progression unit 100, and displays that image on multiple environment displays 50. The image of the virtual space displayed on the multiple environment displays 50 corresponds to an image of the periphery (full circumference) of the player's reference viewpoint position in the virtual space. The player's reference viewpoint position in the virtual space is set to a position corresponding to the center position of the multiple environment displays 50.

[0178] In this way, by displaying images of the virtual space on the multiple environmental displays 50, the player P1 can view the images displayed on the multiple environmental displays 50 through the transparent displays 26 of the AR glasses 25. This allows an image of the virtual space corresponding to the "orientation of the player in the real world" detected by the detection unit 101 to be displayed on the transparent displays 26. In other words, the image displayed on the environmental displays 50 visible through the transparent displays 26 corresponds to an image of the virtual space corresponding to the orientation of the player in the real world.

[0179] The display processing unit 105 of the second embodiment may display a portion of the image of the virtual space on the multiple environmental displays 50, and the remaining portion of the image of the virtual space (for example, an image of an object OB) on the transparent display 26. For example, a background view of the virtual space may be displayed on the multiple environmental displays 50, and an object such as a character may be displayed on the transparent display 26 of the AR glasses 25. In such a case, too, an image of the virtual space according to the orientation of the player in the real world may be displayed on the transparent display 26.

[0180] In the hearing assistance mode, the auxiliary control unit 103 of the second embodiment displays guide information (guide display) on the transparent display 26 that follows the orientation of the player in the real world. As a result, the guide display (see FIGS. 6 to 8) is superimposed on the image 70 displayed on the transparent display 26 (specifically, the image displayed on the environmental display 50 that can be seen through the transparent display 26).

[0181] The sound processing unit 106 of the second embodiment generates sound for the virtual space controlled by the game progression unit 100 and outputs the sound to the multiple environmental speakers 60. The sound for the virtual space output from the multiple environmental speakers 60 corresponds to the sound heard at the player's reference listening position in the virtual space (for example, sound collected by a virtual microphone). The player's reference listening position in the virtual space is set to a position corresponding to the center position of the multiple environmental speakers 60. The center position of the multiple environmental speakers 60 is set to the same position as the center position of the multiple environmental displays 50.

[0182] In this way, by outputting the sound of the virtual space from a plurality of environmental speakers 60, the sound of the virtual space can be provided to the player.

[0183] The assistance control unit 103 of the second embodiment, in the same way as in the first embodiment, outputs guide information (guidance voice) to the speaker 30 that follows the direction of the player in the real world in the vision assistance mode. As a result, the guide voice (see FIGS. 9 to 12) is superimposed on the voices output from the multiple environmental speakers 60.

[0184] [Effects of the Second Embodiment] In the second embodiment, it is possible to obtain the same effects as those in the first embodiment. For example, by providing guide information, it is possible to make the player aware of the situation in the virtual space without using the auditory or visual senses.

[0185] (Other embodiments) In the above description, the following configuration or processing may be performed.

[0186] The game system 1 may be configured to allow multiple players to experience the virtual space simultaneously. In this case, a "head-mounted display 20 (or AR glasses 25)," a "speaker 30," a "controller 40," and the "detection unit 101, provision processing unit 102, and auxiliary control unit 103" of the control unit 16 may be provided for each player. The game progression unit 100 may control the virtual space based on signals from the head-mounted display 20 (or AR glasses 25) and controller 40 worn by each of the multiple players.

[0187] For example, in the second embodiment, multiple players may be simultaneously placed in a space surrounded by multiple environmental displays 50 and multiple environmental speakers 60. In this case, the objects (characters, etc.) displayed on the transparent displays 26 of the AR glasses 25 may be different for each player, or may be the same for each player. Note that the multiple environmental displays 50 display a common image (for example, an image of a background landscape in a virtual space) for the multiple players.

[0188] In the above case, the movement and orientation of the object displayed on the transparent display 26 of the AR glasses 25 may be different for each player or may be the same for each player. For example, when multiple players face a common object in the virtual space, the orientation of the object in the virtual space (the direction in which the object faces) may be different for each player so that the object displayed on the transparent display 26 for each player faces that player.

[0189] The game system 1 may be a game system that has only a hearing assistance mode. In this case, the game system 1 does not need to include the speaker 30.

[0190] The game system 1 may be a game system that has only a visual assistance mode. In this case, the game system 1 does not need to include the display 21.

[0191] The voice data does not necessarily need to be prepared in advance. For example, the voice processing unit 106 may synthesize or generate voice data each time based on some data (for example, voice data stored in the storage unit 15) or information.

[0192] The player character may or may not be displayed on the display. In other words, the player character may or may not be included in an image of the virtual space displayed (played back) on the display. For example, when an image of the virtual space is displayed on the display from a first-person perspective, only a virtual hand (virtual body part), which is a part of the player character, may be displayed, or the virtual hand may not be displayed and only a virtual object held by the virtual hand may be displayed.

[0193] The image of the virtual space may be an image from a first-person perspective, or may be an image from a third-person perspective (for example, an image of the player character as seen from behind the player character).

[0194] The multiple speakers 30 may be arranged on the left and right, front and rear, above, etc. of the listener (player) so as to reproduce three-dimensional sound.

[0195] The sensor that outputs a signal indicating the player's movement, including changes in the player's orientation, is not limited to the motion sensor 22 included in the head-mounted display 20. For example, such a sensor may be a combination of a camera (image sensor) that captures an image of a predetermined part of the player's body (e.g., the head), and a recognition processing unit that recognizes the player's movement, including changes in the player's orientation, based on the "image of the predetermined part" obtained by the camera.

[0196] The game program may be implemented as a game program for a so-called online game. In this case, the processing performed by the control unit 16 of the game device 10 may be performed by a control unit of a server device (not shown), or may be shared between the game device 10 and the control unit of the server device. The game device 10 may be a mobile information terminal such as a smartphone or tablet.

[0197] In the above description, a virtual space in which a game progresses (game space) has been given as an example of a virtual space, but the present invention is not limited to this. For example, the virtual space may be a virtual space in which avatars communicate with each other (metaverse space). In other words, the services provided in the virtual space are not limited to games. For example, the services provided in the virtual space may be the provision of a place for commercial transactions, a place for communication, etc.

[0198] The effects of the present invention can be achieved even when these other embodiments are adopted. Furthermore, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate. The above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses. [Explanation of symbols]

[0199] 1 Game System (System) 10. Gaming Devices 11 Network Interface 12 Graphics Processing Unit 13 Audio Processing Section 14 Operation Processing Section 15 Storage section 16 Control Unit 20 Head-mounted display 21 Display 22 Motion Sensor 30 speakers 40 Controller 41 Controls 42 Motion Sensor 50 Environmental Display 60 Environmental Speaker 70 images 71 Text Display Window R1 surrounding area R2 central area 75 Environmental Images 81 Environmental information sign 82 Object-related display 83 Notification display 84 Text Display 85 Notification Icons 86 Display text of surrounding objects 87 Center object text display 91 Environmental explanation audio 92 object-related sounds 93 Guidance voice 94 explanatory audio 95 Notification Sound 100 Virtual space control unit 101 Detection unit 102 Provision Processing Unit 103 Auxiliary control section 105 Display processing unit 106 Audio processing unit

Claims

1. A computer system having one or more computers working together, a detection unit that detects the orientation of the user in the real world; a providing processing unit that performs processing to provide a three-dimensional virtual space to the user; In order to assist the user in experiencing the virtual space, the virtual space control unit provides the user with guide information that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, and functions as an auxiliary control unit that changes the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. program.

2. In the program of claim 1, the providing processing unit includes a display processing unit, the display processing unit performs processing for providing an image of the virtual space to the user; The assistance control unit displays the guide information on a display that follows the orientation of the user in the real world. program.

3. In the program of claim 2, An object is placed in the virtual space, The guide information displayed on the display includes an object-related display associated with the object arranged in the virtual space. program.

4. In the program of claim 3, the object-related display includes a notification display associated with an out-of-field object, which is an object among the objects arranged in the virtual space that is not included in an image of the virtual space corresponding to an orientation of the user in the virtual space; The notification display changes in response to a change in at least one of a distance and a direction from the user's position to the out-of-field object in the virtual space. program.

5. In the program of claim 4, the notification display includes a notification icon; The notification icon is placed on the screen of the display at a position corresponding to the direction from the user's position in the virtual space to the position of the out-of-field object. program.

6. In the program of claim 4, the notification display includes a notification icon; The notification icon is arranged on the screen of the display, and its size changes depending on the distance from the user's position to the out-of-field object's position in the virtual space. program.

7. In the program of claim 3, the object-related display includes a text display associated with a field-of-view object, which is an object included in an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space; The text display shows text corresponding to the voice output from the object within the field of view. program.

8. In the program of claim 7, the text display includes a peripheral object text display associated with a peripheral object that is an in-field object included in a peripheral region of the image of the virtual space according to an orientation of the user in the virtual space; The peripheral object text display is arranged on the screen of the display at a position corresponding to the position of the peripheral object in the virtual space, and indicates text related to the text of the voice output from the peripheral object. program.

9. In the program of claim 7, the text display includes a central object text display associated with a central object, which is the in-field object included in a central region of the image of the virtual space according to the orientation of the user in the virtual space; The central object text display is arranged in a text display window provided at a predetermined position on the screen of the display for displaying text relating to the situation in the virtual space, and indicates the text of the voice output from the central object. program.

10. In the program of claim 1, the provision processing unit includes a voice processing unit, the audio processing unit performs processing to provide audio of the virtual space to the user; The assistance control unit outputs the guide information to a speaker that follows the orientation of the user in the real world. program.

11. In the program of claim 10, The guide information output from the speaker includes an environment description voice for explaining a situation in an image of the virtual space according to an orientation of the user in the virtual space. program.

12. In the program of claim 10, An object is placed in the virtual space, the guide information output from the speaker includes a guidance voice associated with a peripheral object, which is an object included in a peripheral region of an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space; The guiding sound is a sound for guiding the user to change the orientation of the user in the real world so that the peripheral object in the image of the virtual space corresponding to the orientation of the user in the virtual space moves toward a central region of the image. program.

13. In the program of claim 10, An object is placed in the virtual space, the guide information output from the speaker includes an explanatory voice associated with a central object, which is an object included in a central region of an image of the virtual space according to an orientation of the user in the virtual space, among the objects arranged in the virtual space; The explanatory voice is a voice for explaining the situation of the central object. program.

14. In the program of claim 10, An object is placed in the virtual space, the guide information output from the speaker includes a notification sound associated with the object arranged in the virtual space; The notification sound changes in accordance with a change in at least one of a distance and a direction from the user's position to the object's position in the virtual space. program.

15. a detection unit that detects the orientation of the user in the real world; a providing processing unit that performs processing to provide a three-dimensional virtual space to the user; and an auxiliary control unit that provides the user with guide information that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, in order to assist the user in experiencing the virtual space, and that changes the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. system.

16. 1. A method for assisting a user in experiencing a three-dimensional virtual space by a computer system having one or more computers working together, comprising: a detecting step of detecting an orientation of the user in the real world; and an auxiliary step of providing the user with guide information that enables the user to recognize a situation in the virtual space without using the user's hearing or vision, in order to assist the user in experiencing the virtual space, and changing the guide information in accordance with a change in the user's orientation in the virtual space that accompanies a change in the user's orientation in the real world. method.

Images