Information processing device and sidetone control method

Abstract

A game processing unit 110 generates an image of a space in which a player character is present. A sidetone control unit 124 has a function of controlling sidetone in a headset. The sidetone control unit 124 controls the sidetone in the headset on the basis of behavior of the player character.

Description

Information Processing Apparatus and Sidetone Control Method

[0001] The present disclosure relates to a technique for controlling sidetone in a headset.

[0002] In an online multiplayer game, multiple users can play the game while cooperating with each other by having a voice chat. Patent Document 1 discloses an environment in which users wearing headsets have a voice chat in a game session in which multiple users participate.

[0003] Japanese Patent Application Laid-Open No. 2022-75304

[0004] The sidetone function in a headset is a function that outputs the voice of a user speaking toward the microphone from the speaker. Without this sidetone function, a user wearing a headset tends to speak at an unnecessarily high volume because they cannot hear their own voice. By equipping the headset with a sidetone function, the user can hear their own voice during a voice chat and speak at a natural volume.

[0005] As described above, the sidetone function is provided to enable a user wearing a headset to speak naturally. However, the present inventor has focused on this sidetone function and studied a method for controlling the sidetone function in conjunction with a video world such as a game or a movie.

[0006] An information processing apparatus according to an aspect of the present disclosure includes a processing unit that generates an image of a space in which a character exists, and a sidetone control unit that controls sidetone in a headset based on the behavior of the character.

[0007] A sidetone control method according to another aspect of the present disclosure is a method for controlling sidetone in a headset, the method including a step of generating an image of a space in which a character exists, and a step of controlling sidetone in the headset based on the behavior of the character.

[0008] In addition, any combination of the above components, and those obtained by converting the expression of the present disclosure among a method, an apparatus, a system, a recording medium, a computer program, etc., are also effective as aspects of the present disclosure.

[0009] This figure shows an information processing system according to an embodiment. This figure shows the hardware configuration of the information processing device. This figure shows the functional blocks of the information processing device. This figure shows an example of a game image. This figure shows the functional blocks of the headset. (a) and (b) show the positional relationship between the player character and a fixed object. This figure shows the positional relationship between the player character and a fixed object.

[0010] Figure 1 shows an information processing system 1 according to an embodiment. The information processing system 1 comprises multiple information processing devices 10 operated by multiple users and a server system 12, which are connected via a network 3 such as the Internet or a LAN (Local Area Network). An access point (hereinafter referred to as "AP") 8 has the functions of a wireless access point and a router, and the information processing devices 10 are connected to the AP 8 via wireless or wired connections, enabling communication with the server system 12 on the network 3.

[0011] The information processing device 10 is connected wirelessly or via a wired connection to an input device 6 operated by the user, and the input device 6 outputs information operated by the user to the information processing device 10. When the information processing device 10 receives operation information from the input device 6, it reflects it in the processing of system software and application software, and outputs the processing result from the output device 4. In the information processing system 1, the information processing device 10 is a game device (game console) that runs game software, and the input device 6 may be a device such as a game controller that supplies user operation information to the information processing device 10. The input device 6 may also be an input interface such as a keyboard or mouse.

[0012] The auxiliary storage device 2 is a high-capacity storage device such as an HDD (hard disk drive) or SSD (solid-state drive), and may be an external storage device connected to the information processing device 10 by USB (Universal Serial Bus) or the like, or it may be an internal storage device. The output device 4 may be a television having a display for outputting images and a speaker for outputting sound. The output device 4 may be connected to the information processing device 10 by a wired cable or by wireless connection.

[0013] The imaging device, camera 7, is located near the output device 4 and images the space around the output device 4. Figure 1 shows an example where camera 7 is mounted on the top of the output device 4, but it may also be located on the side or bottom of the output device 4, in which case it is positioned to image a user located in front of the output device 4. Camera 7 may be a stereo camera. In this embodiment, the user wears a headset 9 for voice chat. The headset 9 has a microphone and a speaker and may be connected to the information processing device 10 by a wired cable or by wireless connection.

[0014] The server system 12 comprises a management server 14 and a session server 16, and is maintained and managed by the operator of the information processing system 1. Users have a user account that uniquely identifies them, and by signing in to the management server 14 using their user account, they can use the services provided by the management server 14 and the session server 16.

[0015] The session server 16 manages the state of sessions created by the game. The session server 16 manages information about the sessions created by the game, such as the title of the game being played, the user account of the user who created the session (session leader), the user accounts of the users participating in the session, and the duration of the session. When there is a change in the state of the session, the system software of the information processing device 10 sends information indicating that change to the session server 16, and the session server 16 manages the latest state of the session.

[0016] When a user launches a game and selects the game's multiplayer mode, the game creates a session. Other users can then join this session, allowing multiple users to play the game together and use voice chat. There are various ways to join a game session; for example, one method is to send a message to the session leader requesting to join and receive their permission.

[0017] This embodiment describes a case where multiple users engage in voice chat while playing a game. In this embodiment, each user's information processing device 10 is connected to each other via P2P (Peer to Peer), and voice chat is realized by sending and receiving voice signals between the multiple information processing devices 10. In another example, a chat server may be provided in the server system 12, and the multiple information processing devices 10 may send and receive voice signals via the chat server.

[0018] Figure 2 shows the hardware configuration of the information processing device 10. The information processing device 10 consists of a main power button 20, a power-on LED 21, a standby LED 22, a system controller 24, a clock 26, a device controller 30, a media drive 32, a USB module 34, a flash memory 36, a wireless communication module 38, a wired communication module 40, a subsystem 50, and a main system 60.

[0019] The main system 60 includes a main CPU (Central Processing Unit), main memory (memory and memory controller), and a GPU (Graphics Processing Unit). The GPU is primarily used for processing game programs. The main CPU has the function of starting the OS and executing game programs installed on the auxiliary storage device 2 within the environment provided by the OS. The subsystem 50 includes a sub-CPU, main memory (memory and memory controller), and other components, but does not have a GPU.

[0020] The main CPU has the function of executing game programs installed on the auxiliary storage device 2 and the ROM medium 44, while the sub-CPU does not have such functions. However, the sub-CPU has the function of accessing the auxiliary storage device 2 and sending and receiving data with the server system 12. The sub-CPU is configured to have only these limited processing functions and therefore can operate with less power consumption compared to the main CPU. These functions of the sub-CPU are executed when the main CPU is in standby mode.

[0021] The main power button 20 is an input unit that receives user input and is located on the front of the housing of the information processing device 10. It is operated to turn the power supply of the information processing device 10 to the main system 60 on or off. The power-on LED 21 lights up when the main power button 20 is turned on, and the standby LED 22 lights up when the main power button 20 is turned off. The system controller 24 detects when the user presses the main power button 20.

[0022] Clock 26 is a real-time clock that generates current date and time information and supplies it to the system controller 24, subsystem 50, and main system 60.

[0023] The device controller 30 is configured as a Large-Scale Integrated Circuit (LSI) that performs information transfer between devices, similar to a Southbridge. As shown in the figure, devices such as the system controller 24, media drive 32, USB module 34, flash memory 36, wireless communication module 38, wired communication module 40, subsystem 50, and main system 60 are connected to the device controller 30. The device controller 30 absorbs the differences in electrical characteristics and data transfer speeds of each device and controls the timing of data transfer.

[0024] The media drive 32 is a drive device that drives a ROM medium 44 containing application software such as games and license information, and reads programs and data from the ROM medium 44. The ROM medium 44 is a read-only recording medium such as an optical disc, magneto-optical disc, or Blu-ray disc.

[0025] The USB module 34 is a module that connects to external devices via a USB cable. The USB module 34 may also be connected to the auxiliary storage device 2 and the camera 7 via a USB cable. The flash memory 36 is an auxiliary storage device that constitutes the internal storage. The wireless communication module 38 communicates wirelessly with, for example, the input device 6 using a communication protocol such as the Bluetooth® protocol or the IEEE 802.11 protocol. The wired communication module 40 communicates with external devices via a wired connection and connects to the network 3 via, for example, the AP8.

[0026] Figure 3 shows the functional blocks of the information processing device 10. The information processing device 10 of this embodiment comprises a processing unit 100 and a communication unit 102. The processing unit 100 comprises an operation reception unit 104, a game processing unit 110, a chat management unit 120, a chat signal processing unit 122, a side tone control unit 124, and an output processing unit 126. The game processing unit 110 has a game image generation unit 112, a game sound generation unit 114, and a session management unit 116.

[0027] The functions of the components in the information processing device 10 may be realized in a circuit or processing circuitry, including a general-purpose processor, an application-specific processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), a CPU (a Central Processing Unit), a conventional circuit, and / or a combination thereof, configured or programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry that includes transistors and other circuits. A processor may also be a programmed processor that executes a program stored in memory.

[0028] In this specification, circuits, units, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0029] If the hardware is a processor that is considered to be of the type of circuit, the circuit, means, or unit may be a combination of hardware and software used to constitute the hardware and / or processor.

[0030] The communication unit 102 receives operation information from the user operating the input unit of the input device 6 and provides it to the operation reception unit 104. The communication unit 102 also communicates with the management server 14 and the session server 16 to send and receive various information, signals, or data. The communication unit 102 may also have the functions of a wireless communication module 38 and a wired communication module 40.

[0031] The operation reception unit 104 receives operation information from the input device 6 via the communication unit 102. In this embodiment, the game processing unit 110 executes a game program based on the user's operation information to control the movement of the player character in the virtual game space and generate game images and game sounds. The functions shown as the game processing unit 110 are realized by system software, game programs, hardware such as a GPU (Graphics Processing Unit) that executes rendering processes, etc.

[0032] During gameplay by the user, the operation reception unit 104 supplies operation information from the input device 6 to the game processing unit 110, and the game processing unit 110 performs calculations to move the player character in the virtual space based on the operation information. The game image generation unit 112 includes a GPU and, upon receiving the results of the calculations in the virtual space, generates game images from the viewpoint position (virtual camera) in the virtual space.

[0033] The game image generation unit 112 generates an image of the virtual space in which the player character exists. The generated image of the virtual space may or may not include the player character. For example, in the case of an FPS (First Person Shooter) game, the player character itself is not included in the game image. The game sound generation unit 114 generates game sounds within the virtual space. Figure 4 shows an example of a game image displayed on the output device. The output processing unit 126 displays the game image on the output device 4 and outputs game sounds from the headphones of the headset 9.

[0034] When a user selects the game's multiplayer mode, the session management unit 116 creates a session, provided that the connection environment with network 3 is good, and notifies the session server 16 of the session information, including the user's user ID and the game's identification information (game ID). Alternatively, the game may explicitly provide a menu item for "Create Session," and the session management unit 116 may create a session when the user selects this menu item.

[0035] Once a session is created, the session server 16 returns the session identification information (session ID) to the game and manages the session state. If a user allows the session to be made public, the session server 16 makes the session public on the network 3, allowing other users to find and join the session. The information processing devices 10 of multiple users participating in the session may be connected to each other via P2P communication.

[0036] The session management unit 116 maintains the identification information (user IDs) of multiple users participating in a session and manages participating users as members of the session group. User IDs may be user accounts, but they may also be different identification information, such as online IDs used online. The session management unit 116 manages members who newly join the session or members who leave the session and provides the chat management unit 120 with the user IDs of all members constituting the session group. This allows the chat management unit 120 to recognize all users participating in the session. When the chat management unit 120 receives a chat participation request from a user participating in the session, it allows that user to join the chat room.

[0037] In the information processing device 10 operated by the session leader user, the chat management unit 120 notifies the information processing devices 10 of the members participating in the chat of information for identifying the chat partner (hereinafter also referred to as "chat partner identification information"). The information processing devices 10 of other members connected via P2P to the session leader's information processing device 10 can identify the information processing device 10 of the member to which they should send a chat signal by being notified of the chat partner identification information. This enables chat between members. In this embodiment, the form of chat performed is a voice chat, and the information processing device 10 of the chat member transmits the voice signal it has uttered to all other chat members' information processing devices 10, and receives voice signals uttered by other chat members from the information processing devices 10 of the other chat members.

[0038] In each member's information processing device 10, the chat signal processing unit 122 transmits the member's chat signal (audio signal) input to the headset 9's microphone, along with the user ID, to the chat partner's information processing device 10, and also receives chat signals transmitted from the chat partner and supplies them to the output processing unit 126. If there are multiple chat partners, the chat signal processing unit 122 mixes (combines) the chat signals transmitted from the information processing devices 10 of the multiple chat partners and supplies them to the output processing unit 126. During gameplay, the output processing unit 126 transmits an audio signal, which is a mix of the game's audio signal and the chat audio signal, to the headset 9.

[0039] Figure 5 shows the functional block of a headset 9 with sidetone function. The headset 9 comprises a microphone 200, headphones 202, a communication unit 206, a sidetone setting unit 212, and a synthesis unit 214. The headphones 202 have a right speaker 204a for the right ear and a left speaker 204b for the left ear. The communication unit 206 has a transmitter 208 and a receiver 210. The headset 9 may be connected to the information processing device 10 by a wired cable or by wireless connection.

[0040] During a voice chat, the microphone 200 picks up the user's voice and converts it into an electrical signal (voice signal). The transmission unit 208 sends the user's voice signal to the information processing device 10, where the chat signal processing unit 122 acquires the user's voice signal and sends it to the information processing device 10 of the chat partner.

[0041] The headset 9 of this embodiment has a side tone function, and when the side tone setting unit 212 receives the user's voice signal, it sets the volume and supplies it to the synthesis unit 214 as a side tone signal. As will be described later, the side tone setting unit 212 sets the volume of the side tone signal by attenuating or amplifying the user's voice signal based on the control signal provided by the information processing device 10.

[0042] The receiving unit 210 receives a sound signal obtained by mixing a game sound signal and a chat voice signal transmitted from the information processing device 10, and supplies it to the synthesizing unit 214. The synthesizing unit 214 synthesizes the received sound signal and the sidetone signal whose volume has been adjusted, and supplies the synthesized signal to the right speaker 204a and the left speaker 204b. The right speaker 204a and the left speaker 204b convert the supplied synthesized signal into sound, and output a sound in which game sounds, chat voices from the chat partner, and sidetone are mixed.

[0043] In the embodiment, the volume of the sidetone of the headset 9 is controlled according to the situation in the game world. Specifically, the sidetone control unit 124 in the information processing device 10 adjusts the volume of the sidetone in the headset 9 based on the behavior of the player character in the game virtual space.

[0044] The sidetone control unit 124 determines the volume of the sidetone in the headset 9 based on the behavior of the player character in the game virtual space, and transmits a control signal indicating the determined volume to the headset 9. The control signal may be a signal indicating a volume value to be increased or decreased from the current volume. For example, when the player character is stationary, the sidetone control unit 124 may determine to increase the sidetone volume, while when the player character is moving, the sidetone control unit 124 may determine to decrease the sidetone volume. For example, when the player character is running, the sidetone control unit 124 may decrease the sidetone volume as the running speed increases and increase the sidetone volume as the running speed decreases. As the movement of the player character becomes more intense, the sidetone volume decreases, so that the user can feel the intensity of the movement of the player character.

[0045] In the headset 9, the receiving unit 210 receives the control signal and supplies the received control signal to the sidetone setting unit 212. When receiving the control signal, the sidetone setting unit 212 amplifies or attenuates the user's voice signal so that the volume corresponds to the control signal, and outputs a sidetone signal with the adjusted volume to the synthesizing unit 214. As a result, the user can feel that the sidetone volume changes according to the behavior of the player character, and the sense of immersion in the game can be enhanced.

[0046] FIGS. 6(a) and 6(b) show the state when overlooking the game virtual space, and show the positional relationship between the player character and the fixed object as seen from above the space. In FIGS. 6(a) and 6(b), the fixed object is an object fixedly arranged in the game virtual space, and its height may be higher than the height of the player character. For example, the fixed object may be a wall erected vertically from the floor or ground on which the player character stands, and the player character stands facing the wall.

[0047] FIG. 6(a) shows the state where the distance between the player character and the wall is L1, and FIG. 6(b) shows the state where the player character approaches the wall and the distance between the player character and the wall shrinks to L2 (<L1). For example, in a combat game, assume a scene where the player character approaches a window fitted in the wall to check outside the wall. The game processing unit 110 identifies the distance between the player character and the fixed object, and provides the identified distance to the side sound control unit 124. The side sound control unit 124 may determine the volume of the side sound based on the distance between the player character and the fixed object.

[0048] The side sound control unit 124 may increase the volume of the side sound as the distance between the player character and the fixed object becomes closer. Also, the side sound control unit 124 may decrease the volume of the side sound as the distance between the player character and the fixed object becomes farther. Also in the real world, as the person making a sound approaches the wall, the reflected sound from the wall becomes larger, and as the person moves away from the wall, the reflected sound from the wall becomes smaller. Therefore, by the side sound control unit 124 determining the side sound volume according to the distance between the player character and the fixed object, the user can feel the distance between the player character and the fixed object by the volume of the side sound. For example, when it is dark around the wall and it is difficult to visually tell the distance between the player character and the wall, the user's ability to feel the distance by the side sound volume can have a great impact on the game play.

[0049] The side sound control unit 124 may have the function of separately adjusting the volume of the side sound in the right speaker 204a and the left speaker 204b of the headset 9. The game processing unit 110 identifies the orientation of the player character relative to a fixed object and provides the identified orientation to the side sound control unit 124. The side sound control unit 124 may determine the volume of the side sound in the right speaker 204a and the left speaker 204b based on the orientation of the player character relative to the fixed object.

[0050] Figure 7 shows a situation where the player character is not directly facing a fixed object. In this example, since the fixed object is to the left of the player character, the player character's left ear is closer to the fixed object than their right ear. Therefore, the side sound control unit 124 determines the side sound volume for the left speaker 204b to be greater than the side sound volume for the right speaker 204a. Similarly, if the fixed object is to the right of the player character, the side sound control unit 124 may determine the side sound volume for the right speaker 204a to be greater than the side sound volume for the left speaker 204b.

[0051] The side sound control unit 124 may determine the side sound volume not only based on the orientation of the player character, but also based on the distance between the player character's left and right ears and a fixed object. For example, when the player character is walking down a narrow corridor, the side sound control unit 124 may determine the side sound volume in the left speaker 204b based on the distance between the left wall of the corridor and the player character's left ear, and determine the side sound volume in the right speaker 204a based on the distance between the right wall of the corridor and the player character's right ear. In this way, the side sound control unit 124 may determine the side sound volume for each ear based on the distance between each ear and the nearest fixed object.

[0052] The present disclosure has been described above based on embodiments. These embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications are also within the scope of the present disclosure.

[0053] In this embodiment, the side tone control unit 124 adjusts the volume of the side tone in the headset 9. However, in this modified example, the parameters of an equalizer that adjusts the frequency characteristics of the side tone in the headset 9 may be determined. For example, the side tone control unit 124 may emphasize the high frequencies of the side tone and attenuate the low frequencies as the distance between the player character and the fixed object decreases, and emphasize the low frequencies of the side tone and attenuate the high frequencies as the distance between the player character and the fixed object increases. Alternatively, the side tone control unit 124 may emphasize the low frequencies of the side tone and attenuate the high frequencies as the distance between the player character and the fixed object decreases, and emphasize the high frequencies of the side tone and attenuate the low frequencies as the distance between the player character and the fixed object increases. By changing the frequency characteristics of the side tone, the user can perceive that the distance to the fixed object is changing.

[0054] In this embodiment, the game processing unit 110 controls the character's movements in the virtual space based on user operation information. In a modified example, the game processing unit 110 may function as a content processing unit that plays a video. In this case, the content processing unit generates an image of the space in which the character exists and provides the side sound control unit 124 with information (for example, distance) indicating the relative positional relationship between the character and a fixed object, and the side sound control unit 124 may control the side sound in the headset 9 based on the character's behavior.

[0055] This disclosure may include the following embodiments: [Item 1] An information processing device comprising a circuit configured to the following: The circuit is an information processing device that generates an image of a space in which a character exists, and controls side sound in a headset based on the behavior of the character. [Item 2] The information processing device according to Item 1, wherein the circuit identifies the distance between a character and a fixed object in space, and controls side sound in a headset based on the distance between the character and the fixed object. [Item 3] The information processing device according to Item 2, wherein the circuit determines the volume of side sound in a headset. [Item 4] The information processing device according to Item 3, wherein the circuit increases the volume of side sound as the distance between the character and the fixed object decreases. [Item 5] The information processing device according to Item 3, wherein the circuit decreases the volume of side sound as the distance between the character and the fixed object increases. [Item 6] The information processing device according to Item 2, wherein the circuit determines the parameters of an equalizer that adjusts the frequency characteristics of side sound in a headset. [Item 7] The information processing device according to Item 1, wherein the circuit controls the side sound in the right speaker and the side sound in the left speaker of the headset, respectively. [Item 8] The information processing device according to Item 7, wherein the circuit identifies the orientation of a character relative to a fixed object and controls the side sound in the right speaker and the side sound in the left speaker, respectively, based on the orientation of the character relative to the fixed object. [Item 9] The information processing device according to Item 1, wherein the circuit controls the movement of a character in a virtual space based on user operation information. [Item 10] A method for controlling side sound in a headset, comprising generating an image of the space in which a character exists and controlling the side sound in the headset based on the behavior of the character.[Item 11] A recording medium that stores a program executed on a computer, wherein the program enables the computer to perform the following functions: generate an image of a space in which a character exists, and control the side sound in a headset based on the behavior of the character.

[0056] This disclosure can be used in technology for controlling side sounds in headsets.

[0057] 1... Information processing system, 9... Headset, 10... Information processing device, 100... Processing unit, 102... Communication unit, 104... Operation reception unit, 110... Game processing unit, 112... Game image generation unit, 114... Game sound generation unit, 116... Session management unit, 120... Chat management unit, 122... Chat signal processing unit, 124... Side tone control unit, 126... Output processing unit, 200... Microphone, 202... Headphones, 204a... Right speaker, 204b... Left speaker, 206... Communication unit, 208... Transmission unit, 210... Receiving unit, 212... Side tone setting unit, 214... Synthesis unit.

Claims

1. An information processing device comprising: a processing unit that generates an image of a space in which a character exists; and a side sound control unit that controls side sounds in a headset based on the behavior of the character.

2. The information processing apparatus according to claim 1, wherein the processing unit determines the distance between a character and a fixed object in space, and the side sound control unit controls the side sound in the headset based on the distance between the character and the fixed object.

3. The information processing apparatus according to claim 2, characterized in that the side sound control unit determines the volume of side sounds in the headset.

4. The information processing apparatus according to claim 3, characterized in that the side tone control unit increases the volume of the side tone as the distance between the character and the fixed object decreases.

5. The information processing apparatus according to claim 3, characterized in that the side tone control unit reduces the volume of the side tone as the distance between the character and the fixed object increases.

6. The information processing apparatus according to claim 2, characterized in that the side tone control unit determines the parameters of an equalizer that adjusts the frequency characteristics of side tones in the headset.

7. The information processing apparatus according to claim 1, characterized in that the side sound control unit controls the side sound from the right speaker and the side sound from the left speaker of the headset, respectively.

8. The information processing apparatus according to claim 7, wherein the processing unit identifies the orientation of the character relative to a fixed object, and the side sound control unit controls the side sound in the right speaker and the side sound in the left speaker, respectively, based on the orientation of the character relative to the fixed object.

9. The information processing apparatus according to claim 1, characterized in that the processing unit controls the movement of a character in a virtual space based on user operation information.

10. A method for controlling side sounds in a headset, comprising the steps of: generating an image of the space in which a character exists; and controlling the side sounds in the headset based on the behavior of the character.

11. A program to enable a computer to generate an image of the space in which a character exists, and to control the side sound in a headset based on the character's behavior.

Images