Program and video / audio control device

Abstract

To alleviate a feeling of discomfort of a sound generated when interlocking a virtual microphone with a virtual camera.SOLUTION: A camera control part moves a virtual camera C on the basis of a motion signal according to a user's motion. A sound collection point control part executes interlocking control for interlocking a virtual microphone M with the virtual camera C so as to maintain predetermined positional relationships with respect to the position of the virtual camera C, and invasion avoidance control to maintain the position of the virtual microphone M at the position before the invasion to an invasion impossible area when the virtual microphone M invades the invasion impossible area by the interlocking control.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure relates to a program and an audio-visual control device.

Background Art

[0002] Patent Document 1 discloses a game system that realizes a virtual three-dimensional space in which a plurality of sound source objects, virtual microphones, and virtual cameras are arranged. In this game system, an audio signal is generated based on the volume and localization of each sound when the sound reproduced from a plurality of sound source objects is received by a virtual microphone. Also, the virtual camera is arranged at the same position as the virtual microphone in the virtual three-dimensional space, and an image captured by the virtual camera is displayed on a monitor. The virtual camera and the virtual microphone move as the player character moves.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a game system such as Patent Document 1, when a boundary line that divides a part or all of the space into two regions is provided at a position corresponding to a wall in the virtual three-dimensional space, and a virtual microphone is in one of the two regions divided by the boundary line, there is a technique for reducing the volume of the sound generated in the other region. Thereby, it is possible to make the user feel as if the sound is blocked by a wall.

[0005] By the way, in such a game system, when the virtual camera penetrates inside a thick wall, the virtual microphone moves in conjunction with the virtual camera across the boundary line, and the sound in the region where the virtual microphone was arranged before crossing the boundary line suddenly becomes inaudible, which may cause a sense of discomfort.

[0006] The purpose of this disclosure is to mitigate the unnatural sound that can occur when a virtual microphone is linked to a virtual camera. [Means for solving the problem]

[0007] The first aspect is a computer system having one computer or multiple cooperating computers, A video playback unit that plays back images obtained from a virtual camera placed in a virtual space on a display, A camera control unit that moves the virtual camera based on an operation signal corresponding to the user's actions, A sound playback unit that plays back sound obtained by a virtual microphone placed in the virtual space to a speaker, The virtual microphone is configured to function as a sound collection point control unit that performs linked control to maintain a predetermined positional relationship with the virtual camera, and intrusion avoidance control to maintain the position of the virtual microphone at its position before entering a predetermined inaccessible area if the linked control causes the virtual microphone to enter the inaccessible area. It is a program.

[0008] In the first embodiment, The aforementioned inaccessible area may be an area that cannot be entered by a player character.

[0009] The second embodiment includes a storage unit that stores the program of the first or second embodiment, A control unit that executes the aforementioned program, This is a video and audio control device equipped with [features / equipment]. [Effects of the Invention]

[0010] According to this disclosure, it is possible to mitigate the unnatural sound caused by linking a virtual microphone with a virtual camera. [Brief explanation of the drawing]

[0011] [Figure 1]This is a block diagram illustrating the configuration of the game device according to the embodiment. [Figure 2] This is a schematic diagram illustrating the position of the virtual microphone when performing intrusion avoidance control. [Figure 3] This is a flowchart illustrating the processing of the sound collection point control unit. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals, and their descriptions will not be repeated.

[0013] (Game description) In the following description, "game" refers to a game (video game) played in a virtual space. The virtual space is a three-dimensional game space, displayed on a screen as an image based on VR (Virtual Reality) technology. In the following example, objects are placed within the virtual space. Examples of objects include player characters controlled by the user and non-player characters controlled by the computer. Furthermore, partitions (obstacles) such as walls, rocks, and doors are placed within the virtual space, dividing part or all of the virtual space into two areas (spaces). Of the two areas separated by the partition, the area without a player character becomes an area that cannot be entered by the player character.

[0014] Furthermore, objects include sound source objects that act as virtual sound sources within the virtual space. Examples of sound source objects include objects that operate within the virtual space, and objects that only output sound within the virtual space.

[0015] In the game described below, sounds are played. The sounds played in the game include sounds output from sound source objects, BGM (Background Music), sound effects, and the like. Examples of sounds output from sound source objects include sounds associated with the movement of objects such as footsteps and wind sounds, and sounds emitted by objects such as cries and lines.

[0016] Below, for the sake of convenience of explanation, as an example of a game, a "fighting action game in which a player character attacks and defeats enemy characters" is given. The enemy character is an example of a non-player character. The player character and the enemy character are examples of sound source objects. Note that there is no limitation on the type of game.

[0017] (Game device) FIG. 1 illustrates the configuration of the game device 10 according to the embodiment. The game device 10 executes a game in response to an operation by a user. A display 21, a speaker 22, and a controller 23 are externally connected or built in to the game device 10. The game device 10 is an example of a computer system having one computer or a plurality of cooperating computers. Also, the game device 10 is an example of a video and audio control device.

[0018] Commercially available devices such as, for example, a personal computer, PlayStation (registered trademark), XBox (registered trademark), PlayStation Vita (registered trademark), Nintendo Switch (registered trademark), etc. can be used as the game device 10.

[0019] In the game device 10, the game progresses based on the installed game program and game data. The game devices 10 can also perform data communication with each other using a communication network (not shown) or a short-range wireless communication device (not shown).

[0020] 〔Hardware configuration of game device〕 The game device 10 includes a network interface 11, a graphics processing unit 12, an audio processing unit 13, an operation unit 14, a storage unit 15, and a control unit 16. The network interface 11, graphics processing unit 12, audio processing unit 13, operation unit 14, and storage unit 15 are electrically connected to the control unit 16 via a bus 17.

[0021] The network interface 11 is connected to a communication network (not shown) so as to enable communication in order to send and receive various data with, for example, other game devices 10 or external server devices (not shown).

[0022] The graphics processing unit 12 is connected to the display 21 (for example, an LCD display). The graphics processing unit 12 renders game images, including various objects in the virtual space, in video format according to the game image information output from the control unit 16. The game images rendered in video format are displayed on the display 21 as the game screen.

[0023] The audio processing unit 13 is connected to the speaker 22. The audio processing unit 13 plays digital game sounds according to the instructions of the control unit 16. Specifically, the audio processing unit 13 converts the audio data output by the control unit 16 into an analog signal and outputs it to the speaker 22.

[0024] The audio processing unit 13 is configured to provide multi-channel audio output in order to reproduce three-dimensional sound. Accordingly, multiple speakers 22 are connected to the game device 10. These speakers 22 may be positioned to the left, right, front, back, and above the listener (e.g., the game player) to enable three-dimensional sound reproduction.

[0025] The control unit 14 is connected to the controller 23. The control unit 14 sends and receives operation signals to and from the controller 23 in accordance with the actions (operations) of the user (i.e., the game player). The game player inputs operation signals (commands, data, etc.) to the game device 10 by operating various controls such as buttons on the controller 23. The controller 23 includes multiple controls and can transmit operation signals to the control unit 14 in accordance with the movements of the user's head and body. For example, if the user takes a bent-over posture, that posture will be reflected in the operation signal.

[0026] The storage unit 15 stores various types of information and data. The storage unit 15 is composed of HDD, SSD, RAM, ROM, etc. For example, the storage unit 15 stores various programs, including game data and game programs. Examples of game data include game media and user account information.

[0027] Furthermore, the memory unit 15 stores various types of pre-prepared audio data. These types of audio data include audio data indicating sounds output from sound source objects such as player characters and enemy characters, audio data indicating music (BGM) played in the game, and audio data indicating sound effects.

[0028] The control unit 16 controls the operation of the game device 10. The control unit 16 transmits and receives various information and data, and processes various information and data. The control unit 16 has a CPU (microcomputer) and semiconductor memory. The semiconductor memory stores programs and data for operating the CPU.

[0029] [Functional configuration of the control unit] The control unit 16 includes a game progression unit 100, a camera control unit 101, a video playback unit 102, a sound collection point control unit 103, and an audio playback unit 104. Specifically, the control unit 16 functions as the game progression unit 100, the camera control unit 101, the video playback unit 102, the sound collection point control unit 103, and the audio playback unit 104 by executing a game program.

[0030] <Game Management Team> The game progress unit 100 advances the game being played in the virtual space in accordance with the operation signals received by the operation unit 14.

[0031] In this example, the game progression unit 100 advances the game by moving the player character C1 (see Figure 2) in the virtual space in response to the operation signal. For example, the game progression unit 100 moves the player character C1 in the virtual space in response to the operation signal. The game progression unit 100 also moves non-player characters and predetermined objects (such as vehicles) in the virtual space. For example, the game progression unit 100 controls the movements of non-player characters and predetermined objects using AI (artificial intelligence). The game progression unit 100 then advances the game in response to the movements of the player character and non-player characters.

[0032] <Camera Control Unit> The camera control unit 101 controls the position of the virtual camera C (see Figure 2) placed in the virtual space in accordance with the game progress made by the game progress unit 100. The virtual camera C is a virtual viewer (viewpoint) and is a reference point for reproducing the viewpoint of the player character C1. The virtual camera C is directional.

[0033] In this example, the camera control unit 101 positions the virtual camera C near the player character C1. Specifically, the virtual camera C is positioned at the head of the player character C1. The camera control unit 101 then controls the position of the virtual camera C so that it moves in accordance with the movement of the player character C1.

[0034] As described above, the game progress unit 100 moves the player character C1 in accordance with the operation signal. Therefore, the camera control unit 101 also moves the virtual camera C based on the operation signal.

[0035] <Video Playback Section> The video playback unit 102 generates video data according to the progress of the game. The video playback unit 102 then outputs the video data to the graphics processing unit 12. As a result, video (game images in video format) corresponding to the progress of the game is displayed on the display 21.

[0036] Furthermore, the video playback unit 102 generates video data including the video obtained by the virtual camera C and outputs this video data to the graphics processing unit 12. As a result, the video playback unit 102 displays the video, including the video obtained by the virtual camera C (for example, the video seen from the perspective of the player character C1), on the display 21.

[0037] <Sound Collection Point Control Section> The sound collection point control unit 103 controls the position of the virtual microphone M (see Figure 2) placed in the virtual space according to the progress of the game. The virtual microphone M is a virtual listener (sound collection point) and is the reference point for sound image localization by the sound playback unit 104. The virtual microphone M is directional.

[0038] In this example, the sound collection point control unit 103 normally performs linked control to synchronize the virtual microphone M with the virtual camera C so that the position of the virtual microphone M is the same as the position of the virtual camera C. In other words, the sound collection point control unit 103 normally controls the position of the virtual microphone M so that it moves in accordance with the movement of the virtual camera C. On the left side of Figure 2, the virtual microphone M and the virtual camera C are shown slightly separated for illustrative purposes, but in reality, the virtual microphone M and the virtual camera C are points at the same location.

[0039] <Audio Playback Unit> The audio playback unit 104 generates audio data according to the progress of the game. Specifically, the audio playback unit 104 selects the audio data to be played from the audio data (various pre-prepared audio data) stored in the memory unit 15 according to the progress of the game, and outputs the selected audio data (or new audio data obtained by combining multiple selected audio data) to the audio processing unit 13.

[0040] When the audio playback unit 104 outputs audio data, it instructs the audio processing unit 13 on the volume. The audio processing unit 13 converts the audio data output from the audio playback unit 104 into an analog signal of the instructed volume and outputs it to the speaker 22. As a result, the speaker 22 plays a sound corresponding to the audio data generated by the audio playback unit 104.

[0041] Sound localization processing Furthermore, the audio playback unit 104 performs sound image localization processing. In sound image localization processing, the audio playback unit 104 generates audio data that includes sound obtained from a virtual microphone M (see Figure 2) placed in the virtual space.

[0042] Specifically, in sound image localization processing, the audio playback unit 104 generates audio data according to the sound collection conditions of the virtual microphone M (for example, the position and direction of the virtual microphone M relative to the sound source object in the virtual space, the volume and direction of the sound output from the sound source object, etc.) so that the sound output from the speaker 22 includes the sound obtained by the virtual microphone M.

[0043] The audio data generated by the sound image localization process is processed in the audio processing unit 13, making it possible to output sound from a virtual sound source (sound source object) in the virtual space to the speaker 22 with an acoustic representation as if it were collected by a virtual microphone M. In this way, the audio playback unit 104 reproduces the sound obtained by the virtual microphone M through the speaker 22.

[0044] Furthermore, as shown in Figure 2, a boundary line L is set at a position corresponding to the wall W in the virtual space, dividing part or all of the space into two regions R1 and R2. If a virtual microphone M is located in one of the two regions R1 and R2 separated by the boundary line L, the sound playback unit 104 reduces the volume of sound produced in the other region. This makes the user feel as if the sound is being blocked by the wall W. Note that the boundary line L is not limited to the wall W; it may also be set at a position corresponding to other partitions (obstacles) such as rocks or doors.

[0045] Of the two regions R1 and R2 separated by boundary line L, the region where the player character C1's body (torso) is not located is a region that the virtual camera C can enter, but the player character C1's body cannot enter. In the example in Figure 2, since the player character C1's body is in region R2, the player character C1's body cannot enter region R1. More specifically, the determination of whether or not the player character C1 has hit wall W is made based on the position of the center point of the player character C1's body. Therefore, the player character C1's body cannot enter inside wall W and region R1, but the player character C1's head and virtual camera C can enter inside wall W and region R1.

[0046] [Virtual camera intrusion into the wall] Next, referring to Figure 2, we will explain the control of the position of the virtual microphone M when the virtual camera C enters the interior of wall W. Note that if a boundary line L is set at a location corresponding to a partition (shield) other than wall W, and the virtual camera C enters the interior of that partition (shield), the same control as described below will be performed.

[0047] As shown in Figure 2, the camera control unit 101 may cause the virtual camera C to enter the wall W based on an operation signal corresponding to the user's operation.

[0048] The sound collection point control unit 103 normally performs linked control to synchronize the virtual microphone M with the virtual camera C so that the virtual microphone M is in the same position as the virtual camera C.

[0049] However, for example, as shown on the left side of Figure 2, if the player character C1, virtual camera C, and virtual microphone M are in area R2, and the player character C1 presses its head against wall W, the virtual camera C enters the interior of wall W, crosses the boundary line L, and enters area R1. At this time, if the virtual microphone M crosses the boundary line L in conjunction with the virtual camera C and enters area R1, the sound from area R2 where the virtual microphone M was located before crossing the boundary line L suddenly stops playing, creating an unnatural feeling.

[0050] Therefore, in this embodiment, as shown on the right side of Figure 2, when the linked control causes the virtual microphone M to enter region R1 (the position of the virtual microphone M shown by the virtual line), the sound collection point control unit 103 performs intrusion avoidance control to maintain the position of the virtual microphone M at the position before it entered region R1, i.e., region R2.

[0051] [Sound collection point control processing] Next, with reference to Figure 3, the processing of the sound collection point control unit 103 will be explained.

[0052] <Step S11> When a user performs a predetermined operation (action) on the controller 23, the sound collection point control unit 103 determines whether moving the virtual microphone M by the interlocking control in response to the operation signal corresponding to that operation would cause the virtual microphone M to enter a predetermined inaccessible area. For example, raycasting can be used for this determination. If the virtual microphone M would enter the predetermined inaccessible area, the process proceeds to step S12. On the other hand, if the virtual microphone M does not enter the predetermined inaccessible area, the process proceeds to step S13. Here, the inaccessible area is the other area when the virtual microphone M is originally in one of two areas R1 and R2 separated by a boundary line L, i.e., the area where the volume of the reproduced sound is set to 0. For example, in the example in Figure 2, the virtual microphone M is originally in area R2, so the inaccessible area is area R1.

[0053] <Step S12> In step S12, the sound collection point control unit 103 performs intrusion avoidance control to maintain the position of the virtual microphone M at the position it was at before entering the inaccessible area. In other words, the sound collection point control unit 103 maintains the position of the virtual microphone M in the area where the virtual microphone M originally was, among the two areas R1 and R2 separated by the boundary line L. For example, in the example in Figure 2, the position of the virtual microphone M is maintained in area R2. More specifically, as shown in Figure 2, the position of the virtual microphone M is maintained at the position it was at just before entering the wall W.

[0054] <Step S13> Next, in step S13, the sound collection point control unit 103 moves the virtual microphone M by linked control in response to the operation signal corresponding to the user's operation. In other words, the sound collection point control unit 103 moves the virtual microphone M to the same position as the virtual camera C.

[0055] [Summary of Embodiments] To summarize, the game program of the embodiment functions as a sound collection point control unit 103 that controls the computer (control unit 16) by performing: a video playback unit 102 that plays images obtained by a virtual camera C placed in the virtual space on a display 21; a camera control unit 101 that moves the virtual camera C based on an operation signal corresponding to the user's actions; an audio playback unit 104 that plays sounds obtained by a virtual microphone M placed in the virtual space on a speaker 22; and an interlocking control unit 103 that performs interlocking control to link the virtual microphone M with the virtual camera C to maintain a predetermined positional relationship with the position of the virtual camera C, and intrusion avoidance control to maintain the position of the virtual microphone M at the position it was at before entering the intrusion area if the interlocking control causes the virtual microphone M to enter a predetermined intrusion-free area.

[0056] The game device 10 of this embodiment includes a storage unit 15 for storing the above-mentioned game program and a control unit 16 for executing the above-mentioned game program.

[0057] [Effects of the Embodiment] As described above, in this embodiment, when the virtual microphone M would enter a predetermined inaccessible area due to linked control, the position of the virtual microphone M is maintained at the position it was in before entering the inaccessible area. Therefore, by designating the area where the volume of the reproduced sound is set to 0 as the inaccessible area, it is possible to suppress the virtual microphone M from entering the area where the volume of the reproduced sound is set to 0 through linked control. Thus, sudden changes in volume can be suppressed, and the feeling of unnaturalness in sound can be mitigated.

[0058] (Other embodiments) In the above embodiment, the positional relationship between the virtual camera C and the virtual microphone M maintained during linked control was set so that the virtual camera C and the virtual microphone M are in the same position. However, the virtual microphone M may be set to be separated from the virtual camera C by a predetermined distance greater than 0 in a predetermined direction. For example, the virtual camera C may be at the position of the player character C1's eyes, and the virtual microphone M may be at the position of the player character C1's ears.

[0059] Furthermore, in the above description, the game program may be implemented as a game program for so-called online games. In this case, the processing performed in the control unit 16 of the game device 10 may be performed in the control unit of the server device (not shown), or it may be shared between the control unit of the game device 10 and the control unit of the server device. The game device 10 may be a portable information terminal such as a smartphone or tablet.

[0060] Furthermore, while the above explanation uses a virtual space where a game takes place as an example of a virtual space, it is not limited to this. For example, a virtual space could be a virtual space where avatars communicate with each other (a metaverse space). For example, the services provided in a virtual space could be the provision of a platform for commercial transactions, the provision of a platform for communication, etc.

[0061] The effects and benefits of the present invention will also be achieved when these other embodiments are adopted. Furthermore, it is possible to combine these embodiments with other embodiments, and with other embodiments as appropriate. The above embodiments are essentially preferred examples and are not intended to limit the scope of the present invention, its applications, or its uses. [Explanation of symbols]

[0062] 10. Game equipment (computer system, video and audio control device) 11 Network Interfaces 12 Graphics Processing Unit 13 Audio Processing Unit 14 Control section 15 Storage section 16 Control Unit 17 Bus 21 displays 22 speakers 23 Controllers 100 Game Management Department 101 Camera Control Unit 102 Video Playback Unit 103 Sound Collection Point Control Unit 104 Audio playback unit C Virtual Camera C1 Player Character M Virtual Microphone

Claims

1. A computer system having one computer or multiple cooperating computers A video playback unit that plays back images obtained from a virtual camera placed in a virtual space on a display, A camera control unit that moves the virtual camera based on an operation signal corresponding to the user's actions, A sound playback unit that plays back sound obtained by a virtual microphone placed in the virtual space to a speaker, The virtual microphone functions as a sound collection point control unit that performs interlocking control to link with the virtual camera to maintain a predetermined positional relationship with the virtual camera's position, and intrusion avoidance control to maintain the position of the virtual microphone, which would otherwise enter the intrusion-restricted area, at its position before entering the intrusion-restricted area when the camera control unit causes the virtual camera to enter a predetermined intrusion-restricted area. program.

2. In the program of claim 1, The virtual camera is positioned at the head of the player character. The program is characterized in that the inaccessible area is an area that the player character's head and the virtual camera can enter, but the player character's body cannot enter.

3. A storage unit for storing the program according to claim 1 or 2, The system comprises a control unit that executes the aforementioned program. Video and audio control device.

Images