Sound control system and program

Abstract

To provide a new technology to output a sound according to a state of content.SOLUTION: A server system controls a sound output by content that is playable or viewable. Using a sound determination AI (artificial intelligence) 401, 403, and 405 that has learned to generate sound control information according to a state of content being played, the sever system acquires application sound control information according to the progress state of the content being progressed by a user. The server system executes output control for a sound for the user on the basis of the application sound control information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a sound control system that controls sounds output from content. [Background technology]

[0002] Patent Document 1 discloses a technology that reflects the status of a character in a game in the volume of the sound that is output. The technology in Patent Document 1 makes it possible to control the volume of each melody that is output when content such as a game is played, depending on the status of the content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-17402 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology of Patent Document 1, the melody of the sound to be output is preset, and only the volume changes depending on the game situation. If it were possible to change not only the volume but also the output sound itself depending on the game situation, it would be possible to further increase the interest. Furthermore, if it were possible to change the sound not only when playing a game but also when watching a game commentary video depending on the game situation, it is believed that the interest would be similarly increased. The same applies to content other than games, such as content that allows users to virtually experience virtual spaces such as the Metaverse.

[0005] The problem to be solved by the present invention is to provide a new technology that makes it possible to output sound according to the situation of the content. [Means for solving the problem]

[0006] The first invention for solving the above problem is a sound control system that controls the sound output from playable or viewable content, and includes: an applicable sound control information acquisition means (e.g., the progress determination unit 310 and the AI ​​execution unit 320 in Figure 4) that acquires applicable sound control information according to the progress of the content being played or viewed by the user using a sound determination AI (Artificial Intelligence) that has been trained to determine sound control information according to the status of the content being played or viewed; and a sound output control means (e.g., the sound determination unit 350 in Figure 4) that controls the output of sound directed to the user based on the applicable sound control information.

[0007] "Learning" refers to machine learning, and more specifically, deep learning. According to the first invention, applicable sound control information according to the progress of the content being played or viewed by the user can be obtained from the sound determination AI, and sound output control for the user can be performed. Sound can be output according to the content situation.

[0008] A second invention is a sound control system in which, in the above invention, the sound determination AI is an AI that has been trained for multiple pieces of content based on sound control information that corresponds to the situation of the content.

[0009] According to the second aspect of the present invention, it is possible to control the output of sound for ongoing content using a sound determination AI that has been trained on a plurality of content items.

[0010] A third invention is a sound control system in which, in the above invention, the sound determination AI is an AI that has learned about content other than the content being progressed by the user based on sound control information corresponding to the situation of the content.

[0011] According to the third aspect of the present invention, it is possible to control the output of the sound of the ongoing content by using the sound determination AI that has been trained on content other than the ongoing content.

[0012] A fourth invention is a sound control system in which, in the above invention, the sound determination AI is an AI that re-learns based on the progress status and the applied sound control information.

[0013] According to the fourth invention, sound output control can be performed using a sound determination AI that has been re-learned based on the progress of the ongoing content and the applied sound control information obtained from the sound determination AI.

[0014] A fifth invention is a sound control system in which, in the above invention, the applied sound control information acquisition means has an input information generation means (e.g., progress determination unit 310 of Figure 4) that generates progress input information to be input to the sound determination AI based on the progress of the content being progressed by the user.

[0015] According to the fifth aspect of the present invention, progress input information can be generated based on the progress of the ongoing content and input to the sound determination AI.

[0016] A sixth invention is a sound control system in which, in the above invention, the sound determination AI is an AI that analyzes annotation target information from the context of the content and determines the sound control information based on the analysis results.

[0017] According to the sixth aspect of the present invention, sound output control can be performed using a sound determination AI that determines sound control information based on the analysis result of annotation target information.

[0018] The seventh invention is a sound control system in which, in the above invention, there are multiple types of sound determination AI, and the system further includes an application AI selection means (e.g., application AI selection unit 360 in Figure 10) that selects an application AI from the multiple types of sound determination AI, and the application sound control information acquisition means acquires the application sound control information from the application AI.

[0019] According to the seventh aspect of the present invention, the applicable sound control information is acquired from the applicable AI selected from among a plurality of types of sound determination AI, and sound output control can be performed.

[0020] The eighth invention is a sound control system in which, in the above invention, there are multiple types of sound determination AI, the applied sound control information acquisition means acquires the applied sound control information from each of the multiple types of sound determination AI, and the sound output control means determines final applied sound control information based on the applied sound control information acquired from the multiple types of sound determination AI, and controls the sound output based on the final applied sound control information.

[0021] According to the eighth aspect of the present invention, final applicable sound control information is determined based on applicable sound control information acquired from a plurality of types of sound determination AI, and sound output control can be performed.

[0022] A ninth invention is a sound control system in the above invention, wherein the plurality of types of sound determination AI are AIs trained with different training data.

[0023] According to the ninth aspect of the present invention, sound output control can be performed by using one of multiple types of sound determination AIs trained with different learning data as an applicable AI, or by determining final applicable sound control information based on the applicable sound control information acquired from the multiple types of sound determination AIs, and performing sound output control.

[0024] A tenth aspect of the invention is the sound control system according to the above-mentioned invention, wherein the plurality of types of sound determination AI are AIs corresponding to the types of sounds.

[0025] According to the tenth aspect of the present invention, sound output control can be performed by using one of a plurality of types of sound determination AI corresponding to the sound type as the applicable AI, or by determining final applicable sound control information based on the applicable sound control information acquired from the plurality of types of sound determination AI, and then performing sound output control.

[0026] An eleventh invention is a sound control system in which, in the above invention, the sound determination AI is an AI trained to determine the sound control information that governs each of a plurality of sound types, and the AI ​​is trained to have different sound tendencies based on the sound control information for each type of sound determination AI.

[0027] According to the eleventh aspect, it is possible to determine sound control information that integrates each of a plurality of sound types.

[0028] A twelfth aspect of the invention is the sound control system according to the above-mentioned invention, wherein a character is set for each type of sound determination AI.

[0029] According to the twelfth aspect, a character can be set for each of a plurality of types of sound determination AI.

[0030] A thirteenth invention is a sound control system in which, in the above invention, the sound determination AI is an AI that has been trained to determine the sound control information by referring to sound data stored in a sound data storage unit (e.g., sound source list 540 in Figure 4).

[0031] According to the thirteenth aspect of the present invention, sound output control can be performed using a sound determination AI that has been trained to determine sound control information by referring to stored sound data.

[0032] A fourteenth invention is a sound control system in the above invention, further comprising an additional storage control means (for example, the AI ​​execution unit 320 in Figure 4) for additionally storing sound data based on the applied sound control information in the sound data storage unit.

[0033] According to the fourteenth aspect, sound data based on the applied sound control information can be additionally stored.

[0034] The 15th invention is a sound control system in which, in the above invention, the sound determination AI has a sound generation AI trained to generate new sound data according to the situation of the content, and is an AI trained to determine the sound control information by referring to the new sound data generated by the sound generation AI, and further includes an additional storage control means for additionally storing the new sound data in the sound data storage unit.

[0035] According to the fifteenth aspect of the present invention, new sound data generated by a sound generation AI is additionally stored, and sound output control can be performed using a sound determination AI that has been trained to determine sound control information by referring to the new sound data.

[0036] The 16th invention is a sound control system in the above invention, further comprising an evaluation means for evaluating each of the sound determination AIs, and the applied sound control information acquisition means selects the sound determination AI from which to acquire the applied sound control information based on the evaluation result of the evaluation means.

[0037] According to the sixteenth aspect of the present invention, for example, it is possible to select a sound determination AI with a high evaluation from among a plurality of sound determination AIs as the sound determination AI from which applied sound control information is to be obtained.

[0038] A seventeenth invention is a sound control system in which the content being progressed by the user is game content.

[0039] According to the seventeenth aspect of the present invention, it is possible to obtain applicable sound control information according to the progress of the game and control the output of sounds (game sounds) that are output in the game.

[0040] The 18th invention is a program for causing a computer system to control the sound output from playable or viewable content, the program causing the computer system to function as: an applicable sound control information acquisition means that acquires applicable sound control information according to the progress of the content being played or viewed by a user using a sound determination AI (Artificial Intelligence) that has been trained to determine sound control information according to the status of the content being played or viewed; and a sound output control means that controls the output of sound directed to the user based on the applicable sound control information.

[0041] According to the eighteenth aspect of the present invention, it is possible to realize a program that provides the same effects as the first aspect of the present invention. [Brief explanation of the drawings]

[0042] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of a game system. [Figure 2] FIG. 2 is a diagram showing an example of the device configuration of a user terminal. [Figure 3] FIG. 4 is a diagram showing an example of the data structure of play data. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a sound control unit. [Figure 5] A diagram to explain the learning process of the BGM determination AI. [Figure 6] A diagram to explain the learning process of the sound effect determination AI. [Figure 7] FIG. 2 is a block diagram showing an example of the functional configuration of a server system. [Figure 8]FIG. 2 is a block diagram showing an example of the functional configuration of a user terminal. [Figure 9] 10 is a flowchart showing the flow of processing performed by the server system. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a sound control unit in Modification 1. [Figure 11] FIG. 10 is a diagram for explaining a sound determination AI in Modification 2. [Figure 12] FIG. 10 is another diagram for explaining the sound determination AI in the second modification example. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of a sound control unit in Modification 2. [Figure 14] FIG. 13 is a block diagram showing an example of the configuration of a sound control unit in Modification 4. [Figure 15] FIG. 13 is a block diagram showing an example of the configuration of a sound control unit in Modification 5. [Figure 16] FIG. 20 is a block diagram showing an example of the functional configuration of a user terminal in Modification 8. DETAILED DESCRIPTION OF THE INVENTION

[0043] Preferred embodiments of the present invention will be described below with reference to the drawings. In the following, control of sounds (game sounds) output from game content will be described using game content as an example of playable content. Note that the present invention is not limited to the embodiments described below, and the forms to which the present invention can be applied are not limited to the following embodiments. Furthermore, in the drawings, the same parts are given the same reference numerals.

[0044] [Overall configuration] Fig. 1 is a diagram showing an example of the overall configuration of a game system 1000 according to this embodiment. As shown in Fig. 1, the game system 1000 is a computer system including a server system 1100 having a function as a sound control system and a user terminal 1500 owned by a player (user) 2 of the game according to this embodiment, which are connected via a network N so as to be able to communicate data with each other.

[0045] The network N refers to a communication path that allows data communication. That is, the network N includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) using Ethernet (registered trademark), etc., as well as a communication network such as a telephone communication network, a cable network, or the Internet, and the communication method can be either wired or wireless.

[0046] The server system 1100 is a computer system including a main device 1101, a keyboard 1106, a touch panel 1108, and storage 1140. The main device 1101 incorporates a control board 1150 on which electronic components such as a CPU (Central Processing Unit) 1151, various microprocessors such as a GPU (Graphics Processing Unit) and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM and ROM, and a communication device 1153 are mounted. Note that part or all of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or an SoC (System on a Chip).

[0047] This server system 1100 realizes a user management function related to the user registration of user 2 and a game management function that provides data necessary for playing the game on user terminal 1500 and manages execution control of the game on user terminal 1500, by having CPU 1151 and the like perform calculations based on predetermined programs and data. In other words, the game in this embodiment is realized as a kind of client-server online game. User 2 accesses server system 1100 with his / her own user terminal 1500, logs in using an issued account (user ID), and enjoys the game of this embodiment.

[0048] The server system 1100 also cooperates with an external electronic settlement server operated by an electronic settlement service provider or the like to carry out the purchase procedure (billing process) of game coins, which are in-game currency. During the billing process, the electronic settlement server responds to an inquiry from the server system 1100 and settles the purchase amount of the game coins with User 2's credit card, prepaid card, or the like. Then, the server system 1100 grants User 2 game coins equivalent to the purchase amount settled by the electronic settlement server.

[0049] 1, the server system 1100 may be configured to include multiple blade servers each assigned to one function, connected to each other via an internal bus for data communication. Alternatively, the server system 1100 may be configured to function as a whole as a server system 1100 by having multiple independent servers installed in remote locations communicate with each other via a network N.

[0050] The user terminal 1500 is a computer system that functions as a man-machine interface, and is connected to the network N via a mobile phone base station, a wireless communication base station, or the like, and can perform data communication with the server system 1100. This user terminal 1500 can take the form of, for example, a smartphone, a mobile phone, a portable game device, a stationary home game device, a controller for a stationary home game device, an arcade game device, a personal computer, a tablet computer, a wearable computer, or the like.

[0051] Fig. 2 is a diagram showing an example of the device configuration of a smartphone, which is an example of a user terminal 1500. As shown in Fig. 2, the user terminal 1500 includes a directional input key 1502, a home key 1504, a touch panel 1506 that functions as an image display device and a contact position input device, an internal battery 1509, a speaker 1510, a microphone 1512, a camera 1520, a control board 1550, and a memory card reader 1542 that can read and write data from and to a memory card 1540 that is a computer-readable storage medium. In addition, the user terminal 1500 is provided with a power button, a volume control button, etc. (not shown).

[0052] The control board 1550 is equipped with various microprocessors such as a CPU 1551, a GPU, and a DSP; various IC memories 1552 such as a VRAM, a RAM, and a ROM; a wireless communication module 1553 for wireless communication with a mobile phone base station or a wireless communication base station connected to the network N; and an interface circuit 1557. The interface circuit 1557 includes a circuit for receiving signals from the direction input keys 1502 and the home key 1504, a driver circuit for the touch panel 1506, an output amplifier circuit for outputting audio signals to the speaker 1510, an audio signal generation circuit for generating an audio signal collected by the microphone 1512, a circuit for inputting image data of an image captured by the camera 1520, and a signal input / output circuit for the memory card reader 1542. These elements equipped on the control board 1550 are electrically connected to each other via a bus circuit or the like, and are connected to enable reading and writing of data and sending and receiving of signals. Note that part or all of the control board 1550 may be configured using an ASIC, an FPGA, or an SoC.

[0053] In control board 1550, IC memory 1552 stores a game client program and various setting data required to execute this game client program. The game client program and the like are downloaded from server system 1100 at appropriate times. Alternatively, the program may be read from a separately obtained storage medium such as memory card 1540. CPU 1551 and the like then execute the game client program to perform arithmetic processing, and control each section of user terminal 1500 in response to operational inputs made to touch panel 1506, directional input keys 1502, and home key 1504, thereby enabling user 2 to play the game.

[0054] [overview] The game of this embodiment is, for example, a stage-clear type fighting game in which game stages are cleared in order. The user uses a character they own as a user character (player character) to challenge enemy characters that appear in each game stage. When the user finally wins the battle against the boss character of that stage, the stage is cleared.

[0055] Here, in this embodiment, the game played by the user is a stage-clear type game, and one stage is made up of multiple phases (scenes), such as a phase in which the user character moves within the stage, a phase in which the user character encounters an enemy character and the battle begins, a phase in which the user character battles the enemy character, a phase in which the user character is defeated and the game is over, a phase in which the user character confronts a boss character, a phase in which the user character battles the boss character, a phase in which the user character is defeated and the stage is cleared, or a phase in which the user character is defeated and the game is over, etc. The configuration of multiple phases can also be called a scenario.

[0056] The server system 1100 generates and stores play data 550 (see FIG. 3, etc.) in addition to controlling the progress of the game. The play data 550 is generated for each user. The play data 550 for that user is updated as needed each time the user plays the game, and describes the latest status of the game. For example, as shown in FIG. 3, one piece of play data 550 stores game status information 557 including information indicating the current status of the game, together with a play date and time 553, a user ID 555 of the user playing the game, etc.

[0057] The game status information 557 stores various information for describing and identifying the latest game status and game play history data (logs). For example, the game status information 557 stores basic information such as play time (play start time and elapsed time from that time), game world time, cumulative play time, user level and experience points, information such as the type and attributes of the user character, money in possession, items in possession, billing information, etc.

[0058] The game status information 557 stores information relating to the progress of the game, such as the game stage ID currently being played, the progress of the scenario, information indicating the phase (scene) currently being played in the current stage, and game results.

[0059] The game status information 557 also stores information about the user character, such as various status information such as parameter values ​​indicating the position, posture, and state of the user character, and information about equipment items. Parameter values ​​include, for example, experience points, ability parameter values, skill types, skill levels, and stamina parameter values.

[0060] The game status information 557 also includes information on various statuses of enemy characters and boss characters (NPCs; Non-Player Characters), information on their actions, etc. In addition, the game status information 557 stores play log data, information on various statuses of the game field, information on the relative positions of characters during a battle, information on the content of chats exchanged between users during the game, etc.

[0061] The server system 1100 then controls the sounds (game sounds) output in the game based on the play data 550 using sound determination AI (Artificial Intelligence).

[0062] The sound determination AI is an AI trained to determine sound control information according to the game situation based on information included in the game situation information. The learning uses data from actual past game play as training data. Specifically, given information from the game situation information during a predetermined period before and after a sound is output is used as input data, and sound control information related to the output sound is used as output data. Through training using a large amount of training data, the AI ​​learns what kind of sound to edit and output in what way for what play data (game situation information). Here, in this embodiment, "learning" refers to machine learning, and more specifically, deep learning. Therefore, the AI ​​described in this embodiment is an AI trained by deep learning.

[0063] The actual game play that is the source of the input data is not limited to the same single game. It can include play data of any game. Furthermore, training data may be extracted from play data of games other than the game targeted by this embodiment, and the sound determination AI may learn the training data. Alternatively, training data may be extracted only from play data of the game targeted by this embodiment, and the sound determination AI may learn the training data.

[0064] Furthermore, the sound determination AI of this embodiment determines a sound using a sound source list. Therefore, during learning, sound source data for sounds related to the training data is registered in the sound source list. The sound source list is a database in which all possible sound source data (sound data) is registered in association with unique sound source numbers. The sound control information, which corresponds to the output data of the training data, is control data for outputting a sound using a sound source with a sound source number registered in the sound source list. Therefore, the trained sound determination AI becomes an AI that refers to the sound source list and determines sound control information capable of outputting a sound using a sound source registered in the sound source list. The server system 1100 has a sound source list 540 (see FIG. 4, etc.), which is referenced when the sound determination AI is executed.

[0065] [detail] Fig. 4 is a block diagram showing an example configuration of the sound control unit 300 that controls sound in the server system 1100. As shown in Fig. 4, the sound control unit 300 includes a progress determination unit 310, an AI execution unit 320, and a sound determination unit 350, and the progress determination unit 310 and the AI ​​execution unit 320 constitute an applied sound control information acquisition unit.

[0066] The progress status determination unit 310 determines the progress status of the game based on the game status information 557 (see FIG. 3 ) in the input play data 550, generates progress status input information corresponding to the determined progress status, and outputs the generated information to the AI ​​execution unit 320. The progress status input information generated by the progress status determination unit 310 can be considered a type of prompt input to the AI ​​execution unit 320. Therefore, the progress status determination unit 310 can also be considered to generate a prompt. In this embodiment, the type of game status information that identifies and describes each game phase is classified and defined in advance. Then, the progress status determination unit 310 determines the current phase from various types of information in the game status information 557, extracts the classified and defined type of information for the determined phase from the play data 550, and generates the progress status input information.

[0067] The AI ​​execution unit 320 is a functional unit that executes a sound determination AI. In this embodiment, the AI ​​execution unit 320 is configured to execute multiple types of sound determination AI according to the sound type. For example, the sound determination AI executed by the AI ​​execution unit 320 includes a BGM (Background Music) determination AI 401 and a sound effect determination AI 403.

[0068] The BGM determination AI 401 is an AI that has been trained to output sound control information related to the BGM during game play (hereinafter referred to as "BGM control information").

[0069] 5 is a diagram illustrating the learning process of the BGM determination AI 401. Training data is used for training, with progress input information for each phase from game situation information obtained in past game play as input, and BGM control information for that phase obtained in that game play as output. As described above, in this embodiment, multiple training data are prepared regardless of the type of game.

[0070] The progress input information includes (1) a game stage ID, (2) information indicating a phase, (3) game world time, (4) various status data of the user character, (5) various status data of enemy characters and boss characters, etc. The BGM control information includes (1) a sound source number indicating a sound source, (2) volume, (3) tempo, (4) tone, (5) playback time, (6) playback timing, etc. Specific examples of the contents of the training data include training data in which the game stage ID in the progress input information indicates an underwater game stage and the sound source number in the BGM control information indicates sound source data that evokes an underwater world, or training data in which the game stage ID in the progress input information indicates a game stage in space and the sound source number in the BGM control information indicates sound source data that evokes an outer space world.

[0071] While the user is playing the game, the BGM determination AI 401 determines BGM control information by referring to the sound source list 540, based on the progress input information input from the progress determination unit 310. The BGM control information includes, for example, the sound source number, volume, playback timing, playback time, etc. of the sound source data to be used as BGM.

[0072] Furthermore, the BGM determination AI 401 appropriately re-learns when outputting BGM control information, generating new sound source data that was not previously in the sound source list 540, and can output this data the next time the BGM control information is output. During re-learning, the AI ​​execution unit 320 treats the sound data based on the BGM control information output from the BGM determination AI 401 as new sound source data, associates it with a sound source number, and adds it to the sound source list 540.

[0073] The sound effect determination AI 403 is an AI trained to output sound control information relating to sound effects such as sound effects (hereinafter referred to as "sound effect control information").

[0074] 6 is a diagram for explaining the learning process of the sound effect determination AI 403. For learning, training data is used, which takes as input progress status input information for each phase from game situation information obtained in past game play, and outputs sound effect control information for that phase obtained in that game play. Multiple training data are prepared regardless of the type of game.

[0075] The progress status input information includes (1) a game stage ID, (2) information indicating the phase, (3) game world time, (4) various status data of the user character, (5) various status data of enemy characters and boss characters, etc. The sound effect control information includes (1) a sound source number indicating the sound source, (2) volume, (3) tempo, (4) tone, (5) playback time, (6) playback timing, etc. The sound sources of the sound effect control information include sound sources of sound effects corresponding to the setting and phase of the game stage indicated by the game stage ID of the progress status input information, the status of the battle, etc., such as the sound of water, the sound of wind, the sound of a storm, the sound of rain, the sound of thunder, the sound of a burning fire, cheers, applause, the sound of a city crowd, traffic sounds, and various sound effects (such as "shakin," "kiran," and "pampakapaaan").

[0076] During the user's game play, the sound effect determination AI 403 determines sound effect control information by referring to the sound source list 540 based on progress input information input from the progress determination unit 310. The sound effect control information includes, for example, the sound source number, volume, playback timing, playback time, etc. of the sound source data to be used as a sound effect.

[0077] Similarly to the BGM determination AI 401, the sound effect determination AI 403 performs appropriate re-learning when outputting sound effect control information, generating new sound source data that was not previously in the sound source list 540 and making it available for output the next and subsequent output of sound effect control information. During re-learning, the AI ​​execution unit 320 regards the sound data based on the sound effect control information output from the sound effect determination AI 403 as new sound source data, associates it with the sound source number, and adds it to the sound source list 540.

[0078] The sound determination unit 350 receives as input BGM control information from the BGM determination AI 401 and sound effect control information from the sound effect determination AI 403. The sound determination unit 350 determines final application sound control information based on the BGM control information and sound effect control information, and controls sound output based on the final application sound control information. In this embodiment, the sound determination unit 350 synthesizes BGM and sound effects based on the respective control information to generate game sound data. The generated game sound data is transmitted to the user terminal 1500 via the communication unit 394s, and is played and output by the user terminal 1500.

[0079] [Function Configuration] 1. Server system Fig. 7 is a block diagram showing an example of the functional configuration of the server system 1100. As shown in Fig. 7, the server system 1100 of this embodiment includes an operation input unit 100s, a server processing unit 200s, an image display unit 390s, a sound output unit 392s, a communication unit 394s, and a server storage unit 500s.

[0080] The operation input unit 100s is used to input various operations for system management, maintenance, etc., and can be realized by, for example, a keyboard, a mouse, a touch panel, etc. In FIG. 1, this corresponds to the keyboard 1106 and the touch panel 1108.

[0081] The server processing unit 200s can be realized by electronic components such as a processor, which is an arithmetic circuit such as a CPU, GPU, ASIC, or FPGA, or an IC memory, and controls the input and output of data between the operation input unit 100s and each unit of the device, including the server storage unit 500s. The server processing unit 200s performs various arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100s, data received from the user terminal 1500, etc., and controls the overall operation of the server system 1100. In FIG. 1, this corresponds to the control board 1150 and its CPU 1151.

[0082] The server processing unit 200s includes a user management unit 210, a billing processing unit 220, a game management unit 230, a timekeeping unit 280s, an image generation unit 290s, a sound generation unit 292s, and a communication control unit 294s.

[0083] The user management unit 210 performs processes related to user registration and manages the data of each registered user linked to a user ID. For example, it can execute processes such as assigning a unique user ID to each registered user, registering and managing personal information for each user ID, and managing usage history such as login and logout history. Of course, it can also appropriately include management processes for other data linked to user IDs.

[0084] The billing processing unit 220 performs billing processing in response to a user's operation to purchase game coins, and grants the user game coins equivalent to the purchase amount.

[0085] The game management unit 230 performs various processes related to game execution management. Since the game of this embodiment is a client-server online game, the game management unit 230 controls the provision of data necessary for game play while communicating with the user terminal 1500. In this embodiment, the game management unit 230 includes a game progress control unit 231 and the sound control unit 300 described with reference to FIG. 4.

[0086] The game progress control unit 231 controls the progress of the game for each logged-in user using the game setting data 520, updates play data 550 as needed, and outputs it to the sound control unit 300. Specifically, the game progress control unit 231 executes (1) a process of arranging background objects and the like to set up a game space, (2) a process of arranging a user character in the game space and controlling the user character in response to operation input to the operation input unit 100, (3) a process of controlling a virtual camera, (4) a process of arranging an enemy character in the game space and controlling its behavior, (5) a process related to attack hit determination and damage determination and their reflection, and (6) a process of determining whether a game end condition has been met.

[0087] The timekeeping unit 280s uses a system clock to keep track of the current date and time, the time limit, and the like.

[0088] The image generating unit 290s generates images relating to system management of the server system 1100 and outputs them to the image display unit 390s.

[0089] The sound generation unit 292s generates or decodes sound data such as operation sounds related to system management of the server system 1100, and outputs the generated sound to the sound output unit 392s.

[0090] The communication control unit 294s performs communication connection and data processing for data communication with an external device (for example, the user terminal 1500) via the communication unit 394s, thereby realizing data exchange with the external device.

[0091] The image display unit 390s displays various screens for system management and the like based on the image signals input from the image generation unit 290s. For example, this can be realized by an image display device such as a flat panel display, a projector, or a head-mounted display. In FIG. 1, this corresponds to the touch panel 1108.

[0092] The sound output unit 392s outputs the audio signal input from the sound generation unit 292s. In Fig. 1, this corresponds to a speaker (not shown) provided in the main device 1101 or the touch panel 1108.

[0093] The communication unit 394s establishes communication by connecting to the network N. For example, it can be realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc. In FIG. 1, this corresponds to the communication device 1153.

[0094] The server storage unit 500s stores in advance or temporarily stores each time processing is performed programs for operating the server system 1100 and implementing various functions of the server system 1100, as well as data used during execution of these programs. For example, this can be implemented by an IC memory such as RAM or ROM, a magnetic disk such as a hard disk, or an optical disk such as a CD-ROM or DVD. In FIG. 1, this corresponds to the IC memory 1152 and storage 1140.

[0095] The server storage unit 500s also stores a server program 501, a distribution game client program 503, user management data 510, game setting data 520, sound determination AI model data 530, a sound source list 540, and play data 550. In addition, other necessary data such as timers, counters, various tables, thresholds, and flags are also stored as appropriate.

[0096] The server program 501 is a program for causing the server processing unit 200s to function as the user management unit 210, the billing processing unit 220, and the game management unit 230. Note that the server program 501 may also include programs for causing the server processing unit 200s to function as the image generation unit 290s, the sound generation unit 292s, and the communication control unit 294s, as appropriate.

[0097] The distribution game client program 503 is the original of the game client program 502 (see FIG. 8) downloaded to the user terminal 1500.

[0098] The sound determination AI model data 530 is prepared for each sound type. In this embodiment, model data 531, 533, and 535 are stored for the BGM determination AI 401 and the sound effect determination AI 403 (see FIG. 4), respectively.

[0099] The user management data 510 is prepared for each user who has completed user registration, and stores various management data related to the game play of the corresponding user. For example, one piece of user management data 510 stores, in association with the user ID of the user, the user's username, personal information such as the user's age, gender, and birthday, information on the balance of the electronic payment medium (game coins in this embodiment) linked to the user, and the user's game play history such as the date and time of game play and the play time.

[0100] Game setting data 520 stores various setting data necessary for executing the online game of this embodiment, such as character setting data that defines the type and attributes of each character appearing in the game, initial values ​​of parameter values ​​such as various ability values, model data, motion data used for movement control, object definition data related to various objects such as items that the user can obtain during the game, and stage setting data related to the settings of the game stage.

[0101] The play data 550 is prepared for each game play, and stores various game status information for the game play in association with the user ID of the user who is playing the game (see FIG. 3).

[0102] 2. User Device 8 is a block diagram showing an example of the functional configuration of the user terminal 1500. As shown in FIG. 8, the user terminal 1500 includes an operation input unit 100, a device processing unit 200, an image display unit 390, a sound output unit 392, a communication unit 394, and a device storage unit 500.

[0103] The operation input unit 100 is used by the user to input various operations, and can be realized by, for example, a button switch, a joystick, a touchpad, a trackball, an acceleration sensor, an angular velocity sensor, a CCD module, etc. In Fig. 2, this corresponds to the direction input keys 1502, the home key 1504, and the touch panel 1506.

[0104] The device processing unit 200 can be realized by a processor, which is an arithmetic circuit such as a CPU, GPU, ASIC, or FPGA, or by electronic components such as an IC memory, and controls the input and output of data between the device and each unit including the operation input unit 100 and the device storage unit 500. The device processing unit 200 performs various arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100, data received from the server system 1100, etc., and controls the operation of the user terminal 1500. In FIG. 2, this corresponds to the control board 1550 and its CPU 1551. The device processing unit 200 in this embodiment includes a user terminal arithmetic unit 270, a timer unit 280, an image generation unit 290, a sound generation unit 292, and a communication control unit 294.

[0105] The user terminal calculation unit 270 executes various calculation processes to cause the user terminal 1500 to function as a terminal for the user to play games. For example, the user terminal calculation unit 270 includes an operation signal transmission control unit 271 and a screen display control unit 273.

[0106] The operation signal transmission control unit 271 performs processing for transmitting various data and request information to the server system 1100 in response to an operation input to the operation input unit 100 .

[0107] The screen display control unit 273 controls the display of various screens, such as game screens, based on various data received from the server system 1100. For example, if the online game of this embodiment is realized as a web game, it can be realized using web technology that actively controls screen display using a web browser-based HTML together with Java (registered trademark) and CSS (Cascading Style Sheets), or a plug-in such as Adobe (registered trademark) Flash. Of course, other methods are also acceptable. In addition, in the configuration of this embodiment, the game space image (e.g., 3DCG, etc.) that forms the basis of the game screen is generated by the server system 1100, but it is also possible to configure the game space image to be generated by the user terminal 1500. In this case, the screen display control unit 273 controls objects arranged in a virtual three-dimensional space for generating the 3DCG.

[0108] The image generation unit 290, in cooperation with the screen display control unit 273, generates an image signal for displaying one game screen per frame time (for example, 1 / 60 seconds) based on various data received from the server system 1100, and outputs the generated image signal to the image display unit 390. For example, this can be realized by a processor such as a GPU or a digital signal processor (DSP), a video signal IC, a program such as a video codec, an IC memory for drawing frames such as a frame buffer, etc.

[0109] The sound generation unit 292 generates an audio signal for the game sound based on data relating to sound output received from the server system 1100 (in this embodiment, game sound data generated by the sound determination unit 350 in the sound control unit 300), and outputs the generated audio signal to the sound output unit 392. For example, this can be realized by a processor such as a digital signal processor (DSP) or a voice synthesis IC, an audio codec for playing back audio files, or the like.

[0110] The communication control unit 294 performs communication connection and data processing for data communication with an external device (for example, the server system 1100) via the communication unit 394, thereby realizing data exchange with the external device.

[0111] The image display unit 390 displays various screens, such as a game screen, based on the image signal input from the image generation unit 290. For example, this can be realized by an image display device such as a flat panel display or a head-mounted display. In FIG. 2, this corresponds to the touch panel 1506.

[0112] The sound output unit 392 emits sound effects, background music, etc. related to the game based on the audio signal input from the sound generation unit 292. In FIG.

[0113] The communication unit 394 connects to the network N to realize communication. For example, it can be realized by a wireless communication device, a modem, a TA, a jack for a wired communication cable, a control circuit, etc. In FIG. 2, the wireless communication module 1553 corresponds to this.

[0114] The terminal storage unit 500 stores in advance or temporarily stores each time processing is performed programs for operating the user terminal 1500 and realizing the functions of the user terminal 1500, as well as data used during the execution of these programs. For example, this can be realized by IC memory such as RAM or ROM, a magnetic disk such as a hard disk, or an optical disk such as a CD-ROM or DVD. In FIG. 2, this corresponds to the IC memory 1552 and the memory card 1540.

[0115] The terminal storage unit 500 also stores a game client program 502. The game client program 502 is a program that causes the terminal processing unit 200 to function as the user terminal computing unit 270. This game client program 502 may be a dedicated client program that corresponds to the technical method for realizing the online game, or may be configured by a web browser program and a plug-in that realizes interactive image display. In this embodiment, the game client program 502 is a copy of the distribution game client program 503 (see FIG. 7) provided by the server system 1100.

[0116] [Processing flow] Fig. 9 is a flowchart showing the flow of processing executed by the server system 1100. Fig. 9 shows the flow of processing when one user starts playing a game. The processing described here is realized by the server processing unit 200s reading and executing the server program 501.

[0117] First, the game progression control unit 231 starts controlling the progress of the game (step S1). Specifically, the game progression control unit 231 arranges background objects to set up a game space, and arranges user characters and enemy characters. Then, it starts automatic control of the enemy characters and moves the user characters in response to user operation inputs. It also starts control of the virtual camera. It also starts other controls that are basically required for controlling the progress of the game as appropriate.

[0118] Thereafter, the processing of steps S3 to S13 is repeated based on the game status information 557 (see FIG. 3) of the play data 550, which is updated from time to time by the game progress control by the game progress control unit 231, to control the sound (game sound) output in the game for the user.

[0119] That is, first, the progress determination unit 310 determines the progress of the game from the game status information 557 of the play data 550 (step S3), and extracts information corresponding to the determined game progress from the game status information 557 to generate progress input information (step S5). Next, the AI ​​execution unit 320 executes the BGM determination AI 401 and obtains BGM control information (applied BGM control information) from the progress input information generated in step S5 (step S7). The AI ​​execution unit 320 also executes the sound effect determination AI 403 and obtains sound effect control information (applied sound effect control information) from the progress input information generated in step S5 (step S9).

[0120] Next, the sound determination unit 350 generates final game sound data from the applied BGM control information and applied sound effect control information acquired in step S7 or S9 (step S13).

[0121] Then, the end of the game play is determined, and the process returns to step S3 until the game play is ended (step S15: NO). If the game play is ended (step S15: YES), the process ends.

[0122] As described above, according to this embodiment, the progress of a game can be determined based on the game status information 557 (see FIG. 3), and progress status input information corresponding to the determined progress status can be generated. Then, by using AI trained to determine sound control information based on information that identifies and describes the progress of the game among the game status information, progress status input information can be input as needed to obtain applicable sound control information, and game sound output control can be performed. This allows sound control for a user to be achieved according to the progress of the game the user is playing.

[0123] The forms to which the present invention can be applied are not limited to the above-described embodiments, and constituent elements can be added, omitted, or modified as appropriate.

[0124] [Variation 1] FIG. 10 is a block diagram illustrating an example of the configuration of a sound control unit 300a in this modified example. In FIG. 10, components similar to those in the above embodiment are denoted by the same reference numerals. As shown in FIG. 10, the sound control unit 300a includes an application AI selection unit 360, a progress determination unit 310, an application AI execution unit 330, and a sound determination unit 350a. In this modified example, three sound determination AIs are provided: a creator "A"-style sound determination AI 411, a creator "B"-style sound determination AI 413, and a creator "C"-style sound determination AI 415. The application AI execution unit 330 executes the one selected as the application AI by the application AI selection unit 360. The number of sound determination AIs provided as options is not limited to three; two, four, or more may be used. Furthermore, each sound determination AI may be configured as a sound determination AI for each sound type (BGM, sound effects, etc.) as in the above embodiment.

[0125] Creator "A" style sound determination AI 411 is an AI trained to output sound control information according to the progress of the game based on the sound source data created by creator "A." During the learning process, actual game play data is used as training data. Specifically, game situation information from a predetermined period before and after the sound created by creator "A" is output is used as input data, and sound control information related to the output sound is used as output data. By learning using this training data, the AI ​​learns what kind of sound created by creator "A" should be edited and output, and how, depending on the play data (game situation information).

[0126] Similarly, creator "B" style sound determination AI413 is an AI trained to output sound control information according to the progress of the game based on the sound source data created by creator "B." Creator "C" style sound determination AI415 is an AI trained to output sound control information according to the progress of the game based on the sound source data created by creator "C."

[0127] Creators "A," "B," and "C" could be, for example, creators who create fantastical sounds, creators who create futuristic sounds, creators who create powerful sounds, creators who create light sounds, etc.

[0128] For each game play, the applied AI selection unit 360 selects an applied AI to be applied to that game play from among three sound determination AIs 411, 413, and 415. For example, before the start of a game, the applied AI selection unit 360 accepts a selection operation of a creator (here, "A," "B," or "C") by the user, and selects which of the sound determination AIs 411, 413, and 415 to apply to that user's game play. During that game play, the sound determination AI selected as the applied AI is used to obtain sound control information (applied sound control information), and musical game sounds created by the corresponding creator are output to the user.

[0129] The progress status determination unit 310 generates progress status input information from game status information 557 (see FIG. 3) of play data 550 that is input at any time. The applied AI execution unit 330 executes the applied AI selected by the applied AI selection unit 360 to determine sound control information. Furthermore, the sound determination unit 350a generates game sound data for the user based on the sound control information from the applied AI.

[0130] According to this modified example, an applicable AI is selected from three sound determination AIs 411, 413, and 415 in response to a user's selection operation before the start of game play, and the applicable AI is executed during game play, thereby realizing output control of game sounds in the manner desired by the user.

[0131] [Variation 2] 11 and 12 are diagrams for explaining the sound determination AI 420 (420-1, 2, 3, ...) in this modified example. As shown in FIG. 11, the sound determination AI 420 has an overall AI 421 and sound type-specific AIs shown surrounded by dashed lines. The sound type-specific AIs are configured to include, for example, two AIs, a BGM determination AI 401b and a sound effect determination AI 403b, which have been trained in the same manner as described in the above embodiment. In this modified example, multiple types of sound determination AI 420, each representing a different genre, are prepared in advance. FIG. 12 shows three types: "pop," "classical," and "rock."

[0132] The overall AI 421 constituting each sound determination AI 420 has a prompt generation unit 423 and a final control information determination unit 425. The prompt generation unit 423 inputs a prompt that instructs the sound determination AI 420 to determine sound control information (BGM control information / sound effect control information) by referring to the sound source data of the genre of the sound determination AI 420, and progress status input information input from the progress status determination unit 310 (see FIG. 13), to each of the BGM determination AI 401B and the sound effect determination AI 403b. The progress status input information may be included in the prompt and input.

[0133] For example, if the sound determination AI 420-1 has a "pop" genre, the supervisory AI 421 generates a prompt to instruct the AI ​​401b to determine BGM control information by referencing pop music sound source data from the sound source list 540 (see FIG. 13), and inputs the generated prompt to the BGM determination AI 401b. The BGM determination AI 401b determines the BGM control information by referencing the pop music sound source data in accordance with the input prompt. Similarly, the supervisory AI 421 generates a prompt to instruct the AI ​​403b to determine sound effect control information by referencing pop music sound source data, and inputs the generated prompt to the sound effect determination AI 403b. The sound effect determination AI 403b determines the sound effect control information by referencing the pop music sound source data in accordance with the input prompt.

[0134] Furthermore, the BGM determination AI 401b returns the determined BGM control information to the overall AI 421, and the sound effect determination AI 403b returns the determined sound effect control information to the overall AI 421. The final control information determination unit 425 of the overall AI 421 determines the final applied sound control information based on the determined BGM control information and sound effect control information. For example, the final sound control information is determined by adjusting the volume balance between the volume of the BGM control information and the volume of the sound effect control information based on the progress status input information input from the progress status determination unit 310.

[0135] As shown in FIG. 12, each of the sound determination AIs 420 (420-1, 2, 3, ...) is set with a notification character 7 (7-1, 2, 3, ...) having a different design. In this modified example, the server system 1100 controls the output of sound for a user by using one of the sound determination AIs 420 as the applicable AI while the user is playing the game, and also controls the display of the notification character 7 set as the applicable AI on the game screen. For example, if the user performs a selection operation to set the sound determination AI 420-2 as the applicable AI before the start of the game, the notification character 7-2 will be displayed on the game screen.

[0136] Figure 13 is a block diagram showing an example of the configuration of a sound control unit 300b in this modified example. In Figure 13, the same components as in the above embodiment are assigned the same reference numerals. As shown in Figure 13, the sound control unit 300b includes an applied AI selection unit 360b, a progress determination unit 310, an applied AI execution unit 330b, and a sound determination unit 350b.

[0137] The application AI selection unit 360b selects, for each game play, an application AI to be applied in that game play from among the sound determination AIs 420 (420-1, 2, 3, . . .). For example, before the start of the game, a selection operation by the user is accepted, and a selection is made among the sound determination AIs 420 (420-1, 2, 3, . . .) to be applied during the user's game play.

[0138] The progress determination unit 310 generates progress input information from game status information 557 (see FIG. 3) of play data 550 that is input at any time, and outputs it to the applied AI execution unit 330b. The applied AI execution unit 330b executes the sound determination AI 420 selected as the applied AI by the applied AI selection unit 360, and determines sound control information (final applied sound control information). The sound determination unit 350b generates game sound data for the user based on the sound control information from the applied AI.

[0139] [Variation 3] Furthermore, in the case of a configuration in which an applied AI is selected from a selection of multiple sound determination AIs, as in Modification 1 and Modification 2, a configuration may be adopted in which sound output control by the applied AI is evaluated. In this case, the server system 1100, as an evaluation means, evaluates the applied AI applied in the game play by, for example, accepting input from the user in a questionnaire after the end of the game play. Alternatively, a configuration may be adopted in which the evaluation results are tallied and each of the selected sound determination AIs is ranked, and the applied AI is automatically selected based on the ranking results. For example, in the example of Modification 1, the sound determination AI currently ranked highest based on the evaluation results is selected as the applied AI from among the three sound determination AIs 411, 413, and 415.

[0140] [Variation 4] Fig. 14 is a block diagram showing an example of the configuration of a sound control unit 300c in this modified example. In Fig. 14, the same components as those in the above embodiment are assigned the same reference numerals. As shown in Fig. 14, the sound control unit 300c includes an applied sound control information acquisition unit 340 that executes a sound determination AI 431 to acquire applied sound control information, and a sound determination unit 350c.

[0141] During the user's game play, the applied sound control information acquisition unit 340 executes the sound determination AI 431 to acquire sound control information (applied sound control information). The sound determination AI 431 of this modified example is an AI trained to determine sound control information based on annotation information obtained by performing annotation processing on play data (game situation information), and includes an annotation processing unit 433.

[0142] The annotation processing unit 433 performs annotation processing on the game status information 557 of the play data 550 input as needed during the user's game play. Through the annotation processing, the annotation processing unit 433 extracts, as annotation information, various pieces of information that the progress status determination unit 310 extracts as progress status input information in the above-described embodiment. In addition to the progress status input information, the annotation processing unit 433 also extracts, as annotation information, information such as the relative positional relationship between the user character and an enemy character or a boss character during a battle, the movement trajectory of each character, time-series data of the user character's attacks and defenses, and information on the type and timing of items used. By training the sound determination AI 431 using training data that specifies which information to extract and analyze as annotation information, it is possible to create a sound determination AI 431 (in this case, the annotation processing unit 433 of the sound determination AI 431) that extracts desired annotation information.

[0143] The sound determination AI 431 then determines and outputs sound control information based on the annotation information extracted and analyzed by the annotation process. In other words, according to this modification, the configuration of the progress determination unit 310 shown in FIG. 4 can be incorporated into the sound determination AI 431. Information that could not be determined by the progress determination unit 310 can also be extracted and analyzed as annotation information, making it possible to determine more detailed sounds that are more suited to the progress of the game. The sound determination unit 350c generates game sound data for the user based on the sound control information from the sound determination AI 431.

[0144] The sound determination AI may be configured as separate sound determination AIs for different sound types (BGM, sound effects, etc.) as in the example of the above embodiment.

[0145] [Variation 5] Fig. 15 is a block diagram showing an example of the configuration of a sound control unit 300d in this modified example. In Fig. 15, the same components as those in the above embodiment are assigned the same reference numerals. As shown in Fig. 15, the sound control unit 300d includes a progress determination unit 310, an AI execution unit 320d that executes a sound determination AI 441, and a sound determination unit 350d.

[0146] The sound determination AI441 of this modified example has a sound generation AI443, and is an AI that has been trained to determine sound control information by referring to new sound data generated by the sound generation AI443. The sound generation AI443 is an AI that has been trained to generate new sound source data (new sound data) according to the progress of the game. In this modified example, the sound generation AI443 generates new sound data according to the progress of the game based on game status information 557 (see FIG. 3) of play data 550 that is input at any time. The generated new sound data is registered in the sound source list 540d and is referenced by the sound determination AI441. The sound determination AI441 refers to the sound source list 540d and determines sound control information according to the progress of the game based on the new sound data generated by the sound generation AI443.

[0147] The progress status determination unit 310 generates progress status input information from game status information 557 (see FIG. 3) of play data 550 that is input at any time. The AI ​​execution unit 320d executes the sound determination AI 441 to determine sound control information. The AI ​​execution unit 320d also adds and stores the sound source data newly generated by the sound generation AI 443 upon this determination to the sound source list 540d. The sound determination unit 350d generates game sound data for the user based on the sound control information from the applicable AI.

[0148] The sound determination AI may be configured as separate sound determination AIs for different sound types (BGM, sound effects, etc.) as in the example of the above embodiment.

[0149] [Variation 6] Furthermore, in the above embodiment, game content has been described as an example of content, but the type of content is not particularly limited. For example, the present invention can be applied to content in which an avatar moves to enjoy a virtual experience in a virtual space, or content distributed by a broadcasting user (for example, content in which a broadcasting user performs a performance or a show, or plays a game, and broadcasts the performance live). Sound control for a user can be realized according to the status of the content the user is viewing.

[0150] [Variation 7] In the above embodiment, the server system 1100 has a sound determination AI, but the sound determination AI may be called and used as an external service. For example, the entire sound determination AI (BGM determination AI 401 and sound effect determination AI 403) may be provided by an external server system, or the BGM determination AI 401 and sound effect determination AI 403 may be provided by separate external server systems.

[0151] [Variation 8] Furthermore, in the above embodiment and modified examples, the server system 1100 is described as the entity that processes game management, but the user terminal 1500 may be configured as the entity that processes game management, or the processing related to the game management may be distributed and executed between the server system 1100 and the user terminal 1500. For example, in the configuration of the above embodiment, if the user terminal is configured as the entity that processes game management, an example of the functional configuration of the user terminal 1500A is as shown in Fig. 16. Note that in Fig. 16, the same components as those in the above embodiment are denoted by the same reference numerals.

[0152] As shown in FIG. 16 , in the user terminal 1500A of this modification, the device processing unit 200 includes a game management unit 230, and the screen display control unit 273 is omitted. That is, the user terminal 1500A of this modification does not acquire data for displaying images such as game screens and outputting game sounds from the server system 1100, but rather functions as a computer system in which its own game management unit 230 performs game management processing, controls game execution, and generates images such as game screens and game sounds. A game program 504 is stored in the device storage unit 500 of the user terminal 1500A. The game program 504, when executed by the user terminal 1500A, causes the device processing unit 200 to function as the user terminal calculation unit 270 and the game management unit 230. The device storage unit 500 also stores user management data 510 related to the user of the user terminal 1500A, play data 550 related to the game being played by the user, game setting data 520, sound determination AI model data 530, and a sound source list 540.

[0153] The processing flow of the user terminal 1500A of this modified example is basically the same as the flowchart shown in FIG. 9, and each step can be interpreted as being executed by the game management unit 230 of the user terminal 1500A.

[0154] According to this modification, the same effects as those of the above embodiment can be obtained. Note that the user terminal 1500 may be configured to perform only some of the functions of the game management unit 230, rather than all of them. [Explanation of symbols]

[0155] 1000...Game System 1100...Server system 100s...Operation input section 200s...Server processing section 210...User Management Department 220...Charging processing unit 230...Game Management Department 231...Game progress control unit 300, 300a, 300b, 300c, 300d...Sound control section 310...Progress status determination section 320,320d…AI Execution Department 330, 330b…Application AI Execution Department 340...Applied sound control information acquisition unit 350, 350a, 350b, 350c, 350d...Sound determining section 360, 360b…Application AI selection section 401,401b…BGM decision AI 403, 403b…Sound effect decision AI 411… Creator “A” Wind Sound Decision AI 413… Creator "B" Wind Sound Decision AI 415… Creator “C” Wind Sound Decision AI 420 (420-1, 2, 3, . . .) ... Sound decision AI 421...Overall AI 431...Sound Decision AI 433...Annotation processing unit 441...Sound Decision AI 443…Sound generation AI 290s...Image generation section 292s…sound generation section 294s…Communication control unit 390s...Image display section 392s...Sound output section 394s…Communication Department 500s...Server storage section 501...Server program 503...Streaming game client program 510...User management data 520...Game setting data 530…Sound determination AI model data 531...BGM decision AI model data 533…Sound effect determination AI model data 540,540d…Sound source list 550...Play data 553...Play date and time 555...User ID 557...Game status information 1500, 1500A...User terminal 100...Operation input section 200...Terminal processing section 270...User terminal calculation unit 271...Operation signal transmission control unit 273...Screen display control unit 290...Image generation unit 292...Sound generation section 294...Communication control unit 390...Image display unit 392...Sound output unit 394…Communications Department 500...Device memory section 502...Game client program N...Network 2...User

Claims

1. A sound control system for controlling a sound output from playable or viewable content, an applicable sound control information acquisition means for acquiring applicable sound control information according to the progress of the content being played or viewed by a user, using a sound determination AI (Artificial Intelligence) that has been trained to determine sound control information according to the situation of the content being played or viewed; a sound output control means for controlling the output of sound directed to the user based on the applied sound control information; A sound control system comprising:

2. The sound determination AI is an AI that has been trained based on sound control information corresponding to the situation of the plurality of contents. The sound control system of claim 1 .

3. The sound determination AI is an AI that has been trained based on sound control information corresponding to the situation of content other than the content being progressed by the user. The sound control system of claim 1 .

4. The sound determination AI is an AI that re-learns based on the progress and the applied sound control information; The sound control system of claim 1 .

5. The applied sound control information acquisition means an input information generating means for generating progress status input information to be input to the sound determination AI based on the progress status of the content being progressed by the user; having The sound control system of claim 1 .

6. The sound determination AI is an AI that analyzes annotation target information based on the context of the content and determines the sound control information based on the analysis result. The sound control system of claim 1 .

7. There are multiple types of sound determination AIs, an application AI selection means for selecting an application AI from the plurality of types of sound determination AI; Further provided with The applied sound control information acquisition means acquires the applied sound control information from the applied AI. The sound control system of claim 1 .

8. There are multiple types of sound determination AIs, the applied sound control information acquisition means acquires the applied sound control information from each of the plurality of types of sound determination AIs; The sound output control means determines final application sound control information based on the application sound control information acquired from the plurality of types of sound determination AI, and performs sound output control based on the final application sound control information. The sound control system of claim 1 .

9. The plurality of types of sound determination AI are AIs trained with different training data.

9. A sound control system according to claim 7 or 8.

10. The plurality of types of sound determination AI are AIs corresponding to sound types.

9. A sound control system according to claim 7 or 8.

11. The sound determination AI is an AI trained to determine the sound control information that integrates each of a plurality of sound types, and is an AI trained to have different sound tendencies based on the sound control information for each type of the sound determination AI.

9. A sound control system according to claim 7 or 8.

12. A character is set for each type of sound determination AI. The sound control system of claim 11.

13. The sound determination AI is an AI that has been trained to determine the sound control information by referring to sound data stored in a sound data storage unit. The sound control system of claim 1 .

14. additional storage control means for additionally storing sound data based on the applied sound control information in the sound data storage unit; The sound control system of claim 13 further comprising:

15. The sound determination AI has a sound generation AI that has been trained to generate new sound data according to the situation of the content, and is an AI that has been trained to determine the sound control information by referring to the new sound data generated by the sound generation AI, additional storage control means for additionally storing the new sound data in the sound data storage unit; The sound control system of claim 13 further comprising:

16. evaluation means for evaluating each of said sound determination AIs; Further provided with The applied sound control information acquisition means selects the sound determination AI from which the applied sound control information is to be acquired based on the evaluation result of the evaluation means. The sound control system of claim 11.

17. The content being progressed by the user is game content. The sound control system of claim 1 .

18. A program for causing a computer system to control a sound output from playable or viewable content, an applicable sound control information acquisition means for acquiring applicable sound control information according to the progress of the content being played or viewed by the user, using a sound determination AI (Artificial Intelligence) that has been trained to determine sound control information according to the situation of the content being played or viewed; a sound output control means for controlling the output of sound directed to the user based on the applied sound control information; A program for causing the computer system to function as a

Images