Cloud game audio generation method and device, medium and computer program product

Abstract

The invention relates to the technical field of streaming media, and particularly provides a cloud game audio generation method and device, a medium and a computer program product. The cloud game audio generation method comprises the following steps: acquiring monaural cloud game audio data and user audio preference information; filtering the monaural cloud game audio data based on the user audio preference information to obtain monaural initial audio data; adopting a pre-constructed audio generation model to generate multi-channel audio data based on the cloud game scene; and synthesizing the single-channel initial audio data and the multi-channel audio data to obtain a target multi-channel game audio. According to the cloud game audio generation method provided by the invention, the personal preference of the user can be met, and the output target multi-channel game audio can better fit the current game scene, so that the expressive force and immersion of the game audio are increased.

Description

Technical Field

[0001] The present disclosure relates to the field of streaming media technology, and in particular to a cloud game audio generation method, device, medium and computer program product. Background Art

[0002] Cloud gaming is based on cloud computing. The game is run on a cloud server, which then compresses and encodes the rendered game screen and transmits it to the user's mobile device through the network. During the game, users can play preset audio through the cloud gaming platform to enhance the immersion of the game.

[0003] In related technologies, in order to improve user experience, cloud gaming platforms usually use virtual sound card drivers and corresponding client software to simulate the playback of game audio. This solution captures the audio data output by the virtual sound card through the system application programming interface (API), with the channel being mono, and then uses an encoding method to expand the mono audio data into multi-channel audio data for playback. However, since mono audio itself lacks spatial sense and stereo effects, the generated multi-channel audio has poor realism, resulting in a poor gaming experience for users. Summary of the invention

[0004] In view of this, exemplary embodiments of the present disclosure provide a cloud game audio generation method, device, medium and computer program product to solve the problems existing in the related art.

[0005] In one aspect of an exemplary embodiment of the present disclosure, a method for generating audio for a cloud game is provided, the method comprising:

[0006] Obtain mono cloud game audio data and user audio preference information;

[0007] Filtering the mono-channel cloud game audio data based on the user audio preference information to obtain mono-channel initial audio data;

[0008] Use a pre-built audio generation model to generate multi-channel audio data based on cloud gaming scenarios;

[0009] The mono-channel initial audio data and the multi-channel audio data are synthesized to obtain target multi-channel game audio.

[0010] Another aspect of the exemplary embodiments of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the exemplary embodiments of the present disclosure.

[0011] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in the exemplary embodiments of the present disclosure is implemented.

[0012] According to another aspect of the exemplary embodiments of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the method described in the exemplary embodiments of the present disclosure is implemented.

[0013] As will be described in detail below, according to a cloud game audio generation method of an embodiment of the present disclosure, by obtaining mono cloud game audio data and user audio preference information; filtering the mono cloud game audio data based on the user audio preference information to obtain mono initial audio data; using a pre-constructed audio generation model to generate multi-channel audio data based on the cloud game scene; synthesizing the mono initial audio data and the multi-channel audio data to obtain the target multi-channel game audio. The cloud game audio generation method provided by the present disclosure can not only meet the user's personal preferences, but also make the output target multi-channel game audio more suitable for the current game scene, thereby increasing the expressiveness and immersion of the game audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 A flowchart of a method for generating audio for a cloud game provided by an exemplary embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of a process for generating multi-channel audio data based on a cloud gaming scenario provided by an exemplary embodiment of the present disclosure;

[0017] Figure 3 This is a schematic block diagram of the functional modules of a cloud gaming audio generation device provided as an example in the present disclosure;

[0018] Figure 4 A structural block diagram of an electronic device provided as an example in the present disclosure;

[0019] Figure 5 A schematic diagram of a computer program product provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0022] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0027] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] Cloud gaming is based on cloud computing. The game is run on a cloud server, which will compress and encode the rendered game screen and transmit it to the user's mobile device through the network. During the game, users can play preset audio through the cloud gaming platform to enhance the immersion of the game.

[0029] In related technologies, in order to improve user experience, cloud gaming platforms usually use virtual sound card drivers and corresponding client software to simulate the playback of game audio. This solution captures the audio data output by the virtual sound card through the system application programming interface (API), with the channel being mono, and then uses an encoding method to expand the mono audio data into multi-channel audio data for playback. However, since mono audio itself lacks spatial sense and stereo effects, the generated multi-channel audio has poor realism, resulting in a poor gaming experience for users.

[0030] Therefore, in order to solve the above problems, the exemplary embodiments of the present disclosure provide a cloud game audio generation method, which obtains mono initial audio data by filtering the mono cloud game audio according to the user audio preference information, and generates multi-channel audio data according to the game scene. Finally, the mono initial audio data that meets the user's preference and the multi-channel audio data generated based on the game scene are mixed and superimposed to generate the target multi-channel game audio.

[0031] For example, Figure 1 This is a flow chart of a method for generating audio for a cloud game provided by an exemplary embodiment of the present disclosure, such as Figure 1 As shown, the following steps may be specifically included:

[0032] Step S110: Obtain mono-channel cloud game audio data and user audio preference information.

[0033] Since cloud gaming servers lack physical sound cards, they usually install virtual sound card drivers and corresponding driver clients to simulate game audio playback.

[0034] In the audio and video acquisition module, the audio data in the virtual sound card can be captured through the system's API (Application Programming Interface). The output format is PCM (Pulse Code Modulation) data, the sampling rate is sampleRate, the sampling accuracy is 16 bits, and the channel is mono cloud game audio data.

[0035] Then, obtain the user's audio preference information set U. The audio preference information can be set by the user in the cloud game client, or the user's audio preference information can be extracted through cloud game history data.

[0036] Exemplarily, the user audio preference information may include:

[0037] Music type setting: Game audio can be divided into three types: background music, sound effects, and voice. Users can choose to turn off or on a certain type of music.

[0038] Volume settings: Volume settings can be divided into total volume, sound effect volume, music volume and dialogue volume. Among them, total volume: controls the overall loudness of all sounds in the game; sound effect volume: independently controls the volume of game effect sounds (such as gunshots, explosions, etc.); music volume: independently controls the volume of game background music; dialogue volume: independently controls the volume of character dialogues in the game.

[0039] Surround sound settings: Users can adjust the surround sound settings according to the sound system they use (such as stereo, 5.1 surround sound, 7.1 surround sound, etc.).

[0040] Equalizer settings: Users can adjust the gain of different frequencies to optimize the frequency response characteristics of the game audio, such as enhancing the low or high frequencies to meet personal preferences.

[0041] Dynamic Range Compression: Users can choose whether to enable dynamic range compression, which helps make the maximum and minimum volume levels of sounds closer together, providing a more consistent listening experience in a variety of listening environments.

[0042] Voice chat settings: Users can set the volume of voice chat, whether to enable microphone echo cancellation, noise suppression, etc.

[0043] Special audio modes: For example, Night Mode reduces sudden volume peaks, making it ideal for gaming at night or in quiet environments.

[0044] By obtaining the above-mentioned user audio preference information, the cloud gaming platform can better understand the user's preferences and optimize subsequent audio output based on these preferences to provide a more personalized and satisfying gaming experience.

[0045] Step S120: Filter the mono cloud game audio data based on the user audio preference information to obtain mono initial audio data.

[0046] Step S120 may specifically include:

[0047] Performing feature extraction processing on the user's audio preference information to obtain preference feature information;

[0048] Perform dynamic frame processing on the mono-channel cloud game audio data to obtain multiple short-term audio data segments;

[0049] Based on the preference feature information, the multiple short-term audio data are filtered to obtain the filtered multiple short-term audio data;

[0050] The multiple short-term audio data segments after filtering are concatenated to obtain the mono initial audio data.

[0051] Exemplarily, first extract the key features u of the user audio preference set i , i = 1, 2, ..., n, n is the number of features of set U, u i is the characteristic of set U.

[0052] Then, the monophonic cloud game audio data is dynamically framed to obtain multiple segments of short-term audio data, which may specifically include:

[0053] Select a group of recorded mono-channel cloud game audio, the duration of the audio is T, T can be set to 100 milliseconds. Divide each group of mono-channel cloud game audio into sub-time series of duration t, t can be set to 10 milliseconds.

[0054] Perform dynamic frame processing on each set of audio data to obtain a short-term audio data set Audio init , the set size is μ:

[0055]

[0056] Among them, rate represents the audio sampling rate; T represents the total time series of mono-channel cloud game audio; t represents the sub-time series; peak t 、valley t Respectively represent the number of audio peaks and troughs in the time series t; Represents the ceiling function.

[0057] When there are more peaks and troughs in the audio signal, it indicates that the audio information is more complex or changes frequently. In this case, in order to capture more details, the frame length of the sub-frame is increased so that each frame contains more audio information. In addition, the size of the entire audio data set will be reduced accordingly, so that information-intensive segments can be processed more effectively. On the contrary, when there are fewer peaks and troughs, that is, the amount of information is sparse, the frame length will be reduced and the set size will be increased, so that these relatively simple or less changing time periods can be processed more finely.

[0058] Based on this, dynamic frame processing of monophonic cloud game audio can divide the time periods with concentrated information in the audio data into one group, and the parts with less information into another group, thereby effectively analyzing and processing the audio data and ensuring the accuracy and efficiency of audio processing.

[0059] From the collection Audio init Each short-time audio data is extracted in sequence, and the short-time audio data is obtained by dynamic frame processing. Short-time Fourier transform is performed on each short-time audio data, and the time domain audio data is converted into frequency domain data to obtain short-time audio frequency domain data. The frequency domain data of each short-time audio can be obtained by short-time Fourier transform, including the frequency components of the audio and the corresponding amplitude information.

[0060] In the frequency domain, different types of sounds (such as human voices, background noise, etc.) have obvious differences in frequency distribution, which enables the filtering algorithm to effectively remove or retain certain frequency components based on preset user preferences or specific audio characteristics, thereby performing filtering processing more accurately.

[0061] For example, the key feature u i Filtering the mono-channel cloud game audio collected in step S110 to filter out audio data that does not meet the user audio preference set may specifically include:

[0062] According to the audio preference information set U, the short-term audio frequency domain data is matched. During the matching process, the audio frequency domain data that does not meet the key feature u is identified and filtered out. i For example, if the user prefers not to include high frequency noise, the filter will reduce the impact of these frequency components.

[0063] The filtered short-time audio frequency domain data is inversely transformed by Fourier to generate short-time audio data. After each short-time audio data segment in the monophonic cloud game audio of duration T is filtered, a filtered short-time audio data set Audio is generated. filter .

[0064] Finally, the short-term audio data set Audio filterThe original time series of the mono cloud game audio is spliced ​​together in sequence, and the filtered data is replaced with silent data to form complete mono initial audio data.

[0065] Step S130: Generate multi-channel audio data based on the cloud game scenario using a pre-constructed audio generation model.

[0066] In order to enhance the immersion and sense of reality of the game, multi-channel audio data can be generated according to different game scenes. In this embodiment, a trained audio generation model is used to generate multiple mono audio data corresponding to the cloud game video frame according to different cloud game scenes, and the multiple mono audio data are adjusted and merged to obtain multi-channel audio data.

[0067] Exemplarily, step S130 may include:

[0068] Get game scene images and game mono audio datasets;

[0069] Selecting a plurality of mono audio data from a mono audio data set of a game based on a game scene picture using an audio generation model to be trained;

[0070] Obtain the correlation between the game scene image and multiple mono audio data;

[0071] When the correlation meets the preset conditions, the audio generation model training is completed.

[0072] For example, Figure 2 This is a schematic diagram of a process for generating multi-channel audio data based on a cloud gaming scenario provided by an exemplary embodiment of the present disclosure, such as Figure 2 As shown, the following steps may be specifically included:

[0073] Step S131: pre-process the game scene data.

[0074] First, obtain a dataset of game scene images, analyze the visual elements of the image dataset through a deep learning model, obtain a scene feature set P in the game scene, and design and generate audio content based on the scene feature set. The scene feature set includes but is not limited to the layout of the scene, main objects (such as buildings, vegetation), environmental conditions (such as weather conditions), and dynamic elements of the scene (such as moving vehicles, characters). The scene feature set can help engineers and algorithms decide what types of environmental sounds and special effects should be added, and how the spatial positions of these sounds should be distributed to achieve the most realistic multi-channel audio effect.

[0075] Then, it is necessary to collect and record a mono audio dataset that matches the game scene. Mono audio datasets can be captured from different resources by recording sounds on-site (such as water sounds or gunshot sounds), using digital audio libraries, or through web crawler technology. The audio data in the mono audio dataset should maintain the same sampling rate (such as sampleRate) and sampling accuracy (such as 16bit) as the audio collected by the game to ensure consistency in sound quality. The mono audio dataset includes multiple track types k, such as front left / right channels, center channels, left / right surround channels, and bass effect channels.

[0076] After collecting the mono audio data set, the audio feature set A can be extracted through deep learning technology, including the frequency characteristics, time domain characteristics of the audio and its correlation with the game scene. In addition, according to different game scenes, different audio data are arranged into multiple channels to construct realistic multi-channel audio.

[0077] Step S132: Generate multi-channel audio data according to the game scene data.

[0078] For example, the required number of channels x can be determined according to the game's screen dimension (one-dimensional, two-dimensional or three-dimensional) and the player's perspective (such as first-person or third-person), where x≤k, and k is the maximum number of channels currently supported. For example, a first-person gunfight game requires more channels to provide a deeper immersive experience, while a traditional platform jumping game may only require audio data from four channels: front, back, left, and right.

[0079] Then, the game scene images are processed for feature extraction to obtain scene features. Select images from the game scene image dataset in turn, and extract the scene feature set P of the image through the convolutional neural network. i , i=1,2,…,m.

[0080] Then, feature extraction is performed on multiple mono audio data to obtain audio features. The audio generation model is used to randomly select x mono audio data from the mono audio data set. The feature set of each mono audio data is A j (x), j = 1, 2,…, n.

[0081] Calculate the correlation between the scene feature set of the image and the feature set of randomly selected mono audio data:

[0082]

[0083] Where γ represents the correlation of multiple mono audio data generated by the picture; P i A represents the i-th scene feature value of the image; j (k) represents the j-th eigenvalue of the k-th mono audio.

[0084] Finally, when the relevance meets the preset conditions, the audio generation model training is completed.

[0085] Exemplarily, the discrete case of calculating the correlation between the scene feature set of the picture and the mono audio data if If is less than the threshold ε, the selected x mono audio data are synthesized into the multi-channel audio data of the picture. If it is greater than or equal to the threshold ε, reselect x mono audio data and repeat step S132 until the audio generation model training is completed.

[0086] Exemplarily, step S130 may further include:

[0087] Obtain a cloud game video frame, and input the cloud game video frame into a pre-constructed audio generation model to generate multiple mono audio data corresponding to the cloud game video frame;

[0088] Based on the game scene corresponding to the cloud game video frame, multiple mono audio data are adjusted and merged to obtain multi-channel audio data.

[0089] Step S133: Generate multi-channel audio using a multi-channel generation model in combination with the game scene.

[0090] In the cloud gaming platform, the video frames of the cloud games are collected in real time and input into the trained audio generation model to generate x mono audio data corresponding to the video frames. According to the acoustic characteristics of the human body and the game scene corresponding to the video frame, each mono audio data is adjusted and processed. The adjustment process includes adjusting the time delay, intensity and frequency response of the audio. For example, in a scene of a large open space, distant sounds need to have a larger time delay and lower intensity. This adjustment ensures the directionality and sense of distance of the sound, allowing players to perceive the location and distance of the sound source in the game.

[0091] Finally, the adjusted x mono audio data tracks are merged to generate a multi-channel audio data with x channels. The synthesized multi-channel audio data can provide a rich stereo environment and enhance the player's sense of immersion and spatial positioning.

[0092] Step S140: synthesize the mono-channel initial audio data and the multi-channel audio data to obtain target multi-channel game audio.

[0093] Exemplarily, the mono-channel initial audio data obtained in step S120 and the multi-channel audio data obtained in step S130 may be mixed and superimposed to generate target multi-channel game audio.

[0094] Before mixing and superimposing, the head-related transfer function is used to convert the mono initial audio data into multi-channel initial audio data. First, the mono initial audio data needs to be converted into x-channel audio to obtain multi-channel initial audio data. This conversion can be achieved by using HRTF (Head Related Transfer Function). HRTF is a function that takes into account the influence of the shape of the human head and ears on sound propagation, and can simulate the influence of sounds from different directions on hearing. By convolving the mono initial audio data with the impulse response of HRTF, multi-channel audio with a sense of spatial positioning can be generated, allowing players to feel the directionality and distance of the sound.

[0095] Since the sampling precision of each channel is 16 bits, the mixing process may exceed the 16-bit range, resulting in overflow. Therefore, the adaptive attenuation factor mixing algorithm can be used to synthesize the multi-channel initial audio data and the multi-channel audio data to obtain the target multi-channel game audio. The adaptive attenuation factor mixing algorithm dynamically adjusts the attenuation factor to effectively avoid the dynamic range in the mixing process exceeding the limit that can be represented by the 16-bit sampling precision without sacrificing the sound quality. This not only prevents sound distortion, but also ensures the smoothness and naturalness of the mixed audio. It is suitable for processing high dynamic range audio mixing problems that may cause digital audio overflow.

[0096] Exemplarily, an adaptive attenuation factor mixing algorithm may be used to mix and superimpose the multi-channel initial audio data and the multi-channel audio data generated based on the game scene, which may be expressed as:

[0097] output=(A1(t)+A2(t))*f (3)

[0098] Among them, f represents the initial attenuation factor, which is set to 1; A1(t) represents the multi-channel initial audio data after the mono-channel initial audio data is converted; A2(t) represents the multi-channel audio data generated by the game scene.

[0099] Finally, check whether the mixed target multi-channel game audio (output) exceeds the preset maximum (MAX) or minimum (MIN). If it exceeds the maximum value, the attenuation factor should be appropriately reduced to avoid overflow; if it is lower than the minimum value, the attenuation factor should be increased to improve the signal strength. In addition, after processing each frame of audio data, if the attenuation factor f is less than 1 (meaning that the signal has been attenuated), the attenuation factor should be gradually increased with a small step size adjustment so that the signal gradually returns to a normal level to prevent excessive attenuation of the signal, thereby affecting the overall quality of the mix.

[0100] Through the adaptive mixing and superposition algorithm, the attenuation factor can be dynamically adjusted according to the specific characteristics of the audio signal to ensure that overflow is avoided during the mixing process, while ensuring the smoothness and naturalness of the mixing effect to achieve high-quality audio output.

[0101] One or more technical solutions provided in the exemplary embodiments of the present disclosure filter the mono-channel cloud game audio according to the user audio preference information, obtain the mono-channel initial audio data, and generate multi-channel audio data according to the game scene. Finally, the mono-channel initial audio data that meets the user's preference and the multi-channel audio data generated based on the game scene are mixed and superimposed to generate the target multi-channel game audio.

[0102] Therefore, the cloud game audio generation method provided in the exemplary embodiment of the present disclosure can not only meet the user's personal preferences, but also make the output target multi-channel game audio more in line with the current game scene, thereby increasing the expressiveness and immersion of the game audio.

[0103] The above mainly introduces the solutions provided by the exemplary embodiments of the present disclosure. It is understandable that in order to implement the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0104] The exemplary embodiments of the present disclosure can divide the functional units of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the exemplary embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0105] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a cloud game audio generating device, which can be a server or a chip applied to a server. Figure 3 This is a schematic block diagram of the functional modules of a cloud gaming audio generation device provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the cloud game audio generation device 300 includes:

[0106] A data acquisition module 310 is used to acquire monophonic cloud game audio data and user audio preference information;

[0107] A data filtering module 320, configured to filter the mono-channel cloud game audio data based on the user audio preference information to obtain mono-channel initial audio data;

[0108] A data processing module 330 is used to generate multi-channel audio data based on a cloud game scene using a pre-constructed audio generation model;

[0109] The data synthesis module 340 is used to synthesize the mono-channel initial audio data and the multi-channel audio data to obtain target multi-channel game audio.

[0110] In another embodiment provided by the present disclosure, the data filtering module 320 is further used to perform feature extraction processing on the user audio preference information to obtain preference feature information; perform dynamic frame processing on the mono cloud game audio data to obtain multiple short-time audio data; filter the multiple short-time audio data based on the preference feature information to obtain multiple filtered short-time audio data; and splice the multiple filtered short-time audio data to obtain mono initial audio data.

[0111] In another embodiment provided by the present disclosure, the data filtering module 320 is further used to perform short-time Fourier transform on the short-time audio data to obtain short-time audio frequency domain data; perform matching processing on the short-time audio frequency domain data based on the preference feature information to obtain filtered short-time audio frequency domain data; perform inverse Fourier transform on the filtered short-time audio frequency domain data to generate filtered short-time audio data.

[0112] In another embodiment provided by the present disclosure, the data processing module 330 is also used to obtain cloud game video frames, and input the cloud game video frames into a pre-constructed audio generation model to generate multiple mono-channel audio data corresponding to the cloud game video frames; based on the game scenes corresponding to the cloud game video frames, adjust and merge the multiple mono-channel audio data to obtain the multi-channel audio data.

[0113] In another embodiment provided by the present disclosure, the data processing module 330 is also used to obtain a game scene picture and a game mono audio data set; select multiple mono audio data from the game mono audio data set based on the game scene picture using the audio generation model to be trained; obtain the correlation between the game scene picture and the multiple mono audio data; when the correlation meets a preset condition, the audio generation model training is completed.

[0114] In another embodiment provided by the present disclosure, the data processing module 330 is further used to perform feature extraction processing on the game scene image to obtain scene features; perform feature extraction processing on the multiple mono audio data respectively to obtain audio features; and calculate the correlation between the scene features and the audio features.

[0115] In another embodiment provided by the present disclosure, the data synthesis module 340 is further used to convert the monophonic initial audio data into multi-channel initial audio data by adopting a head-related transfer function; and to synthesize the multi-channel initial audio data and the multi-channel audio data by adopting an adaptive attenuation factor mixing algorithm to obtain the target multi-channel game audio.

[0116] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0117] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0118] Figure 4 The block diagram of the electronic device provided as an example of the present disclosure, the block diagram of the electronic device 400 that can be used as the server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0119] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0120] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0121] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. Each of the methods described above may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409.

[0122] Figure 5This is a schematic diagram of an exemplary computer program product provided by the present disclosure. The exemplary embodiment of the present disclosure also provides a computer program product 500, including a computer program 501, wherein the computer program 501 is used to enable the computer to perform a method according to an embodiment of the present disclosure when executed by a processor of the computer.

[0123] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0128] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0129] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0130] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A cloud game audio generation method, characterized in that: The method comprises: Obtain mono cloud game audio data and user audio preference information; Filtering the mono-channel cloud game audio data based on the user audio preference information to obtain mono-channel initial audio data; Use a pre-built audio generation model to generate multi-channel audio data based on cloud gaming scenarios; The mono-channel initial audio data and the multi-channel audio data are synthesized to obtain target multi-channel game audio.

2. The method according to claim 1, characterized in that The filtering the mono-channel cloud game audio data based on the user audio preference information to obtain mono-channel initial audio data includes: Performing feature extraction processing on the user audio preference information to obtain preference feature information; Performing dynamic frame processing on the monophonic cloud game audio data to obtain multiple segments of short-term audio data; Based on the preference feature information, the multiple segments of short-time audio data are filtered to obtain multiple segments of filtered short-time audio data; The multiple short-term audio data segments after filtering are concatenated to obtain the mono initial audio data.

3. The method according to claim 2, characterized in that The filtering the multiple short-term audio data segments based on the preference feature information includes: Performing short-time Fourier transform on the short-time audio data to obtain short-time audio frequency domain data; Performing matching processing on the short-time audio frequency domain data based on the preference feature information to obtain filtered short-time audio frequency domain data; Perform inverse Fourier transform on the filtered short-time audio frequency domain data to generate filtered short-time audio data.

4. The method according to claim 1, characterized in that: The method of using a pre-built audio generation model to generate multi-channel audio data based on a cloud game scenario includes: Obtain a cloud game video frame, and input the cloud game video frame into a pre-constructed audio generation model to generate multiple mono audio data corresponding to the cloud game video frame; Based on the game scene corresponding to the cloud game video frame, multiple mono-channel audio data are adjusted and merged to obtain the multi-channel audio data.

5. The method according to claim 4, characterized in that The method further comprises: Get game scene images and game mono audio datasets; Selecting a plurality of mono audio data from the game mono audio data set based on the game scene picture using the audio generation model to be trained; Obtaining the correlation between the game scene image and the plurality of mono audio data; When the correlation satisfies a preset condition, the training of the audio generation model is completed.

6. The method according to claim 5, characterized in that The obtaining of the correlation between the game scene image and the plurality of mono audio data comprises: Performing feature extraction processing on the game scene picture to obtain scene features; Performing feature extraction processing on the multiple mono audio data respectively to obtain audio features; The correlation between the scene feature and the audio feature is calculated.

7. The method according to claim 1, characterized in that The synthesizing the mono-channel initial audio data and the multi-channel audio data to obtain target multi-channel game audio includes: Converting the monophonic initial audio data into multi-channel initial audio data using a head-related transfer function; The multi-channel initial audio data and the multi-channel audio data are synthesized by using an adaptive attenuation factor mixing algorithm to obtain the target multi-channel game audio.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method of claim 1.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method of claim 1 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method of claim 1 is implemented.

Images