Audio processing method and device

Abstract

The invention provides an audio processing method and device, and the method comprises the steps: determining a target audio and a scene audio in a target object, adjusting the frequency range of the scene audio based on the key frequency band of the target audio, and obtaining a target scene audio in the target frequency range, the frequency band conflict of the target audio and the scene audio in the frequency dimension is reduced, and the fusion effect of the target audio and the scene audio is improved. The target audio and the target scene audio are integrated into the object audio of the target object, the object audio with high expressive force is obtained, and frequency band interference is reduced. The harmonious matching of the target audio and the scene audio can improve the auditory experience of the user and enhance the immersion of the user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio processing method and device. Background Art

[0002] In modern multimedia applications, games, and virtual reality scenarios, the combination of sound effects and music is a key factor in enhancing the user's immersive experience. When sound effects and music are played at the same time, it will cause audio effect interference. This interference will reduce the user experience and affect the expressiveness of the product.

[0003] In the prior art, interference is usually reduced by manually adjusting the tonality or frequency of sound effects and music, but this method is inefficient and it is difficult to ensure the fusion effect of sound effects and music in dynamic and changing scenes. Therefore, a more effective audio processing method is urgently needed to solve the above problems. Summary of the invention

[0004] In view of this, the embodiment of the present application provides an audio processing method to solve the technical defects existing in the prior art. The embodiment of the present application also provides an audio processing device, a computing device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect of an embodiment of the present application, there is provided an audio processing method, comprising: determining target audio and scene audio in a target object; Adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain target scene audio within a target frequency range; The target audio and the target scene audio are integrated into the object audio of the target object.

[0006] Optionally, after determining the target audio and scene audio in the target object, the method further includes: The sound effect tonality of the scene audio is adjusted based on the tuning range and the audio tonality of the target audio to obtain an initial scene audio with an initial sound effect tonality.

[0007] Optionally, the adjusting the sound effect tonality of the scene audio based on the tuning range and the audio tonality of the target audio to obtain the initial scene audio with the initial sound effect tonality includes: Determining the tuning range, the sound effect tonality of the scene audio, and the audio tonality and timbre of the target audio; Determine a pitch shift direction and a pitch shift degree based on the tuning range and the audio tonality and timbre of the target audio; The sound effect tonality of the scene audio is adjusted according to the pitch shift direction and the pitch shift degree to obtain the initial scene audio with the initial sound effect tonality.

[0008] Optionally, determining the audio tonality of the target audio includes: Identifying the main tonality of the target audio based on a main tonality recognition algorithm, and identifying the auxiliary tonality of the target audio based on an auxiliary tonality recognition algorithm; The main tonality and the auxiliary tonality are used as the audio tonality of the target audio.

[0009] Optionally, determining the key frequency band of the target audio includes: Performing spectrum recognition on the target audio, and determining a main melody frequency band and an audio frequency band in the target audio according to the spectrum recognition result; Audio feature recognition is performed on the main melody frequency band and the audio frequency band, and the key frequency band is determined according to the feature recognition result, the main melody frequency band and the audio frequency band.

[0010] Optionally, before adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain the target scene audio in the target frequency range, the method further includes: Performing frequency band detection on the key frequency band of the target audio and the scene frequency band of the scene audio; When it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range.

[0011] Optionally, the adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain the target scene audio in the target frequency range includes: Determining speech audio and sound effect audio in the scene audio, and determining a frequency range of the speech audio and a frequency range of the sound effect audio; Performing gain adjustment on the frequency range of the speech audio based on the frequency range of the key frequency band to obtain a target speech audio within the gain frequency range; Based on the frequency range of the key frequency band, the frequency range of the sound effect audio is reduced and adjusted to obtain the target sound effect audio with the reduced frequency range; The target voice audio and the target sound effect audio are used as the target scene audio.

[0012] Optionally, after determining the target audio and scene audio in the target object, the method further includes: determining an object scene of the target object; Determining a volume ratio of the target audio and the scene audio according to the object scene; The volume of the target audio and / or the volume of the scene audio is adjusted based on the volume ratio.

[0013] Optionally, after determining the target audio and scene audio in the target object, the method further includes: Determining the volume of the target audio, and determining a volume threshold based on the scene audio; When the volume of the target audio is greater than the volume threshold, the volume of the target audio is reduced.

[0014] According to a second aspect of an embodiment of the present application, there is provided an audio processing device, including: A determination module configured to determine target audio and scene audio in a target object; An adjustment module is configured to adjust the frequency range of the scene audio based on the key frequency band of the target audio to obtain a target scene audio within a target frequency range; An integration module is configured to integrate the target audio and the target scene audio into the object audio of the target object.

[0015] According to a third aspect of an embodiment of the present application, a computing device is provided, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor implements the steps of the audio processing method when executing the computer-executable instructions.

[0016] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the audio processing method are implemented.

[0017] According to a fifth aspect of the embodiments of the present application, a chip is provided, which stores a computer program, and the computer program implements the steps of the audio processing method when executed by the chip.

[0018] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned audio processing method when executed by a processor.

[0019] The audio processing method provided in the present application can adjust the scene audio in the target object to achieve the effect of fusion of the target audio and the scene audio. Determine the target audio and scene audio in the target object, adjust the frequency range of the scene audio based on the key frequency band of the target audio, obtain the target scene audio in the target frequency range, reduce the frequency band conflict between the target audio and the scene audio in the frequency dimension, and enhance the fusion effect of the target audio and the scene audio. Integrate the target audio and the target scene audio into the object audio of the target object, obtain object audio with higher expressiveness, and reduce frequency band interference. The harmonious matching of the target audio and the scene audio can enhance the user's auditory experience and enhance the user's sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a structural diagram of an audio processing method provided by an embodiment of the present application; Figure 2 is a flow chart of an audio processing method provided by an embodiment of the present application; Figure 3 is a processing flow chart of an audio processing method applied to game audio provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of an audio processing device provided by an embodiment of the present application; Figure 5 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0021] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0022] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0024] First, the terms involved in one or more embodiments of the present invention are explained.

[0025] Autocorrelation algorithm: Based on the autocorrelation of the signal, the pitch is determined by calculating the autocorrelation function of the audio signal at different delays. The autocorrelation function will reach a peak at the delay corresponding to the pitch. This algorithm is simple and easy to implement, but it may have a high computational complexity in real-time applications.

[0026] FFT algorithm: Converts the audio signal in the time domain to the frequency domain and determines the pitch by analyzing the peak position on the spectrum. Common methods include techniques based on spectral peaks or pitch estimation. The FFT algorithm can efficiently calculate the spectrum and is suitable for real-time pitch detection.

[0027] Time domain-based algorithms: mainly use autocorrelation functions or difference functions to detect periodic features in audio signals. Suitable for identifying chords or single tone signals with insignificant pitch changes. By analyzing the periodicity of audio signals on the time axis, signal features with specific periods are identified, thereby inferring chord types and structures. Suitable for simple tracks or background music with simple tones.

[0028] Frequency domain-based algorithm: By performing Fourier transform on the audio signal, the time domain signal is converted into a frequency domain signal, its spectrum information is extracted, and the spectrum characteristics of the chord tone are used for detection. The audio signal is processed by FFT to obtain the spectrum information, and the chord recognition is performed by analyzing the amplitude of different frequencies in the spectrum and their combination patterns. It is suitable for complex music signals and can accurately locate the frequency components of various chords.

[0029] In modern multimedia applications, games, and virtual reality scenarios, the combination of sound effects and music is a key factor in enhancing the user's immersive experience. When sound effects and music are played at the same time, audio effects will be interfered with. This interference will reduce the user experience and affect the expressiveness of the product. In the prior art, interference is usually reduced by manually adjusting the tonality or frequency of sound effects and music, but this method is inefficient and it is difficult to ensure the fusion effect of sound effects and music in dynamic and changing scenarios. Figure 1This is a schematic diagram of the structure of an audio processing method provided by an embodiment of the present application. The audio processing method can adjust the scene audio in the target object to achieve the effect of merging the target audio with the scene audio. The target object can be multimedia content such as games, videos, animations, etc.

[0030] Determine the target audio and scene audio in the target object, adjust the frequency range of the scene audio based on the key frequency band of the target audio, obtain the target scene audio in the target frequency range, reduce the frequency band conflict between the target audio and the scene audio in the frequency dimension, and improve the fusion effect of the target audio and the scene audio. Integrate the target audio and the target scene audio into the object audio of the target object, obtain object audio with high expressiveness, and reduce frequency band interference. The harmonious matching of the target audio and the scene audio can enhance the user's auditory experience and enhance the user's sense of immersion.

[0031] In the present application, an audio processing method is provided. The present application also relates to an audio processing device, a computing device, a computer-readable storage medium, and a computer program product. The following embodiments will describe them in detail one by one.

[0032] Figure 2 A flowchart of an audio processing method provided according to an embodiment of the present application is shown, which specifically includes the following steps: Step 202: Determine the target audio and scene audio in the target object.

[0033] Specifically, the target object can be a virtual reality scene such as a game, animation, or video. In a game scene, the target game includes but is not limited to online games, PC games, and VR games. The target audio can be the background music of the game, and the scene audio is the sound effect generated by a specific operation triggered by a game player or a game character in the game scene. For example, operations such as game players selecting characters, selecting weapons, inputting voice or text, and sending voice or text will produce different sound effects, and in the process of performing tasks, actions such as fighting, picking up items, running, and walking by game characters will also trigger special sound effects. Similarly, in an animation scene, the target audio can be the background music in the animation, and the sound effect audio is the sound effects generated by people, animals, cars, and other things in the animation scene.

[0034] Furthermore, considering that the scene audio in the target object is added based on the scene in the target object, there may be a difference in tonality between the scene audio and the target audio. In order to improve the fusion effect of the audio in the target object, the sound effect tonality of the scene audio can be adjusted, which is specifically implemented as follows: The sound effect tonality of the scene audio is adjusted based on the tuning range and the audio tonality of the target audio to obtain an initial scene audio with an initial sound effect tonality.

[0035] Specifically, the tuning range may be a tolerance range, which refers to the pitch difference between the target audio and the scene audio. The tuning range may be set according to actual needs. The audio tonality of the target audio refers to the main tonality of the target audio, such as C major, A minor, etc. The sound effect tonality of the scene audio is the main tonality of the scene audio. The initial sound effect tonality refers to the sound effect tonality obtained by adjusting the sound effect tonality of the scene audio according to the audio tonality of the target audio.

[0036] Based on this, the audio tonality of the target audio and the sound effect tonality of the scene audio are determined. A tuning range is determined, and the sound effect tonality of the scene audio is adjusted based on the tuning range and the audio tonality of the target audio, so that the sound effect tonality of the scene audio matches the audio tonality of the target audio, and an initial scene audio with an initial sound effect tonality is obtained.

[0037] For example, in an animation scene, the background music in the animation is the target audio, and the sound effects produced by the characters' activities in the animation are the scene audio. If there is a tonality mismatch between the target audio and the scene audio, the sound effect tonality of the scene audio can be adjusted with reference to the audio tonality of the target audio, so that the sound effect tonality of the scene audio matches the audio tonality of the target audio, achieving audio tonality harmony.

[0038] In summary, the sound effect tonality of the scene audio is adjusted with reference to the audio tonality of the target audio to ensure the tonality harmony between the scene audio and the target audio, thereby improving the fusion effect of the target audio and the scene audio in the target object.

[0039] Furthermore, it is necessary to reserve degrees of freedom between the target audio and the sound effect audio in the tonality dimension of the audio, so that the fusion between the target audio and the sound effect audio is more natural. The specific implementation is as follows: Determine the tuning range, the sound effect tonality of the scene audio, and the audio tonality and timbre of the target audio; determine the pitch shift direction and the pitch shift degree based on the tuning range and the audio tonality and timbre of the target audio; adjust the sound effect tonality of the scene audio according to the pitch shift direction and the pitch shift degree to obtain the initial scene audio of the initial sound effect tonality.

[0040] Specifically, the timbre of the target audio may be a chord of the target audio. The pitch shift direction refers to the direction of adjusting the sound effect tonality of the scene audio, and the pitch shift direction may be an upward shift or a downward shift based on the sound effect tonality of the scene audio. The pitch shift degree refers to the degree of shift of the sound effect tonality of the scene audio, and may be a shift of one tone, a shift of half a tone, and so on.

[0041] Based on this, determine the tuning range, the sound effect tonality of the scene audio, and the audio tonality and timbre of the target audio. Determine the pitch shift direction and pitch shift degree within the tuning range based on the tuning range and the audio tonality and timbre of the target audio. Determine the shift direction of the sound effect tonality of the scene audio based on the pitch shift direction, determine the tonality shift degree of the sound effect tonality of the scene audio based on the pitch shift degree, and adjust the sound effect tonality of the scene audio based on the shift direction and tonality shift degree of the sound effect tonality of the scene audio to obtain the initial scene audio of the initial sound effect tonality. When determining the pitch shift direction, the range can usually be considered, that is, a direction that does not exceed the available range is selected; timbre consistency, that is, considering the overall timbre of the background music, a direction that is closer to the timbre harmony is selected; interval distance, that is, the smallest adjustment amplitude is usually selected to achieve tonality matching to reduce harmony changes. If the adjustment amplitudes of the two directions are similar, fine-tuning selection is made based on the specific music context.

[0042] Continuing with the above example, in an animation scene, the background music in the animation is the target audio, and the sound effects produced by the character activities in the animation are the scene audio. When the background music is determined to be in A major, even if the sound effects use certain notes in A major, they can be allowed to deviate slightly within a certain tolerance range (such as chords that lean toward C major), but still remain within the scale that is coordinated with the background music. The tuning range is the tolerance range, which is usually set to a small interval, such as a minor second or a major second. This allows for slight pitch deviations when matching the tonality, ensuring that the adjusted sound effects still sound natural. The basis for setting is usually based on the suitability test of the ears and the requirements of the music style.

[0043] In summary, the sound effect tonality of the scene audio is adjusted according to the pitch shift direction and the pitch shift degree to obtain the initial scene audio with the initial sound effect tonality, thereby ensuring the harmony between the scene audio and the target audio and improving the overall sound effect.

[0044] Furthermore, considering that the audio tonality of the target audio is different, a main tonality recognition algorithm can be used when identifying the audio tonality of the target audio, which is specifically implemented as follows: The main tonality of the target audio is identified based on a main tonality recognition algorithm, and the auxiliary tonality of the target audio is identified based on an auxiliary tonality recognition algorithm; and the main tonality and the auxiliary tonality are used as the audio tonality of the target audio.

[0045] Specifically, the main tonality recognition algorithm may be a pitch analysis algorithm, such as an FFT algorithm, an autocorrelation algorithm, etc. The auxiliary tonality recognition algorithm may be a chord recognition algorithm, such as an algorithm based on the time domain, an algorithm based on the frequency domain, etc. The auxiliary tonality of the target audio may be the temporary modulation or harmonic color of the music in addition to the main tonality of the target audio, covering the chord changes and modal transitions in different phrases.

[0046] Based on this, the main tonality of the target audio is identified based on the main tonality recognition algorithm. The auxiliary tonality such as chord changes and modal transitions of the target audio is identified based on the auxiliary tonality recognition algorithm. The main tonality and auxiliary tonality are used as the audio tonality of the target audio to provide a tonality reference for the subsequent adjustment of the scene audio.

[0047] Using the above example, when identifying the background music of an animation, you can use pitch analysis algorithms and chord recognition technology to automatically detect the tonality of the background music. First, perform pitch analysis on the background music to identify the main tonality of the background music (such as C major, A minor, etc.), and then analyze the chords of the background music. Analyze the note sequence of the background music through a time-domain-based algorithm or a frequency-domain-based algorithm, extract the main pitch range and chords, and determine the auxiliary tonality of the background music.

[0048] In summary, the main tonality and the auxiliary tonality are used as the audio tonality of the target audio to provide a tonality reference for subsequent adjustment of the scene audio, which can improve the accuracy of subsequent adjustment of the sound effect tonality of the scene audio.

[0049] Furthermore, during the playback of the target audio and scene audio of the target object, the volume of the target audio and scene audio can be monitored and adjusted so that the fusion of the target audio and scene audio is more harmonious. The specific implementation is as follows: Determine an object scene of the target object; determine a volume ratio of the target audio and the scene audio according to the object scene; and adjust the volume of the target audio and / or the volume of the scene audio based on the volume ratio.

[0050] Based on this, the object scene of the target object is determined. The object scene can be a scene in the target object, such as a fighting scene or a character conversation scene. The volume ratio of the target audio and the scene audio is determined according to the object scene. The volume of the target audio and / or the volume of the scene audio are adjusted based on the volume ratio to achieve a volume balance between the target audio and the scene audio.

[0051] Continuing with the above example, when the scene in the animation is a character adventure or shooting scene, environmental sound effects, action sound effects, and background music will all play a key role at different moments. The volume ratio of background music and sound effects can be 1:1, that is, 50% for background music and 50% for sound effects. In an intense battle scene, the volume of background music can be reduced to 40% and the volume of sound effects increased to 60%. When exploring the environment, the background music is increased to 60% and the sound effects are reduced to 40% to enhance immersion.

[0052] In summary, the volume ratio of the target audio and the scene audio is determined according to the object scene. The volume of the target audio and / or the volume of the scene audio is adjusted based on the volume ratio to achieve volume balance between the target audio and the scene audio. This improves the audio performance of the target object and improves the user experience.

[0053] Furthermore, during the target audio playback of the target object, the volume of the target audio can also be detected in real time to ensure that the volume of the target audio is at an appropriate level, which is specifically implemented as follows: The volume of the target audio is determined, and a volume threshold is determined based on the scene audio; if the volume of the target audio is greater than the volume threshold, the volume of the target audio is reduced.

[0054] Based on this, the volume of the target audio is determined, and the volume threshold is determined based on the scene audio. When the volume of the target audio is greater than the volume threshold, the volume of the target audio is reduced to prevent the scene audio from being submerged in the target audio.

[0055] Using the above example, the background music of the animation is detected in real time. When the volume of the main melody of the background music is detected to be high (when it is more than 3 dB higher than the sound effect volume, it is considered to be high), the volume of the background music is reduced according to the volume of the sound effect. This ensures that the sound effect and background music can coexist clearly and harmoniously in any scene.

[0056] Step 204: adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain target scene audio within a target frequency range.

[0057] Specifically, after the target audio and scene audio in the target object are determined, the frequency range of the scene audio can be adjusted based on the key frequency band of the target audio to obtain the target scene audio in the target frequency range, wherein the key frequency band of the target audio can be an audio band that has a greater impact on the sound quality of the target audio. The frequency range of the scene audio can be the frequency range of the sound effect frequency band in the scene audio that overlaps with the key frequency band.

[0058] Based on this, after determining the target audio and scene audio in the target object, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to achieve frequency band avoidance and avoid audio conflicts between the key frequency bands of the scene audio and the target audio in the frequency dimension. The frequency range of the scene audio is adjusted to obtain the target scene audio in the target frequency range.

[0059] Furthermore, determining the key frequency band of the target audio includes: performing spectrum recognition on the target audio, and determining a main melody frequency band and an audio frequency band in the target audio according to the spectrum recognition result; performing audio feature recognition on the main melody frequency band and the audio frequency band, and determining the key frequency band according to the feature recognition result, the main melody frequency band and the audio frequency band.

[0060] Specifically, spectrum analysis tools can be used to perform spectrum recognition on the target audio, that is, spectrum analysis techniques such as Fourier transform are used to convert the audio signal into a frequency domain representation, and the amplitude of each frequency component is identified. The main melody frequency band refers to the frequency band corresponding to the main melody of the target audio. The audio frequency band includes the low-frequency part, the mid-frequency part and the high-frequency part of the target audio. Audio feature recognition refers to identifying the characteristic frequency bands in the audio signal. For example, the frequency band of human voice is usually between 300Hz and 3kHz, and the frequency band of drum sound is between 50Hz and 200Hz. The key frequency band refers to the frequency band that has a greater impact on the sound quality of the target audio.

[0061] Based on this, spectrum analysis techniques such as Fourier transform are used to perform spectrum recognition on the target audio, and the main melody frequency band and the audio frequency band containing the low frequency part, the mid-frequency part and the high frequency part are determined in the target audio according to the spectrum recognition results. Audio feature recognition is performed on the main melody frequency band and the audio frequency band, and feature recognition results such as human voice, instrument sound, and fighting sound are identified. Based on the feature recognition results, the degree of influence of the melody frequency band and the audio frequency band on the sound quality of the target audio, the key frequency band with a greater degree of influence is determined.

[0062] Following the above example, use the spectrum analysis tool to perform real-time analysis on the audio frequency bands of the animation's background music. Identify the main melody frequency band, low-frequency part (such as bass), mid-frequency part, and high-frequency part of the background music. Based on the results of the frequency band analysis, mark the key frequency bands of the background music, and determine the intensity and influence range of these frequency bands in the entire audio. The influence range can be expressed in frequency bands. The effective sound in a certain background music may be mainly concentrated in the range of 200Hz to 4000Hz. It can also be expressed in amplitude. This range includes the volume and frequency response of the guardian sound effect. The volume change is usually expressed in dB (decibel).

[0063] In summary, based on the feature recognition results, the influence of the melody frequency band and the audio frequency band on the sound quality of the target audio, the key frequency band with a greater influence is determined to ensure the clarity of the key frequency band of the target audio.

[0064] Furthermore, whether the key frequency band in the target audio is clear will be affected by the scene audio. When the scene audio affects the clarity of the key frequency band, the frequency range of the scene audio needs to be adjusted, which is specifically implemented as follows: Frequency band detection is performed on the key frequency band of the target audio and the scene frequency band of the scene audio; when it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range.

[0065] Specifically, the scene frequency band of the scene audio may be a performance audio or a speech audio. The key frequency band matches the scene frequency band, indicating that the scene frequency band and the key frequency band overlap in the time dimension.

[0066] Based on this, frequency band detection is performed on the key frequency band of the target audio and the scene frequency band of the scene audio to determine the frequency band length and occurrence time of the scene frequency band of the scene audio. When it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, it means that the scene frequency band and the key frequency band overlap in the time dimension, and frequency band avoidance is required for the scene audio to avoid the key frequency band of the target audio, that is, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to obtain the target scene audio in the target frequency range.

[0067] Continuing with the above example, before playing the sound effect, adjust the frequency band of the sound effect. If the frequency band of the sound effect overlaps with the main melody frequency band of the background music, modify the frequency range of the sound effect through the frequency band adjustment algorithm to avoid conflict with the main frequency band of the background music. If the low frequency band of the background music (such as 20-200Hz) is dominant, and the low frequency part of the sound effect is also concentrated in this frequency band, transfer the low frequency part of the sound effect to a higher frequency band to avoid overlap. The base tone of a piece of background music is between 300Hz and 1200Hz, of which 300Hz to 600Hz is the main melody part and 700Hz to 1000Hz is the harmony part. If the frequency band of the sound effect is between 500Hz and 800Hz, it is a harmonic interference frequency band, so the frequency band needs to be adjusted to prevent the sound effect from overlapping with these key frequency bands, thereby ensuring the clarity of the sound effect.

[0068] In summary, frequency band detection on the key frequency band of the target audio and the scene frequency band of the scene audio helps to reduce the frequency band conflict between the target audio and the scene audio, and implement frequency band avoidance for the target audio.

[0069] Furthermore, considering that the scene audio of the target object contains multiple types of audio such as voice and sound effects, it is necessary to appropriately adjust the scene audio according to different audio types, which is specifically implemented as follows: Determine speech audio and sound effect audio in the scene audio, and determine the frequency range of the speech audio and the frequency range of the sound effect audio; perform gain adjustment on the frequency range of the speech audio based on the frequency range of the key frequency band to obtain target speech audio within the gain frequency range; perform reduction adjustment on the frequency range of the sound effect audio based on the frequency range of the key frequency band to obtain target sound effect audio within the reduced frequency range; use the target speech audio and the target sound effect audio as the target scene audio.

[0070] Specifically, voice audio can be the sound of a person speaking or talking. Sound effect audio can be the sound of a musical instrument, or the sound of a person running, fighting, or other behaviors. Gain adjustment can be to enhance the frequency range, pitch, and volume of voice audio. Cut adjustment can be to weaken the frequency range, pitch, volume, and other musical attributes of sound effect audio.

[0071] Based on this, the voice audio and sound effect audio are determined in the scene audio, and the frequency range of the voice audio and the frequency range of the sound effect audio are determined. The frequency range of the voice audio is gain-adjusted based on the frequency range of the key frequency band to enhance the volume, frequency range, pitch and other properties of the voice audio, and obtain the target voice audio of the gain frequency range. The frequency range of the sound effect audio is cut-adjusted based on the frequency range of the key frequency band to weaken the volume, frequency range, pitch and other properties of the sound effect audio, and obtain the target sound effect audio of the cut frequency range. The target voice audio and target sound effect audio are used as the target scene audio.

[0072] Using the above example, when an animation includes a musical performance, the microphone receives the voices of the characters and the sounds of musical instruments. The frequency spectrum of the detected audio can be analyzed, and if a certain frequency band (for example, 500Hz-600Hz) is detected to conflict with an important melody in the background music, the gain of that frequency band is reduced, while the gain of other frequency bands is increased to ensure that the voices are clear and the musical instruments do not interfere with the background music, thus achieving a high-quality audio experience.

[0073] In summary, the frequency range of the voice audio is gain-adjusted, and the frequency range of the sound effect audio is cut-adjusted to improve the expressiveness of the audio in the target object.

[0074] Step 206: Integrate the target audio and the target scene audio into the object audio of the target object.

[0075] Specifically, after adjusting the frequency range of the scene audio based on the key frequency band of the target audio and obtaining the target scene audio in the target frequency range, the target audio and the target scene audio can be integrated into the object audio of the target object, wherein the object audio refers to the fused audio of the target object, and the object audio integrates the background music and sound effect music of the target object, so that the target object has a higher musical expressiveness.

[0076] The audio processing method provided in the present application can adjust the scene audio in the target object to achieve the effect of fusion of the target audio and the scene audio. Determine the target audio and scene audio in the target object, adjust the frequency range of the scene audio based on the key frequency band of the target audio, obtain the target scene audio in the target frequency range, reduce the frequency band conflict between the target audio and the scene audio in the frequency dimension, and enhance the fusion effect of the target audio and the scene audio. Integrate the target audio and the target scene audio into the object audio of the target object, obtain object audio with higher expressiveness, and reduce frequency band interference. The harmonious matching of the target audio and the scene audio can enhance the user's auditory experience and enhance the user's sense of immersion.

[0077] The following combination Figure 3 Taking the application of the audio processing method provided in this application to game audio as an example, the audio processing method is further described. Figure 3 A processing flow chart of an audio processing method applied to game audio provided by an embodiment of the present application is shown, which specifically includes the following steps: Step 302: Determine the background audio and sound effect audio in the target object.

[0078] In practical applications, the target object may be a multimedia resource such as a game, video, animation, etc. In this embodiment, the target object is a game as an example, the background audio may be the background music in the game scene, and the sound effect audio may be the sound effect triggered by the player or the game character in the game scene, such as the sound effect triggered by the fighting of the game character, the sound effect triggered by picking up items, and the touch sound effect generated by the player operating the game interface.

[0079] Step 304: Determine the tuning range, the sound effect tonality of the sound effect audio, and the audio tonality and timbre of the background audio.

[0080] Use pitch analysis algorithms and chord recognition technology to detect the tonality of background music. For example, first perform pitch analysis on the background music to identify the main tonality of the background music (such as C major, A minor, etc.), and analyze the chord progression of the background music. Use the algorithm to analyze the note sequence of the background music, extract its main pitch range and chords, and determine the tonality of the background music.

[0081] Step 306: Determine the pitch shift direction and pitch shift degree based on the tuning range and the timbre of the background audio.

[0082] Step 308: Adjust the sound effect tonality of the sound effect audio according to the pitch shift direction and the pitch shift degree to obtain an initial sound effect audio with an initial sound effect tonality.

[0083] If the sound effect already has a fixed key, the key of the sound effect will be adjusted to match the key of the background music. Key matching operates by transposing the pitch of the sound effect. If the key of the background music is C major and the key of the sound effect is D minor, the notes of the sound effect will be transposed to C major. The key matching algorithm transforms the notes of the sound effect to match the key of the background music.

[0084] In addition, you can also set the tolerance range for tonality matching. The purpose of tolerance setting is to avoid too strict tonality matching, so that the sound effects maintain a certain degree of freedom and avoid losing the sense of naturalness due to excessive adjustment. In the case where the background music is in C major, even if the sound effects use some notes in C major, they can be allowed to deviate slightly within a certain tolerance range (such as chords that lean towards G major), but still remain within the scale that is coordinated with the background music. Automatic tonality matching can greatly reduce the workload of manual adjustment, ensure the harmony of the pitch of the sound effects and the background music, and improve the overall audio effect.

[0085] Step 310: Perform spectrum recognition on the background audio, and determine the main melody frequency band and the audio frequency band in the background audio according to the spectrum recognition result.

[0086] Step 312: Perform audio feature recognition on the main melody frequency band and the audio frequency band, and determine the key frequency band according to the feature recognition result, the melody frequency band and the audio frequency band.

[0087] Use the spectrum analysis tool to analyze the audio frequency bands of background music in real time. Identify the main melody frequency band, low frequency part (such as bass), mid-frequency part and high frequency part of the background music. Based on the frequency band analysis results, mark the key frequency bands of the background music and determine the strength and influence range of these frequency bands in the entire audio.

[0088] Step 314: Perform frequency band detection on the key frequency band of the background audio and the scene frequency band of the initial sound effect audio.

[0089] Step 316: When it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, the frequency range of the initial sound effect audio is adjusted based on the key frequency band of the background audio to obtain the target sound effect audio within the target frequency range.

[0090] Before playing the sound effect, adjust the frequency band of the sound effect. If the frequency band of the sound effect overlaps with the main melody frequency band of the background music, the frequency range of the sound effect is modified through the frequency band adjustment algorithm to avoid conflict with the main frequency band of the background music. For example, if the low frequency band of the background music (such as 20-200Hz) is dominant, and the low frequency part of the sound effect is also concentrated in this frequency band, the low frequency part of the sound effect is transferred to a higher frequency band to avoid overlap.

[0091] In addition, when sound effects and background music are playing, the spectrum of the sound effects and background music can be monitored in real time, and the frequency band of the sound effects can be dynamically adjusted as needed. This adjustment will be adaptively adjusted according to the real-time audio input to ensure that the sound effects and background music will not interfere with each other in the key frequency band. Dynamic frequency band avoidance can reduce frequency band conflicts, make the frequency distinction between sound effects and background music clearer, avoid audio interference, and improve the fusion effect of the two.

[0092] Step 318: Determine the object scene of the target object, and determine the volume ratio of the background audio and the sound effect audio according to the object scene.

[0093] Step 320: Adjust the volume of the background audio and / or the volume of the sound effect audio based on the volume ratio.

[0094] Monitor the volume of sound effects and background music in real time with the volume monitoring tool. The volume of the main melody, bass or other important parts of the background music will affect the clarity of the sound effects. When the main melody of the background music is detected to be high, reduce the volume of the sound effects to avoid the sound effects being drowned out by the background music. You can also adjust the volume of the background music and sound effects according to the ratio of the volume of the background music and sound effects to balance the sound of the background music and sound effects, that is, dynamically adjust the volume of the sound effects and background music according to the needs of each scene. In different scenes (such as battle scenes, dialogue scenes, background scenes, etc.), the volume can be automatically adjusted according to the importance of the background music and sound effects to achieve the best audio performance.

[0095] In a battle scene, the volume of the sound effects may need to be increased to highlight the atmosphere of the battle. In a dialogue scene, the volume of the background music needs to be lowered to highlight the dialogue between the characters and the environmental sound effects. Dynamic mixing technology will adjust the volume ratio of sound effects and background music in real time according to the needs of the scene, ensuring that the sound effects and background music can coexist clearly and harmoniously in any scene. Dynamic mixing technology can adjust the volume balance of sound effects and background music in real time to avoid audio interference caused by mismatched volumes between the two, and improve the clarity and layering of audio effects.

[0096] Corresponding to the above method embodiment, the present application also provides an audio processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of an audio processing device provided by an embodiment of the present application. Figure 4As shown, the device comprises: A determination module 402 is configured to determine target audio and scene audio in a target object; An adjustment module 404 is configured to adjust the frequency range of the scene audio based on the key frequency band of the target audio to obtain a target scene audio in a target frequency range; The integration module 406 is configured to integrate the target audio and the target scene audio into the object audio of the target object.

[0097] In an optional embodiment, the determining module 402 is further configured to: The sound effect tonality of the scene audio is adjusted based on the tuning range and the audio tonality of the target audio to obtain an initial scene audio with an initial sound effect tonality.

[0098] In an optional embodiment, the determining module 402 is further configured to: Determining the tuning range, the sound effect tonality of the scene audio, and the audio tonality and timbre of the target audio; Determine a pitch shift direction and a pitch shift degree based on the tuning range and the audio tonality and timbre of the target audio; The sound effect tonality of the scene audio is adjusted according to the pitch shift direction and the pitch shift degree to obtain the initial scene audio with the initial sound effect tonality.

[0099] In an optional embodiment, the determining module 402 is further configured to: Identifying the main tonality of the target audio based on a main tonality recognition algorithm, and identifying the auxiliary tonality of the target audio based on an auxiliary tonality recognition algorithm; The main tonality and the auxiliary tonality are used as the audio tonality of the target audio.

[0100] In an optional embodiment, the adjustment module 404 is further configured to: Performing spectrum recognition on the target audio, and determining a main melody frequency band and an audio frequency band in the target audio according to the spectrum recognition result; Audio feature recognition is performed on the main melody frequency band and the audio frequency band, and the key frequency band is determined according to the feature recognition result, the main melody frequency band and the audio frequency band.

[0101] In an optional embodiment, the adjustment module 404 is further configured to: Performing frequency band detection on the key frequency band of the target audio and the scene frequency band of the scene audio; When it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range.

[0102] In an optional embodiment, the adjustment module 404 is further configured to: Determining speech audio and sound effect audio in the scene audio, and determining a frequency range of the speech audio and a frequency range of the sound effect audio; Performing gain adjustment on the frequency range of the speech audio based on the frequency range of the key frequency band to obtain a target speech audio within the gain frequency range; Based on the frequency range of the key frequency band, the frequency range of the sound effect audio is reduced and adjusted to obtain the target sound effect audio with the reduced frequency range; The target voice audio and the target sound effect audio are used as the target scene audio.

[0103] In an optional embodiment, the determining module 402 is further configured to: determining an object scene of the target object; Determining a volume ratio of the target audio and the scene audio according to the object scene; The volume of the target audio and / or the volume of the scene audio is adjusted based on the volume ratio.

[0104] In an optional embodiment, the determining module 402 is further configured to: Determining the volume of the target audio, and determining a volume threshold based on the scene audio; When the volume of the target audio is greater than the volume threshold, the volume of the target audio is reduced.

[0105] The above is a schematic scheme of an audio processing device of the present embodiment. It should be noted that the technical scheme of the audio processing device and the technical scheme of the audio processing method described above belong to the same concept. For details not described in detail in the technical scheme of the audio processing device, please refer to the description of the technical scheme of the audio processing method described above. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through a computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0106] Figure 5The block diagram of a computing device 500 provided according to an embodiment of the present application is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0107] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0108] In one embodiment of the present application, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0109] Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 500 may also be a mobile or stationary server.

[0110] The processor 520 is used to execute computer executable instructions of the audio processing method.

[0111] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above audio processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above audio processing method.

[0112] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which are used in an audio processing method when executed by a processor.

[0113] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above audio processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above audio processing method.

[0114] An embodiment of the present application further provides a chip storing a computer program, which implements the steps of the audio processing method when executed by the chip.

[0115] An embodiment of the present specification further provides a computer program product, including a computer program or instructions, which implement the steps of the above audio processing method when executed by a processor.

[0116] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above audio processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above audio processing method.

[0117] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0119] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. An audio processing method, characterized in that: include: determining target audio and scene audio in a target object; Adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain target scene audio within a target frequency range; The target audio and the target scene audio are integrated into the object audio of the target object.

2. The audio processing method according to claim 1, characterized in that: After determining the target audio and scene audio in the target object, the method further includes: The sound effect tonality of the scene audio is adjusted based on the tuning range and the audio tonality of the target audio to obtain an initial scene audio with an initial sound effect tonality.

3. The audio processing method according to claim 2, characterized in that: The step of adjusting the sound effect tonality of the scene audio based on the tuning range and the audio tonality of the target audio to obtain the initial scene audio with the initial sound effect tonality includes: Determining the tuning range, the sound effect tonality of the scene audio, and the audio tonality and timbre of the target audio; Determine a pitch shift direction and a pitch shift degree based on the tuning range and the audio tonality and timbre of the target audio; The sound effect tonality of the scene audio is adjusted according to the pitch shift direction and the pitch shift degree to obtain the initial scene audio with the initial sound effect tonality.

4. The audio processing method according to claim 2, characterized in that: The determination of the audio tonality of the target audio includes: Identifying the main tonality of the target audio based on a main tonality recognition algorithm, and identifying the auxiliary tonality of the target audio based on an auxiliary tonality recognition algorithm; The main tonality and the auxiliary tonality are used as the audio tonality of the target audio.

5. The audio processing method according to claim 1, characterized in that: Determining the key frequency band of the target audio includes: Performing spectrum recognition on the target audio, and determining a main melody frequency band and an audio frequency band in the target audio according to the spectrum recognition result; Audio feature recognition is performed on the main melody frequency band and the audio frequency band, and the key frequency band is determined according to the feature recognition result, the main melody frequency band and the audio frequency band.

6. The audio processing method according to claim 2, characterized in that: Before adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range, the method further includes: Performing frequency band detection on the key frequency band of the target audio and the scene frequency band of the scene audio; When it is determined according to the frequency band detection result that the key frequency band matches the scene frequency band, the frequency range of the scene audio is adjusted based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range.

7. The audio processing method according to claim 2, characterized in that: The step of adjusting the frequency range of the scene audio based on the key frequency band of the target audio to obtain the target scene audio within the target frequency range includes: Determining speech audio and sound effect audio in the scene audio, and determining a frequency range of the speech audio and a frequency range of the sound effect audio; Performing gain adjustment on the frequency range of the speech audio based on the frequency range of the key frequency band to obtain a target speech audio within the gain frequency range; Based on the frequency range of the key frequency band, the frequency range of the sound effect audio is reduced and adjusted to obtain the target sound effect audio with the reduced frequency range; The target voice audio and the target sound effect audio are used as the target scene audio.

8. The audio processing method according to claim 1, characterized in that: After determining the target audio and scene audio in the target object, the method further includes: determining an object scene of the target object; Determining a volume ratio of the target audio and the scene audio according to the object scene; The volume of the target audio and / or the volume of the scene audio is adjusted based on the volume ratio.

9. The audio processing method according to claim 1, characterized in that: After determining the target audio and scene audio in the target object, the method further includes: Determining the volume of the target audio, and determining a volume threshold based on the scene audio; When the volume of the target audio is greater than the volume threshold, the volume of the target audio is reduced.

10. An audio processing device, characterized in that: include: A determination module configured to determine target audio and scene audio in a target object; An adjustment module is configured to adjust the frequency range of the scene audio based on the key frequency band of the target audio to obtain a target scene audio within a target frequency range; An integration module is configured to integrate the target audio and the target scene audio into the object audio of the target object.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the audio processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the audio processing method described in any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 9 are implemented.

Images