Audio processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses an audio processing method and device, computer equipment and a storage medium, and the method provided by the invention comprises the steps: obtaining an audio feature and a preset frequency of target media data; obtaining a loudness gain of the target media data based on the audio features, wherein the loudness gain is used for carrying out loudness equalization processing on the target media data; analyzing the target media data based on a preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the to-be-filtered data, the to-be-filtered media data being media data of which the frequency is lower than the preset frequency in the target media data, and the loudness compensation gain corresponding to a target playing terminal in a loudspeaker mode; and the target playing terminal is used for carrying out loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the to-be-filtered data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Since the media data comes from different publishing sides, the same terminal needs to perform loudness equalization before playing the media data to ensure consistency of playback loudness. Summary of the invention

[0003] In view of this, the present disclosure provides an audio processing method, apparatus, computer device and storage medium to solve the problem of playback loudness consistency.

[0004] In a first aspect, the present disclosure provides an audio processing method, the method comprising:

[0005] Acquire audio characteristics and preset frequencies of target media data;

[0006] Obtaining a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data;

[0007] The target media data is analyzed based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, the media data to be filtered is media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0008] In a second aspect, the present disclosure provides an audio processing method, the method comprising:

[0009] If the current playback mode is the speaker mode, receiving target media data, a loudness gain, and a loudness compensation gain, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data;

[0010] Filtering the data to be filtered in the target media data to obtain media data to be processed;

[0011] Based on the loudness gain and the loudness compensation gain, loudness processing is performed on the media data to be processed to obtain media data for playback.

[0012] In a third aspect, the present disclosure provides an audio processing device, the device comprising:

[0013] An acquisition module, used to acquire audio features and preset frequencies of target media data;

[0014] A loudness gain determination module, configured to obtain a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data;

[0015] a loudness compensation gain determination module, configured to analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, wherein the media data to be filtered is media data having a frequency lower than the preset frequency in the target media data, the loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is configured to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0016] In a fourth aspect, the present disclosure provides an audio processing device, the device comprising:

[0017] a receiving module, configured to receive target media data, a loudness gain, and a loudness compensation gain if the current playback mode is a speaker mode, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data;

[0018] A filtering module, used to filter the data to be filtered in the target media data to obtain the media data to be processed;

[0019] The loudness gain module is used to perform loudness processing on the to-be-processed media data based on the loudness gain and the loudness compensation gain to obtain media data for playing.

[0020] In a fifth aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the above-mentioned first aspect or any corresponding implementation method thereof, or executes the above-mentioned second aspect or any corresponding implementation method of the audio processing method by executing the computer instructions.

[0021] In a sixth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the audio processing method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the above-mentioned second aspect or any corresponding embodiment thereof.

[0022] The audio processing method provided by the present disclosure has a large attenuation when playing back media data below a preset frequency at the playback terminal, which results in the loss of signal playback energy of the media data below the preset frequency. Therefore, the loudness compensation gain corresponding to the data to be filtered in the target media data is determined in the server for the target media data. Wherein, when the playback terminal is in speaker mode, before playing the target media data, since the data to be filtered has a large attenuation when playing back, the playback terminal filters the data to be filtered, and performs loudness compensation on the media data obtained after filtering based on the loudness gain and the loudness compensation gain, to obtain the media data for playback. This method can ensure that the purpose of loudness balance is achieved in speaker mode and that the consistency of loudness is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart of another audio processing method according to an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of a process for determining a loudness compensation gain according to an embodiment of the present disclosure;

[0027] Figure 4 is a flowchart of another audio processing method according to an embodiment of the present disclosure;

[0028] Figure 5 is a modular representation of audio processing according to an embodiment of the present disclosure;

[0029] Figure 6 is a structural block diagram of an audio processing device according to an embodiment of the present disclosure;

[0030] Figure 7 is a structural block diagram of an audio processing device according to an embodiment of the present disclosure;

[0031] Figure 8 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure;

[0032] Fig. 9 It is a schematic diagram of the hardware structure of another computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0034] When performing loudness equalization on the playback end, the solution adopted by the relevant technology is to pull media data (audio or video) with different loudness to the same playback loudness through loudness gain and forced limiting processing. However, since this solution does not take into account that the playback terminal is limited by the playback frequency response of the speaker in the speaker playback mode, the playback terminal has a large attenuation of low-frequency signals when using the speaker to play media data. Among them, the low-frequency signal is a signal with a frequency lower than the preset frequency. The preset frequency can be 100Hz, 150Hz or 200Hz, which is set according to actual needs. The low-frequency signal may be attenuated by 20dB to 40dB, or even a larger range of dB values.

[0035] For a large number of short video playback scenarios such as voice, music, and movies, low-frequency signals are an important part that cannot be ignored. For example, the lack of playback energy of human voices, drums, or musical instruments will inevitably lead to problems such as uneven video playback loudness and fluctuating sound when playing back with speakers after digital and loudness equalization, resulting in more frequent volume adjustments in speaker mode.

[0036] Based on this, the audio processing method provided by the present disclosure is used for loudness equalization processing in speaker mode to solve the problem of inconsistent audio loudness when playing audio and video in speaker mode. That is, in speaker mode, it is avoided to the greatest extent possible that the volume of the playback device is turned up to the maximum but still cannot be heard clearly, and the volume is adjusted in the same video or between different videos.

[0037] As an optional application scenario of the embodiment of the present disclosure, a target application is installed on the playback terminal, and the target application is used to play media data, and the media data comes from the server corresponding to the target application. During playback, the playback terminal sends a request to the server, and accordingly, the server sends the corresponding media data to the playback terminal. Among them, the number of playback terminals that the target server can connect to is set according to actual needs, and no limitation is made here. When the target server sends the media data, the corresponding content is sent according to the current playback mode of the playback terminal. The current playback mode includes but is not limited to speaker mode, headphone mode or power amplifier mode, etc., and each playback terminal communicates with the server, and accordingly, the server can know the current playback mode of the playback terminal. In the embodiment of the present disclosure, the playback mode is divided into speaker mode and non-speaker mode, wherein in the speaker mode, the content sent by the server includes media data, loudness gain and loudness compensation gain. Among them, the loudness compensation gain corresponds to the speaker mode, and the loudness compensation gain is used to compensate for the energy of the data to be filtered that is filtered out due to the inability to play back. Since audio data below the preset frequency cannot be played back normally in speaker mode, this part of data is called the data to be filtered. After the playback terminal filters this part of the data to be filtered, the energy of this part of the data is compensated by the loudness compensation gain, and the loudness is equalized by the loudness gain. In non-speaker mode, the server sends media data and loudness gain, and the loudness gain is used to equalize the media data at the playback terminal.

[0038] According to an embodiment of the present disclosure, an audio processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] In this embodiment, an audio processing method is provided, which can be used in a computer device, such as a server. That is, the loudness gain and loudness compensation gain corresponding to the target media data are determined in the server, so as to send the corresponding gain according to the current playback mode of the playback terminal. Specifically, Figure 1 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0040] Step S101: Acquire audio features and preset frequencies of target media data.

[0041] The target media data is the media data sent by the server to the playback terminal, which can be in the form of audio or video. Video can be divided into short video or long video. The length of the video is determined by the duration of the video. For example, videos with a duration less than the duration threshold are called short videos, and the rest are called long videos.

[0042] The audio features of the target media data are obtained by performing feature analysis on the target media data, and the audio features include but are not limited to duration, source loudness, type of the target media data, etc. Among them, the source loudness is the published loudness of the target media data. The type of the target media data can be voice, music, etc.

[0043] The preset frequency corresponds to the frequency that the speaker of the playback terminal cannot play back normally. For example, by testing the playback loudness of various types of playback terminals, the frequency that cannot be played back normally is obtained and used as the preset frequency. Alternatively, the size of the preset frequency is set based on an empirical value, etc. The specific value is not limited to it here, for example, it can be 100Hz, 150Hz or 200Hz, etc.

[0044] The audio feature of the target media data is used to determine a subsequent loudness gain, and the preset frequency is used to determine a subsequent loudness compensation gain.

[0045] Step S102: obtaining a loudness gain of target media data based on the audio feature.

[0046] The loudness gain is used to perform loudness equalization processing on the target media data.

[0047] The loudness gain is determined based on the source loudness and the target loudness. The source loudness is included in the audio features, while the target loudness is obtained based on the audio feature analysis. For example, the audio features include the duration, the type of target media data, and the voice ratio, etc. The target loudness is calculated based on these parameters. Alternatively, the initial loudness set by the user is obtained, and then the initial loudness is adjusted based on the audio features to obtain the target loudness.

[0048] Of course, the target loudness can also be calculated in other ways, which are specifically set according to actual needs, and the calculation method is not limited here. After determining the target loudness, the loudness gain can be determined by the difference between the target loudness and the source loudness.

[0049] In some optional implementations, when calculating the target loudness, the user personalized settings of the playback terminal may also be combined, for example, the user personalized settings, the current environment information of the playback terminal, etc. The specific settings may be made according to actual needs.

[0050] Step S103: Analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered.

[0051] Among them, the media data to be filtered is the media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to the target playback terminal in the speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0052] As described above, when the playback terminal plays back the media data, the media data with a frequency lower than the preset frequency cannot be played back normally. Based on this, the playback terminal filters out the media data that cannot be played back normally, and accordingly, loudness compensation needs to be performed for the energy of the filtered media data. Therefore, it is necessary to obtain the loudness compensation gain corresponding to the signal energy of this part of the data to be filtered.

[0053] For the target media data, it can be converted into the frequency domain to obtain the amplitude of each frequency point. Based on this, the energy of each frequency point is calculated. Due to the limitation of the preset frequency, the media data with a frequency lower than the preset frequency can be filtered out, and this part of the media data is called the media data to be filtered. Since the media data to be filtered needs to be compensated for loudness later, the signal energy of the media data to be filtered is used to determine the loudness compensation gain.

[0054] The loudness compensation gain corresponds to the playback terminal in the speaker mode, that is, the loudness compensation gain is only used for the loudness processing of the playback terminal in the speaker mode. For the playback terminal in the non-speaker mode, the loudness gain is used for loudness processing. For the same playback terminal, if it is currently processing the non-speaker mode, the loudness processing is performed through the loudness gain; if it switches to the speaker mode at the next moment, the loudness processing is performed through the loudness gain and the loudness compensation gain.

[0055] It should be noted that the loudness gain and the loudness compensation gain are both corresponding to the target media data, that is, for different target media data, the loudness gain and the loudness compensation gain are different.

[0056] The audio processing method provided in this embodiment has a large attenuation when playing back media data below the preset frequency at the playback terminal, which results in the loss of signal playback energy of this part of the media data below the preset frequency. Therefore, the loudness compensation gain corresponding to the data to be filtered in the target media data is determined in the server for the target media data. Among them, when the playback terminal is in speaker mode, before playing the target media data, since the data to be filtered has a large attenuation when playing back, the playback terminal filters the data to be filtered, and performs loudness compensation on the media data obtained after filtering based on the loudness gain and the loudness compensation gain to obtain the media data for playback. This method can ensure that the purpose of loudness balance is achieved in speaker mode and the consistency of loudness is ensured.

[0057] In this embodiment, an audio processing method is provided, which can be used in computer devices, such as servers, etc. Figure 2 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0058] Step S201, obtain the audio features and preset frequency of the target media data. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0059] Step S202: obtaining a loudness gain of target media data based on the audio feature.

[0060] The loudness gain is used to perform loudness equalization processing on the target media data. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0061] Step S203: Analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered.

[0062] Among them, the media data to be filtered is the media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to the target playback terminal in the speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0063] Specifically, the above step S203 includes:

[0064] Step S2031: Perform energy analysis on the target media data to obtain signal energy of each frequency point in the target media data.

[0065] The energy analysis of the target media data is performed by converting it to the frequency domain. After the frequency domain conversion, the signal energy of each frequency point is obtained according to the interval of the sampling frequency.

[0066] In some optional implementations, the above step S2031 includes:

[0067] Step a1, performing frequency domain conversion on target media data to obtain a target frequency domain signal.

[0068] Step a2: sampling the target frequency domain signal to obtain the amplitude of each frequency point.

[0069] Step a3, obtaining the signal energy of each frequency point based on the amplitude of each frequency point.

[0070] The frequency domain conversion of the target media data can be implemented by fast Fourier or other methods. After the frequency domain conversion, the target frequency domain signal is obtained, and the target frequency domain signal represents the corresponding relationship between frequency and amplitude. According to the preset sampling interval, the target frequency domain signal is sampled to obtain the amplitude of each frequency point. On this basis, the corresponding signal energy is calculated using the amplitude of each frequency point. That is, the signal energy is obtained based on the square of the amplitude.

[0071] For example, the target frequency domain signal of the target media data after fast Fourier transformation is A, which includes 512 frequency points, that is, A = {a1, a2, ..., a512}, where ai represents the amplitude of the i-th frequency point, i = 1, ..., 512. Correspondingly, the signal energy of each frequency point is E, E = {a1 2 , a2 2 ,…,a512 2}.

[0072] The target audio data is converted into the frequency domain to obtain the target frequency domain signal, and sampling is performed on this basis to obtain the amplitude of each frequency point, thereby obtaining the signal energy of each frequency point. That is, the amplitude of each frequency point is obtained by sampling, and the signal energy of the frequency point is obtained, which improves the accuracy of the signal energy of each frequency point.

[0073] Step S2032: obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency.

[0074] The frequency corresponding to each frequency point can be obtained through the index i of the frequency point. The frequency corresponding to each frequency point is compared with the preset frequency to filter out the data to be filtered that is lower than the preset frequency. Combined with the signal energy of each frequency point obtained in the above step S2031, the signal energy of the frequency point lower than the preset frequency can be obtained, and these signal energies are fused to obtain the signal energy of the data to be filtered.

[0075] In some optional implementations, the above step S2032 includes:

[0076] Step b1, obtaining the weight coefficient of each frequency point.

[0077] Step b2: obtaining the target signal energy of each frequency point based on the weight coefficient of each frequency point and the signal energy of each frequency point.

[0078] Step b3, screening each frequency point based on the preset frequency to obtain the signal energy of each frequency point below the preset frequency.

[0079] Step b4, obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point below the preset frequency.

[0080] The weight coefficient is used to convert the signal energy into the human ear's perception of the sound volume. The weight coefficient can be represented by a weight curve, which is used to represent the corresponding relationship between frequency and weight. As mentioned above, the frequency of each frequency point can be obtained by using the index of each frequency point, and then the weight coefficient of each frequency point can be obtained by querying the weight curve through the frequency of each frequency point.

[0081] The weight coefficient of each frequency point and the signal energy of each frequency point are weighted to obtain the target signal energy of each frequency point. Then the frequency points below the preset frequency are screened out to obtain the signal energy of each frequency point below the preset frequency. The signal energy of each frequency point below the preset frequency is fused to obtain the signal energy of the data to be filtered.

[0082] The weight coefficient of each frequency point is used to convert the calculated signal energy into the human ear's judgment of "sound volume" to obtain the target signal energy of each frequency point, thereby improving the adaptability of the obtained signal energy of the data to be filtered to the human ear.

[0083] Step S2033: obtaining a loudness compensation gain based on the signal energy of the data to be filtered.

[0084] The signal energy of the data to be filtered is in the energy domain, while the loudness is in the loudness domain. Therefore, it is necessary to use the correspondence between energy and loudness to convert the energy domain into the loudness domain to obtain the loudness compensation gain. The loudness compensation gain can be obtained by converting the signal energy into the loudness domain and then combining it with the target loudness; or, it can be obtained by the ratio of the signal energy of the data to be filtered to the total signal energy of the target media data, and so on. Of course, other methods can also be used to obtain the loudness compensation gain, which is set according to actual needs.

[0085] In some optional implementations, the above step S2033 includes:

[0086] Step c1, determining the total signal energy of the target media data based on the sum of the target signal energies at each frequency point.

[0087] Step c2: determining a loudness compensation gain based on a ratio of the signal energy of the data to be filtered to the total signal energy.

[0088] The total signal energy of the target media data is obtained by adding the signal energies of all frequency points, calculating the ratio of the signal energy of the data to be filtered to the total signal energy, and obtaining the energy proportion of the data to be filtered in the target media data. Loudness compensation is performed based on the energy proportion, and accordingly, the loudness compensation gain is obtained.

[0089] In speaker mode, the energy of audio data below the preset frequency cannot be played back normally. The loudness compensation gain is obtained by using the ratio of the signal energy that cannot be played back normally to the total signal energy, thereby improving the accuracy of the loudness compensation gain.

[0090] In some optional implementations, the above step c2 includes:

[0091] Step c21, calculating the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio.

[0092] Step c22: convert the energy proportion into the loudness domain to obtain the loudness compensation gain.

[0093] As described above, the ratio of the signal energy of the data to be filtered to the total signal energy of the target media data is calculated to obtain the energy ratio, and then the energy ratio is converted to the loudness domain using the corresponding relationship between energy and loudness to obtain the loudness compensation gain.

[0094] For example, for the target media data, the preset frequency is 200Hz, and the signal energy of the data to be filtered below 200Hz accounts for 90% of the total signal energy of the target media data. Then convert it to the loudness domain, and get the loudness compensation gain: -10*log10(1-90%)=10dB. Therefore, in the speaker mode, an additional 10dB loudness value needs to be compensated.

[0095] The audio processing method provided in this embodiment obtains the signal energy of each frequency point by performing energy analysis on the target audio data. Based on this, the signal energy of the data to be filtered is obtained using a preset frequency, thereby ensuring the accuracy of the obtained loudness compensation gain.

[0096] In a specific embodiment, Figure 3As shown, the loudness compensation gain is determined by: performing fast Fourier calculation on the overall signal, which corresponds to the target media data described above. After the fast Fourier calculation, the signal energy of each frequency point is calculated, and the target signal energy of each frequency point is calculated in combination with the weight curve. Using the preset frequency, the signal energy of the data to be filtered below the preset frequency and the total signal energy of the overall signal are calculated. The ratio of the signal energy of the data to be filtered to the total signal energy is calculated to obtain the energy ratio, and then the energy ratio is converted to the loudness domain to calculate the loudness compensation gain.

[0097] In some optional implementations, the above audio processing method further includes:

[0098] Step d1, obtaining the playback mode of the playback terminal.

[0099] Step d2: For a target playback terminal whose playback mode is the speaker mode, target media data, loudness gain, and loudness compensation gain are sent.

[0100] Step d3: For other playback terminals whose playback mode is not the speaker mode, the target media data and the loudness gain are sent.

[0101] As mentioned above, the loudness compensation gain corresponds to the playback terminal in speaker mode. Therefore, when sending data, it is necessary to determine the playback mode of the playback terminal. The playback mode is obtained based on the data communication between the playback terminal and the server. The playback mode includes speaker mode and non-speaker mode. For the target playback terminal in speaker mode, the target media data, loudness gain and loudness compensation gain are sent; for other playback terminals in non-speaker mode, the target media data and loudness gain are sent.

[0102] For the target playback terminal corresponding to the speaker mode playback mode, in addition to the target audio data and loudness gain, the server also sends down the loudness compensation gain, which is used to compensate the loudness of the filtered signal. Since the calculation of the loudness compensation gain is processed on the server, the speaker mode playback terminal does not need to perform a secondary loudness scan, ensuring the real-time playback of the target audio data. For the non-speaker mode playback mode, the target audio data and loudness gain are sent down, and there is no need to filter the audio data. Accordingly, there is no need to perform loudness compensation gain.

[0103] In this embodiment, an audio processing method is provided, which can be used in computer devices, such as mobile phones, tablet computers, etc., that is, used in playback terminals. Figure 4 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 4 As shown, the process includes the following steps:

[0104] Step S401: if the current playback mode is the speaker mode, receiving target media data, loudness gain and loudness compensation gain.

[0105] The loudness gain is obtained based on the audio features of the target media data, and the loudness compensation gain is obtained based on the signal energy of the to-be-filtered data with a frequency lower than a preset frequency in the target media data. For the specific determination method of the loudness gain and the loudness compensation gain, please refer to the description of the audio processing method above, which will not be repeated here.

[0106] When the playback terminal needs to play media data, it sends a request to the server, which carries the current playback mode. The server parses the request to obtain the current playback mode. If the current playback mode is the speaker mode, it sends the target media data, loudness gain, and loudness compensation gain. Correspondingly, the playback terminal receives the target media data, loudness gain, and loudness compensation gain.

[0107] Step S402: filtering the data to be filtered in the target media data to obtain the media data to be processed.

[0108] After receiving the target media data, the target media data is filtered by a preset frequency to filter out the data to be filtered that is lower than the preset frequency. The filtering of the data to be filtered can be achieved by a filter, or by converting the target media data into a frequency domain to obtain the frequency of each frequency point, thereby determining the data to be filtered that is lower than the preset frequency, and filtering out this part of the data to be filtered, that is, filtering the data to be filtered in the target media data to obtain the media data to be processed.

[0109] Step S403: loudness processing is performed on the media data to be processed based on the loudness gain and the loudness compensation gain to obtain media data for playback.

[0110] By calculating the sum of the loudness gain and the loudness compensation gain, and using the calculation result to perform loudness processing on the media data to be processed, the media data for playback is obtained. After the loudness processing, further loudness equalization processing can be performed in combination with other loudness information. Other loudness information includes a curve of dynamic range control (DRC), another loudness compensation gain (i.e., make-up gain) and a peak limit. The specific processing methods of the DRC curve, another loudness compensation and peak limit are not limited here, and can be set according to actual needs.

[0111] In the audio processing method provided in this embodiment, when the playback terminal is in speaker mode, before playing the target audio data, since the to-be-filtered data has a large attenuation during playback, the playback terminal filters the to-be-filtered data, and performs loudness compensation on the audio data obtained after filtering based on the loudness gain and the loudness compensation gain to obtain the audio data for playback. This method can ensure that the purpose of loudness balance is achieved in speaker mode and that the loudness consistency is ensured.

[0112] In some optional implementations, the above audio processing method further includes:

[0113] Step e1: if the current playback mode is non-speaker mode, receiving target media data and loudness gain.

[0114] Step e2: loudness processing is performed on the target media data based on the loudness gain to obtain media data for playback.

[0115] For a playback terminal in non-speaker mode, what is received is the target media data and the loudness gain. In this case, there is no need to filter the data to be filtered that is lower than the preset frequency. After receiving the target media data, the target media data is loudness processed based on the loudness gain, and then further processed in combination with the above-mentioned other loudness information to obtain the media data for playback.

[0116] As a specific application embodiment of the embodiment of the present disclosure, the playback terminal is a mobile phone, and the mobile phone is installed with a target application, which is used to play short videos. After receiving the short video uploaded by the publishing side, the server calculates the loudness information and loudness compensation gain corresponding to the short video. The loudness information includes loudness gain, DRC curve, another loudness compensation gain (make-up gain) and peak limit. Figure 5 As shown, the playback terminal communicates with the server corresponding to the target application, sends the current playback mode to the server, and the server determines the content to be sent based on the current playback mode. If the current playback mode is the speaker mode, a short video, loudness information, and loudness compensation gain are sent. At the playback terminal, the data to be filtered that is lower than the preset frequency in the short video is first filtered, and then the loudness information and the loudness compensation gain are combined for subsequent processing to obtain the target short video for playback. If the current playback mode is not the speaker mode, the short video and the loudness information are sent. At the playback terminal, the short video is processed in combination with the loudness information to obtain the target short video for playback.

[0117] like Figure 5As shown, the audio processing of the server and the playback terminal is divided into a parameter preparation stage and a streaming processing stage. The parameter preparation stage is completed on the server to obtain loudness gain, loudness compensation gain, DRC curve and another loudness compensation gain. The streaming processing stage is the processing of the sent media data. Depending on the current playback mode, it is determined whether filtering is required before loudness processing.

[0118] This audio processing method supports real-time processing during short video playback. This is because after filtering in speaker mode, there is no need to perform a second loudness scan on the filtered signal at the playback terminal. Instead, the loudness information and loudness compensation information of the short video are scanned by the server and sent directly to the playback terminal. During playback, audio processing can be performed directly through the entire sound effect link. Compared with the related art of performing a second loudness scan after filtering, this method can ensure the real-time playback of short videos and reduce problems such as freezes in short video playback.

[0119] In this embodiment, an audio processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0120] This embodiment provides an audio processing device, such as Figure 6 As shown, including:

[0121] The acquisition module 601 is used to acquire the audio features and preset frequency of the target media data.

[0122] The loudness gain determination module 602 is used to obtain the loudness gain of the target media data based on the audio features, and the loudness gain is used to perform loudness equalization processing on the target media data.

[0123] The loudness compensation gain determination module 603 is used to analyze the target media data based on a preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered, the media data to be filtered is the media data with a frequency lower than the preset frequency in the target media data, the loudness compensation gain corresponds to the target playback terminal in the speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0124] In some optional implementations, the loudness compensation gain determination module 603 includes:

[0125] The energy analysis unit is used to perform energy analysis on the target media data to obtain the signal energy of each frequency point in the target media data.

[0126] The signal energy determination unit is used to obtain the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency.

[0127] The loudness compensation gain determination unit is used to obtain the loudness compensation gain based on the signal energy of the data to be filtered.

[0128] In some optional embodiments, the energy analysis unit includes:

[0129] The frequency domain conversion subunit is used to perform frequency domain conversion on the target media data to obtain a target frequency domain signal.

[0130] The sampling subunit is used to sample the target frequency domain signal to obtain the amplitude of each frequency point.

[0131] The signal energy determination subunit is used to obtain the signal energy of each frequency point based on the amplitude of each frequency point.

[0132] In some optional implementations, the signal energy determination unit includes:

[0133] The weight coefficient acquisition subunit is used to obtain the weight coefficient of each frequency point.

[0134] The target signal energy determination subunit is used to obtain the target signal energy of each frequency point based on the weight coefficient of each frequency point and the signal energy of each frequency point.

[0135] The first signal energy determination subunit is used to screen each frequency point based on a preset frequency to obtain the signal energy of each frequency point below the preset frequency.

[0136] The second signal energy determination subunit is used to obtain the signal energy of the data to be filtered based on the signal energy of each frequency point below a preset frequency.

[0137] In some optional implementations, the loudness compensation gain determination unit includes:

[0138] The signal total energy determination subunit is used to determine the signal total energy of the target media data based on the sum of the target signal energies at each frequency point.

[0139] The loudness compensation gain determination subunit is used to determine the loudness compensation gain based on the ratio of the signal energy of the data to be filtered to the total signal energy.

[0140] In some optional implementations, the loudness compensation gain determination subunit includes:

[0141] The energy ratio determination subunit is used to calculate the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio.

[0142] The loudness domain conversion subunit is used to convert the energy proportion into the loudness domain to obtain the loudness compensation gain.

[0143] In some optional embodiments, the device further comprises:

[0144] The playback mode acquisition module is used to acquire the playback mode of the playback terminal.

[0145] The first data sending module is used to send target media data, loudness gain and loudness compensation gain to a target playback terminal whose playback mode is speaker mode.

[0146] In some optional embodiments, the device further comprises:

[0147] The second data sending module is used to send target media data and loudness gain to other playback terminals whose playback mode is not the speaker mode.

[0148] This embodiment provides an audio processing device, such as Figure 7 As shown, including:

[0149] The receiving module 701 is used to receive target media data, loudness gain and loudness compensation gain if the current playback mode is the speaker mode, wherein the loudness gain is obtained based on the audio features of the target media data, and the loudness compensation gain is obtained based on the signal energy of the data to be filtered whose frequency is lower than the preset frequency in the target media data.

[0150] The filtering module 702 is used to filter the data to be filtered in the target media data to obtain the media data to be processed.

[0151] The loudness gain module 703 is used to perform loudness processing on the media data to be processed based on the loudness gain and the loudness compensation gain to obtain media data for playback.

[0152] In some optional embodiments, the device comprises:

[0153] The loudness gain receiving module is used to receive target media data and loudness gain if the current playback mode is non-speaker mode.

[0154] The loudness processing module is used to perform loudness processing on the target media data based on the loudness gain to obtain media data for playback.

[0155] The audio processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0156] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0157] The present disclosure also provides a computer device having the above Figure 6 or Figure 7 The audio processing device shown.

[0158] See also Figure 8 or Fig. 9 , Figure 8 or Fig. 9 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 8 or Fig. 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 or Fig. 9 A processor 10 is taken as an example.

[0159] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0160] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0161] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0163] If the computer equipment includes Figure 6 The audio processing device shown, Figure 8 As shown, the computer device further includes a communication interface 50 for the computer device to communicate with other devices or a communication network.

[0164] If the computer equipment includes Figure 7 The audio processing device shown, Fig. 9 As shown, the computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0165] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0166] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0167] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0168] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0169] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0170] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0171] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Acquire audio characteristics and preset frequencies of target media data; Obtaining a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data; The target media data is analyzed based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, the media data to be filtered is media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

2. The method according to claim 1, characterized in that The analyzing the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered includes: Performing energy analysis on the target media data to obtain signal energy of each frequency point in the target media data; Based on the signal energy of each frequency point and the preset frequency, obtaining the signal energy of the data to be filtered; The loudness compensation gain is obtained based on the signal energy of the data to be filtered.

3. The method according to claim 2, characterized in that The performing energy analysis on the target media data to obtain the signal energy of each frequency point in the target media data includes: Performing frequency domain conversion on the target media data to obtain a target frequency domain signal; Sampling the target frequency domain signal to obtain the amplitude of each frequency point; Based on the amplitude of each frequency point, the signal energy of each frequency point is obtained.

4. The method according to claim 2, characterized in that: The obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency includes: Obtaining weight coefficients of each frequency point; Based on the weight coefficient of each frequency point and the signal energy of each frequency point, obtaining the target signal energy of each frequency point; Screening the frequency points based on the preset frequency to obtain signal energy of each frequency point below the preset frequency; Based on the signal energy of each frequency point below the preset frequency, the signal energy of the data to be filtered is obtained.

5. The method according to claim 4, characterized in that The obtaining the loudness compensation gain based on the signal energy of the data to be filtered includes: Determining the total signal energy of the target media data based on the sum of the target signal energies of the respective frequency points; The loudness compensation gain is determined based on a ratio of the signal energy of the data to be filtered to the total signal energy.

6. The method according to claim 5, characterized in that The step of determining the loudness compensation gain based on the ratio of the signal energy of the data to be filtered to the total signal energy includes: Calculating the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio; The energy proportion is converted into a loudness domain to obtain the loudness compensation gain.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Get the playback mode of the playback terminal; For the target playback terminal whose playback mode is the speaker mode, the target media data, the loudness gain, and the loudness compensation gain are delivered.

8. The method according to claim 7, characterized in that The method further comprises: For other playback terminals whose playback mode is not the loudspeaker mode, the target media data and the loudness gain are delivered.

9. An audio processing method, characterized in that: The method comprises: If the current playback mode is the speaker mode, receiving target media data, a loudness gain, and a loudness compensation gain, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data; Filtering the data to be filtered in the target media data to obtain media data to be processed; Based on the loudness gain and the loudness compensation gain, loudness processing is performed on the media data to be processed to obtain media data for playback.

10. The method according to claim 9, characterized in that The method comprises: If the current playback mode is a non-speaker mode, receiving the target media data and the loudness gain; Loudness processing is performed on the target media data based on the loudness gain to obtain the media data for playback.

11. An audio processing device, characterized in that: The device comprises: An acquisition module, used to acquire audio features and preset frequencies of target media data; A loudness gain determination module, configured to obtain a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data; a loudness compensation gain determination module, configured to analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, wherein the media data to be filtered is media data having a frequency lower than the preset frequency in the target media data, the loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is configured to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

12. An audio processing device, characterized in that: The device comprises: a receiving module, configured to receive target media data, a loudness gain, and a loudness compensation gain if the current playback mode is a speaker mode, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data; A filtering module, used to filter the data to be filtered in the target media data to obtain the media data to be processed; The loudness gain module is used to perform loudness processing on the to-be-processed media data based on the loudness gain and the loudness compensation gain to obtain media data for playing.

13. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the audio processing method according to any one of claims 1 to 10 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the audio processing method according to any one of claims 1 to 10.

Images