Audio processing method and apparatus, computer device, and storage medium

Abstract

The present invention relates to the technical field of computers. Disclosed are an audio processing method and apparatus, a computer device, and a storage medium. The method provided in the present invention comprises: acquiring an audio feature of target media data and a preset frequency; obtaining a loudness gain of the target media data on the basis of the audio feature, wherein the loudness gain is used for performing loudness equalization processing on the target media data; and analyzing the target media data on the basis of the preset frequency to obtain a loudness compensation gain corresponding to signal energy of data to be filtered, wherein media data to be filtered is media data with the frequency lower than the preset frequency in the target media data, the loudness compensation gain corresponds to a target playback terminal in a loudspeaker mode, and the target playback terminal is used for performing loudness processing on the target media data on the basis of the loudness gain and the loudness compensation gain when the data to be filtered is filtered.

Description

Audio processing method, device, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 14, 2023, with application number 202311189797.X and invention name “Audio processing method, device, computer equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an audio processing method, apparatus, computer equipment, and storage medium. Background Art

[0004] Because media data comes from different publishing sides, the same terminal needs to perform loudness equalization before playing the media data to ensure consistency of playback loudness.

[0005] Summary of the Invention

[0006] In view of this, the present disclosure provides an audio processing method, apparatus, computer device, and storage medium to solve the problem of playback loudness consistency.

[0007] In a first aspect, the present disclosure provides an audio processing method, the method comprising:

[0008] Obtaining audio characteristics and preset frequencies of target media data;

[0009] obtaining a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data;

[0010] The target media data is analyzed based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, where the media data to be filtered is media data in the target media data having a frequency lower than the preset frequency. The loudness compensation gain corresponds to a target playback terminal in speaker mode. The target playback terminal is configured to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0011] In a second aspect, the present disclosure provides an audio processing method, the method comprising:

[0012] If the current playback mode is speaker mode, receiving target media data, a loudness gain, and a loudness compensation gain, wherein the loudness gain is obtained based on audio features of the target media data, and the loudness compensation gain is obtained based on signal energy of to-be-filtered data having a frequency lower than a preset frequency in the target media data;

[0013] Filtering the target media data to obtain the media data to be processed;

[0014] Based on the loudness gain and the loudness compensation gain, loudness processing is performed on the media data to be processed to obtain media data for playback.

[0015] In a third aspect, the present disclosure provides an audio processing device, the device comprising:

[0016] An acquisition module, used to obtain the audio characteristics and preset frequency of target media data;

[0017] a loudness gain determination module, configured to obtain a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data;

[0018] The loudness compensation gain determination module is configured to analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the to-be-filtered data, wherein the to-be-filtered media data is media data in the target media data having a frequency lower than the preset frequency. The loudness compensation gain corresponds to a target playback terminal in speaker mode. The target playback terminal is configured to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the to-be-filtered data.

[0019] In a fourth aspect, the present disclosure provides an audio processing device, the device comprising:

[0020] a receiving module configured to, if the current playback mode is speaker mode, receive target media data, a loudness gain, and a loudness compensation gain, wherein the loudness gain is obtained based on audio features of the target media data, and the loudness compensation gain is obtained based on signal energy of to-be-filtered data having a frequency lower than a preset frequency in the target media data;

[0021] A filtering module, configured to filter the target media data to obtain the media data to be processed;

[0022] The loudness gain module is configured to perform loudness processing on the media data to be processed based on the loudness gain and the loudness compensation gain to obtain media data for playback.

[0023] In a fifth aspect, the present disclosure provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the computer instructions to thereby execute the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the above-mentioned second aspect or any corresponding embodiment thereof, in an audio processing method.

[0024] In a sixth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the audio processing method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the above-mentioned second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] FIG1 is a schematic flow chart of an audio processing method according to an embodiment of the present disclosure;

[0027] FIG2 is a flow chart of another audio processing method according to an embodiment of the present disclosure;

[0028] FIG3 is a schematic diagram of a process for determining a loudness compensation gain according to an embodiment of the present disclosure;

[0029] FIG4 is a flow chart of another audio processing method according to an embodiment of the present disclosure;

[0030] FIG5 is a modular representation of audio processing according to an embodiment of the present disclosure;

[0031] FIG6 is a structural block diagram of an audio processing device according to an embodiment of the present disclosure;

[0032] FIG7 is a structural block diagram of an audio processing device according to an embodiment of the present disclosure;

[0033] FIG8 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure;

[0034] FIG9 is a schematic diagram of the hardware structure of another computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0036] When performing loudness equalization on the playback end, the solution adopted by the relevant technology is to pull media data (audio or video) with different loudness to the same playback loudness through loudness gain and forced limiting processing. However, this solution does not take into account that the playback terminal is limited by the playback frequency response of the speaker in the speaker playback mode, resulting in a large attenuation of low-frequency signals when the playback terminal uses the speaker to play media data. Among them, the low-frequency signal is a signal with a frequency lower than the preset frequency, and the preset frequency can be 100Hz, 150Hz or 200Hz, which is set according to actual needs. The low-frequency signal may be attenuated by 20dB to 40dB, or even a larger range of dB values.

[0037] In scenarios where a large number of short videos, such as voice, music, and movies, are played, low-frequency signals are a significant component. For example, vocals, drum beats, or instruments, for example, lack the playback energy of these signals. This inevitably leads to uneven loudness and fluctuating sound levels when playing back through speakers, even after digital loudness equalization. This results in more frequent volume adjustments when using speakers.

[0038] Based on this, the present disclosure provides an audio processing method for loudness equalization in speaker mode to address the issue of inconsistent audio loudness when playing audio and video in speaker mode. Specifically, in speaker mode, this method minimizes the problem of inaudible audio even when the volume on the playback device is turned up to its maximum. This method allows for adjusting the volume within the same video or between different videos.

[0039] As an optional application scenario of the embodiment of the present disclosure, a target application is installed on the playback terminal, and the target application is used to play media data, and the media data comes from the server corresponding to the target application. During playback, the playback terminal sends a request to the server, and accordingly, the server sends the corresponding media data to the playback terminal. The number of playback terminals that the target server can connect to is set according to actual needs and is not limited to any restrictions here. When sending media data, the target server sends the corresponding content based on the current playback mode of the playback terminal. The current playback mode includes but is not limited to speaker mode, headphone mode or power amplifier mode, etc. Each playback terminal communicates with the server, and accordingly, the server can know the current playback mode of the playback terminal. In the embodiment of the present disclosure, the playback mode is divided into speaker mode and non-speaker mode. In the speaker mode, the content sent by the server includes media data, loudness gain and loudness compensation gain. The loudness compensation gain corresponds to the speaker mode, and the loudness compensation gain is used to compensate for the energy of the filtered data that is filtered out due to the inability to play back. Because audio data below the preset frequency cannot be played back properly in speaker mode, this data is called filtered data. After filtering this data, the playback terminal uses the loudness compensation gain to compensate for its energy and balance its loudness. In non-speaker mode, the server sends media data and loudness gain, and the playback terminal uses the loudness gain to balance the media data.

[0040] The audio processing method provided by the present disclosure determines a loudness compensation gain for the target media data in the server, corresponding to the data to be filtered in the target media data, because the playback terminal experiences significant attenuation during playback of media data below a preset frequency. When the playback terminal is in speaker mode, before playing the target media data, the data to be filtered experiences significant attenuation during playback. Therefore, the playback terminal filters the data to be filtered and performs loudness compensation on the filtered media data based on the loudness gain and loudness compensation gain, obtaining media data for playback. This method ensures that loudness balancing is achieved in speaker mode, ensuring loudness consistency.

[0041] According to an embodiment of the present disclosure, an embodiment of an audio processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] In this embodiment, an audio processing method is provided that can be used in a computer device, such as a server. Specifically, the server determines the loudness gain and loudness compensation gain corresponding to the target media data, so that the corresponding gains are issued based on the current playback mode of the playback terminal. Specifically, Figure 1 is a flow chart of the audio processing method according to an embodiment of the present disclosure. As shown in Figure 1, the process includes the following steps:

[0043] Step S101: Acquire audio features and preset frequencies of target media data.

[0044] The target media data is the media data sent from the server to the playback terminal. It can be in the form of audio or video. Videos can be classified as short or long, and the length of a video is determined by its duration. For example, videos with a duration less than the duration threshold are considered short videos, and the rest are considered long videos.

[0045] The audio features of the target media data are obtained by analyzing the target media data. These features include, but are not limited to, duration, source loudness, and target media data type. Source loudness refers to the published loudness of the target media data. The target media data type can be speech, music, or other types.

[0046] The preset frequency corresponds to a frequency at which the speaker of the playback terminal cannot properly play back the sound. For example, by testing the playback loudness of various types of playback terminals, the frequency at which normal playback cannot be properly played back is determined and used as the preset frequency. Alternatively, the preset frequency may be set based on empirical values, etc. The specific value of the preset frequency is not limited herein and may be, for example, 100Hz, 150Hz, or 200Hz, etc.

[0047] The audio feature of the target media data is used to determine a subsequent loudness gain, and the preset frequency is used to determine a subsequent loudness compensation gain.

[0048] Step S102: obtaining a loudness gain of target media data based on the audio features.

[0049] The loudness gain is used to perform loudness equalization processing on the target media data.

[0050] Loudness gain is determined based on the source loudness and target loudness. Source loudness is included in audio features, while target loudness is derived from audio feature analysis. For example, audio features include duration, target media data type, and speech ratio. The target loudness is calculated based on these parameters. Alternatively, the initial loudness set by the user is obtained and then adjusted based on the audio features to determine the target loudness.

[0051] Of course, the target loudness can also be calculated using other methods, which are set according to actual needs, and the calculation method is not limited here. After determining the target loudness, the loudness gain can be determined by the difference between the target loudness and the source loudness.

[0052] In some optional implementations, when calculating the target loudness, the user personalized settings of the playback terminal may also be combined, for example, the user personalized settings, the current environment information of the playback terminal, etc. The specific settings may be made according to actual needs.

[0053] Step S103 : Analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered.

[0054] The media data to be filtered is the media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to the target playback terminal in speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0055] As mentioned above, when a playback terminal plays back media data, media data below a preset frequency cannot be properly played back. Therefore, the playback terminal filters out this media data that cannot be properly played back. Accordingly, loudness compensation is performed on the energy of the filtered media data. Therefore, it is necessary to obtain the loudness compensation gain corresponding to the signal energy of this filtered data.

[0056] The target media data can be converted to the frequency domain to obtain the amplitude at each frequency point. Based on this, the energy at each frequency point is calculated. Due to the preset frequency limit, media data with frequencies below the preset frequency can be filtered out. This portion of media data is referred to as the to-be-filtered media data. Since loudness compensation will subsequently be performed on the to-be-filtered media data, the loudness compensation gain is determined using the signal energy of the to-be-filtered media data.

[0057] The loudness compensation gain is specific to the playback terminal in speaker mode; that is, it is only used for loudness processing on playback terminals in speaker mode. For playback terminals in non-speaker mode, the loudness gain is used for loudness processing. For the same playback terminal, if it is currently in non-speaker mode, loudness processing is performed using the loudness gain. If it switches to speaker mode next, loudness processing is performed using both the loudness gain and the loudness compensation gain.

[0058] It should be noted that both the loudness gain and the loudness compensation gain correspond to the target media data, that is, the loudness gain and the loudness compensation gain are different for different target media data.

[0059] The audio processing method provided in this embodiment uses a server to determine a loudness compensation gain for the target media data, corresponding to the to-be-filtered data within the target media data, because media data below a preset frequency experiences significant attenuation during playback on a playback terminal. This results in a loss of signal playback energy for this portion of the media data. When the playback terminal is in speaker mode, before playing the target media data, the to-be-filtered data experiences significant attenuation during playback. Therefore, the playback terminal filters the to-be-filtered data and performs loudness compensation on the filtered media data based on the loudness gain and the loudness compensation gain, obtaining media data for playback. This method ensures loudness balancing and loudness consistency in speaker mode.

[0060] In this embodiment, an audio processing method is provided, which can be used in a computer device, such as a server. FIG2 is a flow chart of the audio processing method according to an embodiment of the present disclosure. As shown in FIG2 , the flow chart includes the following steps:

[0061] Step S201: Acquire the audio characteristics and preset frequency of target media data. Please refer to step S101 of the embodiment shown in FIG1 for details, which will not be repeated here.

[0062] Step S202: obtaining a loudness gain of target media data based on the audio features.

[0063] The loudness gain is used to perform loudness equalization processing on the target media data. Please refer to step S102 of the embodiment shown in FIG1 for details, and will not be described in detail here.

[0064] Step S203 : Analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered.

[0065] The media data to be filtered is the media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to the target playback terminal in speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

[0066] Specifically, the above step S203 includes:

[0067] Step S2031 : Perform energy analysis on the target media data to obtain the signal energy of each frequency point in the target media data.

[0068] The energy analysis of the target media data is performed by converting it into the frequency domain. After the frequency domain conversion, the signal energy of each frequency point is obtained according to the interval of the sampling frequency.

[0069] In some optional implementations, the above step S2031 includes:

[0070] Step a1: Perform frequency domain conversion on target media data to obtain a target frequency domain signal.

[0071] Step a2: sampling the target frequency domain signal to obtain the amplitude of each frequency point.

[0072] Step a3: obtaining the signal energy of each frequency point based on the amplitude of each frequency point.

[0073] Frequency domain conversion of the target media data can be achieved using Fast Fourier Transformation (FFT) or other methods. After frequency domain conversion, the target frequency domain signal is obtained. The target frequency domain signal represents the correspondence between frequency and amplitude. The target frequency domain signal is sampled at a preset sampling interval to obtain the amplitude at each frequency point. Based on this, the corresponding signal energy is calculated using the amplitude at each frequency point. In other words, the signal energy is calculated based on the square of the amplitude.

[0074] For example, the target frequency domain signal of the target media data after fast Fourier transformation is A, which includes 512 frequency points, that is, A = {a1, a2, ..., a512}, where ai represents the amplitude of the i-th frequency point, i = 1, ..., 512. Correspondingly, the signal energy of each frequency point is E, E = {a1 2 , a2 2 ,…,a512 2}.

[0075] The target audio data is converted to the frequency domain to obtain the target frequency domain signal. Based on this, sampling is performed to obtain the amplitude of each frequency point, thereby obtaining the signal energy of each frequency point. In other words, the amplitude of each frequency point is obtained by sampling, and the signal energy of the frequency point is obtained, which improves the accuracy of the signal energy of each frequency point.

[0076] Step S2032: obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency.

[0077] The frequency corresponding to each frequency point can be obtained using the frequency point index i. By comparing the frequency corresponding to each frequency point with the preset frequency, the data to be filtered that is below the preset frequency can be screened out. Combined with the signal energy of each frequency point obtained in step S2031 above, the signal energy of the frequency points below the preset frequency can be obtained. These signal energies are then fused to obtain the signal energy of the data to be filtered.

[0078] In some optional implementations, the above step S2032 includes:

[0079] Step b1: Obtain the weight coefficient of each frequency point.

[0080] Step b2: obtaining the target signal energy of each frequency point based on the weight coefficient of each frequency point and the signal energy of each frequency point.

[0081] Step b3: screening each frequency point based on a preset frequency to obtain signal energy of each frequency point below the preset frequency.

[0082] Step b4: obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point below the preset frequency.

[0083] The weight coefficient is used to convert signal energy into the human ear's perception of sound volume. The weight coefficient can be represented by a weight curve, which shows the relationship between frequency and weight. As described above, the frequency of each frequency point can be obtained by using the index of each frequency point. The weight coefficient of each frequency point can then be obtained by querying the weight curve using the frequency of each frequency point.

[0084] The weight coefficients of each frequency point are weighted together with the signal energy at each frequency point to obtain the target signal energy at each frequency point. Frequency points below the preset frequency are then filtered out to obtain the signal energy of each frequency point below the preset frequency. The signal energy of each frequency point below the preset frequency is then combined to obtain the signal energy of the data to be filtered.

[0085] The weight coefficient of each frequency point is used to convert the calculated signal energy into the human ear's judgment of "sound volume" to obtain the target signal energy of each frequency point, thereby improving the adaptability of the obtained signal energy of the filtered data to the human ear.

[0086] Step S2033: obtaining a loudness compensation gain based on the signal energy of the data to be filtered.

[0087] The signal energy of the filtered data is in the energy domain, while the loudness is in the loudness domain. Therefore, the corresponding relationship between energy and loudness is used to convert the energy domain into the loudness domain to obtain the loudness compensation gain. The loudness compensation gain can be obtained by converting the signal energy into the loudness domain and then combining it with the target loudness. Alternatively, it can be obtained by calculating the ratio of the signal energy of the filtered data to the total signal energy of the target media data, and so on. Of course, other methods can also be used to obtain the loudness compensation gain, and the specific setting depends on actual needs.

[0088] In some optional implementations, the above step S2033 includes:

[0089] Step c1: determining the total signal energy of the target media data based on the sum of the target signal energies at each frequency point.

[0090] Step c2: determining a loudness compensation gain based on a ratio of the signal energy of the data to be filtered to the total signal energy.

[0091] The total signal energy of the target media data is obtained by adding the signal energies of all frequency points. The ratio of the signal energy of the data to be filtered to the total signal energy is calculated to obtain the energy proportion of the data to be filtered in the target media data. Loudness compensation is performed based on the energy proportion, and accordingly, the loudness compensation gain is obtained.

[0092] In speaker mode, the energy of audio data below the preset frequency cannot be played back normally. The ratio of the energy of this part of the signal that cannot be played back normally to the total signal energy is used to obtain the loudness compensation gain, thereby improving the accuracy of the loudness compensation gain.

[0093] In some optional implementations, the above step c2 includes:

[0094] Step c21 : Calculate the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio.

[0095] Step c22: convert the energy proportion into the loudness domain to obtain the loudness compensation gain.

[0096] As described above, the ratio of the signal energy of the data to be filtered to the total signal energy of the target media data is calculated to obtain the energy ratio. Then, using the corresponding relationship between energy and loudness, the energy ratio is converted to the loudness domain to obtain the loudness compensation gain.

[0097] For example, let's assume the target media data has a preset frequency of 200 Hz. The signal energy of the data to be filtered below 200 Hz accounts for 90% of the total signal energy of the target media data. Converting this to the loudness domain yields a loudness compensation gain of -10*log10(1-90%) = 10 dB. Therefore, in speaker mode, an additional 10 dB of loudness compensation is required.

[0098] The audio processing method provided in this embodiment performs energy analysis on the target audio data to obtain the signal energy of each frequency point. Based on this, the signal energy of the data to be filtered is obtained using a preset frequency, thereby ensuring the accuracy of the obtained loudness compensation gain.

[0099] In one specific embodiment, as shown in FIG3 , the loudness compensation gain is determined by performing a Fast Fourier Transformation (FFT) calculation on the entire signal, which corresponds to the target media data described above. After the FFT calculation, the signal energy at each frequency point is calculated, and the target signal energy at each frequency point is calculated using a weighting curve. Using a preset frequency, the signal energy of the data to be filtered below the preset frequency and the total signal energy of the entire signal are calculated. The ratio of the signal energy of the data to be filtered to the total signal energy is calculated to obtain an energy percentage, which is then converted to the loudness domain to calculate the loudness compensation gain.

[0100] In some optional implementations, the above audio processing method further includes:

[0101] Step d1, obtaining the playback mode of the playback terminal.

[0102] Step d2: Send target media data, loudness gain, and loudness compensation gain to the target playback terminal whose playback mode is speaker mode.

[0103] Step d3: For other playback terminals whose playback mode is not the loudspeaker mode, target media data and loudness gain are delivered.

[0104] As mentioned above, the loudness compensation gain corresponds to the playback terminal in speaker mode. Therefore, when sending data, the playback mode of the playback terminal must be determined. The playback mode is determined based on data communication between the playback terminal and the server. Playback modes include speaker mode and non-speaker mode. For target playback terminals in speaker mode, the target media data, loudness gain, and loudness compensation gain are sent; for other playback terminals in non-speaker mode, the target media data and loudness gain are sent.

[0105] For target playback terminals in speaker mode, in addition to the target audio data and loudness gain, the server also delivers a loudness compensation gain to compensate for the filtered signal. Because the loudness compensation gain calculation is handled on the server, the speaker mode playback terminal does not need to perform a secondary loudness scan, ensuring real-time playback of the target audio data. For playback modes other than speaker mode, only the target audio data and loudness gain are delivered, eliminating the need for audio data filtering and, accordingly, loudness compensation gain calculation.

[0106] In this embodiment, an audio processing method is provided, which can be used in computer devices such as mobile phones, tablet computers, etc., that is, for playback terminals. FIG4 is a flow chart of the audio processing method according to an embodiment of the present disclosure. As shown in FIG4 , the flow chart includes the following steps:

[0107] Step S401 : If the current playback mode is the speaker mode, target media data, loudness gain, and loudness compensation gain are received.

[0108] The loudness gain is determined based on the audio characteristics of the target media data, and the loudness compensation gain is determined based on the signal energy of the target media data whose frequencies are below a preset frequency. For details on how to determine the loudness gain and loudness compensation gain, please refer to the description of the audio processing method above and will not be repeated here.

[0109] When a playback terminal needs to play media data, it sends a request to the server, including the current playback mode. The server parses the request to determine the current playback mode. If the current playback mode is speaker mode, it sends the target media data, loudness gain, and loudness compensation gain. In response, the playback terminal receives the target media data, loudness gain, and loudness compensation gain.

[0110] Step S402: Filter the target media data to obtain the media data to be processed.

[0111] After receiving the target media data, the target media data is filtered using a preset frequency to remove data below the preset frequency. The filtering of the data can be achieved through a filter, or by converting the target media data into the frequency domain to obtain the frequencies of each frequency point, thereby determining data below the preset frequency and removing this data. This filtering of the data in the target media data is completed, resulting in the media data to be processed.

[0112] Step S403 : performing loudness processing on the media data to be processed based on the loudness gain and the loudness compensation gain to obtain media data for playback.

[0113] Loudness processing is performed on the media data to be processed by calculating the sum of the loudness gain and the loudness compensation gain, and the calculated result is used to obtain media data for playback. After loudness processing, further loudness equalization can be performed in combination with other loudness information. This other loudness information includes a dynamic range control (DRC) curve, another loudness compensation gain (i.e., make-up gain), and peak limiting. The specific processing methods for the DRC curve, another loudness compensation gain, and peak limiting are not limited here and can be set according to actual needs.

[0114] The audio processing method provided in this embodiment filters the target audio data before playing it, because the filtered data experiences significant attenuation during playback. The filtered audio data is then loudness compensated based on a loudness gain and a loudness compensation gain, resulting in audio data for playback. This method ensures loudness equalization and loudness consistency in speaker mode.

[0115] In some optional implementations, the above audio processing method further includes:

[0116] Step e1: If the current playback mode is non-speaker mode, receive target media data and loudness gain.

[0117] Step e2: performing loudness processing on the target media data based on the loudness gain to obtain media data for playback.

[0118] For playback terminals in non-speaker mode, the target media data and loudness gain are received. In this case, there is no need to filter data below the preset frequency. After receiving the target media data, loudness processing is performed on the target media data based on the loudness gain. Subsequently, the target media data can be further processed in combination with the other loudness information described above to obtain media data for playback.

[0119] As a specific application example of the disclosed embodiments, a playback terminal is a mobile phone installed with a target application for playing short videos. After receiving the short video uploaded by the publisher, the server calculates the loudness information and loudness compensation gain corresponding to the short video. The loudness information includes the loudness gain, DRC curve, another loudness compensation gain (make-up gain), and peak limiter. As shown in FIG5 , the playback terminal communicates with the server corresponding to the target application and sends the current playback mode to the server. The server then determines the content to be delivered based on the current playback mode. If the current playback mode is speaker mode, the short video, loudness information, and loudness compensation gain are delivered. The playback terminal first filters the data to be filtered below a preset frequency in the short video. Then, the loudness information and loudness compensation gain are combined for subsequent processing to obtain the target short video for playback. If the current playback mode is not speaker mode, the short video and loudness information are delivered. The playback terminal processes the short video based on the loudness information to obtain the target short video for playback.

[0120] As shown in Figure 5, audio processing on the server and playback terminal is divided into a parameter preparation phase and a streaming phase. The parameter preparation phase is completed on the server and is used to obtain the loudness gain, loudness compensation gain, DRC curve, and another loudness compensation gain. The streaming phase processes the delivered media data and determines whether filtering is required before loudness processing, depending on the current playback mode.

[0121] This audio processing method supports real-time processing during short video playback. This is because after filtering in speaker mode, there's no need to perform a second loudness scan on the playback terminal. Instead, the server scans the short video's loudness information and loudness compensation information, which is then sent directly to the playback terminal. During playback, audio processing is performed directly across the entire audio chain. Compared to related techniques that require a second loudness scan after filtering, this method ensures real-time short video playback and reduces issues like playback lag.

[0122] This embodiment also provides an audio processing device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides an audio processing device, as shown in FIG6 , including:

[0124] The acquisition module 601 is configured to acquire audio features and a preset frequency of target media data.

[0125] The loudness gain determination module 602 is configured to obtain a loudness gain of target media data based on audio features, and the loudness gain is used to perform loudness equalization processing on the target media data.

[0126] Loudness compensation gain determination module 603 is configured to analyze target media data based on a preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the to-be-filtered data. The to-be-filtered media data is media data in the target media data having a frequency lower than the preset frequency. The loudness compensation gain corresponds to a target playback terminal in speaker mode. After filtering the to-be-filtered data, the target playback terminal is configured to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain.

[0127] In some optional implementations, the loudness compensation gain determination module 603 includes:

[0128] The energy analysis unit is used to perform energy analysis on the target media data to obtain the signal energy of each frequency point in the target media data.

[0129] The signal energy determination unit is used to obtain the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency.

[0130] The loudness compensation gain determining unit is configured to obtain a loudness compensation gain based on signal energy of the data to be filtered.

[0131] In some optional embodiments, the energy analysis unit includes:

[0132] The frequency domain conversion subunit is used to perform frequency domain conversion on the target media data to obtain a target frequency domain signal.

[0133] The sampling subunit is used to sample the target frequency domain signal to obtain the amplitude of each frequency point.

[0134] The signal energy determination subunit is used to obtain the signal energy of each frequency point based on the amplitude of each frequency point.

[0135] In some optional implementations, the signal energy determination unit includes:

[0136] The weight coefficient acquisition subunit is used to obtain the weight coefficient of each frequency point.

[0137] The target signal energy determination subunit is configured to obtain the target signal energy of each frequency point based on the weight coefficient of each frequency point and the signal energy of each frequency point.

[0138] The first signal energy determination subunit is configured to screen each frequency point based on a preset frequency to obtain the signal energy of each frequency point below the preset frequency.

[0139] The second signal energy determination subunit is configured to obtain the signal energy of the data to be filtered based on the signal energy of each frequency point below a preset frequency.

[0140] In some optional implementations, the loudness compensation gain determination unit includes:

[0141] The total signal energy determination subunit is configured to determine the total signal energy of the target media data based on the sum of the target signal energies at each frequency point.

[0142] The loudness compensation gain determination subunit is configured to determine the loudness compensation gain based on a ratio of signal energy of the data to be filtered to the total signal energy.

[0143] In some optional implementations, the loudness compensation gain determination subunit includes:

[0144] The energy ratio determination subunit is used to calculate the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio.

[0145] The loudness domain conversion subunit is used to convert the energy proportion into the loudness domain to obtain the loudness compensation gain.

[0146] In some optional embodiments, the device further comprises:

[0147] The playback mode acquisition module is used to obtain the playback mode of the playback terminal.

[0148] The first data sending module is configured to send target media data, loudness gain, and loudness compensation gain to a target playback terminal in a speaker mode.

[0149] In some optional embodiments, the device further comprises:

[0150] The second data sending module is used to send target media data and loudness gain to other playback terminals whose playback mode is not the loudspeaker mode.

[0151] This embodiment provides an audio processing device, as shown in FIG7 , including:

[0152] Receiving module 701 is configured to receive target media data, a loudness gain, and a loudness compensation gain if the current playback mode is speaker mode. The loudness gain is obtained based on the audio characteristics of the target media data, and the loudness compensation gain is obtained based on the signal energy of the to-be-filtered data in the target media data having a frequency lower than a preset frequency.

[0153] The filtering module 702 is configured to filter the target media data to obtain the media data to be processed.

[0154] The loudness gain module 703 is configured to perform loudness processing on the media data to be processed based on the loudness gain and the loudness compensation gain to obtain media data for playback.

[0155] In some optional embodiments, the device comprises:

[0156] The loudness gain receiving module is configured to receive target media data and loudness gain if the current playback mode is non-speaker mode.

[0157] The loudness processing module is used to perform loudness processing on the target media data based on the loudness gain to obtain media data for playback.

[0158] The audio processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0159] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0160] An embodiment of the present disclosure further provides a computer device having the audio processing apparatus shown in FIG. 6 or FIG. 7 .

[0161] Please refer to Figure 8 or Figure 9, which is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present disclosure. As shown in Figure 8 or Figure 9, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, with each device providing some of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 or Figure 9 uses a processor 10 as an example.

[0162] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0163] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0164] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0166] If the computer device includes the audio processing apparatus shown in FIG6 , as shown in FIG8 , the computer device further includes a communication interface 50 for the computer device to communicate with other devices or a communication network.

[0167] If the computer device includes the audio processing device shown in Figure 7, as shown in Figure 9, the computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 9 takes the bus connection as an example.

[0168] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0169] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0170] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0171] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0172] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0173] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0174] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An audio processing method, comprising: Acquire audio characteristics and preset frequencies of target media data; Obtaining a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data; The target media data is analyzed based on the preset frequency to obtain a loudness compensation gain corresponding to signal energy of the data to be filtered, the media data to be filtered is media data in the target media data whose frequency is lower than the preset frequency, the loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

2. The method according to claim 1, wherein the step of analyzing the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the data to be filtered comprises: Performing energy analysis on the target media data to obtain signal energy of each frequency point in the target media data; Based on the signal energy of each frequency point and the preset frequency, obtaining the signal energy of the data to be filtered; The loudness compensation gain is obtained based on the signal energy of the data to be filtered.

3. The method according to claim 2, wherein the performing energy analysis on the target media data to obtain the signal energy of each frequency point in the target media data comprises: Performing frequency domain conversion on the target media data to obtain a target frequency domain signal; Sampling the target frequency domain signal to obtain the amplitude of each frequency point; Based on the amplitude of each frequency point, the signal energy of each frequency point is obtained.

4. The method according to claim 2, wherein the obtaining the signal energy of the data to be filtered based on the signal energy of each frequency point and the preset frequency comprises: Obtaining weight coefficients of each frequency point; Based on the weight coefficient of each frequency point and the signal energy of each frequency point, obtaining the target signal energy of each frequency point; Screening the frequency points based on the preset frequency to obtain signal energy of each frequency point below the preset frequency; Based on the signal energy of each frequency point below the preset frequency, the signal energy of the data to be filtered is obtained.

5. The method according to claim 4, wherein obtaining the loudness compensation gain based on the signal energy of the data to be filtered comprises: Determining the total signal energy of the target media data based on the sum of the target signal energies of the respective frequency points; The loudness compensation gain is determined based on a ratio of the signal energy of the data to be filtered to the total signal energy.

6. The method according to claim 5, wherein determining the loudness compensation gain based on the ratio of the signal energy of the data to be filtered to the total signal energy comprises: Calculating the ratio of the signal energy of the data to be filtered to the total signal energy to obtain the energy ratio; The energy proportion is converted into a loudness domain to obtain the loudness compensation gain.

7. The method according to any one of claims 1 to 6, further comprising: Get the playback mode of the playback terminal; For the target playback terminal whose playback mode is the speaker mode, the target media data, the loudness gain, and the loudness compensation gain are delivered.

8. The method according to claim 7, further comprising: For other playback terminals whose playback mode is not the loudspeaker mode, the target media data and the loudness gain are delivered.

9. An audio processing method, comprising: If the current playback mode is the speaker mode, receiving target media data, a loudness gain, and a loudness compensation gain, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data; Filtering the data to be filtered in the target media data to obtain media data to be processed; Based on the loudness gain and the loudness compensation gain, loudness processing is performed on the media data to be processed to obtain media data for playback.

10. The method according to claim 9, comprising: If the current playback mode is a non-speaker mode, receiving the target media data and the loudness gain; Loudness processing is performed on the target media data based on the loudness gain to obtain the media data for playback.

11. An audio processing device, comprising: An acquisition module, used to acquire audio features and preset frequencies of target media data; A loudness gain determination module, configured to obtain a loudness gain of the target media data based on the audio feature, wherein the loudness gain is used to perform loudness equalization processing on the target media data; a loudness compensation gain determination module, configured to analyze the target media data based on the preset frequency to obtain a loudness compensation gain corresponding to the signal energy of the to-be-filtered data, wherein the to-be-filtered media data is the media data in the target media data whose frequency is lower than the preset frequency; The loudness compensation gain corresponds to a target playback terminal in a speaker mode, and the target playback terminal is used to perform loudness processing on the target media data based on the loudness gain and the loudness compensation gain after filtering the data to be filtered.

12. An audio processing device, comprising: a receiving module, configured to receive target media data, a loudness gain, and a loudness compensation gain if the current playback mode is a speaker mode, wherein the loudness gain is obtained based on an audio feature of the target media data, and the loudness compensation gain is obtained based on a signal energy of data to be filtered having a frequency lower than a preset frequency in the target media data; A filtering module, used to filter the data to be filtered in the target media data to obtain the media data to be processed; The loudness gain module is used to perform loudness processing on the to-be-processed media data based on the loudness gain and the loudness compensation gain to obtain media data for playing.

13. A computer device comprising: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the audio processing method according to any one of claims 1 to 10 by executing the computer instructions. 14 . A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the audio processing method according to claim 1 .

Images