Method, apparatus, medium and program product for loudness equalization processing

Abstract

The invention provides a method and device for loudness equalization processing, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: for to-be-processed audio data which needs to be subjected to loudness equalization processing, acquiring a volume equalization parameter corresponding to the to-be-processed audio data; performing loudness equalization processing on the to-be-processed audio based on the obtained volume equalization parameter, wherein the loudness equalization processing controls the output loudness of the to-be-processed audio and video through a gain node; and carrying out loudness measurement based on the audio data after loudness equalization processing. According to the method and the device, the loudness balance processing is performed on the played audio data at the WEB end, and the loudness output by the to-be-processed audio and video is controlled through the gain node, so that the audio loudness can be dynamically adjusted in different client devices to adapt to different playing environments and user preferences, and the video playing quality and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium, and computer program product for loudness equalization processing. Background Art

[0002] On video media platforms, users upload a variety of video content. If users watch multiple videos in a row, the difference in audio loudness of each video may affect the user experience. For example, the background music of one video may be very loud, while the dialogue sound of the next video is very low. This sudden change in loudness may make users feel uncomfortable, and they may even need to adjust the volume frequently to adapt to different audio levels.

[0003] In addition, videos on video media platforms can be viewed on a variety of devices, such as smartphones, tablets, TVs, and computers. Since these devices have different requirements for audio loudness, if unified audio loudness equalization is performed in the cloud, the audio effect on some devices may not be ideal. For example, if the audio loudness is increased to accommodate the high volume requirements of a TV, it may be too loud when played on a smartphone, and vice versa. Summary of the invention

[0004] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for loudness equalization processing.

[0005] In one aspect of the present application, a method for loudness equalization processing is provided, wherein the method comprises:

[0006] For the audio data to be processed that needs to be processed with loudness equalization, obtaining a volume equalization parameter corresponding to the audio data to be processed;

[0007] Based on the obtained volume equalization parameters, loudness equalization processing is performed on the processed audio, and the loudness equalization processing controls the loudness of the processed audio and video output through the gain node;

[0008] Loudness measurement is performed based on the audio data after loudness equalization processing.

[0009] In one aspect of the present application, a device for loudness equalization processing is provided, wherein the device comprises:

[0010] A device for obtaining, for audio data to be processed that needs to be processed with loudness equalization, volume equalization parameters corresponding to the audio data to be processed;

[0011] A device for performing loudness equalization processing on the audio to be processed based on the acquired volume equalization parameters, wherein the loudness equalization processing controls the loudness of the audio and video output to be processed through a gain node;

[0012] A device for performing loudness measurement based on audio data after loudness equalization processing.

[0013] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the embodiment of the application.

[0017] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the application.

[0018] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0019] In the solution provided by the embodiment of the present application, by performing loudness equalization processing on the played audio data on the WEB side and controlling the loudness of the processed audio and video output through the gain node, the audio loudness can be dynamically adjusted in different client devices to adapt to different playback environments and user preferences, thereby improving the quality of video playback and user experience; the loudness of the audio data after the loudness equalization processing is measured, and the audio data is processed through the worker and AudioWorklet during the loudness measurement process to achieve real-time loudness detection; by calculating the audio energy, audio visualization information including an audio energy percentage display diagram is presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0022] Figure 1 A schematic flow chart of a method for loudness equalization processing provided in an embodiment of the present application is shown;

[0023] Figure 2 A schematic diagram of the structure of a device for loudness equalization processing provided by an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0025] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0028] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0029] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0030] Figure 1 A schematic flow chart of a method for switching audio tracks provided in an embodiment of the present application is shown. The method at least includes step S101, step S102 and step S103.

[0031] The execution subject of the method may be a client device or an application running on the client device. The client device includes but is not limited to computers, mobile phones, tablet computers, smart watches, bracelets and other terminal devices.

[0032] Preferably, the method is implemented based on a browser on the WEB side, or based on an application of a browser kernel.

[0033] The concepts involved in the embodiments of the present application are explained below.

[0034] Gain: Gain is a basic concept in audio signal processing, which is used to control the volume of audio signals.

[0035] MSE: MSE (Media Source Extensions) is a protocol for streaming media content to <audio>or <video>This technology allows developers to add audio or video data to a media element in the form of a stream, chunk by chunk, rather than loading the entire file at once.

[0036] Web Audio API: Web Audio API is a powerful browser API that allows developers to implement complex audio effects and processing on web pages. Web Audio API provides methods for processing and synthesizing audio, which can be used to create music players, game sound effects, audio visualization and other innovative audio applications.

[0037] FFmpeg: FFmpeg is an open source multimedia framework that can be used to process video and audio data. It includes a very powerful set of command-line tools that can be used to record, convert, and stream audio and video data. FFmpeg supports a variety of audio and video codecs and container formats, so it is very flexible and can be used for a variety of different audio and video processing tasks.

[0038] According to one embodiment, before step S101, the method further includes step S104 and step S105.

[0039] In step S104, it is determined whether the loudness of the audio or video currently being played in the client device meets a predetermined equalization processing condition.

[0040] For example, it is detected whether the loudness of the currently played audio or video exceeds a predetermined threshold, and if so, it is determined that the equalization processing condition is met.

[0041] In step S105, if the equalization processing condition is met, the corresponding audio data is obtained as the audio data to be processed that needs to be processed by loudness equalization processing.

[0042] Refer to the following Figure 1 To illustrate, in step S101, for the audio data to be processed that needs to be processed with loudness equalization, volume equalization parameters corresponding to the audio data to be processed are obtained.

[0043] The volume equalization parameters include measured loudness, target loudness and high dynamic target loudness.

[0044] The measured loudness (Measured_I) indicates the overall loudness level of the audio file and can be used for subsequent loudness equalization processing.

[0045] Target loudness (Target_I) indicates the loudness level you want to achieve during audio processing. During the transcoding process, the target loudness is set as a reference value to guide subsequent loudness equalization.

[0046] High dynamic target loudness (Dynamic_I) is a loudness target set to maintain the dynamic range of the audio content. Dynamic range refers to the difference between the loudest and quietest sounds in the audio. High dynamic target loudness allows the audio to retain more dynamic changes while maintaining a certain loudness, avoiding auditory fatigue caused by over-compression.

[0047] The method can obtain the original measured loudness of the video to be processed through a variety of audio analysis tools or software.

[0048] Optionally, the method measures the audio loudness through FFmpeg during transcoding, obtains the original measured loudness of the video to be processed, and obtains a preset target loudness and a high dynamic target loudness.

[0049] In step S102, loudness equalization processing is performed on the audio to be processed based on the acquired volume equalization parameters, and the loudness equalization processing controls the loudness of the audio and video output to be processed through a gain node.

[0050] Specifically, the step S102 includes step S1021, step S1022 and step S1023.

[0051] In step S1021 , an average loudness difference is calculated based on the measured loudness and the target loudness / high dynamic target loudness.

[0052] Specifically, the average loudness difference is calculated based on the following formula:

[0053] D_I = Measure_I - (Target_I or Dynamic_I) (1)

[0054] Among them, D_I represents, Measure_I represents measured loudness, Target_I represents target loudness, and Dynamic_I represents high dynamic target loudness.

[0055] In step S1022, a corresponding target gain value is obtained based on the loudness difference.

[0056] Specifically, the obtained loudness difference is converted into units and converted into a corresponding target gain value based on the following formula:

[0057] Gain = Math.pow( 10, D_I / 20 ) (2)

[0058] Wherein, Gain represents the target gain value.

[0059] In step S1023, the obtained target gain value is applied to the gain node of Web Audio to perform average loudness gain in the gain node.

[0060] The gain node (GainNode) is a node in the Web Audio API for implementing gain control.

[0061] By applying the target gain value to GainNode, the audio in the embodiment of the present application is originally played using MSE, and the audio after loudness equalization processing is played using WebAudio.

[0062] Optionally, the process of creating a GainNode in Web Audio and performing average loudness gain control through the GainNode in the embodiment of the present application is as follows:

[0063] Initialize AudioContext: Create an AudioContext instance, which is the starting point for all audio processing;

[0064] Create GainNode: Create a GainNode in AudioContext for subsequent volume control;

[0065] Loading and decoding audio: If you use MSE, the audio data will be loaded and decoded block by block. If you use the WebAudio API, you can directly decode the audio file into AudioBuffer. AudioBuffer is an interface in the Web Audio API. The object of this interface is a short-term audio resource stored in memory.

[0066] Connect audio source: connect the decoded audio data to GainNode;

[0067] Calculate the target gain value: Calculate the target gain value to be applied based on the measured loudness and the target loudness / high dynamic target loudness, and set the gain property of the GainNode;

[0068] Connect to output: Connect the GainNode to the AudioContext's destination (AudioContext.destination), which is usually a speaker, or to other audio nodes for processing;

[0069] Play audio: Start playing the audio. At this time, the audio data after loudness equalization processing will be adjusted to the loudness corresponding to the target gain value through GainNode.

[0070] Continue to refer to Figure 1 To explain, in step S103, loudness measurement is performed based on the audio data after the loudness equalization processing.

[0071] The loudness measurement includes measurement of overall loudness, short-term loudness and / or instantaneous loudness.

[0072] The integrated loudness refers to the average loudness from the start point to the end point of the audio, and the integrated loudness is used to indicate the overall loudness level of the entire audio clip.

[0073] Among them, short-term loudness refers to the loudness within a short time window (usually 3 seconds). Short-term loudness helps to identify loudness changes in audio.

[0074] Among them, momentary loudness refers to the loudness within a very short time window (usually 400 milliseconds), and momentary loudness can capture the instantaneous changes of audio.

[0075] According to one embodiment, the method uses AudioWorkletNode or WebWorker to process audio data on a background thread during the loudness measurement in step S103.

[0076] Since the amount of audio data is usually large, performing calculations directly on the main thread may affect the responsiveness of the user interface. Using AudioWorkletNode or WebWorker to process audio data on a background thread can avoid this situation.

[0077] AudioWorkletNode is part of the Web Audio API, which allows developers to run custom JavaScript code in the context of audio processing, thereby achieving more efficient audio data processing. WebWorker allows scripts to be run outside the main thread, so that even if the processing task is heavy, the user interface will not be frozen.

[0078] Optionally, the method performs loudness measurement and loudness control based on the EBU-R128 standard, wherein the EBU-R128 standard generally includes the following steps:

[0079] Pre-processing of audio data: This usually involves applying a pre-filter and a high-pass filter (RLB-filter) to simulate the human ear's sensitivity to sounds of different frequencies;

[0080] Mean square summation: After filtering, the audio signal is mean squared to calculate its energy;

[0081] Numerical processing conversion: Convert the calculated energy value into loudness units (LU or LUFS).

[0082] According to one embodiment, the method implements the filter of the EBU-R128 standard through the IIR filter or the biquad filter (BiquadFilterNode) of the AudioContext on the Web side.

[0083] According to one embodiment, the method further includes step S106.

[0084] In step S106, measurement result information corresponding to the loudness measurement is displayed.

[0085] The measurement result information includes measurement result values ​​of overall loudness, short-term loudness and / or instantaneous loudness displayed in various types such as charts. Optionally, the measurement result information may also include other information obtained based on the overall loudness, short-term loudness and / or instantaneous loudness, such as loudness range or loudness growth. By observing the displayed loudness measurement results, the user can perceive whether the loudness of the audio content meets his or her auditory habits or specific playback standards.

[0086] According to one embodiment, the method further includes step S107.

[0087] In step S107, in response to the user's adjustment operation on the current equalization process, the equalization process strategy is adjusted or closed.

[0088] For example, if the user finds that the loudness of the currently playing audio does not meet their expectations based on real-world measurement results, they can use the system-provided tools to adjust the equalizer settings. The equalizer allows the user to enhance or attenuate the sound of a specific frequency range to achieve an ideal loudness balance. If the user wants to turn off the equalizer completely so that they can hear the original unprocessed audio, the equalization strategy can be easily enabled or disabled based on the system-provided switch options.

[0089] According to one embodiment, the method further includes step S108.

[0090] In step S108, the loudness change of the played audio is monitored and analyzed based on the measurement result information.

[0091] For example, by using the player's built-in analysis tools to monitor audio loudness changes in real time, it ensures that the audio output always remains within the range set by the user and helps identify possible audio problems such as sudden loudness jumps or persistent loudness inconsistencies.

[0092] Optionally, the method may collect user feedback information on loudness perception and the effects of equalizer adjustments to further optimize the audio processing algorithm and user interface.

[0093] According to one embodiment, the method further includes step S109.

[0094] In step S109, the audio visualization information of the played video is displayed.

[0095] According to one embodiment, the audio visualization information includes a spectrum diagram, a time-domain PCM waveform diagram, and an amplitude waveform diagram.

[0096] The spectrum diagram is used to display the distribution of audio signals at different frequencies, which is helpful for analyzing the frequency components and equalization effects of the audio.

[0097] The time-domain pulse code modulation (PCM) waveform diagram shows the waveform changes of the audio signal in the time dimension, which is helpful for observing the dynamic characteristics of the audio signal and possible noise interference.

[0098] The amplitude waveform diagram reflects the change of the audio signal amplitude over time, which is helpful for evaluating the loudness and dynamic range of the audio.

[0099] According to an embodiment, the audio visualization information includes an audio energy percentage display diagram. The method of this embodiment estimates audio energy based on PCM data through step 110 and step S111.

[0100] In step S110, the PCM absolute value average amplitude is obtained.

[0101] Among them, in the PCM data with a bit depth of 16 bits, the maximum value is 32767. By calculating the absolute value mean amplitude of the PCM data, the basic amplitude information of the audio signal can be obtained.

[0102] In step S111 , the PCM energy is estimated by calculating the area of ​​the sampling point amplitudes.

[0103] Specifically, the decibel calculation formula shown below is used to calculate the loudness of the audio signal relative to the maximum possible amplitude:

[0104] 20×log10(x / y)(3)

[0105] Where x represents the amplitude of the sampling point and y represents the reference value.

[0106] Next, the PCM data less than 1250 is taken as the 10% energy line by the following formula:

[0107] Math.round((power / 1250)*10)(4)

[0108] One-third (about 11,000) of the maximum value (32,767) is taken as the upper limit of energy. PCM data exceeding this value may cause discomfort to the human ear.

[0109] Next, use the following formula to calculate the energy percentage to ensure that the loudness level is within the acceptable range:

[0110] Math.round(Math.min(100,Math.max(0,(1+Math.log10(power / 11000)))* 100)))(5)

[0112] Based on the audio energy percentage display graph, developers can analyze in real time whether there are any abnormalities in the current equalization or sound effects, and adjust the loudness and quality of the audio signal accordingly.

[0113] According to the method of the embodiment of the present application, by performing loudness equalization processing on the played audio data on the WEB side and controlling the loudness of the processed audio and video output through the gain node, the audio loudness can be dynamically adjusted in different client devices to adapt to different playback environments and user preferences, thereby improving the quality of video playback and user experience; the loudness of the audio data after the loudness equalization processing is measured, and the audio data is processed through the worker and AudioWorklet during the loudness measurement process to achieve real-time loudness detection; by calculating the audio energy, audio visualization information including an audio energy percentage display diagram is presented.

[0114] In addition, the embodiment of the present application also provides a device for loudness equalization processing, the structure of which is as follows: Figure 2 The device comprises: a device for acquiring volume equalization parameters corresponding to the audio data to be processed (hereinafter referred to as "parameter acquisition device 101") for the audio data to be processed that needs to be processed with loudness equalization, a device for performing loudness equalization processing on the audio data to be processed based on the acquired volume equalization parameters (hereinafter referred to as "loudness equalization device 102"), and a device for performing loudness measurement based on the audio data after the loudness equalization processing (hereinafter referred to as "loudness measurement device 103").

[0115] Wherein, the device is included in the client equipment.

[0116] Preferably, the device is implemented based on a browser on the WEB side, or based on an application of a browser kernel.

[0117] According to one embodiment, the device further includes a condition determination device and an audio acquisition device.

[0118] The condition determination device determines whether the loudness of the audio or video currently played in the client device meets a predetermined equalization processing condition.

[0119] For example, it is detected whether the loudness of the currently played audio or video exceeds a predetermined threshold, and if so, it is determined that the equalization processing condition is met.

[0120] If the equalization processing condition is met, the audio acquisition device acquires the corresponding audio data as the audio data to be processed that needs to be processed with loudness equalization.

[0121] Refer to the following Figure 2 To illustrate, the parameter acquisition device 101 acquires the volume equalization parameters corresponding to the audio data to be processed for the audio data to be processed that needs to be processed with loudness equalization.

[0122] The volume equalization parameters include measured loudness, target loudness and high dynamic target loudness.

[0123] The measured loudness (Measured_I) indicates the overall loudness level of the audio file and can be used for subsequent loudness equalization processing.

[0124] Target loudness (Target_I) indicates the loudness level you want to achieve during audio processing. During the transcoding process, the target loudness is set as a reference value to guide subsequent loudness equalization.

[0125] High dynamic target loudness (Dynamic_I) is a loudness target set to maintain the dynamic range of the audio content. Dynamic range refers to the difference between the loudest and quietest sounds in the audio. High dynamic target loudness allows the audio to retain more dynamic changes while maintaining a certain loudness, avoiding auditory fatigue caused by over-compression.

[0126] The parameter acquisition device 101 may acquire the original measured loudness of the video to be processed through a variety of audio analysis tools or software.

[0127] Optionally, the parameter acquisition device 101 measures the audio loudness through FFmpeg during transcoding, obtains the original measured loudness of the video to be processed, and obtains a preset target loudness and a high dynamic target loudness.

[0128] The loudness equalization device 102 performs loudness equalization processing on the audio to be processed based on the acquired volume equalization parameters. The loudness equalization processing controls the loudness of the audio and video output to be processed through a gain node.

[0129] The loudness equalization device 102 further includes a difference calculation device, a gain calculation device and a gain output device.

[0130] The difference calculation device calculates an average loudness difference based on the measured loudness and the target loudness / high dynamic target loudness.

[0131] Specifically, the difference calculation device calculates the average loudness difference based on the above formula (1).

[0132] The gain calculation is based on the loudness difference to obtain a corresponding target gain value.

[0133] Specifically, the obtained loudness difference is converted into units and converted into a corresponding target gain value based on the above formula (2).

[0134] The gain output device applies the obtained target gain value to the gain node of the Web Audio to perform an average loudness gain in the gain node.

[0135] The gain node (GainNode) is a node in the Web Audio API for implementing gain control.

[0136] By applying the target gain value to GainNode, the audio in the embodiment of the present application is originally played using MSE, and the audio after loudness equalization processing is played using WebAuio.

[0137] Optionally, the process of the device of the embodiment of the present application creating a GainNode in Web Audio and performing average loudness gain control through the GainNode is as follows:

[0138] Initialize AudioContext: Create an AudioContext instance, which is the starting point for all audio processing;

[0139] Create GainNode: Create a GainNode in AudioContext for subsequent volume control;

[0140] Loading and decoding audio: If you use MSE, the audio data will be loaded and decoded block by block. If you use the WebAudio API, you can directly decode the audio file into AudioBuffer. AudioBuffer is an interface in the Web Audio API. The object of this interface is a short-term audio resource stored in memory.

[0141] Connect audio source: connect the decoded audio data to GainNode;

[0142] Calculate the target gain value: Calculate the target gain value to be applied based on the measured loudness and the target loudness / high dynamic target loudness, and set the gain property of the GainNode;

[0143] Connect to output: Connect the GainNode to the AudioContext's destination (AudioContext.destination), which is usually a speaker, or to other audio nodes for processing;

[0144] Play audio: Start playing the audio. At this time, the audio data after loudness equalization processing will be adjusted to the loudness corresponding to the target gain value through GainNode.

[0145] Continue to refer to Figure 2 To explain, the loudness measurement device 103 performs loudness measurement based on the audio data after the loudness equalization process.

[0146] The loudness measurement includes measurement of overall loudness, short-term loudness and / or instantaneous loudness.

[0147] The integrated loudness refers to the average loudness from the start point to the end point of the audio, and the integrated loudness is used to indicate the overall loudness level of the entire audio clip.

[0148] Among them, short-term loudness refers to the loudness within a short time window (usually 3 seconds). Short-term loudness helps to identify loudness changes in audio.

[0149] Among them, momentary loudness refers to the loudness within a very short time window (usually 400 milliseconds), and momentary loudness can capture the instantaneous changes of audio.

[0150] According to one embodiment, the loudness measurement device 103 uses AudioWorkletNode or WebWorker to process audio data on a background thread during loudness measurement.

[0151] Since the amount of audio data is usually large, performing calculations directly on the main thread may affect the responsiveness of the user interface. Using AudioWorkletNode or WebWorker to process audio data on a background thread can avoid this situation.

[0152] AudioWorkletNode is part of the Web Audio API, which allows developers to run custom JavaScript code in the context of audio processing, thereby achieving more efficient audio data processing. WebWorker allows scripts to be run outside the main thread, so that even if the processing task is heavy, the user interface will not be frozen.

[0153] Optionally, the method performs loudness measurement and loudness control based on the EBU-R128 standard, wherein the EBU-R128 standard generally includes the following steps:

[0154] Pre-processing of audio data: This usually involves applying a pre-filter and a high-pass filter (RLB-filter) to simulate the human ear's sensitivity to sounds of different frequencies;

[0155] Mean square summation: After filtering, the audio signal is mean squared to calculate its energy;

[0156] Numerical processing conversion: Convert the calculated energy value into loudness units (LU or LUFS).

[0157] According to one embodiment, the method implements the filter of the EBU-R128 standard through the IIR filter or the biquad filter (BiquadFilterNode) of the AudioContext on the Web side.

[0158] According to one embodiment, the device further comprises a result display device.

[0159] The result display device displays the measurement result information corresponding to the loudness measurement.

[0160] The measurement result information includes measurement result values ​​of overall loudness, short-term loudness and / or instantaneous loudness displayed in various types such as charts. Optionally, the measurement result information may also include other information obtained based on the overall loudness, short-term loudness and / or instantaneous loudness, such as loudness range or loudness growth. By observing the displayed loudness measurement results, the user can perceive whether the loudness of the audio content meets his or her auditory habits or specific playback standards.

[0161] According to one embodiment, the device further comprises a balance adjustment device.

[0162] The equalization adjustment device adjusts or closes the equalization processing strategy in response to the user's adjustment operation on the current equalization processing.

[0163] For example, if the user finds that the loudness of the currently playing audio does not meet their expectations based on real-world measurement results, the equalization adjustment device uses the tools provided by the system to adjust the equalizer settings. The equalizer allows the user to enhance or attenuate the sound of a specific frequency range to achieve an ideal loudness balance. If the user wants to turn off the equalizer completely so that the original unprocessed audio can be heard, the equalization strategy can be easily enabled or disabled based on the switch options provided by the system.

[0164] According to one embodiment, the device further comprises a measurement and analysis device.

[0165] The measurement and analysis device monitors and analyzes loudness changes of the played audio based on the measurement result information.

[0166] For example, the measurement and analysis device monitors the loudness changes of the audio in real time by using the player's built-in analysis tools, thereby ensuring that the audio output always remains within the range set by the user and helping to identify possible audio problems such as sudden loudness jumps or continuous loudness inconsistencies.

[0167] Optionally, the device may collect user feedback information on loudness perception and the effects of equalizer adjustments to further optimize the audio processing algorithm and user interface.

[0168] According to one embodiment, the device further comprises information visualization means.

[0169] The information visualization device displays audio visualization information of the playing video.

[0170] According to one embodiment, the audio visualization information includes a spectrum diagram, a time-domain PCM waveform diagram, and an amplitude waveform diagram.

[0171] The spectrum diagram is used to display the distribution of audio signals at different frequencies, which is helpful for analyzing the frequency components and equalization effects of the audio.

[0172] The time-domain pulse code modulation (PCM) waveform diagram shows the waveform changes of the audio signal in the time dimension, which is helpful for observing the dynamic characteristics of the audio signal and possible noise interference.

[0173] The amplitude waveform diagram reflects the change of the audio signal amplitude over time, which is helpful for evaluating the loudness and dynamic range of the audio.

[0174] According to an embodiment, the audio visualization information includes an audio energy percentage display diagram. The visualization device of this embodiment estimates audio energy based on PCM data through the following operations.

[0175] The visualization device obtains the PCM absolute value average amplitude. Among them, in the PCM data at a bit depth of 16 bits, the maximum value is 32767. By calculating the absolute value average amplitude of the PCM data, the basic amplitude information of the audio signal can be obtained.

[0176] Next, the visualization device estimates the PCM energy by calculating the area of ​​the sampling point amplitude. Specifically, the decibel calculation formula (3) is used to calculate the loudness of the audio signal relative to the maximum possible amplitude.

[0177] Next, the visualization device uses formula (4) to take the PCM data less than 1250 as the 10% energy line.

[0178] Next, the visualizer uses formula (5) to calculate the energy percentage to ensure that the loudness level is within an acceptable range.

[0179] Based on the audio energy percentage display graph, developers can analyze in real time whether there are any abnormalities in the current equalization or sound effects, and adjust the loudness and quality of the audio signal accordingly.

[0180] According to the device of the embodiment of the present application, by performing loudness equalization processing on the played audio data on the WEB side and controlling the loudness of the processed audio and video output through the gain node, the audio loudness can be dynamically adjusted in different client devices to adapt to different playback environments and user preferences, thereby improving the quality of video playback and user experience; the loudness of the audio data after the loudness equalization processing is measured, and the audio data is processed through the worker and AudioWorklet during the loudness measurement process to achieve real-time loudness detection; by calculating the audio energy, audio visualization information including an audio energy percentage display diagram is presented.

[0181] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for loudness equalization processing in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0182] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0183] Figure 3 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0184] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0185] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0186] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0187] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0188] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0189] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0190] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0196] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0198] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.< / video> < / audio>

Claims

1. A method for loudness equalization processing, wherein: The method comprises: For the audio data to be processed that needs to be processed with loudness equalization, obtaining the volume equalization parameter corresponding to the audio data to be processed; Based on the obtained volume equalization parameters, loudness equalization processing is performed on the audio to be processed, and the loudness equalization processing controls the loudness of the audio and video output to be processed through a gain node; Loudness measurement based on audio data after loudness equalization processing.

2. The method according to claim 1, wherein: The loudness equalization processing of the audio to be processed based on the acquired volume equalization parameter includes: Calculate the average loudness difference based on the measured loudness and the target loudness / high dynamic target loudness; Based on the loudness difference, obtaining a corresponding target gain value; The resulting target gain value is applied to the Web Audio gain node to perform an average loudness gain in the gain node.

3. The method according to claim 1, wherein: The loudness measurement based on the audio data after the loudness equalization processing comprises: The loudness measurement process uses an AudioWorkletNode or a WebWorker to process the audio data on a background thread.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: Displays the measurement result information corresponding to the loudness measurement.

5. The method according to any one of claims 1 to 3, wherein: The method further comprises: In response to the user's adjustment operation on the current equalization process, the equalization process strategy is adjusted or closed.

6. The method according to any one of claims 1 to 3, wherein: The method further comprises: The audio visualization information of the played video is displayed, wherein the audio visualization information includes a spectrum diagram, a time domain PCM waveform diagram, and an amplitude waveform diagram.

7. The method according to claim 6, wherein: The audio visualization information includes an audio energy percentage display diagram, and the method estimates the audio energy based on the PCM data by the following steps: Get the PCM absolute value average amplitude; The PCM energy is estimated by calculating the area of ​​the sampling point amplitude.

8. The method according to any one of claims 1 to 3, wherein: The method further comprises: Determine whether the loudness of the audio and video currently played in the client device meets the predetermined equalization processing condition; If the equalization processing condition is met, the corresponding audio data is obtained as the audio data to be processed that needs to be processed with loudness equalization processing.

9. A device for loudness equalization processing, wherein: The device comprises: A device for obtaining, for audio data to be processed that needs to be processed with loudness equalization, volume equalization parameters corresponding to the audio data to be processed; A device for performing loudness equalization processing on the audio to be processed based on the acquired volume equalization parameters, wherein the loudness equalization processing controls the loudness of the audio and video output to be processed through a gain node; A device for performing loudness measurement based on audio data after loudness equalization processing.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images