Video internal loudness dynamic compensation method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of audio processing and playing control, and discloses a video internal loudness dynamic compensation method and device, electronic equipment and storage medium. The method calculates the overall loudness of the current video file and the short-time loudness in the sliding time window, generates an overall loudness compensation gain according to the difference between the overall loudness and the reference loudness; when the short-time loudness is lower than the reference loudness and the difference exceeds the trigger threshold, a short-time loudness compensation gain is generated. The system synthesizes the two compensation gains into a candidate compensation gain, then obtains a safe gain through upper and lower limits, anti-shake and maximum step constraints, and generates a final gain by using an asymmetric smoothing mode of fast descent and slow ascent. The method keeps the overall loudness of the video close to the reference level while performing real-time compensation on the low-loudness segments with short duration inside the video, thereby reducing the gain jitter caused by frequent adjustment.

Description

Technical Field

[0001] This invention relates to the field of audio processing and playback control technology, specifically to a method, apparatus, electronic device, and storage medium for dynamic compensation of video internal loudness that controls audio gain based on the overall loudness of the video and the short-term loudness during playback. Background Technology

[0002] The audio in video files typically includes dialogue, music, and ambient sounds. The loudness of the same video may vary at different times of playback due to factors such as recording distance, changes in sound sources, and post-production techniques.

[0003] Overall loudness reflects the combined loudness level of a video over a longer period. Static normalization can set a fixed gain for the entire video based on the overall loudness, but a fixed gain cannot reflect short-duration low-loudness segments during playback. During playback, the volume may still need to be repeatedly increased and then decreased when normal loudness content returns.

[0004] Therefore, the technical problem to be solved by this invention is how to maintain the overall loudness level of the video while providing real-time compensation for low-loudness segments based on short-term loudness during playback, and to limit frequent changes in gain. Summary of the Invention

[0005] This invention provides a method for dynamic loudness compensation within a video, used to perform real-time gain compensation on low-loudness segments within a video.

[0006] Therefore, the first aspect of the present invention provides a method for dynamic loudness compensation within a video. The method includes: acquiring audio data of a current video file; determining the overall loudness of the current video file based on the audio data, and determining a short-term loudness that changes with playback time according to a sliding time window; determining an overall loudness compensation gain based on the difference between the overall loudness and a reference loudness; when the short-term loudness is lower than the reference loudness and the difference between them exceeds a short-term compensation trigger threshold, determining a short-term loudness compensation gain based on the difference between the reference loudness and the short-term loudness; combining the overall loudness compensation gain and the short-term loudness compensation gain into a candidate compensation gain; applying upper and lower limit constraints to the candidate compensation gain, and... When the change in gain relative to the previous safe gain is less than the anti-shake threshold, the previous safe gain is kept constant, and the maximum gain change between adjacent time windows is limited to obtain the current safe gain. When the safe gain is less than the final gain of the previous time, a fast response coefficient is used to smooth the safe gain. When the safe gain is not less than the final gain of the previous time, a slow recovery coefficient is used to smooth the safe gain to obtain the current final gain, wherein the fast response coefficient is greater than the slow recovery coefficient. The audio playback output of the current video file is controlled according to the final gain. Overall loudness compensation is used to correct the overall loudness level of the video, and short-time loudness compensation only intervenes when the low loudness deviation exceeds the threshold. Anti-shake, step size, and asymmetric smoothing limit the rate of change of the final gain to avoid frequent start and stop of compensation.

[0007] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the overall loudness compensation gain satisfies G_overall = clip(L_ref - L_overall, G_min_overall, G_max_overall), where L_ref is the reference loudness, L_overall is the overall loudness, G_min_overall and G_max_overall are the lower limit and upper limit of the overall loudness compensation, respectively, and clip(x, a, b) indicates that x is restricted to [a, b]. The upper and lower limits are used to limit the gain adjustment range of the entire video.

[0008] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the short-time loudness compensation gain satisfies: G_ST(t) = clip(L_ref - L_S(t), 0, G_max_ST); where L_S(t) is the short-time loudness at time t, and G_max_ST is the upper limit of short-time loudness compensation; when L_ref - L_S(t) is not greater than the short-time compensation trigger threshold, G_ST(t) is set to 0. This condition causes the short-time compensation to focus on the lower loudness segments.

[0009] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the overall loudness is determined by integral loudness measurement; the length of the sliding time window is 3 seconds. The two loudnesses correspond to different time scales.

[0010] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the candidate compensation gain satisfies: G_cand(t) = G_overall + G_ST(t); where G_cand(t) is the candidate compensation gain at time t, G_overall is the overall loudness compensation gain, and G_ST(t) is the short-time loudness compensation gain at time t. The overall loudness compensation forms a relatively stable gain basis, while the short-time loudness compensation changes with the video content.

[0011] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the anti-shake constraint satisfies: when |G_cand(t)-G_safe(t-1)| is less than the anti-shake threshold δ, G_safe(t)=G_safe(t-1); the maximum gain change is limited by the maximum step size Δ_max. These two constraints are used to filter small changes and limit the magnitude of a single adjustment.

[0012] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the final gain satisfies: G_final(t)=α_a·G_safe(t)+(1-α_a)·G_final(t-1), G_safe(t)<G_final(t-1); G_final(t) = α_r·G_safe(t) + (1-α_r)·G_final(t-1), G_safe(t) ≥ G_final(t-1); where 0 < α_r < α_a < 1, α_a is the fast response coefficient, and α_r is the slow recovery coefficient. When the safety gain decreases, the final gain follows quickly; when the safety gain increases, the final gain recovers slowly.

[0013] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the overall loudness and the short-term loudness are pre-calculated by an upstream device cooperating with the end-side device and stored together with the current video file in the form of metadata; the end-side device reads the metadata and generates the final gain when playing the current video file. Reading the metadata when the end-side device plays the video can reduce the amount of computation for real-time loudness analysis.

[0014] In the video internal loudness dynamic compensation method according to the first aspect of the present invention, optionally, the method runs offline on the end-side device. The end-side device may be a digital photo frame, smartphone, tablet computer, smart TV, or in-vehicle entertainment system.

[0015] A second aspect of the present invention provides a dynamic loudness compensation device for video, comprising: an audio acquisition module for acquiring audio data of a current video file; a loudness determination module for determining the overall loudness of the current video file based on the audio data, and determining the short-term loudness that changes with playback time according to a sliding time window; a compensation gain generation module for determining an overall loudness compensation gain based on the difference between the overall loudness and a reference loudness, determining a short-term loudness compensation gain when the short-term loudness is lower than the reference loudness and the difference between the two exceeds a short-term compensation trigger threshold, and combining the overall loudness compensation gain and the short-term loudness compensation gain into a candidate compensation gain; a safety control module for applying upper and lower limit constraints, anti-shake constraints, and maximum step size constraints to the candidate compensation gain to obtain a safety gain; a smoothing processing module for smoothing the safety gain by using a fast response coefficient when the safety gain decreases and a slow recovery coefficient smaller than the fast response coefficient when the safety gain increases to obtain a final gain; and a playback control module for controlling the audio playback output of the current video file according to the final gain. The device processes the audio data of the current video file according to the method of the first aspect.

[0016] A third aspect of the present invention provides an electronic device, including a processor and a memory, wherein a computer program stored in the memory is executed by the processor to implement the method described in the first aspect.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium on which a computer program stored is executed by a processor to implement the method described in the first aspect. Attached Figure Description

[0018] Figure 1 This is a flowchart of a video internal loudness dynamic compensation method according to an embodiment of the present invention.

[0019] Figure 2 This is a structural diagram of a video internal loudness dynamic compensation device according to an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention are described below with reference to the accompanying drawings. The drawings are used to illustrate the relationships between steps or modules and do not represent actual dimensions. The same steps or modules are referred to by the same reference numerals in different drawings.

[0021] It should be noted that the terms "comprising" and "having" in the specification and claims indicate that steps or units not explicitly listed may also be included, but this does not change the technical relationships already explicitly described. Unless the context otherwise requires, the steps may be performed in the order that satisfies the data dependencies.

[0022] This invention aims to provide a method for dynamic loudness compensation within a video, achieving the following objectives: generating an overall loudness compensation gain based on the overall loudness of the current video file, identifying low-loudness segments within the video based on short-term loudness within a sliding time window, and generating short-term loudness compensation gain when trigger conditions are met to compensate for local low-loudness fluctuations during the playback of a single video; limiting abrupt changes and frequent fluctuations in the compensation gain through gain upper and lower limits, anti-shake, maximum step size, and asymmetric smoothing with fast descent and slow ascent, thereby ensuring consistent audio output across different playback periods within the video.

[0023] In some examples, the video internal loudness dynamic compensation method involved in this invention can also be called a single video loudness adaptive compensation method, a video internal dynamic volume control method, a video audio gain control method based on dual time scale loudness, etc.

[0024] See Figure 1 In some examples, the method of this embodiment may include step 100, obtaining the audio data of the current video file.

[0025] In some examples, the audio data can be an audio signal decoded by the end device, or it can be decoded and analyzed by the upstream device during video upload or storage.

[0026] In some examples, the method of this embodiment may include step 200, determining the overall loudness L_overall of the current video file based on audio data, and determining the short-term loudness L_S(t) that varies with playback time according to a sliding time window.

[0027] In some examples, overall loudness can represent the combined loudness level of the current video within the overall playback range. Overall loudness can be determined using integral loudness measurement. Specifically, it can be calculated using the integral loudness measurement method specified in ITU-R BS.1770. Short-time loudness varies with playback time. The sliding time window can be 3 seconds long. Overall loudness describes the long-term level of the video, while short-time loudness describes local variations during playback.

[0028] In some examples, the method of this embodiment may include step 300, determining the overall loudness compensation gain based on the difference between the overall loudness L_overall and the reference loudness L_ref.

[0029] In some examples, the reference loudness L_ref represents the target loudness level that the system expects the content to approach. It is a system-wide parameter that can be configured based on device type, usage scenario, etc.

[0030] In some examples, the overall loudness compensation gain satisfies formula (1): G_overall=clip(L_ref-L_overall, G_min_overall, G_max_overall) (1) Where L_ref is the reference loudness, L_overall is the overall loudness, G_min_overall and G_max_overall are the lower limit and upper limit of overall loudness compensation, respectively, and clip(x, a, b) means restricting x to [a, b].

[0031] In some examples, the method of this embodiment may include step 400, determining whether the short-time loudness is lower than the reference loudness and whether the difference between the two exceeds the short-time compensation trigger threshold θ_ST.

[0032] In some examples, when the short-time loudness is lower than the reference loudness and the difference between the two exceeds the short-time compensation trigger threshold, the short-time loudness compensation gain is determined based on the difference between the reference loudness and the short-time loudness.

[0033] Determine the short-time loudness compensation gain according to formula (2): G_ST(t)=clip(L_ref-L_S(t), 0, G_max_ST), |L_S(t) - L_ref| > θ_ST (2) Where L_S(t) is the short-term loudness at time t, G_max_ST is the upper limit of short-term loudness compensation, and θ_ST is the short-term compensation trigger threshold. Short-term compensation is only performed when the short-term loudness deviates from the reference loudness by more than this threshold, avoiding frequent adjustments to normal volume fluctuations. When |L_ref-L_S(t)| is not greater than the short-term compensation trigger threshold, G_ST(t) is set to 0. The lower limit of short-term loudness compensation is 0, therefore this component is used to increase the short-term loudness of low-volume segments, and not to actively suppress short-term high-volume segments.

[0034] In some examples, the method of this embodiment may include step 500, which combines the two compensation components into a candidate compensation gain.

[0035] The candidate compensation gain satisfies formula (3): G_cand(t)=G_overall+G_ST(t) (3) Where G_cand(t) is the candidate compensation gain at time t, G_overall is the overall loudness compensation gain, and G_ST(t) is the short-time loudness compensation gain at time t.

[0036] Overall loudness compensation is relatively stable during the playback of a video file, while short-term loudness compensation is intervened or withdrawn based on the calculation results of the sliding window.

[0037] In some examples, the method of this embodiment may include step 600, applying upper and lower limit constraints on the candidate compensation gain, keeping the safety gain unchanged at the previous time when the change in the candidate compensation gain relative to the safety gain at the previous time is less than the anti-shake threshold, and limiting the maximum gain change between adjacent time windows to obtain the safety gain at the current time.

[0038] In some examples, upper and lower bounds, sparring, and maximum step size constraints can be applied to the candidate compensation gain. When the absolute value of the difference between the candidate compensation gain and the previous time-step safety gain is less than the sparring threshold δ: G_safe(t)=G_safe(t-1) (4) When the change reaches or exceeds δ, the system updates the safety gain according to the candidate compensation gain and limits the maximum gain change between adjacent time windows to Δ_max. This filters out small loudness changes and avoids frequent gain jumps between adjacent windows.

[0039] In some examples, the method of this embodiment may include step 700: when the security gain is less than the final gain at the previous time step, smoothing the security gain using a fast response coefficient; when the security gain is not less than the final gain at the previous time step, smoothing the security gain using a slow recovery coefficient to obtain the final gain at the current time step. The fast response coefficient is greater than the slow recovery coefficient.

[0040] In some examples, the final gain satisfies: G_final(t)=α_a·G_safe(t)+(1-α_a)·G_final(t-1), G_safe(t)<G_final(t-1) (5a) G_final(t)=α_r·G_safe(t)+(1-α_r)·G_final(t-1), G_safe(t)≥G_final(t-1) (5b) Where 0 < α_r < α_a < 1, α_a is the fast response coefficient, and α_r is the slow recovery coefficient. A larger α_a is used when the safety gain decreases, so that the final gain follows the decrease more quickly; a smaller α_r is used when the safety gain increases, so that the final gain recovers slowly.

[0041] In some examples, the overall loudness and short-term loudness can be pre-calculated by an upstream device working with the end-side device and stored as metadata along with the current video file. The end-side device can then read the metadata and generate the final gain while playing the current video file.

[0042] In some examples, the method of this embodiment may include step 800, controlling the audio playback output of the current video file according to the final gain. Specifically, the final gain can be applied in real time to the audio gain control unit of the end device to control the audio playback output of the current video file.

[0043] In some examples, the above methods can run offline on the edge device. The edge device can be a digital photo frame, smartphone, tablet, smart TV, or in-vehicle entertainment system, etc.

[0044] See Figure 2 In some examples, the video internal loudness dynamic compensation device may include an audio acquisition module 10. The audio acquisition module 10 may provide the loudness determination module 20 with the audio data of the current video file.

[0045] In some examples, the video internal loudness dynamic compensation device may include a loudness determination module 20. The loudness determination module 20 may output the overall loudness and short-term loudness.

[0046] In some examples, the video internal loudness dynamic compensation device may include a compensation gain generation module 30. The compensation gain generation module 30 can generate and synthesize two compensation components.

[0047] In some examples, the video internal loudness dynamic compensation device may include a safety control module 40. The safety control module 40 may output a safety gain.

[0048] In some examples, the video internal loudness dynamic compensation device may include a smoothing module 50. The smoothing module 50 can generate the final gain.

[0049] In some examples, the video internal loudness dynamic compensation device may include a playback control module 60. The playback control module 60 can control the audio output accordingly.

[0050] Specifically, the audio acquisition module 10 can be used to acquire the audio data of the current video file.

[0051] The loudness determination module 20 can be used to determine the overall loudness of the current video file based on the audio data, and to determine the short-term loudness that changes with the playback time according to the sliding time window.

[0052] The compensation gain generation module 30 can be used to determine the overall loudness compensation gain based on the difference between the overall loudness and the reference loudness; when the short-time loudness is lower than the reference loudness and the difference between the two exceeds the short-time compensation trigger threshold, the short-time loudness compensation gain is determined based on the difference between the reference loudness and the short-time loudness; and the overall loudness compensation gain and the short-time loudness compensation gain are combined into a candidate compensation gain.

[0053] The safety control module 40 can be used to apply upper and lower limit constraints on the candidate compensation gain, and keep the safety gain of the previous moment unchanged when the change of the candidate compensation gain relative to the safety gain of the previous moment is less than the anti-shake threshold, and limit the maximum gain change between adjacent time windows to obtain the safety gain of the current moment.

[0054] The smoothing module 50 can be used to smooth the security gain using a fast response coefficient when the security gain is less than the final gain of the previous moment; and to smooth the security gain using a slow recovery coefficient when the security gain is not less than the final gain of the previous moment, so as to obtain the final gain of the current moment, wherein the fast response coefficient is greater than the slow recovery coefficient.

[0055] The playback control module 60 can be used to control the audio playback output of the current video file according to the final gain.

[0056] Overall loudness and short-term loudness can also be pre-calculated by upstream devices and stored as metadata along with the video file. When playing the video, the end device reads the loudness metadata to perform compensation gain calculations, safety controls, and smoothing.

[0057] The embodiments of the present invention have been described above with reference to the accompanying drawings. Those skilled in the art can adjust the specific parameters or implementation forms without changing the technical relationships between overall loudness compensation, threshold-triggered short-term loudness compensation, anti-jitter and step size constraints, and asymmetric smoothing.

Claims

1. A method for dynamic loudness compensation within a video, characterized in that, include: Get the audio data of the current video file; The overall loudness of the current video file is determined based on the audio data, and the short-term loudness that changes with playback time is determined according to a sliding time window; The overall loudness compensation gain is determined based on the difference between the overall loudness and the reference loudness; When the short-time loudness is lower than the reference loudness and the difference between the two exceeds the short-time compensation trigger threshold, the short-time loudness compensation gain is determined based on the difference between the reference loudness and the short-time loudness. The overall loudness compensation gain and the short-time loudness compensation gain are combined into a candidate compensation gain; Apply upper and lower limit constraints to the candidate compensation gain, and keep the safety gain of the previous time unchanged when the change of the candidate compensation gain relative to the safety gain of the previous time is less than the anti-shake threshold, and limit the maximum gain change between adjacent time windows to obtain the safety gain at the current time. When the security gain is less than the final gain at the previous moment, the security gain is smoothed using a fast response coefficient. When the safety gain is not less than the final gain at the previous moment, the safety gain is smoothed using a slow recovery coefficient to obtain the final gain at the current moment, wherein the fast response coefficient is greater than the slow recovery coefficient. The audio playback output of the current video file is controlled according to the final gain.

2. The video internal loudness dynamic compensation method according to claim 1, characterized in that, The overall loudness compensation gain satisfies: G_overall=clip(L_ref-L_overall, G_min_overall, G_max_overall); Where L_ref is the reference loudness, L_overall is the overall loudness, G_min_overall and G_max_overall are the lower limit and upper limit of overall loudness compensation, respectively, and clip(x, a, b) means restricting x to [a, b].

3. The video internal loudness dynamic compensation method according to claim 2, characterized in that, The short-time loudness compensation gain satisfies: G_ST(t)=clip(L_ref-L_S(t), 0, G_max_ST), |L_S(t) - L_ref| > θ_ST; Wherein, L_S(t) is the short-term loudness at time t, and G_max_ST is the upper limit of short-term loudness compensation; when L_ref-L_S(t) is not greater than the short-term compensation trigger threshold, G_ST(t) is set to 0.

4. The video internal loudness dynamic compensation method according to claim 1, characterized in that, The overall loudness is determined using an integral loudness measurement method; the length of the sliding time window is 3 seconds.

5. The video internal loudness dynamic compensation method according to claim 1, characterized in that, The candidate compensation gain satisfies: G_cand(t)=G_overall+G_ST(t); Wherein, G_cand(t) is the candidate compensation gain at time t, G_overall is the overall loudness compensation gain, and G_ST(t) is the short-time loudness compensation gain at time t.

6. The video internal loudness dynamic compensation method according to claim 1, characterized in that, The anti-shake constraint satisfies the following: when |G_cand(t)-G_safe(t-1)| is less than the anti-shake threshold δ, G_safe(t)=G_safe(t-1); the maximum gain change is limited by the maximum step size Δ_max.

7. The video internal loudness dynamic compensation method according to claim 1, characterized in that, The final gain satisfies: G_final(t)=α_a·G_safe(t)+(1-α_a)·G_final(t-1), G_safe(t)<G_final(t-1); G_final(t)=α_r·G_safe(t)+(1-α_r)·G_final(t-1), G_safe(t)≥G_final(t-1); Where 0 < α_r < α_a < 1, α_a is the fast response coefficient, and α_r is the slow recovery coefficient.

8. A dynamic loudness compensation device for video, characterized in that, include: The audio acquisition module is used to acquire the audio data of the current video file; The loudness determination module is used to determine the overall loudness of the current video file based on the audio data, and to determine the short-term loudness that changes with the playback time according to a sliding time window; The compensation gain generation module is used to determine the overall loudness compensation gain based on the difference between the overall loudness and the reference loudness. When the short-time loudness is lower than the reference loudness and the difference between the two exceeds the short-time compensation trigger threshold, the module determines the short-time loudness compensation gain based on the difference between the reference loudness and the short-time loudness, and combines the overall loudness compensation gain and the short-time loudness compensation gain into a candidate compensation gain. The safety control module is used to apply upper and lower limit constraints on the candidate compensation gain, and to keep the safety gain of the previous time unchanged when the change of the candidate compensation gain relative to the safety gain of the previous time is less than the anti-shake threshold, and to limit the maximum gain change between adjacent time windows, so as to obtain the safety gain at the current time. A smoothing module is used to smooth the security gain using a fast response coefficient when the security gain is less than the final gain at the previous moment. When the safety gain is not less than the final gain at the previous moment, the safety gain is smoothed using a slow recovery coefficient to obtain the final gain at the current moment, wherein the fast response coefficient is greater than the slow recovery coefficient. The playback control module is used to control the audio playback output of the current video file according to the final gain.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the video internal loudness dynamic compensation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video internal loudness dynamic compensation method according to any one of claims 1 to 7.

Images