Audio detection method and device, computer equipment and storage medium

Abstract

The invention relates to an audio detection method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a dialogue signal by separating a dialogue and background sound in a stereo audio file, and determining the signal attribute of each frame signal in left and right sound channel signals through time-frequency domain analysis so as to determine the frame signal of which the signal attribute is the dialogue in the left and right sound channel signals, according to the loudness difference between the loudness value of the frame signal of which the signal attribute is dialogue in the left channel signal and the loudness value of the frame signal of which the timestamp is the same in the right channel signal, whether the dialogue bias of the left and right channels exists in the dialogue signal or not is judged, so that the problem that the dialogue bias in movie and television play works cannot be detected by the existing detection technology is solved.

Description

Technical Field

[0001] The present application relates to the field of audio detection, and in particular to an audio detection method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of the film and television entertainment industry, users can watch different types of film and television works through film and television platforms. Users have gradually improved their viewing requirements for film and television dramas, not only for picture quality but also for audio. Stereo audio is a sound reproduction method that uses two or more channels to give people a wider and richer sound field. Usually, in order to ensure a better listening experience, the overall strength of the left and right channels of stereo audio must be relatively balanced. In addition, in film and television works accompanied by the front picture, the character dialogues in the left and right channels must also be relatively balanced to ensure that the character dialogues match the picture.

[0003] However, due to the limitations of mixing technology and methods, some film and television works have different degrees of channel imbalance, which leads to audio problems such as sound field offset and dialogue bias that seriously affect the audio-visual experience. Most of the existing detection technologies are aimed at sound field offset, but there is a lack of effective and accurate detection solutions for the audio problem of dialogue bias. Summary of the invention

[0004] The present application provides an audio detection method, apparatus, computer device and storage medium to solve the problem that existing detection technologies are unable to detect dialogue bias in film and television works.

[0005] In a first aspect, the present application provides an audio detection method, the method comprising:

[0006] Get a stereo audio file;

[0007] Separating the dialogue and the background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal;

[0008] Performing frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal respectively to obtain a corresponding left attribute set and a corresponding right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal;

[0009] The dialogue bias detection result is determined according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding time stamp in the right attribute set.

[0010] In a second aspect, the present application provides an audio detection device, the device comprising:

[0011] An acquisition module, used to acquire a stereo audio file;

[0012] A processing module, used for separating the dialogue and the background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal;

[0013] An analysis module, used to perform time-frequency domain analysis on the left channel signal and the right channel signal frame by frame, respectively, to obtain a corresponding left attribute set and a right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal;

[0014] The determination module is used to determine the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding time stamp in the right attribute set.

[0015] In a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned audio detection method when executing the computer program.

[0016] In a fourth aspect, the present application also provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the above-mentioned audio detection method.

[0017] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains a stereo audio file; separates the dialogue and background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal; performs frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal to obtain a corresponding left attribute set and right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal; determines the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal with the signal attribute of dialogue in the left attribute set and the loudness value of the frame signal of the corresponding timestamp in the right attribute set.

[0018] Based on the above method, the dialogue signal is obtained by separating the dialogue and background sound in the stereo audio file, and then the signal attribute of each frame signal in the left and right channel signals is determined by time-frequency domain analysis to determine the frame signal with the signal attribute of dialogue in the left and right channel signals. According to the loudness difference between the loudness value of the frame signal with the signal attribute of dialogue in the left channel signal and the loudness value of the frame signal of the same timestamp in the right channel signal, it is judged whether there is a dialogue bias between the left and right channels in the dialogue signal, thereby solving the problem that the existing detection technology cannot detect the dialogue bias in film and television works. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0022] Figure 1 A flowchart of an audio detection method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of an audio detection method provided in an embodiment of the present application;

[0024] Figure 3 A structural block diagram of an audio detection device provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0028] In one embodiment, Figure 1 FIG. 1 is a flow chart of an audio detection method in an embodiment, referring to FIG. Figure 1 , provides an audio detection method. This embodiment mainly uses the method applied to a server as an example, and the audio detection method specifically includes the following steps:

[0029] Step S210, obtaining a stereo audio file.

[0030] Specifically, stereo audio files Audio can be obtained through different film and television platforms, or through a resource database, that is, there is no restriction on the source of obtaining stereo audio files. Stereo audio files are audio files containing left and right channels. The audio type of stereo audio files can be film and television dramas or novels, etc.

[0031] Step S220, performing a process of separating the dialogue and the background sound on the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal.

[0032] Specifically, when the audio type of a stereo audio file is a film or television drama or novel, there will be background sound. In order to more clearly detect the dialogue bias problem, it is necessary to separate the dialogue and the background sound of the stereo audio file to obtain a separated dialogue signal Audio_v. The dialogue signal includes a left channel signal vl and a right channel signal vr, and both the left and right channel signals are composed of multiple frame signals with time stamps arranged in sequence.

[0033] Step S230, performing frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal, respectively, to obtain corresponding left attribute set and right attribute set, wherein the left attribute set includes signal attributes of each frame signal in the left channel signal, and the right attribute set includes signal attributes of each frame signal in the right channel signal.

[0034] Specifically, a time-frequency domain analysis is performed frame by frame on the left channel signal to determine the signal attributes of each frame signal in the left channel signal, and a time-frequency domain analysis is performed frame by frame on the right channel signal to determine the signal attributes of each frame signal in the right channel signal. The signal attributes include dialogue, background sound, silence, background noise, etc. Under normal circumstances, the signal attributes of a specified frame signal in the left channel signal are the same as the signal attributes of a corresponding frame signal of the same timestamp in the right channel signal, but there may be deviations in audio features between frame signals in the left and right channels at the same timestamp. The audio features include spectral smoothness, zero-crossing rate, audio energy, loudness value, etc.

[0035] Step S240 , determining a dialogue bias detection result according to a loudness difference between a loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and a loudness value of a frame signal of a corresponding time stamp in the right attribute set.

[0036] Specifically, the difference in loudness values ​​between frame signals with the same timestamp and whose signal attributes are dialogue in the left and right channel signals is used to detect whether the frame signal has a dialogue bias, and the dialogue bias is divided into dialogue bias towards the left channel or dialogue bias towards the right channel. Dialogue bias towards the left channel means that the signal strength of the left channel is greater than the signal strength of the right channel, and dialogue bias towards the right channel means that the signal strength of the right channel is greater than the signal strength of the left channel, that is, the dialogues of the left and right channels are not in a balanced state. Therefore, the dialogue bias detection result is determined according to the difference in loudness values ​​of all frame signals with the signal attributes of dialogue in the left and right channel signals, thereby realizing the detection of the audio problem of dialogue bias, so as to solve the problem that the existing detection technology cannot detect dialogue bias in film and television works.

[0037] In one embodiment, obtaining a stereo audio file includes:

[0038] Get original audio and video materials;

[0039] Extracting stereo audio material from the original audio and video material;

[0040] The overall volume of the stereo audio material is adjusted to a preset volume through a loudness control algorithm to obtain the stereo audio file.

[0041] Specifically, the original audio and video material (mkv / mp4) is the initial material obtained through different data sources. At this time, the original audio and video material includes video material and stereo audio material. Therefore, only the stereo audio material related to the audio is extracted from the original audio and video material, and the overall volume of the stereo audio material is adjusted to the preset volume through the loudness control algorithm (EBUR.128) to avoid the original stereo audio material volume being too small to affect the judgment of the dialogue and background sound, thereby obtaining a stereo audio file Audio with a unified volume, so that the background sound and the dialogue can be separated and processed more accurately to obtain the dialogue signal Audio_v, thereby improving the accuracy of the dialogue signal and the accuracy of the subsequent dialogue detection results.

[0042] In one embodiment, the separating the dialogue and the background sound of the stereo audio file to obtain the dialogue signal includes:

[0043] When the playing time of the stereo audio file is greater than or equal to a preset time, splitting the stereo audio file into a plurality of audio segments;

[0044] Using a deep learning network to separate the dialogue and background sound of each of the audio clips, to obtain a dialogue clip corresponding to each of the audio clips;

[0045] The dialogue segments are sequentially spliced ​​together according to the sequence of the playback timestamps to form a dialogue signal.

[0046] Specifically, when the playback time of a stereo audio file is greater than or equal to a preset time, it indicates that the playback time of the stereo audio file is relatively long. If the background sound of the stereo audio file is separated according to the playback time, it will take a long time. In order to improve the audio processing speed, the stereo audio file is split into multiple audio clips, and the audio clip splitting rule can be to split the clips according to a specified time, and the specified time is less than the playback time of the stereo audio file; or the stereo audio file can be split according to a preset number of splits, that is, to obtain audio clips with a preset number of splits; or the stereo audio file can be split according to preset split points marked by different timestamps in the stereo audio file. The preset split points can be set on different timestamps of the stereo audio file to divide the clips of the specified plot. The specified plot can be key plot content or embedded advertising content.

[0047] A deep learning network is used to separate the dialogue and background sound of each audio clip, so as to obtain the dialogue clip corresponding to each audio clip. Finally, the dialogue clips are spliced ​​together in sequence according to the sequence of playback timestamps to form a dialogue signal. By simultaneously separating the background sound of multiple audio clips, the separation processing efficiency is improved and the time spent on obtaining the dialogue signal is shortened.

[0048] In one embodiment, the performing frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal to obtain the corresponding left attribute set and right attribute set includes:

[0049] Performing time-frequency domain analysis on the left channel signal and the right channel signal frame by frame respectively to obtain audio features of each frame of the left channel signal and audio features of each frame of the right channel signal;

[0050] Based on the audio features of each frame signal in the left channel signal and the audio features of each frame signal in the right channel signal, a left attribute set corresponding to the left channel signal and a right attribute set corresponding to the right channel signal are determined.

[0051] Specifically, the left channel signal is analyzed frame by frame in the time-frequency domain to obtain the audio features of each frame of the left channel signal, and the right channel signal is analyzed frame by frame in the time-frequency domain to obtain the audio features of each frame of the right channel signal. The audio features include spectral smoothness, zero-crossing rate, audio energy, loudness value, etc.

[0052] According to the matching results between the reference audio features corresponding to different signal attributes and the audio features obtained by analyzing the left and right channels, the signal attributes of each frame of the left and right channel signals are determined frame by frame, and the signal attribute set of each frame of the left channel signal forms a left attribute set, and the signal attribute set of each frame of the right channel signal forms a right attribute set.

[0053] In one embodiment, determining the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding timestamp in the right attribute set includes:

[0054] When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the right attribute set exceeds a first preset difference range, it is determined that the frame signal corresponding to the target timestamp has a dialogue bias;

[0055] Accumulating the timestamps of all frame signals with dialogue bias in the dialogue signal to form a dialogue bias duration;

[0056] Determine the total duration of the dialogue according to the timestamps of the frame signals whose signal attributes are dialogues in the left attribute set and the right attribute set;

[0057] When the ratio between the duration of dialogue bias and the total duration of dialogue is greater than or equal to a preset ratio, it is determined that there is a dialogue bias in the entire stereo audio file, where the dialogue bias detection result includes that there is a dialogue bias in the entire stereo audio file.

[0058] Specifically, the signal attribute in the left attribute set is dialogue, and the frame signal corresponding to the target timestamp is the frame signal at the nth second in the dialogue signal. If the audio feature obtained based on the above time-frequency domain analysis does not include the loudness value and the audio energy is obtained, the audio energy is converted into the corresponding loudness value. The audio energy of the left channel signal at the nth second is denoted as Rms_vl, and the corresponding loudness value is denoted as Loud_vl; the audio energy of the right channel signal at the nth second is denoted as Rms_vr, and the corresponding loudness value is denoted as Loud_vr. The loudness difference between the left and right channels at the nth second is: Diff_Loud(n) = Loud_vl(n) - Loud_vr(n)

[0059] = 20lg(Rms_vl(n)) - 20lg(Rms_vr(n))

[0060] The loudness difference between the left and right channels at the nth second is compared with the first preset difference range to determine whether there is a dialogue bias and the type of dialogue bias in the frame signal at the nth second. If the loudness difference exceeds the first preset difference range, it is determined that there is a dialogue bias in the frame signal at the nth second; if the loudness difference is within the first preset difference range, it is determined that there is no dialogue bias in the frame signal at the nth second. The first preset difference range can be [-3, 3] dB, [-4, 4] dB, [-5, 5] dB, etc.

[0061] Based on the above method, it is judged whether there is a dialogue bias in each frame signal in the dialogue signal, and the timestamps of all frame signals with dialogue bias are accumulated to form the dialogue bias duration T1. The dialogue bias duration is used to indicate the total duration of the segments with dialogue bias in the dialogue signal. Also, the total dialogue duration T0 is accumulated according to the timestamps of the frame signals with the signal attribute of dialogue in the dialogue signal. The total dialogue duration is used to indicate the total duration of the segments with the signal attribute of dialogue in the dialogue signal. The ratio between the dialogue bias duration and the total dialogue duration is T1 / T0, and the preset ratio is T. Here, T takes the value of 30%, but this value can be customized according to actual needs. Refer to Figure 2 If T1 / T0 ≥ T, it means that most of the duration in the dialogue segments of the dialogue signal has a dialogue bias, so it is determined that there is a dialogue bias in the entire stereo audio file; if T1 / T0 < T, it means that a relatively small duration in the dialogue segments of the dialogue signal has a dialogue bias, so it is determined that the entire stereo audio file has balanced dialogue and there is no dialogue bias problem. In this way, the overall dialogue bias problem of the stereo audio file can be accurately detected.

[0062] In one embodiment, after determining the total duration of the dialogue according to the timestamps of the frame signals whose signal attributes are dialogues in the left attribute set and the right attribute set, the method further includes:

[0063] When the ratio between the dialogue bias duration and the total dialogue duration is less than a preset ratio, determining that the overall dialogue of the stereo audio file is balanced;

[0064] Selecting a target dialogue segment from the dialogue signal according to a time window corresponding to a specified duration, wherein the target dialogue segment includes a left channel segment and a right channel segment;

[0065] Determine a loudness difference between a loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and a loudness value of a frame signal of a corresponding time stamp in the right channel segment, and a weighted average of the loudness differences corresponding to each frame signal in the target dialogue segment, to obtain a loudness difference mean corresponding to the target dialogue segment;

[0066] When the loudness difference mean value exceeds a second preset difference range, it is determined that the target dialogue segment has a dialogue bias, and a time window corresponding to the specified time length is slid backward by a specified time length to form a new target dialogue segment by a plurality of frame signals framed therein, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment is continued based on the new target dialogue segment to obtain the loudness difference mean value corresponding to the target dialogue segment, until there is no frame signal in the dialogue signal that has not been detected by the segment, wherein the dialogue bias detection result also includes the existence of a dialogue bias in the target dialogue segment; or,

[0067] When the loudness difference mean is within a second preset difference range, the target dialogue segment is determined to be dialogue balanced, and a plurality of frame signals framed by sliding the time window corresponding to the specified duration backward by one frame form a new target dialogue segment, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment are continued based on the new target dialogue segment to obtain the loudness difference mean corresponding to the target dialogue segment until no frame signal that has not been detected by the segment exists in the dialogue signal.

[0068] Specifically, when determining the overall dialogue balance of the stereo audio file, a target dialogue segment is selected from multiple frame signals in the dialogue signal that have not been detected according to the specified duration. The specified duration can be 1, 2, 3 minutes, etc. At this time, the specified duration of 2 minutes is taken as an example, that is, a target dialogue segment with a duration of 2 minutes is selected from the dialogue signal, that is, the playback duration of the target dialogue segment is 2 minutes. The target dialogue segment can be understood as a dialogue segment dynamically selected from the dialogue signal according to the time window corresponding to the specified duration.

[0069] Dialogue bias detection is performed on the target dialogue segment, thereby realizing local dialogue bias detection on the stereo audio file. The average value of the loudness difference between the left and right channels of each frame signal in the target dialogue segment is used to determine whether the dialogue segment has dialogue bias, that is, the loudness difference corresponding to each frame signal in the target dialogue segment is weighted and averaged to obtain the loudness difference mean value Mean_Diff_Loud corresponding to the target dialogue segment (the weighted average value of Diff_Loud in 120 consecutive seconds). The second preset difference range can be the same as or different from the first preset difference range mentioned above. For example, the second preset difference range is [-0.5, 0.5]. Figure 2 If the mean loudness difference of the target dialogue segment exceeds the second preset difference range, it is determined that there is dialogue bias in the target dialogue segment, that is, dialogue imbalance; if the mean loudness difference of the dialogue segment is within the second preset difference range, it is determined that the dialogue in the dialogue segment is balanced and there is no dialogue bias problem. In this way, it can be accurately determined whether there is dialogue bias in different target dialogue segments in the dialogue signal, that is, whether there is a local dialogue bias problem in the dialogue signal.

[0070] When the current target dialogue segment does not have a dialogue bias problem, the multiple frame signals framed by sliding the time window corresponding to the specified duration backward by one frame form a new target dialogue segment. For example, the time period corresponding to the current target dialogue segment is t—t+120s, and the time period corresponding to the updated target dialogue segment is t+1—t+121s.

[0071] When the current target dialogue segment has a dialogue bias problem, the time window corresponding to the specified duration is extended backward by multiple frame signals framed by the specified duration to form a new target dialogue segment. For example, the time period corresponding to the current target dialogue segment is t—t+120s, and the time period corresponding to the updated target dialogue segment is t+120—t+240s. In this way, the time window can slide smoothly in different sliding modes under different dialogue segment detection results, and detect whether the dialogue segments composed of different time periods have dialogue imbalance problems, thereby ensuring that no dialogue segments are missed.

[0072] In one embodiment, when a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the right attribute set exceeds a first preset difference range, determining that the frame signal corresponding to the target timestamp has a dialogue bias includes:

[0073] When the loudness difference between the loudness value of the frame signal corresponding to the target timestamp and the signal attribute of the left attribute set being dialogue is greater than the upper limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased towards the left channel; or,

[0074] When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the right attribute set is less than a lower limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased toward the right channel, wherein the dialogue bias detection result further includes a distribution of frame signals with dialogue bias in the dialogue signal and a bias type, and the bias type is dialogue biased toward the left channel or dialogue biased toward the right channel.

[0075] Specifically, if the loudness difference between the loudness value of the frame signal of the left channel whose signal attribute is dialogue and the loudness value of the frame signal of the right channel at the corresponding time is greater than the upper limit threshold of the first preset difference range, it means that the left channel loudness is stronger than the right channel strength, so it is determined that the frame signal has dialogue biased to the left channel; if the loudness difference between the loudness value of the frame signal of the left channel whose signal attribute is dialogue and the loudness value of the frame signal of the right channel at the corresponding time is less than the lower limit threshold of the first preset difference range, it means that the right channel loudness is stronger than the left channel strength, so it is determined that the frame signal has dialogue biased to the left channel. There is dialogue bias towards the right channel in the frame signal. In this way, the dialogue bias type of each frame signal with the signal attribute of dialogue in the dialogue signal can be accurately detected. The distribution and bias type of the frame signal with dialogue bias in the dialogue signal can be determined through the detection result of each frame signal with the signal attribute of dialogue in the dialogue signal. This allows the backend personnel to accurately find the cause of the dialogue bias and perform dialogue bias elimination processing based on the dialogue bias distribution and bias type in the dialogue bias detection result, thereby improving the efficiency of dialogue bias elimination.

[0076] Figure 1 and Figure 2 FIG. 1 is a flow chart of an audio detection method in an embodiment. It should be understood that although Figure 1 and Figure 2The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 and Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0077] In one embodiment, Figure 3 As shown, an audio detection device is provided, comprising:

[0078] An acquisition module 310 is used to acquire a stereo audio file;

[0079] The processing module 320 is used to separate the dialogue and the background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal;

[0080] An analysis module 330 is used to perform time-frequency domain analysis on the left channel signal and the right channel signal frame by frame respectively to obtain a corresponding left attribute set and a right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal;

[0081] The determination module 340 is used to determine the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding time stamp in the right attribute set.

[0082] In one embodiment, the acquisition module 310 is further used for:

[0083] Get original audio and video materials;

[0084] Extracting stereo audio material from the original audio and video material;

[0085] The overall volume of the stereo audio material is adjusted to a preset volume through a loudness control algorithm to obtain the stereo audio file.

[0086] In one embodiment, the processing module 320 is further configured to:

[0087] When the playing time of the stereo audio file is greater than or equal to a preset time, splitting the stereo audio file into a plurality of audio segments;

[0088] Using a deep learning network to separate the dialogue and background sound of each of the audio clips, to obtain a dialogue clip corresponding to each of the audio clips;

[0089] The dialogue segments are sequentially spliced ​​together according to the sequence of the playback timestamps to form a dialogue signal.

[0090] In one embodiment, the analysis module 330 is further configured to:

[0091] Performing time-frequency domain analysis on the left channel signal and the right channel signal frame by frame respectively to obtain audio features of each frame of the left channel signal and audio features of each frame of the right channel signal;

[0092] Based on the audio features of each frame signal in the left channel signal and the audio features of each frame signal in the right channel signal, a left attribute set corresponding to the left channel signal and a right attribute set corresponding to the right channel signal are determined.

[0093] In one embodiment, the determining module 340 is further configured to:

[0094] When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the right attribute set exceeds a first preset difference range, it is determined that the frame signal corresponding to the target timestamp has a dialogue bias;

[0095] Accumulating the timestamps of all frame signals with dialogue bias in the dialogue signal to form a dialogue bias duration;

[0096] Determine the total duration of the dialogue according to the timestamps of the frame signals whose signal attributes are dialogues in the left attribute set and the right attribute set;

[0097] When the ratio of the dialogue bias duration to the total dialogue duration is greater than or equal to a preset ratio, it is determined that the entire stereo audio file has a dialogue bias, wherein the dialogue bias detection result includes that the entire stereo audio file has a dialogue bias.

[0098] In one embodiment, the determining module 340 is further configured to:

[0099] When the ratio between the dialogue bias duration and the total dialogue duration is less than a preset ratio, determining that the overall dialogue of the stereo audio file is balanced;

[0100] Selecting a target dialogue segment from the dialogue signal according to a time window corresponding to a specified duration, wherein the target dialogue segment includes a left channel segment and a right channel segment;

[0101] Determine a loudness difference between a loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and a loudness value of a frame signal of a corresponding time stamp in the right channel segment, and a weighted average of the loudness differences corresponding to each frame signal in the target dialogue segment, to obtain a loudness difference mean corresponding to the target dialogue segment;

[0102] When the loudness difference mean value exceeds a second preset difference range, it is determined that the target dialogue segment has a dialogue bias, and a time window corresponding to the specified time length is slid backward by a specified time length to form a new target dialogue segment by a plurality of frame signals framed therein, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment is continued based on the new target dialogue segment to obtain the loudness difference mean value corresponding to the target dialogue segment, until there is no frame signal in the dialogue signal that has not been detected by the segment, wherein the dialogue bias detection result also includes the existence of a dialogue bias in the target dialogue segment; or,

[0103] When the loudness difference mean is within a second preset difference range, the target dialogue segment is determined to be dialogue balanced, and a plurality of frame signals framed by sliding the time window corresponding to the specified duration backward by one frame form a new target dialogue segment, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment are continued based on the new target dialogue segment to obtain the loudness difference mean corresponding to the target dialogue segment until no frame signal that has not been detected by the segment exists in the dialogue signal.

[0104] In one embodiment, the determining module 340 is further configured to:

[0105] When the loudness difference between the loudness value of the frame signal corresponding to the target timestamp and the signal attribute of the left attribute set being dialogue is greater than the upper limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased towards the left channel; or,

[0106] When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the right attribute set is less than a lower limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased toward the right channel, wherein the dialogue bias detection result further includes a distribution of frame signals with dialogue bias in the dialogue signal and a bias type, and the bias type is dialogue biased toward the left channel or dialogue biased toward the right channel.

[0107] like Figure 4 As shown, an embodiment of the present application provides a computer device, including a processor 711, a communication interface 712, a memory 713 and a communication bus 714, wherein the processor 711, the communication interface 712, and the memory 713 communicate with each other through the communication bus 714;

[0108] Memory 713, used for storing computer programs;

[0109] The processor 711 is used to implement the audio detection method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 713.

[0110] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, the audio detection device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 4 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the audio detection device, for example, Figure 3 The acquisition module 310, the processing module 320, the analysis module 330 and the determination module 340 are shown. The computer program composed of various program modules enables the processor to execute the audio detection method of each embodiment of the present application described in this specification.

[0112] Figure 4 The computer device shown can be Figure 3The acquisition module 310 in the audio detection device shown acquires a stereo audio file. The computer device can perform a separation process of the dialogue and the background sound on the stereo audio file through the processing module 320 to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal. The computer device can perform a frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal through the analysis module 330 to obtain a corresponding left attribute set and a right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal. The computer device can determine the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left attribute set and the loudness value of the frame signal of the corresponding timestamp in the right attribute set through the determination module 340.

[0113] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the audio detection method provided in any of the aforementioned method embodiments is implemented.

[0114] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0116] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative can be used.

[0117] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An audio detection method, characterized in that: The method comprises: Get a stereo audio file; Separating the dialogue and the background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal; Performing frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal respectively to obtain a corresponding left attribute set and a corresponding right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal; The dialogue bias detection result is determined according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding time stamp in the right attribute set.

2. The method according to claim 1, characterized in that The step of obtaining a stereo audio file comprises: Get original audio and video materials; Extracting stereo audio material from the original audio and video material; The overall volume of the stereo audio material is adjusted to a preset volume through a loudness control algorithm to obtain the stereo audio file.

3. The method according to claim 1, characterized in that The step of separating the dialogue and the background sound of the stereo audio file to obtain a dialogue signal includes: When the playing time of the stereo audio file is greater than or equal to a preset time, splitting the stereo audio file into a plurality of audio segments; Using a deep learning network to separate the dialogue and background sound of each of the audio clips, to obtain a dialogue clip corresponding to each of the audio clips; The dialogue segments are sequentially spliced ​​together according to the sequence of the playback timestamps to form a dialogue signal.

4. The method according to claim 1, characterized in that The performing frame-by-frame time-frequency domain analysis on the left channel signal and the right channel signal to obtain a corresponding left attribute set and right attribute set includes: Performing time-frequency domain analysis on the left channel signal and the right channel signal frame by frame respectively to obtain audio features of each frame of the left channel signal and audio features of each frame of the right channel signal; Based on the audio features of each frame signal in the left channel signal and the audio features of each frame signal in the right channel signal, a left attribute set corresponding to the left channel signal and a right attribute set corresponding to the right channel signal are determined.

5. The method according to claim 1, characterized in that The determining of the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding timestamp in the right attribute set includes: When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the right attribute set exceeds a first preset difference range, it is determined that the frame signal corresponding to the target timestamp has a dialogue bias; Accumulating the timestamps of all frame signals with dialogue bias in the dialogue signal to form a dialogue bias duration; Determine the total duration of the dialogue according to the timestamps of the frame signals whose signal attributes are dialogues in the left attribute set and the right attribute set; When the ratio of the dialogue bias duration to the total dialogue duration is greater than or equal to a preset ratio, it is determined that the entire stereo audio file has a dialogue bias, wherein the dialogue bias detection result includes that the entire stereo audio file has a dialogue bias.

6. The method according to claim 5, characterized in that After determining the total duration of the dialogue according to the timestamps of the frame signals whose signal attributes are dialogues in the left attribute set and the right attribute set, the method further includes: When the ratio between the dialogue bias duration and the total dialogue duration is less than a preset ratio, determining that the overall dialogue of the stereo audio file is balanced; Selecting a target dialogue segment from the dialogue signal according to a time window corresponding to a specified duration, wherein the target dialogue segment includes a left channel segment and a right channel segment; Determine a loudness difference between a loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and a loudness value of a frame signal of a corresponding time stamp in the right channel segment, and a weighted average of the loudness differences corresponding to each frame signal in the target dialogue segment, to obtain a loudness difference mean corresponding to the target dialogue segment; When the loudness difference mean value exceeds a second preset difference range, it is determined that the target dialogue segment has a dialogue bias, and a time window corresponding to the specified time length is slid backward by a specified time length to form a new target dialogue segment by a plurality of frame signals framed therein, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment is continued based on the new target dialogue segment to obtain the loudness difference mean value corresponding to the target dialogue segment, until there is no frame signal in the dialogue signal that has not been detected by the segment, wherein the dialogue bias detection result also includes the existence of a dialogue bias in the target dialogue segment; or, When the loudness difference mean is within a second preset difference range, the target dialogue segment is determined to be dialogue balanced, and a plurality of frame signals framed by sliding the time window corresponding to the specified duration backward by one frame form a new target dialogue segment, and the step of determining the loudness difference between the loudness value of each frame signal with a signal attribute of dialogue in the left channel segment and the loudness value of the frame signal of the corresponding timestamp in the right channel segment, and the weighted average of the loudness difference values ​​corresponding to each frame signal in the target dialogue segment are continued based on the new target dialogue segment to obtain the loudness difference mean corresponding to the target dialogue segment until no frame signal that has not been detected by the segment exists in the dialogue signal.

7. The method according to claim 6, characterized in that When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and having a signal attribute of dialogue in the right attribute set exceeds a first preset difference range, determining that the frame signal corresponding to the target timestamp has a dialogue bias includes: When the loudness difference between the loudness value of the frame signal corresponding to the target timestamp and the signal attribute of the left attribute set being dialogue is greater than the upper limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased towards the left channel; or, When a loudness difference between a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the left attribute set and a loudness value of a frame signal corresponding to a target timestamp and a signal attribute of dialogue in the right attribute set is less than a lower limit threshold of the first preset difference range, it is determined that the frame signal corresponding to the target timestamp has dialogue biased toward the right channel, wherein the dialogue bias detection result further includes a distribution of frame signals with dialogue bias in the dialogue signal and a bias type, and the bias type is dialogue biased toward the left channel or dialogue biased toward the right channel.

8. An audio detection device, characterized in that: The device comprises: An acquisition module, used to acquire a stereo audio file; A processing module, used for separating the dialogue and the background sound of the stereo audio file to obtain a dialogue signal, wherein the dialogue signal includes a left channel signal and a right channel signal; An analysis module, used to perform time-frequency domain analysis on the left channel signal and the right channel signal frame by frame, respectively, to obtain a corresponding left attribute set and a right attribute set, wherein the left attribute set includes the signal attributes of each frame signal in the left channel signal, and the right attribute set includes the signal attributes of each frame signal in the right channel signal; The determination module is used to determine the dialogue bias detection result according to the loudness difference between the loudness value of each frame signal whose signal attribute is dialogue in the left attribute set and the loudness value of the frame signal of the corresponding time stamp in the right attribute set.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images